A verified assessment of whether OrchestKit's 114 skills, 36 agents and rule surface are Opus-5-era dead weight.
"Anthropic cut 80 percent of its own rules for Opus 5 and lost nothing. Yours are holding it back too."
| Claim | Verdict | Evidence |
|---|---|---|
| The always-loaded index is a heavy tax | FALSE | Real session tax is 9,977 tokens total, of which the skill index is 5,867. That is not a problem worth a redesign. |
| 97 unused skills are dead weight | FALSE | 55 are intentionally hidden (documented 2-tier design, CONTRIBUTING-SKILLS.md:124). 87 are statically wired via agent skills:. 9 are genuinely orphaned. |
| 36 specialists lose to Explore because descriptions are vague | FALSE | The repo already A/B tested description rewrites: delta-0 routing gain (audit-activation/SKILL.md:92). Descriptions sit at 249 of a hard 250 B CI cap. |
| Cut ~80 percent like Anthropic did | 20% TRUE | Defensible cut is 1,868 tok of 9,977 (19%) per session and 529 KB of 9.99 MB (5.3%) of source. |
| The scaffolding is actively harmful | TRUE | 5 shipped bugs sat undetected inside 500-line ceremony files. This is the real finding. |
Each of these I confirmed myself against the files, not from the audit's report.
| # | Defect | Location | Proof |
|---|---|---|---|
| 1 | Tells an Opus 5 user to downgrade to Opus 4.8 | doctor/SKILL.md:171,178-182 |
Ships literal text Fix: Either switch to Opus 4.8 (claude --model opus-4-8) or lower effort to high. This session is Opus 5 at xhigh. The check fires a false failure. |
| 2 | --base dev against a repo with no dev branch |
create-pr/SKILL.md:213 |
git ls-remote --heads origin dev returns 0 refs. The command cannot succeed. |
| 3 | /ork:help greps a path installed users do not have, then silently falls back to a hardcoded 18-of-114 table |
help/SKILL.md:55-66 |
Greps src/skills; marketplace installs only have ${CLAUDE_PLUGIN_ROOT}/skills. |
| 4 | cover mandates a manual worktree workaround that its own cited doc marks superseded |
cover/SKILL.md:263-290 |
Cited manual-worktree-pattern.md:3 literally reads ⚠ SUPERSEDED. Contradicts implement:340-345. |
| 5 | TaskGet instructed in 22 skills, permitted in 0 |
22 SKILL.md files |
grep -rl TaskGet src/skills/*/SKILL.md = 22; allowed-tools containing it = 0. |
user-invocable: truedmi != true
tests/performance/test-token-overhead.sh:195 selects on user-invocable: true and multiplies by a flat 80 tokens (line 239). Both halves are wrong: wrong population, and an estimate instead of a measurement. The gate reports 6,235 against a 5,500 budget while the truth is ~9,977. It is failing, and it is failing at the wrong number.
Every BEFORE is copied from HEAD. Every AFTER preserves the non-derivable information and states what was deliberately kept.
| # | Surface | Before | After | Cut | Loaded |
|---|---|---|---|---|---|
| 1 | CLAUDE.md:62-72 rule | 499 | 375 | -24.8% | every session |
| 2 | assess/SKILL.md:163-199 | 1,813 | 252 | -86.1% | on invoke (#1 skill) |
| 3 | monitoring-engineer.md:60-87 | 1,616 | 348 | -78.5% | on spawn |
| 4 | cover/SKILL.md:5 description | 596 | 309 | -48.2% | every session |
| 5 | doctor/SKILL.md:166-185 | 1,112 | 796 | -28.4% | on invoke |
## Critical Rules **DO**: Edit `src/`, run `npm run build`, commit to feature branches, use TaskCreate for 3+ step work. **DON'T**: Edit `plugins/`, commit to `main`/`dev` directly, skip security tests, bypass hooks with `--no-verify`, commit secrets. **Before committing**: 1. `npm test` — all suites must pass 2. `npm run test:security` — MUST pass (blocks push) 3. `npm run typecheck` — if hooks were changed 4. `git diff` — verify changes are real
## Before committing `npm test`, plus `npm run typecheck` if you touched `src/hooks/`. `main`/`dev` are protected, so work on a branch. `bin/git-hooks/pre-push:322` runs `tests/security/run-security-tests.sh` and rejects the push, so `--no-verify` only relocates the failure to CI. Nothing enforces TaskCreate any more; use it when the operator gains from watching progress.
--no-verify is futile, with the exact gate location. The old text asserted "MUST pass (blocks push)" without saying where the gate lives, so it was a prohibition with no content.
## 🚨 Task Management (CC 2.1.16)
# 1. Create main task IMMEDIATELY
TaskCreate(subject="Assess: {target}", ...)
# 2. Create subtasks for each assessment phase
TaskCreate(subject="Understand target...") # id=2
TaskCreate(subject="Discover scope...") # id=3
TaskCreate(subject="Rate quality...") # id=4
TaskCreate(subject="Analyze pros/cons") # id=5
TaskCreate(subject="Compare alternatives") # id=6
TaskCreate(subject="Generate suggestions") # id=7
TaskCreate(subject="Compile report") # id=8
# 3. Set dependencies for sequential phases
TaskUpdate(taskId="3", addBlockedBy=["2"])
TaskUpdate(taskId="4", addBlockedBy=["3"])
TaskUpdate(taskId="5", addBlockedBy=["4"])
TaskUpdate(taskId="6", addBlockedBy=["4"])
TaskUpdate(taskId="7", addBlockedBy=["5","6"])
TaskUpdate(taskId="8", addBlockedBy=["7"])
# 4. Verify unblocked
task = TaskGet(taskId="2")
# 5. Update status as you progress
TaskUpdate(taskId="2", status="in_progress")
## Task tracking Track the phases below as tasks. Phase 1 to 1.5 to 2 is genuinely sequential (each consumes the previous phase's output), but Phases 3-7 all fan out from the Phase 2 scores, so do not chain those with `addBlockedBy` just to look tidy.
Generate tests that do not exist yet. Analyzes coverage gaps, then writes and runs new test files across three tiers (unit, integration against real services via testcontainers/docker-compose, and Playwright E2E), spawning one test-generator agent per tier and healing failures for up to 3 iterations. Use when code has no tests, when raising coverage after implementation, or when building a suite from scratch. Chains naturally after /ork:implement. Do NOT use to grade or score tests that already exist (use /ork:verify), or to run a suite without writing anything new (use npm test directly).
Writes tests that do not exist yet: unit, integration against real services (testcontainers/docker-compose), and Playwright E2E, then runs them. Use when code has no tests or coverage needs raising. Do NOT use to grade tests that already exist (that is /ork:verify), or just to run a suite (that is npm test).
cover and verify are a measured router collision (TF-IDF cosine 0.202, sharing grade, unit, integration, e2e, coverage). The two negative clauses are the only thing separating them, so they stay intact. This is the counter-intuitive result: negative clauses are ~23× the value per byte of "Use when" tails, so the trim should expand them and cut the positives.
### Category 14: Effort/Model Compatibility (CC 2.1.111+) CC 2.1.111 added `xhigh` effort (Opus 4.8; since CC 2.1.154 it defaults to `high`...). **Detection**: - If the active model does NOT support `xhigh` (i.e. not Opus 4.8), check whether... **Warning format**: WARNING: xhigh effort requires Opus 4.8. Fix: Either switch to Opus 4.8 (`claude --model opus-4-8`) or lower effort to `high`.
### Category 14: Effort/Model Compatibility `xhigh` effort degrades to `high` on a model that does not implement it: no error, no log line, the skill just silently loses its extra deepening pass. That silence is the only reason this check has to exist. **Detection**: read `.claude/settings.json` -> `effort`, and `$ORCHESTKIT_EFFORT`.
models.vocab.json, which has no xhigh field at all. That needs a schema addition first.
| Wave | Content | Saving | Risk |
|---|---|---|---|
| 0 | The 5 shipped bugs | ~0 B | ship first |
| 1 | 9 true orphans (281 KB) · agent Task-Management tutorial, 31 of 34 byte-identical (11 KB) · CC-changelog archaeology below the 2.1.220 floor (12 KB) | -322 KB | low, 2 orphans need a grep first |
| 2 | product-frameworks (31 of 49 files byte-identical to its own split-outs) · 5 testing-* into one · design-ship into a flag | -505 KB | medium, rewrite agent skills: in the same commit |
| 3 | 8 longest index descriptions toward ~300 B · "Use when" tails across 54 skills | -1,600 tok | medium, keep all 16 negative clauses |
| 4 | Ban-voice to judgement-voice in CLAUDE.md | -76 tok | low |
hooks:, 8 declare disallowedTools: [Write,Edit,MultiEdit], 29 scope mcpServers. disallowedTools on a reviewer is what makes the review honest. None of it is derivable.skills: blocks. All 36 declare them and 87 skills are reachable only through them. One audit dimension claimed "grep returns EMPTY, no agent declares any skill". I verified: 36 of 36 declare it. Acting on that fabricated finding would have deleted 30 wired skills.design-import/SKILL.md:139 literally contains subagent_type="ork:claude-design-orchestrator". Telemetry silence is not death.dream/quickviz. Written as checkable postconditions, and every guard names the incident that produced it. Use dream's shape (algorithm, guard, incident) as the rewrite target for the bloated skills.cc-native-first.md documents a measured 60% error rate on the alternative. A ban backed by a real incident is informative: keep the ban, and make sure the incident is stated, because the incident is the information.allowed-tools covers what the body callsaudit-skill-permissions.sh:56 checks only that the key exists. That gap shipped TaskGet in 22 files and a commit skill declaring [Bash] while calling three other tools. One assertion kills an 11-name, 10-file class permanently.
dmi:true + ui:false + zero inbound refs = fail. This is the check that finds the 9 real orphans, and critically, the check whose absence let an auditor claim 39. Make dmi:true + ui:false legal only with a declared consumer.
Swap user-invocable for dmi != true, measure actual name+description bytes instead of count × 80, then re-baseline honestly. It is red today at the wrong number.
test-model-recency.sh to read proseIt scans 1,698 files and reports 0 superseded while 79 Opus 4.8 prose mentions sit in the tree, including a doctor check that prescribes a downgrade. Without this, every Wave 3 fix re-rots at Opus 6.
The adversarial pass paid for itself. It killed a fabricated grep claiming 39 unreachable skills (would have deleted 30 wired ones), a recommendation to strip state.json pointers from 25 agents (the file exists and is schema-validated in CI), and a proposal to replace an inline 208 B protocol with a 2,238 B file read across 1,135 spawns, a 10× pessimisation sold as a saving.
Known gaps, stated rather than hidden: the deadweight dimension failed to return structured output, so orphan classification came from the adjacent dimensions and my own greps. One byte claim in 87 findings (1.35 MB from pattern-library consolidation) could not be reproduced and is excluded from all totals above. Telemetry has a real April-to-June hole where the tracker hook was dead (#959), so "never invoked" was cross-checked against static wiring rather than trusted alone.
Generated 2026-07-26 · OrchestKit v9.1.0 · branch chore/opus5-trim-assessment
Full plan: .audit/plan.md · Full exhibits: .audit/examples.md