Are our skills & sub-agents actually good — and can we prove it?

The root virtue is Predictability — "the degree to which a skill makes the agent behave the same way on every run — the same process, not the same output." Everything hangs off four axes and five failure modes.

mindmap
  root((Predictability))
    Invocation
      Model-invoked · has description
      User-invoked · no description
      Tight descriptions · leading words
      One trigger per branch
    Information Hierarchy
      Steps first
      In-file reference
      Disclosed reference · pointers
      Progressive disclosure
    Steering
      Leading words · pretrained concepts
      Branches
      Completion criteria
      Guard premature completion
    Pruning
      Single source of truth
      Kill duplication
      Kill sediment · stale
      Delete no-ops
      

Four axes of predictability. Failure modes to hunt: premature completion · duplication · sediment · sprawl · no-ops.

The eight most actionable techniques from the guide itself. Three (④⑤⑥) sharpen or correct fixes proposed elsewhere in this audit — the guide is more precise than a summary of it.

The guide is a worked example of itself: it's disable-model-invocation: true (user-invoked, zero context load), declares "this skill is all reference", and discloses every definition to GLOSSARY.md behind one pointer. The terms are its leading words.

Two front-door routers sit over a shared library. The eval layer should close the loop back onto skills+agents — today it mostly doesn't (dashed = weak/structural-only).

flowchart TB
  U([plain-english goal]) --> HX["/hq-ext:auto
cross-plugin conductor"]
  U --> OA["/ork:auto
dev-verb intent router"]
  HX -->|dev-verbs, one-directional| OA
  OA -->|classify → confirm → hand off| SK

  subgraph LIB["shared library · src/"]
    SK["113 skills
32 user-invocable · 30 with triggers"]
    AG["38 agents
37/38 skills-wired"]
    HK["hooks
guardrails + telemetry"]
    SK -. spawns .-> AG
    AG -. wired to .-> SK
  end

  subgraph EVAL["eval layer"]
    ES["112 eval specs
query → expected_behavior"]
    RB["routing-benchmark.json
50 labeled pairs"]
    GT["golden tests
agent routing"]
  end

  SK -.->|covered by| ES
  OA -.->|gated by?| RB
  AG -.->|covered by| GT

  ES -. "CI: STRUCTURAL ONLY" .-> CI{{"PR gate
schema + scaffold"}}
  ES ==>|"weekly cron + auth"| LLM[["claude -p --bare
real grade"]]
  RB -. "manual only" .-> LLM

  classDef weak stroke-dasharray:5 5,stroke:#e0654f,color:#e0654f;
  classDef strong stroke:#3ecf8e;
  class ES,RB,GT,CI weak;
  class SK,AG,HK strong;
      

Green = strong & real. Red-dashed = the proof gap: specs exist but the real graded loop only fires weekly, and routing accuracy is never gated on a PR.

Scores are the panel's verified figures (post-adversarial-check), grounded in the real counts. Authoring axes score high; the eval axis drags the aggregate down.

Each row: what the framework prescribes, what we already do well (with a real number), and the concrete gap that survived verification.

Gaps shown are the ones the verification pass confirmed against a real file. Rejected/inflated claims were dropped.

Both are textbook Router Skills: classify → confirm → hand off, and never do the work themselves. hq-ext:auto conducts across both plugins and delegates every dev-verb one-directionally to ork:auto, the single source of truth for fix/build/review/test.

sequenceDiagram
  participant U as User
  participant H as /hq-ext:auto
  participant O as /ork:auto
  participant S as specialist skill
  U->>H: "get me ready & unblock the promote"
  Note over H: decompose → assign from catalog.json → compose waves
  H->>H: ops reads (∥) + gated mutate
  H->>O: dev-verb clause (one-directional)
  Note over O: classify (CoT) → confirm route
  O->>S: hand off + follow ITS phases
  S-->>U: result (specialist owns its report)
  Note over O,S: routing-benchmark.json · 50 pairs · target ≥95%
but only graded manually via /ork:bare-eval
      

Verdict: the routers are our best-authored skills (~8.5/10). The single fix that matters: wire routing-benchmark.json into a PR-time gate so a taxonomy edit can't silently regress routing accuracy.

Today's pipeline validates that specs are well-formed, not that skills behave. The target closes the loop affordably: sample-grade changed skills on every PR; full-grade weekly; and — the real blind spot — give agents their own activation + output-quality evals.

flowchart TB
  P[PR opened] --> V["validate eval schema
+ scaffold"]
  P --> K["keyword-match +
collision test
(deterministic · real)"]
  V --> G{{green}}
  K --> G
  W[weekly cron] -.->|needs auth| R["rate DESCRIPTIONS 1-10
on 5 RANDOM skills
NOT query→behavior"]
  R -.-> D[(one-shot json ·
no trend ledger)]
  B["routing-benchmark
50 pairs"] -.->|zero CI refs| X["manual bare-eval only"]
  A["31 agent specs"] -.-> N["never consumed"]
  SN["schema should_not[]"] -.-> NE["never evaluated"]
  classDef bad stroke:#e0654f,color:#e0654f,stroke-dasharray:4 4;
  classDef ok stroke:#3ecf8e;
  class R,X,N,NE bad;
  class K ok;
        

flowchart TB
  P[PR opened] --> C["changed-skills
detector"]
  C --> S["sample-grade
via bare-eval
(only changed)"]
  S --> M{{"accuracy ≥ threshold?"}}
  M -->|no| BLK[block merge]
  M -->|yes| OK[green]
  RB[routing-benchmark] --> S
  AG[agent-activation
eval set] --> S
  W[weekly cron] --> FULL["full 112-spec grade
+ trend to telemetry"]
  FULL --> TR[(activation precision/recall
misroute rate over time)]
  classDef good stroke:#3ecf8e;
  class S,M,FULL,AG good;
        

The unlock is changed-only sampling: grading every skill on every PR is too costly, but grading just the touched skill (plus the router benchmark) is cheap enough to gate.

Ranked by (impact on real quality signal) ÷ (build cost). The top three are all eval-layer; the authoring fixes are cheap polish.