# Common Mistakes Catalog

Full catalog of eval design mistakes. The top entries are also inline in SKILL.md.

| # | Mistake | What Happens | Fix |
|---|---------|-------------|-----|
| 1 | Writing the scenario before naming the eval question | You end up mixing adherence, correctness, and toolkit effects in one noisy test | State the question first: behavior change, task outcome, or environment/toolkit effect |
| 2 | Putting baseline/treatment structure inside the scenario file | The scenario no longer matches the harness contract; A/B logic is duplicated and confusing | Keep the scenario single-condition and let `arc eval ab` vary skill or environment |
| 3 | One overloaded skill-eval scenario with no trap | Baseline and treatment both look similar; delta is uninformative | Isolate one behavior per scenario and add a discriminative trap or bait |
| 4 | Baseline already near ceiling | Both conditions pass, delta stays tiny, and repeated runs remain inconclusive | Run 2-3 pilot trials first; if baseline already looks likely to exceed ~0.8, redesign before full A/B |
| 5 | Skill formalizes behavior the agent already exhibits | Skill teaches a process but the behavior it produces is generic agent competence — any agent does it without the skill. A/B delta is zero. Unlike competence proxy (where one *artifact* is non-discriminative), here the *entire behavior* is non-discriminative. | Before designing a skill A/B: ask "would a baseline agent, given the same prompt and files, behave differently without this skill?" If no, consider workflow eval (environment differs) or agent eval (quality measurement) instead. |
| 6 | Prompt leaks the repair pattern | Baseline follows the template and scores high without the skill | Remove explicit grader split, required decomposition, or named repair structure from the prompt |
| 7 | Code-grading skill adherence via competence proxy | Code grader checks an artifact (e.g., "uses scryptSync") that baseline produces through general competence, not through the skill. Both conditions pass, delta is zero. | Before finalizing a code grader for skill A/B, mentally run it against a baseline agent (no skill). If it still passes, the artifact isn't discriminative. Either strengthen the trap or switch to model grading for behavioral assertions. |
| 8 | Testing infrastructure artifacts instead of Claude's behavior | Eval checks "log file exists" or "JSON is valid" — passes trivially but doesn't prove the component affected Claude's session | Ask: "does my eval measure how Claude **behaved**, or just whether a side-effect file appeared?" If the latter, use unit/integration tests |
| 9 | Assertions with weak or arguable ground truth | Grading becomes subjective noise, and disagreements look like regressions | Make assertions defensible from the given context or split the scenario |
| 10 | Comprehension scenario labeled as agent eval | Measures text quality, not tool-using behavior; misleading pass rates | Add `**Eval type: comprehension**` to Context, or redesign with Setup + file artifacts |
| 11 | Model grader for deterministic output | Noisy scores, false positives from LLM hallucination | Use code grader with `$TRIAL_DIR` — verify files, run tests, grep patterns |
| 12 | Empty trial dir without Setup or Context | Agent spends 5+ minutes searching nothing, then times out | Add Setup to copy needed files, or provide sufficient Context for text-only responses |
| 13 | Assertions that can't be verified by chosen grader | Code grader can't check "well-structured"; model grader is overkill for "file exists" | Match grader to assertion type — code for existence/correctness, model for judgment |
| 14 | Scenario allows an escape hatch | Agent changes the task so the target tension disappears instead of solving it | Preserve the contract or artifact under test, but do not prescribe the exact repair |
| 15 | Rewriting assertions to fit preferred grader | "Well-structured code" becomes "name > 10 chars" — measures surface proxy, not actual quality (validity sacrificed for reliability) | Choose grader to match assertion's nature. If the assertion requires judgment, use model/human grader — don't reshape it into a grep pattern |
| 16 | Code-grading semantic quality because output is structured JSON | `grep "security"` on a JSON review finding checks the label, not whether the analysis is correct — a hallucinated fix still passes | Structure checks (field exists, type correct) → code grader. Content quality (analysis is sound, fix is correct) → model grader. Split when both aspects matter. |
| 17 | Scenario response shape is too large | Model grading gets noisier, and failures reflect output length more than target behavior | Require a short structured artifact instead of full markdown rewrites |
| 18 | Running k=1 and trusting the result | No variance information, single lucky/unlucky trial dominates | Use scenario-driven defaults (k=3-10 depending on eval type and grader); CI shown at k >= 5 |
| 19 | Grading stdout claims without checking artifacts | Agent says "I created the file" but didn't; grader scores the claim | Use code grader against `$TRIAL_DIR`, or model grader (artifacts auto-included) |
| 20 | Using `--skill-file` for workflow eval | Varies the prompt instead of the environment — measures the wrong thing | Workflow A/B varies the environment. Use `eval ab <name>` without `--skill-file` for workflow scope |
| 21 | Workflow eval with no plugins installed | Baseline and treatment are identical, delta is always 0 | Ensure toolkit plugin is installed: `claude plugin list` should show active plugins |
| 22 | Workflow scenario too skill-specific | Biases toward the toolkit — tests one skill, not systemic value | Use realistic, diverse tasks where the toolkit's value is non-obvious |
| 23 | Choosing the easiest-to-measure aspect over the meaningful one | Eval tests a side-effect hook (easy: check file) instead of a context-injection hook (meaningful: check Claude's awareness) — results are technically correct but answer the wrong question | For each component, first identify HOW it affects Claude's behavior, then test THAT — not the convenient side-effect |
