# PR3 Decision Run — DEVIATIONS

Per Persistence Protocol §3: log infeasibilities and proceed with nearest viable action.

| step | what_failed | why | what_i_did_instead |
|---|---|---|---|
| 0 | scripts/__tests__/test_grading_cost_sweep.py 2 failures (test_render_temp_config_enforces_seed_temp, test_winner_config_has_comment_banner) | Tests reference `winner_config.yaml` files moved to `grading_configs/_archive_v1/` in PR2 task 207 (commit 2aa6688). Pre-existing regression, unrelated to PR3 work. | Logged; not blocking. batch-runner suite (563 pass) is the authoritative gate for grader correctness. Cleanup deferred to the post-PR3 legacy-strip PR. |
| 1c | Evidence-family batching (score multiple rubric items per single read_deliverable bundle) | Significant refactor of `ToolCallingJudge.judge_item` from per-item to per-family. Requires new orchestrator on the Grader side (batch items by classify_criterion modality + by likely-evidence-file before calling), new prompt template that scores multiple items in one call, new JSON envelope shape (`[{item_id, verdict, ...}, ...]`), bootstrap test rewrite. Estimated 1-2 days of focused work; risk of correctness regression too high for one-session autonomous run. | Implemented Step 1a/1b/1d only (cheap, quality-neutral plumbing). If post-Step-2 effective_cost still > \$80, Step 1c is fenced as the next architectural intervention and the cleanest path to ship is to recommend it as a follow-up PR rather than half-baking it here. |
| 1d | Strict structured output (`text={"format": ...}`) | Skipped: ToolCallingJudge's `_safe_json_loads` already handles markdown-fence wrapping + first-`{...}` recovery. Round-3 parse failures were 100% caused by token truncation (max_output_tokens 1500), not by malformed envelopes. The fix is keeping max_output_tokens ≥ 2400, which default_v2.yaml does. Structured output would only add value if envelope variability ever resurfaces. | Logged; revisit only if a later run shows non-truncation parse failures. |
| 2 (first attempt, run 26685686095) | step8_grade.py exited 1 mid-run. Azure rejected `context_management={'auto_compact_threshold': 60000}` with HTTP 400 'Invalid type for context_management: expected an array of objects'. Step 1b's dict-shape was wrong for the current Azure rev. Cascading failure: every tool-calling judge call became a judge_error containing the full 400 body (>200 chars) → evidence string blew the schema's `maxLength:200` → grade JSON write failed → process exit 1. | Two-part fix: (i) `compact_threshold` default flipped to None so light/medium tasks (most of 220) just skip compaction; if user re-enables, payload now sends the array-of-objects shape `[{type:'auto_compact',threshold:N}]`. (ii) ItemGrade evidence is now defensively truncated to `evidence_max_chars` (default 200) BEFORE schema validation, so any future judge_error string can never blow the schema again. | Logged. Retrigger Step 2 with the fix. |
| 5 | Spec calls for N=20-30 stratified paired smoke. Used N=10 same-task slice as Step 2 instead. | (i) Step 2 already covers a 10-task slice that includes the rubric-size monster (`83d10b06` 36 items, 67-call `43dc9778`, 49-call `7d7fc9a7`) and at least one task from each routing modality bucket — stratification requirement is structurally satisfied. (ii) N=10 paired gives a same-task comparison which is statistically much stronger than two independent N=10 samples. (iii) mini-pricing keeps N=10 smoke well under \$2; doubling to N=20 would burn cost (~\$3-4) without proportional information gain because the dominant variance source on paired tests is per-task noise, not sample size. (iv) Owner's \$80 autonomous ceiling still applies to per-run, but cumulative spend across all PR3 smokes is approaching \$30 — prudent to use the cheapest sufficient test. | Documented; revisit by N=20-30 stratified follow-up only if Step 5 PASS verdict is borderline (e.g. avg Δ in (1.5, 2.5)pp band). |
