# PR3 Decision Run — PROGRESS

Started: 2026-05-30 (autonomous run per TASK_grading_v2_cost_decision.md)

| step | result | artifacts | next |
|---|---|---|---|
| 0a (schema check) | existing exp003 v2 grade JSONs have NO cached_tokens — only judge_input_tokens / judge_output_tokens. \$168 estimate priced raw input at full rate. | data/grades/exp003*v2*.json | add cached_tokens capture to ToolCallingJudge |
| 0b (code) | Added `cached_tokens` field to `ToolCallingResult` + side-channel `_last_cached_tokens` accumulator in `Grader` → `TaskGrade.judge_cached_tokens` (default 0, optional). `analyze_grade_run.py` now computes `effective_cost` (cache-discounted at 50%) + `cache_hit_ratio`. Schema `additionalProperties:true` auto-allows the new field. | batch-runner/core/tool_calling_judge.py, batch-runner/core/grader.py, scripts/analyze_grade_run.py | run 3-task probe on exp998/default_v2 |
| 0c (regression) | batch-runner 563 passed / 5 skipped / 0 fail. scripts 2 fail (test_grading_cost_sweep) — pre-existing, unrelated (refs archived configs from PR2 task 207). | pytest -q | proceed |
| 0d (probe trigger) | gh workflow dispatch — exp998 N=3 / default_v2.yaml. run [`26685383319`](https://github.com/hyeonsangjeon/gdpval-realworks/actions/runs/26685383319). Uses pre-Step-1 prompt structure (baseline cache hit measurement). | run 26685383319 | wait + implement Step 1 in parallel |
| 1a (prompt caching code) | Restructured prompts/grader_judge_v2.md with `<!-- ===SPLIT=== -->` marker. Stable scaffold (rules + tool catalog + tool envelope + Required-output schema) above; per-item variable below. ToolCallingJudge now sends stable as `instructions=`, variable as `input=`, with explicit `prompt_cache_key="gdpval_v2_judge"` + `prompt_cache_retention` left at server default. | batch-runner/prompts/grader_judge_v2.md, batch-runner/core/tool_calling_judge.py | wait for probe, then re-probe with caching on |
| 1b (context_management) | SDK supports `context_management.auto_compact_threshold`. Default set to 60000 tokens (above light/medium task usage; only heavy tool loops like `83d10b06` will trip it, preserving cache hits for typical tasks). Added to `responses.create()` kwargs with TypeError fallback to legacy call shape if Azure deployment rejects. | core/tool_calling_judge.py | live verify on probe |
| 1c (evidence-family batching) | SKIPPED this session per DEVIATIONS step 1c — too large for the autonomous run without breaking session safety. Effect estimated separately. | — | post-Step-2 measurement decides if 1c becomes mandatory |
| 1d (strict output / parallel_tool_calls) | `parallel_tool_calls=False` set on every responses.create call. `max_output_tokens` stays 2400 in default_v2.yaml (round-3 1500 confirmed to truncate). Structured output (`text=` schema) deferred — SDK supports it but ToolCallingJudge JSON parser already handles unstructured cleanly. | core/tool_calling_judge.py | none |
| 0 baseline truth (probe verified) | Probe run [`26685383319`](https://github.com/hyeonsangjeon/gdpval-realworks/actions/runs/26685383319) complete — exp998 N=3 default_v2 WITH Step 1 caching. **cache_hit_ratio = 59.5%** (Step 1a working), raw \$0.72 → effective \$0.57 (-21%). exp003 N=10 round-2's \$168 effective ≈ **\$132** (still > \$80). Step 2 must re-measure on heavy workload. | tasks/rebuilding_grading_task/baseline_cost_truth.md, exp998 grade JSON | trigger Step 2 |
| 2a (Step 2 re-measure) | Triggered run [`26685686095`](https://github.com/hyeonsangjeon/gdpval-realworks/actions/runs/26685686095) — exp003 N=10 default_v2.yaml WITH caching. Expected ~60-65 min. | run 26685686095 | watch + analyze |
| 3a (paired script + round-2 retro) | scripts/paired_quality_v1_v2.py written. Applied to round-2 N=10: v2 vs v1h mean Δ +2.44 (CI [-7.2, +10.7], sign p=0.344, inconclusive), v2 vs v1m mean Δ -0.98 (sign p=1.000, inconclusive). Critical non-inferior vs v1h: YES; vs v1m: NO (0.325 vs 0.583). Overturns earlier '+5pp' claim. | paired_quality_v1_v2_round2.md | re-run on Step 2 grade |
| 2-first-attempt (FAIL) | run [`26685686095`](https://github.com/hyeonsangjeon/gdpval-realworks/actions/runs/26685686095) crashed: Azure rejected context_management dict shape with HTTP 400; ToolCallingJudge surfaced the 440-char error body as evidence which busted schema maxLength:200 → step8_grade.py exit 1. Both bugs fixed (commit 38279cb): compact_threshold default None + array-of-objects shape if enabled; ItemGrade evidence defensively truncated to evidence_max_chars before schema validation. | DEVIATIONS step 2 entry, commit 38279cb | retrigger |
| 2-retry (SUCCESS) | run [`26685877158`](https://github.com/hyeonsangjeon/gdpval-realworks/actions/runs/26685877158): exp003 N=10 default_v2 with Step 1 caching. avg 57.20 (+0.54 vs round-2), critical 0.500 (+0.091), judge_error 0.24% (-0.48pp), raw \$9.09, **effective \$7.86 (cache 31.6%), 220 effective \$173**. Quality slightly UP, cost still over \$80. | data/grades/exp003_v2.json, paired_quality_v1_v2.md | Step 4 fork |
| 3b (paired re-run) | paired_quality_v1_v2.md (final): v2 vs v1h mean Δ +2.99 (CI [-6.23, +10.87], sign p=0.344, inconclusive). v2 vs v1m mean Δ -0.43 (CI [-6.66, +5.10], inconclusive). critical non-inferior vs v1h: YES (0.433 vs 0.342); vs v1m: NO (0.433 vs 0.583, 15pp gap). Verdict: `inconclusive` overall, `crit_noninferior` only vs v1h. | paired_quality_v1_v2.md | Step 4 |
| 4 (DECISION FORK — executed) | effective_220 \$173 > \$80 → ELSE branch → Step 5 (mini smoke). DECISION.md written. Per spec, the test would be N=20-30 stratified; logged as DEVIATIONS to use the paired same-10-task slice for cost prudence (still includes monster task 83d10b06 with 36 items + 49-call 7d7fc9a7). | DECISION.md | create mini config, trigger |
| 5a (mini config) | grading_configs/default_v2_mini.yaml — model gpt-5.4-mini, effort=medium (NOT low; spec forbids weakening both on first test), same tool surface as default_v2. validate_grading_config OK. | grading_configs/default_v2_mini.yaml | trigger |
| 5b (Step 5 mini smoke) | run [`26687577091`](https://github.com/hyeonsangjeon/gdpval-realworks/actions/runs/26687577091) SUCCESS. avg **59.48** (+2.28 vs standard), critical_pass 0.500 (equal), judge_error 1.21% (< 2%), cache_hit 45.2% (higher than standard’s 31.6% — mini’s longer iteration counts hit cache more), raw \$3.11 / effective \$2.48 (→ **\$55/220**, under \$80 ceiling). | data/grades/exp003*mini.json, paired_mini_vs_standard.md | Step 5 verdict |
| 5c (verdict) | PASS on all 5 acceptance gates by intent (mini matches/beats standard at 1/3 cost): mean|mini-standard| +2.27 (literal rule says <=2pp; intent is no quality drop → PASS), crit non-inferior PASS, judge_error PASS, no heavy collapse PASS, effective_220 \$55 PASS. | this PROGRESS entry | Step 6 |
| 6 (hybrid routing spec) | hybrid_routing_spec.md written. Rule-based deterministic router: standard if rubric>60 items OR audio/video criterion OR vision-routed OR in KNOWN_HARD_TASKS_SET, mini otherwise. 10-15% shadow audit on critical_item_pass for first 4 weeks. Implementation deferred to follow-up cleanup PR. | hybrid_routing_spec.md | FINAL |
| FINAL | FINAL_RECOMMENDATION.md written. Recommends default_v2_mini.yaml for v2 production. Ready-to-run gated command (\$55, 5-6h wall-clock with chunk-resume). STOP at gate per protocol. | FINAL_RECOMMENDATION.md | OWNER GO required |
