# α-5 Matrix Closure Runbook

> **Audience**: operator finishing the α-5 cycle.
> **When to run**: after the in-flight T0 smoke (and any subsequent T1 / T2
> tiers) completes. Do NOT run mid-flight.
> **Why this exists**: the cycle accumulated 30+ PRs and 5 helper scripts.
> Without a runbook, the post-matrix sequence has to be re-derived from PR
> descriptions and memory entries, which is brittle. This runbook is the
> single ordered list.

---

## 0. Pre-flight check (5 min)

```powershell
# Verify the matrix is actually done — no running bench subprocess.
$running = Get-Process | Where-Object {$_.ProcessName -like '*python*' -and $_.CommandLine -like '*qvt_ablation_matrix*'}
if ($running) { Write-Host "Matrix still running. Wait." }

# Confirm cell JSONs landed.
ls workspaces/hotpot_eval/reports/research-runs/qvt-ablation-cells/
```

You should see at minimum (T0 smoke shape):

```
qvt-ablation-cell-L1-M_M.json
qvt-ablation-cell-L3-M_M.json       # or whichever the matrix ran
qvt-ablation-cell-L5-M_M.json
qvt-ablation-cell-L1-M_M-thinkON.json   # sanity cell
```

---

## 1. Rescore all cells against the correct bench files (~30 s)

QQ (#638) means every cell JSON written during T0 references a stale
step7 bench instead of the freshly-captured multihop_rag bench. The
rescore wrapper detects + fixes this in one pass.

```powershell
$env:JAMES_WORKSPACE = "./workspaces/hotpot_eval"
$env:PYTHONIOENCODING = "utf-8"

python scripts/qvt_rescore_all_cells.py `
    --workspace ./workspaces/hotpot_eval `
    --suite multihop_rag `
    --summary reports/research-runs/qvt-ablation-rescore-summary.md
```

**Expected output**: per-cell classification (`needs` for QQ-bugged,
`ok` for clean), then per-`needs` cell:

```
[auto] qvt-ablation-cell-L1-M_M.json -> reports/bench_<sha>_multihop_rag_<ts>.json
[backup] qvt-ablation-cell-L1-M_M.before-rescore.json
[wrote] qvt-ablation-cell-L1-M_M.json
```

Side-copies (`.before-rescore.json`) preserve the diagnostic trail.

**4-step rule check (mandatory)**: after the rescore, open one rescored
cell JSON. The `aggregate` block should show mid-range numbers
(path≈0.4, graded≈0.3, abst_f1≈0.6 for L1/M_M baseline). If any axis
is still saturated (0.0 or 1.0) on a cell that ought to be mid-range,
apply the 4-step rule (`memory/feedback_oracle_phrase_artifacts.md`)
before moving to step 2.

---

## 2. Re-render the matrix report (~5 s)

```powershell
python scripts/qvt_ablation_matrix.py --render-report
```

Reads every cell JSON in the workspace's cell dir, computes Δ vs
baseline per axis + per question_type, applies the Pareto verdict
classifier, writes the report to
`reports/promo-assets/v0.4-qvt-ablation-matrix-<ts>.md`.

**4-step rule check**: open the report's per-cell verdict table. The
L1 baseline cell should classify as `zero` (it IS the baseline; Δ=0
by construction). Any cell classifying as `zero` with a non-baseline
row should be re-inspected — Δ at noise band is normal, but a
literal 0/0/0 Δ might still be a stale-bench artifact that step 1
missed.

---

## 3. Fill the T0 analysis template (~10 s)

```powershell
python scripts/qvt_fill_t0_analysis.py `
    --workspace ./workspaces/hotpot_eval `
    --template reports/research-runs/qvt-ablation-T0-smoke-analysis-TEMPLATE.md `
    --output reports/research-runs/qvt-ablation-T0-smoke-analysis.md
```

Auto-fills the template's `<FILL>` placeholders from the rescored cell
JSONs. Open the output and add the prose-level commentary by hand —
the script handles numbers, not interpretation.

---

## 4. Promote mechanism-candidate findings (~5 s)

```powershell
python scripts/qvt_promote_findings.py
```

Walks `reports/research-runs/qvt-ablation-findings.md`. Anything tagged
`mechanism-candidate` / `universal-law` lands as a draft under
`reports/research-runs/promoted-findings/`. Review each draft, edit the
`description:` line + body where needed, then move accepted drafts
into the real memory dir (`~/.claude/projects/<id>/memory/`) and add a
one-line entry under `MEMORY.md`.

Record the promotion in the `## Promoted to memory` table of
`qvt-ablation-findings.md`.

---

## 5. Update the cycle summary doc (manual, ~20 min)

Open `reports/research-runs/alpha-5-cycle-summary-DRAFT.md` and fill
the sections that depend on the matrix verdict:

- §"What the corrected matrix says" — paste the 5-axis verdict table
  from step 2's report.
- §"Routing-flag default-flip recommendations" — for each `strong-adopt`
  or `adopt` verdict, log which layer + tier the recommendation
  applies to. These become individual default-flip PRs after this
  cycle closes (per ROADMAP §v0.4.x α-5 cycle "Routing-flag default-
  flip PRs per layer").
- §"Wrong-fix avoided" — confirm the count is 4 (#618 / #619+#623 /
  #625 / #638). If step 1's 4-step rule check surfaced a 5th, add it
  here with the bucket tag.

---

## 6. Publish the cycle closure (~10 min)

Single PR carrying:

- The filled cycle summary doc (step 5 output)
- The rendered matrix report (step 2 output, committed to repo)
- The filled T0 analysis (step 3 output)
- Updates to `ROADMAP.md §v0.4.x α-5 cycle` "Cycle deliverables"
  checkboxes (all four should now check)
- Updates to `docs/handovers/v0.4.x-backlog-reconciliation-2026-05-30.md`
  marking A1 (the α-5 ablation matrix line) as ✅ closed

PR title format: `docs(α-5): cycle closure — N PRs, M wrong-fix-averted, X-axis verdict ready`.

Quality delta: exempt (label: docs).

---

## 7. After-closure follow-up tracks

Each gated on the cycle closure PR above:

| Track | Trigger | First PR |
|---|---|---|
| Default-flip per-layer (D5 / D1 / LEO) | strong-adopt verdict in step 2 report | one per layer; cite the cell as evidence |
| A2 default-flip (`JAMES_GEMMA4_E4B_THINK_OFF=1` default) | G1+G2+G3 gates in pre-template | `docs/design/v0.4-a2-default-flip-quality-delta-card-pretemplate.md` |
| Oracle package split | post-closure clean | `docs/design/v0.4-qvt-oracle-package-split.md` |
| Bucket-(c) LLM-judge | post oracle split | `docs/design/v0.4-qvt-abstention-llm-judge.md` |
| Publishable narrative numbers | post-closure | `reports/research-runs/alpha-5-publishable-narrative-DRAFT.md` |
| Joint piece negotiation | mid-June collab rendezvous | see backlog Lane C |

---

## 8. Sanity check — what NOT to do

- Do NOT kill the running matrix to apply a fix mid-flight. The bench
  JSONs are written correctly; only cell aggregates are stale. The
  rescore step handles it post-hoc.
- Do NOT skip the 4-step rule check after step 1. The cycle's 4
  wrong-fix-averted corrections all came from applying it; assume the
  5th is one cell away.
- Do NOT file routing default-flip PRs from a non-rescored cell. The
  Δ values would be against a stale baseline.
- Do NOT publish the publishable narrative numbers from an `aggregate`
  block whose `runs[*].bench_output` field has not been verified.

---

## 9. Cross-references

- QQ post-mortem: `reports/research-runs/alpha-5-diagnostic-chain-post-mortem.md` §"Correction 4"
- Backlog ledger: `docs/handovers/v0.4.x-backlog-reconciliation-2026-05-30.md` §7.9
- 4-step rule: `memory/feedback_oracle_phrase_artifacts.md`
- Rescore single-cell script: `scripts/qvt_rescore_ablation_cell.py`
- Rescore wrapper: `scripts/qvt_rescore_all_cells.py`
- T0 analysis filler: `scripts/qvt_fill_t0_analysis.py`
- Findings promote: `scripts/qvt_promote_findings.py`
- Matrix renderer: `scripts/qvt_ablation_matrix.py --render-report`
