# α-5 T0 Smoke Result Analysis — TEMPLATE

> **Copy this file** to `qvt-ablation-T0-smoke-result-<YYYYMMDD>.md` and
> fill in the values once T0 smoke completes. Sections marked `[FILL]`
> are quantitative; sections marked `[interpret]` are short prose.
>
> **Plan reference**: `~/.claude/plans/quiet-hugging-iverson.md`.
> **Companion docs**:
> - Backlog reconciliation §7.3-§7.4 — cycle framing + corrections
> - ROADMAP §v0.4.x α-5 cycle — sizing + dual purpose
> - Memory `feedback_oracle_phrase_artifacts` — 4-step verification rule
> - Findings log `reports/research-runs/qvt-ablation-findings.md` — bucket-tagged entries

---

## 0. Run metadata

| Field | Value |
|---|---|
| Date | [FILL: YYYY-MM-DD HH:MM] |
| Workspace | `workspaces/hotpot_eval/` |
| Suite | `multihop_rag` |
| Fixture | `balanced-100` (25 per question_type) |
| Baseline | `baseline_3a961a3_rescored.json` (path 0.404, F1 0.609) |
| Cells run | L1/M_M, L5/M_M + L1/M_M sanity (think=ON) |
| N runs/cell | 1 (smoke) |
| Total compute | [FILL] min |
| Tier | M_M only (gemma4:e4b, A2 think=OFF in workspace .env) |

---

## 1. 5-axis Δ table (vs baseline `baseline_3a961a3_rescored`)

Baseline: path=0.404, graded=0.343, abst_f1=0.609, token=1150, latency=64s.

| Cell | path Δ | graded Δ | abst_f1 Δ | token Δ | latency Δ | Verdict |
|---|---|---|---|---|---|---|
| **L1/M_M** primary | [FILL] | [FILL] | [FILL] | [FILL] | [FILL] | [FILL] |
| **L5/M_M** full stack | [FILL] | [FILL] | [FILL] | [FILL] | [FILL] | [FILL] |
| **L1/M_M-thinkON** sanity | [FILL] | [FILL] | [FILL] | [FILL] | [FILL] | [FILL] |

**Pareto verdict reading** (rule from `scripts/qvt_ablation_matrix.py::_classify_five_axis_delta`):

- `strong-adopt` — quality+ AND cost+ (both improved)
- `adopt` — quality+ only, cost flat
- `efficiency-adopt` — quality flat, cost+ only
- `tier-gated` — quality+ but cost regressed
- `reject` — quality- on any axis
- `zero` — all within noise band

---

## 2. Per-question-type cross-tab

The MultiHop-RAG fixture carries `question_type ∈ {comparison, inference,
temporal, null}`. Cross-tab tests whether routing layers help different
query types differently (user requirement #2: routing-policy evidence).

### L1/M_M primary (think=OFF baseline)

| Type | path | graded | abst_f1 | token | latency |
|---|---|---|---|---|---|
| comparison | [FILL] | [FILL] | [FILL] | [FILL] | [FILL] |
| inference | [FILL] | [FILL] | [FILL] | [FILL] | [FILL] |
| temporal | [FILL] | [FILL] | [FILL] | [FILL] | [FILL] |
| null | n/a (0) | [FILL] | [FILL] | [FILL] | [FILL] |

### L5/M_M full stack (think=OFF)

| Type | path Δ | graded Δ | abst_f1 Δ | token Δ | latency Δ |
|---|---|---|---|---|---|
| comparison | [FILL] | [FILL] | [FILL] | [FILL] | [FILL] |
| inference | [FILL] | [FILL] | [FILL] | [FILL] | [FILL] |
| temporal | [FILL] | [FILL] | [FILL] | [FILL] | [FILL] |
| null | n/a | [FILL] | [FILL] | [FILL] | [FILL] |

### L1/M_M-thinkON sanity

| Type | path Δ vs primary | graded Δ | abst_f1 Δ | token Δ | latency Δ |
|---|---|---|---|---|---|
| comparison | [FILL] | [FILL] | [FILL] | [FILL] | [FILL] |
| inference | [FILL] | [FILL] | [FILL] | [FILL] | [FILL] |
| temporal | [FILL] | [FILL] | [FILL] | [FILL] | [FILL] |
| null | n/a | [FILL] | [FILL] | [FILL] | [FILL] |

---

## 3. via_graph vs via_sources breakdown

PR #618 unified the path axis across `graph_paths` entities AND
`sources` filenames. The per-query row tracks both. This section
tells whether JAMES is winning the path axis via concept traversal
(its graph-RAG layer) or via document citation (its source layer):

| Cell | via_graph hits | via_sources hits | total unique hits |
|---|---|---|---|
| L1/M_M | [FILL] | [FILL] | [FILL] |
| L5/M_M | [FILL] | [FILL] | [FILL] |

[interpret: which contribution dominates? does L5 routing help one
more than the other? — this is the "reasoning capability evidence"
part of the dual purpose, user requirement #4.]

---

## 4. Sanity cell — A2 default-flip evidence

Compares L1/M_M with `JAMES_GEMMA4_E4B_THINK_OFF=1` (primary) vs the
same cell with the env unset (sanity, think=ON, production default):

| Axis | think=OFF (primary) | think=ON (sanity) | Δ (ON − OFF) |
|---|---|---|---|
| path_coverage | [FILL] | [FILL] | [FILL] |
| graded_answer | [FILL] | [FILL] | [FILL] |
| abstention_f1 | [FILL] | [FILL] | [FILL] |
| token_cost | [FILL] | [FILL] | [FILL] |
| latency_cost | [FILL] | [FILL] | [FILL] |

**A2 default-flip rule** (from #609 follow-up gate):
- If sanity quality Δ ≤ 0 on **all** axes AND cost Δ ≥ 0 (worse) → default-flip
  to think=OFF is justified.
- If sanity quality Δ > 0 on any axis → think=OFF default-flip waits for
  more data; A3 v2 (#611) judgment confirmed.

[interpret: which case did we land?]

---

## 5. Routing-policy recommendation (auto-generated by render_report)

The matrix render emits this table per question_type — copy verbatim
from `reports/promo-assets/v0.4-qvt-ablation-matrix-<ts>.md`:

| Query type | Recommended (model, layer-stack) | Verdict | Source cell |
|---|---|---|---|
| comparison_query | [FILL] | [FILL] | [FILL] |
| inference_query | [FILL] | [FILL] | [FILL] |
| temporal_query | [FILL] | [FILL] | [FILL] |
| null_query | [FILL] | [FILL] | [FILL] |

[interpret: this is user requirement #2 deliverable. Do the per-type
recommendations agree (same recipe everywhere) or disagree
(per-type routing buys evidence)?]

---

## 6. T1 / stop decision

T0 verdict rule (from #613 §3, extended by the cost axes in #615):

- **Stop** — both quality (`|Δ_graded| < 0.05`) AND cost
  (`|Δ_token| < 10% of L1 median`, `|Δ_latency| < 10% of L1 median`)
  are within noise. Matrix is null at production tier; skip T1/T2
  (~18 h saved).
- **Quality signal** — at least one quality axis Δ ≥ 0.05. Proceed to
  T1 (`--tiers M_M`, all 6 rows, ~5.5 h).
- **Cost-only signal** — quality flat, cost Δ ≥ 10%. Proceed to T1;
  likely efficiency-adopt verdict.
- **Quality + cost signal** — both moved. Proceed to T1 + likely T2
  (tier-gating evidence required, ~13 h additional).

**Decision**: [FILL — stop / T1 only / T1+T2]
**Rationale**: [interpret]

---

## 7. Findings (mandatory bucket tag — per #621)

Add an entry to `reports/research-runs/qvt-ablation-findings.md` for
every surprise in this run. Each entry MUST carry a `bucket:` line.

Quick guide (full rule in memory `feedback_oracle_phrase_artifacts`):

| Bucket | Use when |
|---|---|
| (a) architecture | layer's design boundary is wrong shape |
| (b) LLM model | model capability ceiling (tier change / swap) |
| (c) feature gap | no layer covers this need (new layer) |
| (d) measurement | oracle / fixture / bench misses real JAMES output |

**Diagnosis order**: (d) → (b) → (a) → (c). Reverse it → wrong layer added.

### Findings drafted from this run

[FILL — list at least 3 by bucket. Two prior bucket-(d) finds were
path_recall=0 → #618 source-recall fix; 76% null hallucination → #619
phrase coverage. New findings here.]

---

## 8. Cycle-closure update

After this analysis lands, update:

- [ ] `reports/research-runs/qvt-ablation-findings.md` — new entries with bucket tags
- [ ] `docs/handovers/v0.4.x-backlog-reconciliation-2026-05-30.md` §7.4 — corrected baseline + matrix verdict
- [ ] `ROADMAP.md` §v0.4.x α-5 cycle — replace "(in flight)" status with closure state
- [ ] If routing default-flip verdicts: file separate PRs per layer with Quality Delta Card pointing at the cell here
- [ ] Memory entry promoting the strongest mechanism-candidate finding (Step 10 of plan)
- [ ] If T0 said "stop": draft a 1-paragraph "matrix found no layer effect" note + close routing-flag decision PRs as "stay opt-in indefinitely"

---

## 9. Reading order for reviewers (5 min)

1. §1 5-axis Δ table — does anything move?
2. §6 T1 / stop decision — what did the matrix conclude?
3. §3 via_graph vs via_sources — the publishable reasoning-capability signal
4. §7 findings (bucket-tagged) — what did we learn that needs follow-up?
5. (skip §2/§4/§5/§8 unless interpreting specific layers / A2 flip / publishing)
