# Design Memo — QVT α-5: 18-Cell Ablation Matrix

> **Status**: design — α-5 from `docs/design/v0.4-qvt-alpha-non-saturating-oracle.md`
> §8 "Roadmap". α-1/α-2/α-3/α-4 closed (#550–#556); α-5 is the keystone
> measurement that converts the QVT 3-axis oracle into routing-policy
> decisions for D1 (adaptive budget) / D5 (auto router) / LEO (scope
> routing).
>
> **작성일**: 2026-05-30
> **선결 의존**: α-3 baseline JSON (`eval/qvt/baseline_<sha>.json`,
> captured at production env: ENTITY_ANCHOR=1 + bge-m3 + QUERY_REWRITE=1,
> routing layers OFF). α-2 step7 v5 fixture (16 queries × atomic
> gold_signals + abstention_truth + expected_path.edges).

---

## 0. 한 줄 요약

> **18 셀 (6 레이어 조합 × 3 모델 티어) × 3-axis QVT × paired N=3**
> = 라우팅 정책 6 결정 (각 셀이 baseline 대비 양/0/음 델타). 0-셀은
> 라우터에서 삭제, 양수 셀은 tier-gated 정책의 evidence. 운영자
> 단독 실행 (~20h compute, prep 끝).

---

## 1. Why now

A3/A2/A3 v2 closed the think-mode track (#608/#609/#611): no oracle
shows thinking-on producing better answers, but the OFF-wins
magnitude A3 grader reported also doesn't survive paired LLM-judge
replay. **A2 default-flip is off the table** — the new gate for any
default-flip is a real-query Quality Delta Card. Until that gate has
data, the think-mode axis is closed for matrix design.

α-5 is the next-best-leverage measurement:
- Routing layers (D1/D5/LEO) all shipped as opt-in (default OFF).
- No operator knows when to flip them on. That decision is
  evidence-bound and the evidence is exactly the matrix below.
- α-3 baseline already exists, α-2 step7 v5 fixture already exists,
  α-3 oracle module already exists. The remaining work is the
  matrix runner + report.

The think-mode result feeds α-5 indirectly: the small-tier model
gemma4:e4b is silently spending ~85% of budget on the thinking
trace, but A3 v2 showed this doesn't measurably hurt answers, so
we leave it alone for α-5 (run with default behaviour). A future
"real-query Quality Delta Card" cycle measures whether flipping
think-OFF on the small tier shifts any α-5 cell.

---

## 2. The 18 cells

### 2.1 Layer combos (6 rows)

Layer flags (all env vars, applied via the `_BASELINE_ENV`-style dict
the existing `scripts/qvt_capture_baseline.py` uses):

| Flag | Default | Module |
|---|---|---|
| `JAMES_ENABLE_ENTITY_ANCHOR` | 1 (prod) | `core/retrieval/entity_anchor_expander.py` |
| `JAMES_ENABLE_QUERY_REWRITE` | 1 (prod) | `core/retrieval/query_rewriter.py` |
| `JAMES_AUTO_ROUTER` (D5) | 0 | `core/reasoning/router.py` |
| `JAMES_ADAPTIVE_BUDGET` (D1) | 0 | `core/reasoning/budget.py` |
| `JAMES_SCOPE_ROUTING` (LEO) | 0 | `core/reasoning/evidence_scope.py` |

Other production env (held fixed across all cells): `JAMES_EMBEDDING_MODEL=BAAI/bge-m3`,
`JAMES_GEMMA4_E4B_THINK_OFF` left at its current setting (operator's
`.env`; A2 opt-in is orthogonal).

Each row is one (env-set, env-clear) pair; everything else is
production-default.

| # | Row label | ENTITY | REWRITE | AUTO_ROUTER | ADAPTIVE_BUDGET | SCOPE_ROUTING | Purpose |
|---|---|---|---|---|---|---|---|
| L0 | **floor** | 0 | 0 | 0 | 0 | 0 | true minimum — pure retrieval baseline. The "what does layer-zero look like?" cell. |
| L1 | **baseline (production)** | **1** | **1** | 0 | 0 | 0 | matches α-3 baseline JSON — Δ ≈ 0 by construction (validates pipeline reproduces the captured baseline). |
| L2 | + AUTO_ROUTER (D5) | 1 | 1 | **1** | 0 | 0 | isolates D5 contribution above baseline. |
| L3 | + ADAPTIVE_BUDGET (D1) | 1 | 1 | 0 | **1** | 0 | isolates D1 contribution above baseline. |
| L4 | + SCOPE_ROUTING (LEO) | 1 | 1 | 0 | 0 | **1** | isolates LEO contribution above baseline. |
| L5 | **full stack** | 1 | 1 | **1** | **1** | **1** | the "everything on" upper bound — measures interaction effects. |

This isolates each routing layer's marginal contribution at L2/L3/L4,
brackets the routing space with L0 (floor) and L5 (full), and pins L1
as the validation cell.

### 2.2 Model tiers (3 cols)

| # | Tag | Family / Size | Role |
|---|---|---|---|
| M_S | `gemma3:4b` | Gemma 3 / 3.3 GB | small — sub-production tier baseline. Same family as M_L for clean scale axis. |
| M_M | `gemma4:e4b` | Gemma 4 / 9.6 GB | medium — current JAMES production default; the cell most operators see. Thinking-capable (A3 finding); think-mode setting inherited from operator `.env`. |
| M_L | `gemma3:12b` | Gemma 3 / 8.1 GB | large — bigger context room without crossing the 16 GB local-VRAM boundary (gemma3:27b at 17.4 GB is too large for many operator boxes). |

Tier choice rationale:
- **Gemma 3 4b / 12b** brackets gemma4:e4b architecturally on the
  scale axis without dragging in a different family (Qwen / Llama).
- M_M is the production default — landing the matrix without it would
  be measuring something other than "what should JAMES default to."
- Avoid gemma3:27b — many operator boxes won't run it without
  swapping; the matrix has to be reproducible on a typical dev box.

### 2.3 The 18 cells = L0..L5 × M_S/M_M/M_L

```
              M_S (gemma3:4b)   M_M (gemma4:e4b)   M_L (gemma3:12b)
L0 floor          [ ]               [ ]               [ ]
L1 baseline       [ ]               [ ]               [ ]      ← α-3 baseline lives here on M_M
L2 +AUTO_ROUTER   [ ]               [ ]               [ ]
L3 +ADAPTIVE_BUD  [ ]               [ ]               [ ]
L4 +SCOPE_ROUTING [ ]               [ ]               [ ]
L5 full stack     [ ]               [ ]               [ ]
```

---

## 3. Per-cell measurement

Per cell, paired N=3 reruns of `scripts/bench.py --suite=step7 --mode=retrieval`,
applying `eval/qvt/oracle.py` `score_three_axis(...)` to each run's
bench JSON. Aggregate = median + noise-band (max − min) per axis.

Output per cell:

```jsonc
{
  "cell": "L2/M_M",
  "env": { "ENTITY_ANCHOR": "1", "REWRITE": "1",
           "AUTO_ROUTER": "1", "ADAPTIVE_BUDGET": "0", "SCOPE_ROUTING": "0" },
  "model": "gemma4:e4b",
  "n_runs": 3,
  "path_coverage":  { "median": 0.81, "min": 0.79, "max": 0.83, "noise_band": 0.04 },
  "graded_answer":  { "median": 0.62, ... },
  "abstention_f1":  { "median": 0.71, ... }
}
```

Then per-cell Δ vs L1/M_M baseline:

```jsonc
{
  "cell": "L2/M_M",
  "delta_vs_baseline": {
    "path_coverage": +0.02,
    "graded_answer": -0.01,
    "abstention_f1": +0.05
  },
  "verdict": "tier-gated keep (positive at M_M but Δ < noise band at M_S)",
}
```

### 3.1 Verdict rule (5-axis Pareto, post-#615)

The original 3-axis rule (positive/mixed/zero/negative/regression) was
replaced when cost axes were added (PR #615). The current Pareto-aware
6-verdict rule lives at
`scripts/qvt_ablation_matrix.py::_classify_five_axis_delta` and is
applied during `--render-report`.

#### Definitions

| Term | Means |
|---|---|
| **quality axes** | `path_coverage`, `graded_answer`, `abstention_f1` — higher = better |
| **cost axes** | `token_cost`, `latency_cost` — *lower* = better |
| **noise band** | per-axis `max − min` across the baseline's paired-N=3 runs (canonical baseline is `eval/qvt/baseline_<sha>.json`) |
| **quality+** | at least one quality axis Δ > +noise_band |
| **quality−** | at least one quality axis Δ < −noise_band |
| **cost+** | at least one cost axis Δ < −noise_band (lower is better — improvement) |
| **cost−** | at least one cost axis Δ > +noise_band (worse) |

#### The 6 verdicts

| # | Verdict | When | Policy implication |
|---|---|---|---|
| 1 | **strong-adopt** | quality+ AND cost+ AND no cost− | both axes win — best case, default on |
| 2 | **adopt** | quality+ AND cost flat (no Δ > band on cost axes) | quality wins, cost neutral — default on |
| 3 | **efficiency-adopt** | quality flat AND cost+ | free efficiency, no quality cost — default on |
| 4 | **tier-gated** | quality+ AND cost− | quality wins but cost regressed — accept on small tier (cost amortized), refuse on large tier |
| 5 | **reject** | quality− on any axis | no cost gain redeems quality loss — keep opt-in indefinitely |
| 6 | **zero** | all axes within noise | layer is inert at production tier — ablation evidence supports deprecation |

#### Worked examples (synthetic, noise = 0.05 quality / 100 token / 5s latency)

**Example A — strong-adopt**:
```
Δ = { path: +0.08, graded: +0.02, abst: +0.01, token: -150, latency: -3 }
quality+: yes (path moves past 0.05 band)
cost+:    yes (token down 150 > 100 band)
cost−:    no
→ strong-adopt
```

**Example B — efficiency-adopt** (the routing layer didn't help quality
but burned fewer tokens):
```
Δ = { path: 0.0, graded: 0.0, abst: +0.02, token: -200, latency: 0 }
quality+: no (abst Δ inside band)
cost+:    yes (token down 200)
→ efficiency-adopt
```

**Example C — tier-gated** (quality boost is real but at higher cost
— accept on small tier where it amortizes, refuse on large where the
small model is already cheap):
```
Δ = { path: +0.10, graded: 0.0, abst: 0.0, token: +250, latency: +12 }
quality+: yes
cost−:    yes (both cost axes worse)
→ tier-gated
```

**Example D — reject** (any quality regression is the disqualifier;
cost gains do not redeem it):
```
Δ = { path: -0.08, graded: +0.03, abst: 0.0, token: -500, latency: -20 }
quality−: yes (path regressed past band)
→ reject
```

**Example E — zero** (the most common verdict for layers that look
busy but don't move the needle):
```
Δ = { path: +0.01, graded: -0.02, abst: 0.0, token: +30, latency: +1 }
quality+: no, quality−: no, cost+: no, cost−: no
→ zero
```

### 3.2 Routing policy decisions per-tier pattern (the actual α-5 output)

| Per-tier pattern | Policy decision |
|---|---|
| `strong-adopt` or `adopt` on ≥ 2 tiers incl. M_M | **enable as default** — ship `=1` in `.env.example` |
| `efficiency-adopt` on all tiers | **enable as default** — free efficiency, no quality cost |
| only one tier `*-adopt` | **tier-gated** — router policy keys on `JAMES_LLM_MODEL` |
| `tier-gated` verdict on any cell | **per-tier evaluation** — small ≤ large? (small accepts cost overhead, large rejects) |
| `zero` on all tiers | **delete or document as inert** — deprecation PR with `architecture` label |
| `reject` on ≥ 1 tier | **keep opt-in indefinitely** — layer ships behind `=0` |

Per-question-type cross-tab (PR #615 §6): when a layer is `tier-gated`
on overall verdict but unanimously `*-adopt` on a single
`question_type`, the layer becomes **type-gated** — routed only for
queries the IntentClassifier labels with that type. The matrix renderer
(`scripts/qvt_ablation_matrix.py::_render_report`) emits the per-type
recommendation table automatically — copy verbatim into the cycle
analysis (template at
`reports/research-runs/qvt-ablation-T0-smoke-analysis-TEMPLATE.md`).

This is the exact mechanism CLAUDE.md §2 (Quality Delta Card) was
designed to feed.

---

## 4. Cost & runtime

Per cell: N=3 reruns × ~16 queries × ~80s/query ≈ 64 min for bench
alone. Server boot (~30s × 3) + oracle scoring (~ms) = ~66 min/cell.
18 cells × 66 min ≈ **20 hours** total compute. Operator-side, can be
chunked (6 rows in parallel batches, or 3 tiers in one weekend).

Suggested chunking:
- Day 1 (M_M, 6 cells × 66 min ≈ 6.5 h): production tier matrix, the
  load-bearing decision row.
- Day 2 (M_L, 6 cells ≈ 6.5 h): large-tier comparison.
- Day 3 (M_S, 6 cells ≈ 6.5 h): small-tier comparison.

Day 1 alone gives the production-tier routing decision. Days 2/3
refine to tier-gating.

---

## 5. Implementation

### 5.1 New script
`scripts/qvt_ablation_matrix.py` — extends `qvt_capture_baseline.py`'s
server-spawn-and-bench loop. CLI:

```bash
# Full matrix
python scripts/qvt_ablation_matrix.py

# Subset (operator chunks day 1)
python scripts/qvt_ablation_matrix.py --tiers M_M --rows L1,L2,L3,L4,L5

# Resume from interruption (skip cells with existing JSON)
python scripts/qvt_ablation_matrix.py --resume
```

Output: `reports/research-runs/qvt-ablation-<sha>-<ts>.json` (full
matrix) + per-cell JSONs at `reports/research-runs/qvt-ablation-cells/
qvt-ablation-cell-<L>-<M>-<ts>.json` (so a partial run is salvageable).

Final-report writer: same script, `--render-report` mode, reads the
per-cell JSONs and writes `reports/promo-assets/v0.4-qvt-ablation-
matrix.md` with the 18-cell verdict table + routing policy
recommendations.

### 5.2 No core/ change

α-5 is pure measurement. The routing policy decisions land in
**follow-up PRs** (one per layer, each as a small `.env.example`
default flip with Quality Delta Card citing the α-5 cell).

### 5.3 Reuse points
- `_BASELINE_ENV` pattern from `scripts/qvt_capture_baseline.py`
- `_spawn_server` / `_wait_for_healthz` / `_shutdown_server`
- `_run_single_bench` (parameterised by env + model tag)
- `score_three_axis` from `eval/qvt/oracle.py`
- `_mint_employee_jwt` for retrieval-mode bypass

### 5.4 Model tag plumbing

`JAMES_LLM_MODEL=<tag>` env var (already supported per
`feedback_ali_demo_model_checkpoint` memory). Set per row before
spawning the server.

---

## 6. CLAUDE.md alignment

| rule | impact |
|---|---|
| #1 (no domain) | ✅ — platform infra |
| #2 (bench numbers on core/) | ✅ — α-5 IS the bench. Future routing-flag default-flip PRs cite α-5 cells in their Quality Delta Card. |
| #3 (self-evolution opt-in) | ✅ — α-5 is read-only measurement; no auto-deploy. |
| #4 (architecture changes) | ⚠️ — none; α-5 itself ships no code change in core/. Follow-up flag-flip PRs may need `architecture` label if removing a layer. |
| #5 (≤ 20 KB) | ✅ — driver split is mechanical; per-cell JSONs keep the final report small. |

---

## 7. Follow-up sequence

After the matrix lands:

1. **Day-1 routing decisions** — for each L2/L3/L4/L5 cell that is
   `positive` on M_M, file a small PR flipping the corresponding
   `.env.example` default with Quality Delta Card citing that cell.
2. **Tier-gating PRs** — for layers that are `positive` on one tier
   but not others, update `core/reasoning/router.py` to gate on
   `JAMES_LLM_MODEL` tag.
3. **Inert-layer cleanup** — for any `zero` row across all tiers,
   open a deprecation PR (with `architecture` label).
4. **Real-query Quality Delta Card** — separate cycle, lands the
   pending A2 default-flip decision (#609 follow-up).

---

## 8. Related

- `docs/design/v0.4-qvt-alpha-non-saturating-oracle.md` — parent QVT
  design (this memo is the α-5 expansion)
- `eval/qvt/oracle.py` — 3-axis scorer
- `eval/qvt/baseline_<sha>.json` — α-3 baseline
- `scripts/qvt_capture_baseline.py` — α-3 capture script (template
  for α-5 matrix runner)
- `eval/regression/step7_queries.json` v5 — α-2 fixture
- PR #608 (A3) / #609 (A2) / #611 (A3 v2) — think-mode track that
  established the "no measurable difference" finding (axis closed
  pending real-query Quality Delta Card)

---

## 9. 한 줄 요약

> **L0..L5 × M_S/M_M/M_L × paired N=3 → 18 cells × 3-axis Δ vs
> α-3 baseline → routing-policy decision rule (default-on / tier-gated
> / inert / opt-in-only). 운영자 ~20h, prep 코드 변경 0.**
