# Design Memo — QVT α-5 Prereq Plan (4 Known Risks)

> **Status**: planning — addresses the 4 known risks called out during
> the matrix-design discussion (#612 prep) before α-5 is actually run.
> Each risk gets: diagnosis from current state, candidate solutions
> with cost, recommendation, gate.
>
> **작성일**: 2026-05-30 (post-#612)
> **선결 의존**: α-5 driver landed (#612). α-3 baseline JSON
> (`eval/qvt/baseline_2a31b20.json`) inspected as-of this memo.

---

## 0. Why this memo before the run

α-5 driver shipped in #612 with the 18-cell matrix locked. But running
the matrix is only worthwhile if its **measurement floor** is below the
expected layer effect size. Inspecting the captured α-3 baseline:

```
path_coverage:  median=1.0000  min=1.0000  max=1.0000  noise=0.0000
graded_answer:  median=0.5833  min=0.5417  max=0.6458  noise=0.1041
abstention_f1:  median=0.6667  min=0.5714  max=0.8571  noise=0.2857
```

Two of three axes are **structurally compromised**:
- Path coverage is at the ceiling (1.0) on its 4 annotated queries —
  axis cannot register improvement, only regression.
- Abstention F1 noise band (0.29) exceeds any plausible layer effect
  size (literature suggests retrieval-layer Δ on abstention F1 ≤ 0.1
  in routing studies).

If we ran the matrix as shipped, **two-thirds of the verdict bandwidth
is gone before the first cell runs**. The likely outcome is "all 18
cells zero" — but that would be ambiguous: did the layer not help, or
did the oracle not see? This memo addresses the 4 risks ahead of the
run so the result is interpretable.

---

## 1. Risk 1 — Measurement weakness (paraphrase intolerance)

### Diagnosis

`eval/qvt/oracle.py:_matches_signal` is literal substring match:

```python
if term.lower() in answer.lower(): return True
```

Failure modes observed / expected:
- **Paraphrase miss**: gold_signal `"AI safety research"` does not match
  answer phrasing `"Anthropic safely develops AI"`. A3 v2 (#611)
  observed the same class of fragility in the synthesis HARD grader.
- **Negation false positive**: gold_signal `"Anthropic"` matches answer
  `"Anthropic does not develop this model"` — both contain the literal
  term. No negation guard.
- **Quantity miss**: gold_signal `"0.03%"` does not match
  `"0.03 percent"` or `"3 basis points"`. No numeric normalization.
- **Antonym admit**: gold_signal alias `"Constitutional AI"` matches an
  answer that calls it "anti-Constitutional approach" — pure substring.

### Candidate solutions

| # | Solution | Cost | Catches |
|---|---|---|---|
| (a) | Targeted alias expansion in `gold_signals` for known paraphrases | data work, ~30 min for 17 queries | paraphrase miss only |
| (b) | Negation guard — if `not`/`doesn't`/`못`/`아니`/`없` within 12 chars before signal, suppress match | code, ~50 LOC + tests | negation false positive |
| (c) | LLM-judge augmentation only when grader score falls in the gray zone (0.33 ≤ score ≤ 0.67) | code, ~80 LOC + judge calls (~2× cost on gray-zone queries) | paraphrase, quantity, antonym |
| (d) | Embedding-based signal match (cosine sim ≥ 0.7) instead of substring | code, ~30 LOC, embedding cost per query | paraphrase, quantity |

### Recommendation

**(b) negation guard + (a) targeted alias expansion** are the cheap,
robust additions and land in this prereq cycle. **(c) LLM-judge
gray-zone augmentation** is reserved for v0.5 if the matrix Δ pattern
turns out to be borderline (most cells 0.04-0.08 graded Δ). **(d)
embedding match** is out of scope — it adds an embedding dependency
and re-introduces the saturation problem (cosine ceiling at 1.0).

### Gate before α-5 run

Implement (b) + at least one round of (a) on the 17 fixture queries
before running α-5 Tier 0 (§3 below).

---

## 2. Risk 2 — Fixture saturation

### Diagnosis

Per the inspection in §0:

- **Path axis**: 4/17 queries annotated (q2/q3/q4/q15), all 4 score
  recall=1.0 at production baseline. Axis median pinned at 1.0 with
  zero noise. **The path axis cannot detect layer improvements** — it
  can only register regressions (a layer that *broke* retrieval). For
  measuring "did this routing layer add value?" the axis is dead.
- **Abstention axis**: 5 queries are `absent`, 12 are `present`. With
  only 5 absent queries the F1 denominator is small → F1 variance is
  high (max − min = 0.29 across 3 paired runs). Any layer Δ smaller
  than 0.3 is statistical noise.
- **Graded answer**: 17 queries × 3 atomic claims = 51 atomic matches.
  Baseline 0.58 (29.6/51 matched on average), noise 0.10. This is the
  only axis with both room to move AND a measurable noise band.

### Candidate solutions

| # | Solution | Cost | Effect |
|---|---|---|---|
| (a) | Annotate `expected_path` on the remaining 12 queries (q1, q5–q14, q16, q17) | data work, ~30 min | path axis denominator 4 → 17, but if all 12 also hit 1.0 the axis stays saturated |
| (b) | Add 5–8 NEW harder queries where production baseline scores < 0.85 path_recall | data work + manual GT, ~2 h | adds movable headroom on path axis specifically |
| (c) | Expand `abstention_truth=absent` queries from 5 to 10+ | data work, ~30 min | reduces abstention F1 noise band |
| (d) | Replace abstention F1 with a separate (precision, recall) pair so noise band is reported per metric | code, ~40 LOC | exposes the actual variance source but doesn't reduce it |
| (e) | Re-port hard fixtures from `scripts/research/v3prime_e_mode_split_complex.py` / planner/reflect drivers into step7 v6 | data work, ~1 h | bridges A3 hard-fixture intuition into the production oracle |

### Recommendation

**(b) + (e) are the load-bearing fixes** but are 1-3 hours of
operator/user work. **(a) + (c) are 30-min data tasks** the user can
do incrementally.

Critically, **(b) determines whether path axis is alive at all**. The
existing q1/q5–q14/q16/q17 queries probably also hit ≈ 1.0 path_recall
under bge-m3 + entity-anchor, since they're easy (q1 = "RAG이 무엇인가?"
is a definitional question). So (a) alone likely doesn't rescue the
axis — only (b) (new hard queries) does.

### Gate before α-5 run

- **Minimum**: (a) annotate path on existing 12 queries → measure
  what the path axis looks like with 17 denominator. If 14+/17 still
  at 1.0, path axis stays dead and α-5 effectively measures 2 axes.
- **Better**: (a) + (b) — add ≥ 3 hard queries. If those bring
  baseline path_recall to ≈ 0.7-0.9, axis is alive.
- **Sequence**: do (a) and (b) BEFORE re-capturing baseline JSON. The
  baseline must be against the v6+ fixture so the matrix Δ is computed
  against the same denominator.

---

## 3. Risk 3 — Workload scoping

### Diagnosis

18 cells × N=3 × ~80s/query × ~17 queries = ~20 h. This assumes the
oracle has signal to detect. If §1 and §2 are not addressed first, the
20 h is wasted on a measurement that mostly registers "zero".

### Adaptive tiering (replaces the original full-run plan)

| Tier | Cells | Cost | Decision rule (graded_answer is the load-bearing axis post-§1/§2) |
|---|---|---|---|
| **T0 — Smoke** | M_M × {L1, L5} only | ~2.2 h | If `|L5 − L1| < 0.05` on graded_answer → STOP. The full stack doesn't move the production oracle; the matrix is null. Save 18 h. |
| **T1 — Production tier isolation** | M_M × {L1, L2, L3, L4, L5} | ~5.5 h | Identifies WHICH layer(s) move the signal at production tier. Outputs per-layer marginal Δ. |
| **T2 — Tier extension** | M_S × all 6 + M_L × all 6 | ~13 h | Only run if T1 found ≥ 1 non-zero layer. Maps tier-gating policy. |

This makes the matrix evidence-bound: **only do the work proportional
to the signal observed in the cheaper cell**.

### Implementation

Driver shipped in #612 already supports `--tiers` and `--rows`
subsetting. T0 invocation: `python scripts/qvt_ablation_matrix.py
--tiers M_M --rows L1,L5`. T1: `--tiers M_M`. T2: `--tiers M_S,M_L`.

### Gate before α-5 run

T0 is the actual α-5 gate. Run T0 first. If T0 null → write a
1-paragraph "α-5 found no layer effect at production tier" note and
close the routing-flag decision PRs as "stay opt-in indefinitely."

---

## 4. Risk 4 — Null result

### Diagnosis

Even with §1 + §2 fixed, the matrix may still produce mostly-null cells.
This is genuinely possible if:
- D5 router decisions never actually re-route the small-tier default
- D1 budget heuristic saturates within noise of fixed cap
- LEO scope routing's chat-mode finding (small effect) doesn't
  translate to retrieval mode

A null result is **informative**, not a failure — but only if
**pre-registered**. Without pre-registration there's risk of ad-hoc
rescue ("the layer wasn't supposed to help on graded_answer, just on
latency") that drains the rigor.

### Pre-registered hypotheses (per layer)

| Layer | Expected direction | Minimum effect size to register positive | Rescue allowed if null |
|---|---|---|---|
| L2 — AUTO_ROUTER (D5) | Δ_graded ≥ +0.05 (router picks better backend per query) | 0.05 on graded_answer (above noise 0.10 requires ≥ 0.10 for "positive"; 0.05-0.10 = "mixed") | No. D5's value claim is "better-backend routing"; if not visible on graded, it has no production value. |
| L3 — ADAPTIVE_BUDGET (D1) | Δ_graded ≥ 0 (design explicitly says "no quality regression, latency only") | n/a — null is the *expected* outcome | Yes. D1 is a latency play; null on graded is acceptance, not failure. But latency must be measured separately (out of scope for α-5). |
| L4 — SCOPE_ROUTING (LEO) | Δ_graded ≥ +0.03 or Δ_path ≥ +0.03 | 0.03 on either axis | No. LEO's value claim is "narrow scope = better retrieval"; if neither moves, it has no production value. |
| L5 — full stack | Δ_graded ≥ +0.07 (additive across layers) | 0.07 on graded_answer | No. Full stack is the upper bound; if even the upper bound is null, the entire routing-layer family is null in production. |

### Decision rules from null

- **L2 null → "AUTO_ROUTER inert in production"** → opt-in
  indefinitely, deprecation PR opened (with `architecture` label).
- **L3 null → "ADAPTIVE_BUDGET works as designed"** → keep opt-in;
  latency measurement is a separate v0.5 cycle.
- **L4 null → "SCOPE_ROUTING inert in production"** → opt-in
  indefinitely, deprecation PR.
- **L5 null → "entire routing family inert in production"** → halt
  routing-track investment, refocus on retrieval-quality / fixture
  improvement.

### Gate before α-5 run

Pre-registration table above stays in this memo. After T0/T1 runs, the
matrix report writer cites this table for each cell's verdict — no
re-interpretation post-hoc.

---

## 5. Action plan (what changes before α-5 actually runs)

| # | Action | Owner | Cost | Status |
|---|---|---|---|---|
| 5.1 | Implement oracle negation guard (§1.b) + at least 5 alias expansions per query (§1.a) | code (Claude) | ~1 h | this PR |
| 5.2 | Annotate `expected_path` on q1, q5–q9 (cheap subset of §2.a) | data (user) | ~20 min | this PR |
| 5.3 | Add 3 hard queries (§2.b) where production baseline path_recall < 0.85 | data (user) | ~1 h | this PR |
| 5.4 | Re-capture α-3 baseline against the v6+ fixture | operator (user, ~70 min compute) | ~70 min | after 5.1–5.3 land |
| 5.5 | Run T0 smoke (§3) — M_M × {L1, L5} | operator (user, ~2.2 h compute) | ~2.2 h | after 5.4 |
| 5.6 | Decide T1 / abort based on T0 graded Δ (§3 rule) | user decision | – | after 5.5 |
| 5.7 | If T1: M_M × all 6 rows (~5.5 h) | operator | ~5.5 h | after 5.6 |
| 5.8 | If T1 found ≥ 1 non-zero: T2 (~13 h) for tier-gating | operator | ~13 h | after 5.7 |

Items 5.1 are code/data changes I can make in this session. 5.2/5.3
need the user (they require domain knowledge about which queries to
annotate / which hard queries to add). 5.4–5.8 are operator compute.

---

## 6. CLAUDE.md alignment

| rule | impact |
|---|---|
| #1 (no domain) | ✅ — platform infra |
| #2 (Quality Delta Card) | n/a — measurement infra. The matrix itself produces Quality Delta Cards for the *next* round of PRs (each routing-flag flip). |
| #4 (architecture changes) | ⚠️ — deprecation PRs from §4 decisions may need `architecture` label. |
| #5 (≤ 20 KB) | ✅ |

---

## 7. Related

- `docs/design/v0.4-qvt-alpha-non-saturating-oracle.md` — parent α design
- `docs/design/v0.4-qvt-alpha-5-ablation-matrix.md` — 18-cell matrix
  definition (#612)
- `eval/qvt/oracle.py` — current scoring impl (§1 target)
- `eval/qvt/baseline_2a31b20.json` — α-3 baseline showing saturation
- `eval/regression/step7_queries.json` v6 — fixture (§2 target)
- PR #611 (A3 v2) — established the "grader keyword-match artifacts"
  failure mode; same risk profile as §1
