# Design Memo — Phase 3b Pre-Work: Cross-Family Refusal Phrase Inventory

> **Status**: design (pre-PR; gated on α-7 closure).
> **Trigger**: 2026-06-01 strategy handover §4 sequencing decision —
> Phase 3b prerequisite. Cross-tier 4-step audit at α-6 close found
> 1 oracle phrase gap (`doesn't link`) at gemma3:12b. Other families
> may have different refusal styles.
> **Cycle position**: parallel-able with α-7 5-tier mode
> re-measurement; runs after α-7 PR #680 merges. Goal: catch oracle
> phrase coverage gaps **before** main Phase 3b measurement runs, so
> cross-family abst_f1 numbers are not corrupted by missed-phrase
> false negatives.
> **작성일**: 2026-06-02 (during α-7 baseline capture)

---

## 0. TL;DR

Before running Phase 3b proper (5 families × n=3 ≈ 45h compute), do
a cheap pre-work pass per family:

1. Send 5-10 null queries to each pure family model (no JAMES context)
2. Inspect responses; identify refusal patterns
3. Compare against current `_ABSTENTION_PHRASES`
4. Add narrow new phrases via bucket-(d) PR (same shape as α-7's
   `doesn't link` add)
5. Re-run α-6 cells if any tier's abst_f1 shifts measurably

**Compute estimate**: 5-10 queries × 5 families × ~10-60s per query
≈ **~30-60 min total Ollama compute** + ~30-60 min analysis. The
expensive part is human inspection; the bench part is cheap.

**Why this prevents wrong-fix-averted #10**: without phrase coverage
calibration, Phase 3b cross-family abst_f1 numbers could read 0 across
all 5 families and the "non-gemma families inert" finding would be
fully measurement-side artifact. α-5 #619 (gemma4:e4b grounding
refusals) and α-7's `doesn't link` (gemma3:12b) are the precedents.

---

## 1. The 5 families

Already downloaded locally as of α-6 close (per
`ollama list` 2026-06-01):

| Tier label (proposed) | Model | Size | Family |
|---|---|---|---|
| **Q_S** | `qwen2.5:7b` | 4.7 GB | Qwen 2.5 |
| **L_S** | `llama3.1:8b` | 4.9 GB | Llama 3.1 |
| **D_M** | `deepseek-v2:16b` | 8.9 GB | DeepSeek V2 |
| **G2_XS** | `gemma2:2b` | 1.6 GB | Gemma 2 (older Gemma arch) |
| **Q_CS** | `qwen2.5-coder:7b` | 4.7 GB | Qwen Coder (fine-tune of base) |

Plus reference (already measured in α-6):
- Gemma 3 family: M_XS / M_S / M_L / M_XL (1b/4b/12b/27b)
- Gemma 4 family: M_M (e4b)

### 1.1 Why these 5 specifically

- **Architecture diversity**: Qwen / Llama / DeepSeek / Gemma 2 / Qwen-Coder cover 4 distinct base architectures + 1 instruction-tuned variant
- **Size diversity within available**: 2b / 7b / 8b / 16b
- **Comparable size band to gemma3:4b ~ 12b**: covers the
  "small-to-medium" inflection zone where α-6 found mode transitions
- **All locally available**: zero download cost; immediate launch

Models **NOT** in Phase 3b minimal scope (deferred):
- `gemma3:1b/4b/12b/27b` (already in α-6)
- `gemma4:e4b` (already in α-6)
- API-only models (Claude / GPT / Gemini) — deferred to v0.5+
- Coder-only base models (qwen2.5-coder:32b is too large for laptop GPU)

---

## 2. Tier label convention (composite `<family>_<size>`)

`qvt_ablation_matrix.py::_TIER_MODELS` currently has:
```python
{"M_XS": "gemma3:1b", "M_S": "gemma3:4b", "M_M": "gemma4:e4b",
 "M_L": "gemma3:12b", "M_XL": "gemma3:27b"}
```

Phase 3b adds:
```python
"Q_S":   "qwen2.5:7b",          # Qwen Small
"L_S":   "llama3.1:8b",         # Llama Small
"D_M":   "deepseek-v2:16b",     # DeepSeek Medium
"G2_XS": "gemma2:2b",           # Gemma 2 Extra-Small (older Gemma)
"Q_CS":  "qwen2.5-coder:7b",    # Qwen Coder Small
```

Cell JSON filenames stay distinct:
- `qvt-ablation-cell-C_minus-Q_S.json`
- `qvt-ablation-cell-C_rag-full-L_S.json`
- etc.

No collision with existing M_XS/M_S/M_M/M_L/M_XL.

**Update sequencing**: this `_TIER_MODELS` extension lands as part of
the Phase 3b prep PR alongside the bucket-(d) phrase adds. Single
small PR per α-5/α-6 cycle precedent.

---

## 3. Per-family inventory procedure

For each family, in order of inspection cost (lowest first):

### 3.1 Step A — pure-model null-query probe

Send 5 null queries from the multihop_rag balanced-100 fixture (the
ones starting with id 51-75 are the `null_query` type). Direct
`ollama run` invocation (skip JAMES; we want pure-model behaviour
isolated from any retrieval/grounding context):

```powershell
# Example for Q_S = qwen2.5:7b
$queries = Get-Content -Raw workspaces/hotpot_eval/eval/multihop_rag_queries.json |
    ConvertFrom-Json | % { $_.queries } |
    ? { $_.question_type -eq 'null_query' } | Select-Object -First 5

foreach ($q in $queries) {
    Write-Host "Q$($q.id): $($q.text.Substring(0, 100))..."
    $q.text | ollama run qwen2.5:7b
    Write-Host "---"
}
```

ETA: ~5 queries × ~30-60s = ~3-5 min per family × 5 families = **~25 min**.

### 3.2 Step B — refusal-pattern inventory

For each response, manually classify:
- 🟢 **Refusal-shape** (model recognises insufficient info)
  - Extract the refusal phrase verbatim
  - Note its position (early/late in response)
- 🔴 **Hallucination** (model fabricates an answer)
  - Note any false confidence markers

Per family, output a 5-row table:

```markdown
### Q_S (qwen2.5:7b)
| id | classification | phrase / hallucination marker | already in oracle? |
|---|---|---|---|
| 51 | hallucination | "Sridevi's role was..." | n/a |
| 52 | refusal | "I don't have information on..." | ✅ (oracle has "no information") |
| 53 | refusal | "based on the given data, I cannot determine" | ✅ (oracle has "cannot determine") |
| 54 | refusal | "the provided context does not include" | ❌ (proposal: add) |
| 55 | hallucination | "Egypt's GDP rise..." | n/a |
```

### 3.3 Step C — narrow phrase proposal

For each "❌ not in oracle" entry, decide:
1. Is the phrase narrow enough (appears ONLY in unhedged refusal)?
2. Or could it FP-flood partial-answer rows (`"the context does not
   include exact dates, but..."`)?

Narrow phrases pass; broader ones get rejected per α-5 #619 lesson.

### 3.4 Step D — synthesis into bucket-(d) PR

Single PR adds all collected narrow phrases to
`eval/qvt/oracle.py::_ABSTENTION_PHRASES`. Same shape as α-7's
`doesn't link` add. Quality delta: exempt (label: fix —
measurement-side debt fix).

---

## 4. Expected output deliverables

After pre-work runs:

1. **`reports/research-runs/phase-3b-pre-work-refusal-inventory-<date>.md`**
   — 5 family tables + synthesis. Sealed artifact for audit trail.
2. **PR `fix(phase-3b): bucket-(d) oracle phrase add — cross-family refusal styles`**
   — adds 5-15 narrow phrases (estimate; could be fewer if families
   share styles). One bench rescore on existing α-6 cells to confirm
   no FP-flood on present-truth queries.
3. **PR `feat(phase-3b): _TIER_MODELS extension for cross-family tiers`**
   — adds Q_S / L_S / D_M / G2_XS / Q_CS to the matrix runner. Same
   shape as PR #677 (M_XS / M_XL add).
4. **(optional) Recovery curve doc annotation** — if any pre-work
   phrase add shifts existing α-6 tier numbers measurably, annotate
   in the recovery curve doc § 5 withdrawn-claims registry.

---

## 5. Compute + time budget

| Step | Compute | Wall-clock |
|---|---|---|
| Step A — 5 family × 5 queries × ~30s | ~12 min Ollama | ~30 min (sequential per family + inspection) |
| Step B — manual classification | none | ~30-60 min |
| Step C — phrase proposal review | none | ~15 min |
| Step D — bucket-(d) PR draft + review | minimal | ~30 min |
| Bench rescore on α-6 cells | minimal (rescore not re-bench) | ~5 min |
| **Total** | **~12 min compute** | **~2-3 hours wall-clock** |

This is **parallel-able with α-7 5-tier mode re-measurement** —
the mode re-measurement uses Gemma family (M_XS / M_S / M_M / M_L /
M_XL); the pre-work uses Qwen / Llama / DeepSeek / Gemma2 / Qwen-Coder.
Ollama can serve one model at a time; sequence them so each finishes
before the next starts. (Two background Ollama queues would VRAM-thrash.)

Sequencing if running in parallel-shape (= time-sliced, not literal parallel):
- α-7 5-tier remeasurement: ~8-10h (Gemma models)
- Phase 3b pre-work: ~30 min Ollama + ~2h analysis (other families)

Total: still ~8-10h, but the pre-work analysis time overlaps with
Gemma remeasurement compute time → no critical-path extension.

---

## 6. Risks + mitigations

| Risk | Probability | Mitigation |
|---|---|---|
| Family refusal styles vary too much for narrow phrases → broad phrase needed | medium | Step C narrow/broad decision per phrase; broader phrases deferred to LLM-judge variant (#632 design) |
| Pre-work bench answers differ from Phase 3b proper bench answers (different timestamp / random seed) | medium | use deterministic seed where possible; flag any new phrases caught in proper but missed in pre-work as bucket-(d) follow-up |
| Family-specific phrase add causes FP-flood at existing Gemma tiers | medium | re-score existing α-6 cells with new phrases; if any cell's TP count rises on **present-truth** queries (FP), reject the phrase |
| 2-3h "parallel" wall-clock turns into critical path because operator-side analysis | low | analysis can extend into α-7 5-tier re-measurement compute window without slowing total cycle |
| New tier labels (Q_S / L_S / etc.) collide with future planned tiers | low | reserved letters Q/L/D/G2/Q_C are domain-distinct (initials of company / model brand) |

---

## 7. Decisions to confirm before launch

| # | Decision | Recommendation | Owner |
|---|---|---|---|
| 1 | Run pre-work as 1 PR (combined phrase-add + tier-extension) or 2 PRs? | 2 PRs — separate concerns | operator |
| 2 | Pre-work compute starts immediately after α-7 PR merge, or after α-7 5-tier re-measurement? | immediately after merge; parallel-shape with re-measurement | recommend immediate |
| 3 | Include Korean refusal phrases (none of the 5 families train Korean primarily)? | yes — non-zero chance Qwen / Llama emit Korean refusal in cross-lingual queries; cost is small | recommend yes |
| 4 | Pre-work output doc goes into `reports/research-runs/` or `docs/handovers/`? | `reports/research-runs/` (measurement artifact, not session handover) | recommend research-runs |
| 5 | If Step B classifies a query as "refusal but with hedge" (partial-answer-shape), what bucket? | label as "hedge" + skip from phrase proposal (would FP-flood); flag for #632 LLM-judge follow-up | recommend skip + log |

---

## 8. Cycle dependencies

### 8.1 Predecessor

- **α-7 PR #680 must merge first** — pre-work uses the updated oracle
  phrase list as starting point + the α-7 baseline as reference for
  any re-score.

### 8.2 Successor

- **Phase 3b proper measurement** — gated on pre-work completion + any
  bucket-(d) PR landing first. Uses the new tier labels Q_S / L_S /
  D_M / G2_XS / Q_CS.

### 8.3 Parallel

- **α-7 5-tier mode re-measurement** — same wall-clock window, different
  Ollama models (so they serialize naturally). No conflict.
- **T3 Evidence Aging** — orthogonal cycle; can land anywhere.

---

## 9. Out of scope (this PR / cycle)

- API-only models (Claude / GPT / Gemini) — deferred to v0.5+ (network
  costs + non-local prohibits the local-first frame)
- Coder-only large models (qwen2.5-coder:32b — too large for consumer
  GPU even with CPU split)
- Korean-corpus version of MultiHop-RAG — backlog handover §7.5 open
  question; v0.5+
- Cross-fixture sanity (HotpotQA / MuSiQue) — S4 universal-law
  promotion prereq; after Phase 3b
- LLM-judge bucket-(c) — #632 design memo; orthogonal

---

## 10. References

- α-6 cycle PR index: `reports/research-runs/alpha-6-cycle-pr-index.md`
- α-7 bucket-(d) sub-finding (precedent): `reports/research-runs/alpha-7-bucket-d-oracle-phrase-gap.md`
- α-7 closure analysis: `reports/research-runs/alpha-7-closure-analysis-20260602.md`
- 4-step audit script (re-usable): `scripts/research/audit_12b_null_query_refusal_shape.py`
- α-5 #619 narrow phrase add precedent: commit history
- Honest framing rule: `memory/feedback_finding_size_honest_framing.md`
- Mode context-dependence: `memory/feedback_james_mode_taxonomy_context_dependent.md`
- P3 sequencing: `memory/project_alpha_7_alpha_8_ontology_track_sequencing.md`
- Oracle phrase list: `eval/qvt/oracle.py:_ABSTENTION_PHRASES`
- Matrix runner: `scripts/qvt_ablation_matrix.py`
- α-6 design memo (matrix shape): `docs/design/v0.4-alpha-6-sector-llm-ablation-matrix.md`
