# Cycle γ Phase C.2 — MuSiQue paired ablation → retrieval bottleneck

**Date**: 2026-06-10
**Status**: Phase C.2 **closed as honest negative + diagnostic finding**.
The pre-registered tradeoff measurement (hyp2) could NOT be run — MuSiQue
R0 floored — but the floor diagnosis produced a **stronger** result: a
decisive, externally-grounded confirmation that **retrieval (not the
component knobs, not the model) is the dominant multi-hop bottleneck**.

---

## 1. Headline

Pre-registration (`docs/research/cycle-gamma-phase-c2-preregistration-2026-06-10.md`)
set out to measure the GAIN half of the rerank / cognitive_stages
tradeoff on MuSiQue-ans (n=25, 3 models). The R0 cell-validity gate
(§5) caught a floor and we entered diagnostic mode. Three measurements
**isolated the cause**:

| Measurement | Result | What it rules out |
|---|---|---|
| R0 (top_k 8/5, full JAMES) | em=0, f1≈0.06, **abstain 76%** | — (floor observed) |
| R0 (top_k 16/10, env-gated) | em=0, f1≈0.06, **abstain 80%** | retrieval **breadth** ❌ (2× wider = no change) |
| **oracle (supporting paras only)** | **gold-in-answer 72%**, abstain 24% | model **ceiling** ❌ (model solves with evidence) |

**Conclusion: JAMES's single-shot dense retrieval fails to surface the
2nd-hop supporting paragraph. That is the floor. Not breadth, not the
model, not rerank/cog_stages.**

---

## 2. The measurement chain (logical order)

### 2.1 Setup
- Data: MuSiQue-ans dev, downloaded from HF mirror `dgslibisey/MuSiQue`
  (2417 rows) via new `scripts/research/download_musique.py`.
- Workspace: `workspaces/cycle_gamma_musique_ans`, first-25 rows
  ingested per-paragraph (`musique-ans-<rowid>-p<idx>`), 500 paragraphs.
- All 25 queries are **2-hop**.

### 2.2 R0 cell-validity gate (pre-registration §5)
- 3 models (mixtral:8x7b / gemma4:e4b / llama3.1:8b), n=25 each.
- f1 = 0.060 / 0.085 / 0.071 — **barely** over the §5 "≤ 0.05 on all 3 =
  wiring suspicion" line, but **em = 0.000 on all 75 queries** forced a
  diagnostic read rather than a pass.
- mxtral raw-answer inspection: **19/25 abstain** ("insufficient
  information"), answers explicitly say *"context states A but does not
  provide B"* → 2nd hop evidence missing.
- Retrieval coverage of gold supporting paragraphs (sources top-3
  shown): **0.60**; both-supporting-caught only **5/25**.

### 2.3 Breadth test (env-gate, exploratory)
- New env-gate `JAMES_RETRIEVE_TOP_K` / `JAMES_RERANK_TOP_K` in
  `core/reasoning/pipeline_loops.py`. **Default 8/5 byte-identical**
  (verified: unset → 8/5; malformed/negative → default).
- mxtral R0 at 16/10: abstain **80%** (vs 76%), f1 0.057 (vs 0.060),
  em 0. **Widening retrieval 2× changed nothing** → breadth is not the
  cause.

### 2.4 Oracle-retrieval test (the decider)
- `scripts/research/_phase_c2_oracle_diag.py`: feed each query ONLY its
  gold supporting paragraphs (perfect evidence, retrieval bypassed),
  score em/f1 + abstention.
- mxtral, n=25: **gold-substring-in-answer 18/25 (72%)** vs R0's 2/25
  (8%) — a **9× jump**. abstain 6/25 (24%) vs 76%.
- → The model solves the 2-hop **once it has the evidence**. The gap is
  retrieval, not the LLM.

---

## 3. What this means

### 3.1 Pre-registered hyp2 (tradeoff) — not measurable here
The rerank/cognitive_stages GAIN cannot be measured on this fixture:
`cognitive_stages` (multi-step reasoning) needs the evidence delivered
to fire, and 80% of queries never get the 2nd-hop supporting paragraph.
Per `mechanism_layer_intent_axis_alignment`, the layer is **"not in
evidence"** — running the 6 ablation cells on the floor would measure
noise. This is an **honest negative**, and the pre-registration §5 gate
worked exactly as designed (it stopped us before the knob runs).

The RGB-en LOSS half (Phase E-min cross-model) stays locked but
**unpaired** — the MuSiQue GAIN half is structurally unavailable until
retrieval can surface multi-hop evidence. **No cycle β default revision
is licensed** (unchanged from the Phase E handover).

### 3.2 The real finding — retrieval is the dominant multi-hop lever
Three independent measurements converge:
- breadth 2× → no change (not a top_k problem)
- oracle evidence → 9× gold-in-answer jump (not a model problem)
- ∴ **single-shot dense retrieval structurally misses the 2nd hop**.
  MuSiQue's 2-hop shape ("Who is the spouse of the Green performer?" →
  hop1 Green→Steve Hillage is query-similar; hop2 Steve
  Hillage→spouse is NOT query-similar) is exactly the case single-shot
  retrieval cannot reach.

This **externally confirms Phase D's "retrieval = dominant lever
(−0.173)"** on a published standard benchmark with an official scorer
(passes the self-eval-trap bar). Honest tier: ⭐⭐ (externally
confirmed, single model on the oracle leg — gemma4/llama oracle
replication would lift to cross-model).

### 3.3 Secondary — em=0 is a response-style artifact
Even with oracle evidence em=0 while gold-in-answer=72%: the model knows
the answer but replies in NATURAL prose (173–410 chars), so SQuAD-style
EM/F1 collapse. This re-confirms cycle β's "short-answer penalty" — a
terse response_style would be needed for EM-graded short-answer benches.
**Not** a JAMES defect; a measurement-mode mismatch.

---

## 4. Artifacts

| Purpose | Path |
|---|---|
| Pre-registration (decision thresholds, LOCKED) | `docs/research/cycle-gamma-phase-c2-preregistration-2026-06-10.md` |
| MuSiQue downloader (HF mirror → official JSONL) | `scripts/research/download_musique.py` |
| Corpus build (per-paragraph ingest) | `scripts/research/cycle_gamma_musique_corpus_build.py` |
| R0 gate runner | `scripts/research/_phase_c2_r0_gate.sh` |
| Oracle diagnostic | `scripts/research/_phase_c2_oracle_diag.py` |
| Verdict combiner (RGB LOSS + MuSiQue GAIN) | `scripts/research/_phase_c2_verdict.py` (unused — GAIN unmeasurable) |
| retrieval/rerank env-gate | `core/reasoning/pipeline_loops.py` (`JAMES_RETRIEVE_TOP_K` / `JAMES_RERANK_TOP_K`) |
| Result JSONs | `reports/cycle_gamma/phase-c2/musique-ans-mxtral-{R0,R0-topk16,ORACLE}.json` + gemma4/llama R0 |

---

## 5. Next session — D1 multi-hop-aware retrieval

The lever is now identified and externally grounded. The next work is
**D1: make retrieval reach the 2nd hop** — single-shot dense retrieval
is the wrong tool for multi-hop. Candidate approaches (all local-first,
no new cloud dep):
- **iterative / multi-step retrieval** (retrieve → read → re-query with
  hop1 answer → retrieve hop2). JAMES already has a query-rewrite layer
  (`core/reasoning/pipeline_query_expansion.py`) — the question is
  whether an answer-conditioned 2nd retrieval round lands hop2.
- **query decomposition** (MuSiQue's own `question_decomposition` field
  shows the 2 sub-questions — a decomposer that splits then retrieves
  each sub-q is the textbook fix).
- Compare against **LazyGraphRAG / iterative-RAG** baselines (review R7)
  before committing build effort.

Measurement design for D1 (pre-register again): R0 (single-shot) vs
D1 (iterative/decomposed) on the SAME 25 MuSiQue queries, primary axis
= **gold-substring-in-answer** (em/f1 are response-style-confounded —
use the substring metric or add terse mode). Target: lift R0's 8% toward
the oracle ceiling of 72%.

Also worth pairing: terse response_style on MuSiQue so em/f1 become
usable (secondary, cycle β follow-up).

### Carry-forward rules
- env-gate default 8/5 stays until a multi-axis measurement (MuSiQue
  gain vs RGB abstention loss) licenses a flip — top_k↑ pulls more noise
  docs into abstention-prone queries (the same tradeoff this whole cycle
  studies).
- `feedback_evidence_grounded_validity_check` (oracle test = the clean
  separator of evidence-delivery vs model-capability) — reuse this
  pattern whenever a RAG floor appears.
- Honest framing held: hyp2 reported as **unmeasurable**, not as a
  fabricated verdict.
