# Cycle γ — multi-hop arc DEFINITIVE closure: evidence-selection is the wall

**Date**: 2026-06-10 (closes the entire MuSiQue multi-hop investigation)
**Status**: Multi-hop arc **definitively closed**. Six measurements
explored every layer (retrieval / rerank / decomposition / iterative /
synth-context / evidence-selection). All real-system interventions land
at ≤12% vs the oracle's 72%. The binding constraint is **identifying
which retrieved candidates are the supporting paragraphs without knowing
the gold** — and that is hard by nature, not a JAMES defect.

---

## 1. The closing measurement (D3 evidence selection)

Pre-registered (`58b14ee`, before impl). Reuse the decomposer to SELECT
evidence (top docs per sub-question → small clean context), the layer
D1/D1b had backwards.

| | gold-in-answer | abstain | selection supporting recall | both |
|---|---|---|---|---|
| R0 (rerank top-5) | 8% | 19/25 | 0.600 | 5/25 |
| **D3 (evidence-select)** | **8%** | 17/25 | **0.540** | 3/25 |
| oracle (gold paras) | 72% | 6/25 | 1.000 | 25/25 |

**§3 verdict: INSUFFICIENT** (gold 8%, Δ +0). The diagnostic sub-rule
fires: **selection supporting recall is LOW (0.54)** — the per-sub-q
selection did not surface the supporting set. The model never got the
clean context the whole approach depended on.

---

## 2. The full multi-hop chain — six measurements, one wall

| Measurement | gold | What it ruled out / showed |
|---|---|---|
| breadth (top_k 8→16) | 8% | not search breadth |
| **oracle (gold paras)** | **72%** | **model CAN do 2-hop — but only knowing the gold** |
| D1 static decompose | 12% | not query decomposition (hop-2 anaphora) |
| D1b iterative self-ask | 12% | not iterative bridging (extract imprecise) |
| D2 rerank-OFF + wide | 12% | not rerank (synth drowns in distractors) |
| **D3 evidence-select** | **8%** | **selection can't find supporting (recall 0.54)** |

Plus the mid-cycle correction: retrieve(top-8) supporting recall 0.76 —
retrieval was never the bottleneck (Phase C.2 sources-top-3 artifact).

### The wall, stated precisely

```
retrieve(top-8)  → supporting IN candidates: recall 0.76  (it's there)
   ↓ but which 2 of the 8 are the supporting paragraphs?
   rerank (original query)     → recall 0.66, misses query-dissimilar hop-2
   sub-q selection (D3)        → recall 0.54, sub-q top-k ≠ supporting
   oracle (knows the gold)     → recall 1.00 → 72%
```

The 60pp gap between oracle (72%) and every real method (≤12%) is the
cost of **not knowing which candidates are the supporting paragraphs**.
Retrieval puts them in the pool; no unsupervised selector reliably picks
them out. That is the multi-hop evidence-selection problem — a genuine
research-grade difficulty, well-documented in the literature (dense
retrieval struggles with hop-2 query-dissimilarity; this is why
iterative/graph methods exist).

---

## 3. What this means

- **JAMES's multi-hop weakness is NOT retrieval, synth, or rerank
  individually.** It is unsupervised supporting-paragraph selection.
- **oracle 72% is an upper bound that assumes gold knowledge** — not a
  target a real system reaches by tuning these layers.
- Six interventions, all default-OFF byte-identical, all honest nulls.
  The infrastructure (decomposer / iterative resolver / evidence
  selector / extra_queries / retrieve+rerank env-gates) is preserved for
  reuse but **none flips the default** — none earned it.

---

## 4. Next — out of the retrieval/selection rabbit hole

The arc is closed. Productive directions (each its own pre-registration,
none a continuation of "tune the selector"):

1. **Graph traversal for hop-2** — JAMES has a graph layer. Multi-hop's
   query-dissimilar hop-2 is exactly what entity-link traversal (hop-1
   answer → graph edge → hop-2 paragraph) is for. This is the
   literature's answer (GraphRAG / LazyGraphRAG, review R7) and the one
   layer this cycle did NOT test. Compare against LazyGraphRAG baseline.
2. **Accept the finding and re-point the cycle** — multi-hop graded
   answer is a known-hard external axis; JAMES's contribution is the
   **reproducible measurement + the per-query overlap method**, not
   beating MuSiQue. Pivot to the moat axis (review R1: replayable audit /
   provenance benchmarks) or platform hygiene (R3/R4), per the
   3-track options.
3. **Synth-side abstention tuning** — connect to Phase E-min: the
   model's abstention on noisy multi-hop context is the same synth
   behaviour. But this is tuning a known-hard axis; weigh against (1)/(2).

Recommendation: **(2) or (1)** — stop tuning selectors (diminishing,
fitting-prone), either bank the honest finding and move to the moat
(R1) / hygiene (R3/R4), or make ONE graph-traversal attempt (the
untested layer) before declaring multi-hop closed for the platform.

---

## 5. Artifacts

| Purpose | Path |
|---|---|
| D3 pre-registration | `docs/research/cycle-gamma-d3-evidence-selection-preregistration-2026-06-10.md` |
| Evidence selector | `core/retrieval/evidence_selector.py` |
| Wiring | `core/reasoning/pipeline_loops.py` |
| All multi-hop JSONs | `reports/cycle_gamma/phase-c2/musique-ans-mxtral-{R0,R0-topk16,ORACLE,D1,D1b,D2-rerankoff,D3-evselect}.json` |
| Retrieval-arc closure (D2) | `docs/handovers/v0.4-cycle-gamma-retrieval-arc-closure-synth-is-the-lever-2026-06-10.md` |

### Carry-forward
- Six pre-registered measurements, six honest nulls, one self-correction
  (Phase C.2). The discipline held end-to-end.
- The decomposer was used three ways (retrieval D1, iterative D1b,
  selection D3) — all default-OFF, all preserved, none promoted.
- v0.5 reframing: "improve multi-hop" is not a retrieval/synth tuning
  task — it needs graph traversal or is a known-hard axis to measure-not-
  beat. Don't let a future session re-open the selector rabbit hole
  without graph traversal or a fundamentally different mechanism.
