# Cycle γ Phase C.4 — 2WikiMultiHopQA smoke (dev, N=20) handover (2026-06-11)

> Companion to the locked pre-registration
> ([`docs/research/cycle-gamma-c4-2wiki-smoke-preregistration-2026-06-11.md`](../research/cycle-gamma-c4-2wiki-smoke-preregistration-2026-06-11.md),
> committed before any download script, smoke driver, or measurement).
> The single most important reading: **infrastructure validates ✓**,
> and the closed-corpus producer reproduces the MuSiQue Phase C.2
> multi-hop floor pattern (single-shot dense / direct injection
> doesn't crack the 2nd hop without supervision). Pre-registration
> §1.3 first row: **⭐ infrastructure validated + cross-bench multi-hop
> floor reproduced**.

## ⭐ Honest tier — locked landing

| Pre-registered result | Required evidence | Met? |
|---|---|---|
| **⭐ infrastructure + multi-hop floor cross-bench reproduced** | em/f1 emit + magnitude in known floor (em < 0.2, f1 < 0.3) | **✓ — landing here** |
| ⭐⭐ surprising magnitude (em > 0.3 or f1 > 0.5) | n/a |
| exploratory | exception / no axes | n/a (0 / 20 errors) |

Pre-registration §1.3 was explicit: the smoke does not promise
publication-grade magnitudes. It promises that the plumbing works
and that the *cross-bench floor pattern* found on MuSiQue (Phase
C.2) shows up on a different multi-hop fixture, in the right shape.
That promise is met.

## Measurement record

```
spec           Pre-registered protocol (commit cf935b2)
fixture        eval/external/_fixtures/wikimulti/dev.json
fixture_sha    (recorded in result.json; >32 MB sha256)
loader         WikiMultiLoader(split='dev')  — 12,576 dev queries total
producer       closed-corpus-gemma (Ollama, gemma4:e4b, all 10 paras)
scorer         WikiMultiScorer  (em / f1 / f1_by_type;
                                  support_fact_f1 = "not measured")
n_samples      20  (pre-registration §1.1 locked)
n_errors       0
elapsed        21.6 s  (~1 s / query)
producer_mode  closed-corpus — 2Wiki's published 10 paragraphs as-is;
               JAMES retrieval NOT used per §1.2 self-eval-trap rule
```

## Axes

| Axis | Score | n_queries | meaning |
|---|---|---|---|
| **em** | **0.1000** | 20 | 2 / 20 exact matches after SQuAD normalisation |
| **f1** | **0.1637** | 20 | token-level F1 (SQuAD-normalised), max over gold + aliases |
| **f1_by_type** | **0.2213** | 14 | mean of per-type F1 means |
| **support_fact_f1** | **(not measured)** | **0** | producer doesn't emit `predicted_supporting_facts`; scorer surfaces this honestly per pre-reg §1.2 — D-2wiki cycle gates this axis |

## Distribution shape

```
"Insufficient Information" abstain count : 11 / 20  (55%)
Non-abstain rows                          :  9 / 20  (45%)
Per-type f1 (n=14, smoke; magnitude per-type framing forbidden by
pre-reg §1.2 — recorded here as distribution sanity only):
  comparison              0.4815   (multiple in sample; the "easy"
                                    multi-hop type — direct binary
                                    relation)
  compositional           0.1111   (hop1 → hop2 → answer)
  inference               0.0714   (relational inference)
  bridge-comparison       (not in sample of 20)
```

The abstain count (11 / 20) is the headline structural finding: the
LLM, given all the gold paragraphs but no supervision, chooses
"Insufficient Information" most of the time rather than chaining
hop-2 supporting facts together. This is the *same* finding the
MuSiQue Phase C.2 measurement closed the multi-hop arc on
([`project_cycle_gamma_phase_c2_retrieval_bottleneck`](../../../../Users/karu-/.claude/projects/C--Project-James-RAG-Evol-v010/memory/project_cycle_gamma_phase_c2_retrieval_bottleneck.md);
PR #756): the wall is *unsupervised supporting-paragraph selection*,
not retrieval, not breadth, not model capability (oracle gives 72%
on MuSiQue).

The per-type signature is consistent with the floor reading: the
`comparison` type (binary relation between two named entities) is
the one the LLM can answer from the context block; `compositional`
and `inference` (which require chaining facts across paragraphs) are
where the floor sits. **N=20 is too small to do statistics on the
type-level magnitudes** — the pre-reg locks the type-magnitude
interpretation out (§1.2). The shape is the finding, not the numbers.

## What this validates

1. **Loader + scorer wiring works.** `WikiMultiLoader(split='dev')`
   pulls the 12,576-row dev split, slices `n_samples=20`
   deterministically; `WikiMultiScorer` emits em / f1 / f1_by_type;
   `support_fact_f1` honestly surfaces as "not measured" rather than
   silently scoring 0.
2. **Closed-corpus producer reuses cleanly across multi-hop
   benches.** The same `ClosedCorpusGemmaProducer` that ran Phase
   B/C MuSiQue (and Phase E-min RGB) drives 2Wiki without
   modification — no per-bench adapter code needed for this tier.
3. **Cross-bench multi-hop floor reproduces.** The "Insufficient
   Information" abstain pattern that dominated MuSiQue's R0
   (abstain 76%) shows up on 2Wiki dev at 55% in this smoke,
   with the same per-question-type signature (comparison easier,
   compositional/inference floor).
4. **Per-call latency is tractable.** ~1 s / query for `gemma4:e4b`
   on 10 paragraphs; future full N=1000 single-model ≈ 17 min.

## What this does NOT validate (per pre-registration §1.2)

- **NOT publication-tier magnitudes.** N=20 is the smoke gate; the
  pre-reg locks magnitude framing (any type / any axis) out of this
  cycle.
- **NOT support-fact prediction.** The closed-corpus producer
  doesn't emit `predicted_supporting_facts`. A JAMES adapter that
  does is the D-2wiki cycle, with its own pre-reg.
- **NOT JAMES retrieval / engine.** Closed-corpus producer uses
  2Wiki's published 10 paragraphs; JAMES retrieval is out of scope.
- **NOT cross-model.** Only `gemma4:e4b` ran.
- **NOT direct MuSiQue magnitude comparison.** The two benches have
  different question-type compositions (2Wiki has explicit
  `bridge-comparison`, MuSiQue 2-hop / 3-hop / 4-hop split); the
  cross-bench claim is the *qualitative floor pattern only*, not
  the numbers.

## Synergy with the multi-hop arc closure (Phase C.2)

Phase C.2 closed the JAMES multi-hop arc with 7 probes (R0 / breadth
/ oracle / D1 / D1b / D2 / D3 + D4 graph) on MuSiQue. The final
reading: "the wall is unsupervised supporting-paragraph selection;
retrieval is sufficient (top-8 supporting recall 0.76); oracle (gold
paragraphs) gives 72% so the model can do the reasoning when handed
the right supporting set; *no* method (decompose / iterative /
rerank-off / evidence-select) closes the 60pp gap from `oracle` to
real methods."

This 2Wiki C.4 smoke is the cross-bench reproduction of the same
floor. The same pattern (abstain-default in the absence of
supervision; comparison-type easier than compositional/inference; no
LLM-side fix without supporting-fact prediction) appears in the
smoke. **Cross-bench evidence at smoke tier; D-2wiki cycle gates
the publication-tier confirmation.**

## Re-verification

```bash
git clone https://github.com/Hashevolution/James-RAG-Evol
cd James-RAG-Evol && git checkout <merge-commit>
python scripts/research/download_2wiki.py --split dev
python scripts/research/wiki2_smoke_run.py --n 20 --model gemma4:e4b
# → reports/external/2wiki/dev-smoke-<ts>.{result.json,bench.jsonl}
```

The smoke is deterministic given the same model checkpoint; run-to-
run variance comes only from the LLM's sampling.

## Closes / opens

**Closes**:
- Cycle γ Phase C.4 pre-registration §4 steps 1–3 (download →
  smoke driver → smoke run with locked scope).
- The cycle γ design memo's 4-bench promise is now **4/4 measured**:
  RGB ✓, MuSiQue ✓, ALCE ⭐ smoke ✓, **2Wiki ⭐ smoke ✓**.
  (Publication-tier on ALCE + 2Wiki = D-alce / D-2wiki cycles,
  separate pre-regs.)
- The multi-hop arc closure's cross-bench evidence is in. The
  Phase C.2 finding (unsupervised supporting-paragraph selection
  as the wall) is no longer single-bench.

**Opens**:
- **D-2wiki cycle**: JAMES adapter that emits
  `predicted_supporting_facts`, full N=1000+ dev, cross-model,
  publication-tier honest-tier ladder. Separate pre-reg.
- **D-alce cycle** (from C.3): real NLI verifier, qampari + eli5
  variants, full N. Separate pre-reg.
- **Paper v1.4 §"Multi-hop QA cross-bench reproduce"**: gated on
  D-2wiki, NOT on this smoke. MuSiQue + 2Wiki at publication tier.

## Pointers

- Pre-registration (locked): `docs/research/cycle-gamma-c4-2wiki-smoke-preregistration-2026-06-11.md`
- Cycle γ design memo: `docs/design/v0.4-cycle-gamma-external-benchmark-integration.md`
- Phase C.2 MuSiQue arc closure: `docs/handovers/v0.4-cycle-gamma-phase-c2-musique-retrieval-bottleneck-2026-06-10.md`
- Phase C.3 ALCE smoke: `docs/handovers/v0.4-cycle-gamma-phase-c3-alce-smoke-2026-06-11.md` (sister smoke)
- 2WikiMultiHopQA upstream: https://github.com/Alab-NII/2wikimultihop (Ho et al., COLING 2020)
- Re-verification artifacts: `reports/external/2wiki/dev-smoke-20260611T071605Z.{result.json,bench.jsonl}`

## Related rules

- [[project_cycle_gamma_phase_c2_retrieval_bottleneck]] — MuSiQue
  Phase C.2 closure that this smoke cross-bench reproduces
- [[project_cycle_gamma_phase_c3_alce_smoke]] — ALCE C.3 sister
  smoke (same closed-corpus pattern, different axis)
- [[feedback_self_evaluation_trap]] — closed-corpus producer with
  published retrieval as-is is the positive pattern
- [[feedback_finding_size_honest_framing]] — ⭐ infrastructure tier
  vocabulary
- [[feedback_n1_verdict_inflation_n3_caught]] — N=20 smoke is below
  publication-tier n-paired rule by design
- [[feedback_evidence_grounded_validity_check]] — abstain-pattern
  distribution as the *shape* finding (not the numbers)
- [[mechanism_layer_intent_axis_alignment]] — support_fact_f1's
  intent axis = supporting-fact prediction, which is in the producer
  not the scorer; the smoke producer doesn't emit, so the axis is
  honestly "not measured"
- [[feedback_eval_cycle_vs_collab_arc_separation]] — internal cycle
  evidence, not joint piece linkage
