# Cycle γ Phase C.3 — ALCE smoke (asqa, N=20) handover (2026-06-11)

> Companion to the locked pre-registration
> ([`docs/research/cycle-gamma-c3-alce-smoke-preregistration-2026-06-11.md`](../research/cycle-gamma-c3-alce-smoke-preregistration-2026-06-11.md),
> committed before any download, post-processor, or measurement). The
> single most important reading of this handover: **the infrastructure
> validates ✓**, the citation axes both emit, and the magnitude
> numbers are **NOT ALCE-grade** (StringContainmentVerifier fallback)
> — pre-registration §1.3 first row, ⭐ infrastructure validated.

## ⭐ Honest tier — locked landing

| Pre-registered result | Required evidence | Met? |
|---|---|---|
| **⭐ infrastructure validated** | adapter + scorer emit both citation axes | **✓ — landing here** |
| ⭐ infra + citation-emit gap finding | 0.0/0.0 because no `[N]` tokens appeared | n/a (100% rows emitted citations) |
| exploratory | exception or no axes | n/a (0 / 20 errors) |

Pre-registration §1.3 was explicit: the smoke does not promise
publication-grade numbers. It promises that the *plumbing* works.
That promise is met.

## Measurement record

```
spec           Pre-registered protocol (commit 00bfab0)
fixture        eval/external/_fixtures/alce/asqa_eval_gtr_top100.json
fixture_sha    d72737ce7d46629d3504fc508f29ec0c270e0ddd5cff3fad69625d2c0e4be35b
loader         ALCELoader(variant='asqa')
producer       alce-closed-corpus (Ollama, gemma4:e4b, n_docs=5, max_tokens=1024)
verifier       StringContainmentVerifier(min_overlap=0.5)
               is_alce_grade = False  (NOT publication)
n_samples      20  (pre-registration §1.1 locked)
n_errors       0
elapsed        41.1 s  (~2 s / query)
producer_mode  closed-corpus (ALCE-published gtr retrieval as-is;
               JAMES retrieval NOT used per §1.2 self-eval-trap rule)
```

## Citation axes (fallback verifier)

| Axis | Score | n_queries | per-query min / median / max |
|---|---|---|---|
| **citation_precision** | **0.8168** | 20 | 0.00 / 1.00 / 1.00 |
| **citation_recall**    | **0.9067** | 20 | 0.00 / 1.00 / 1.00 |

**MAGNITUDE WARNING (pre-registration §1.3 + §5.2)**:
`StringContainmentVerifier(min_overlap=0.5)` is a *generous baseline*
— it flags entailment when ≥ 50% of the hypothesis content tokens
appear in the premise. ALCE's published numbers come from T5-XXL
TRUE NLI Mixture; the two are not comparable. Do **not** cite these
scores against ALCE's published baselines. Cite them only against
themselves (consistency across cycles, infrastructure validation).

## What this validates (the honest readings)

1. **Adapter contract works.** `ALCEClosedCorpusProducer` emits the
   ALCE-formatted prompt verbatim from `princeton-nlp/ALCE`'s
   `asqa_default.json` instruction; `gemma4:e4b` emits `[N]` citations
   on every row (20 / 20 = 100%), so the scorer's citation-extraction
   path is exercised.
2. **Scorer wiring works.** Both `citation_precision` and
   `citation_recall` axes emit — the verifier interface, the citation
   regex (`[1]`, `[2, 3]`, etc.), the sentence-splitter, and the
   per-query bookkeeping all flow end-to-end.
3. **Fixture handling works.** The 80.8 MB `asqa_eval_gtr_top100.json`
   loads, validates, and the loader's `n_samples=20` slicing returns
   a deterministic prefix.
4. **Per-call latency is small.** ~2 seconds per question for
   `gemma4:e4b` with top-5 passages (mean answer ~550 chars). Future
   full-N runs (N=300 / N=1000) at this rate would take ~10 min / ~33
   min single-model — feasible.

## What this does NOT validate (per pre-registration §1.2)

- **NOT ALCE-grade scores.** The StringContainmentVerifier is the
  pre-registered fallback; ALCE-grade requires T5-XXL TRUE NLI Mixture
  or an equivalent NLI model. That step is the D-alce cycle, with its
  own pre-registration.
- **NOT JAMES retrieval quality.** The producer is closed-corpus
  (ALCE's published gtr top-5 passages, exactly the prompt ALCE's own
  baselines see). Comparing this number against any JAMES-retrieval
  experiment would be a self-eval-trap.
- **NOT multi-variant generalisation.** Only `asqa` ran. `qampari` and
  `eli5` are deliberately out of smoke scope (see pre-reg §6).
- **NOT cross-model.** Only `gemma4:e4b` ran. Cross-model comparison
  is also out of smoke scope.
- **NOT publication-tier sample size.** N=20 is the pre-registered
  smoke gate; the n=3-paired / n=20-confirm rules apply to magnitude
  claims at publication tier only ([[feedback_n1_verdict_inflation_n3_caught]]).

## Distribution shape (sanity)

The per-query precision/recall are bimodal: most queries score 1.0 on
both, a small tail scores 0.0. This is the expected shape under the
generous fallback (most sentences contain enough content tokens to
flag the cited passage as entailing; the few zero-scoring queries are
ones whose sentences cited an index that did not contain the claim
tokens, even at the 50% overlap threshold).

Interpreting whether a 1.0 means "the citation actually supports the
claim" requires the real NLI verifier — that is the line between this
⭐ smoke and the D-alce publication-tier work.

## Synergy with RAB (paper context, NOT a claim)

RAB SPEC v0.1.1 §2.3 defines `PC = traceable citations / total
citations` as a fully *deterministic* metric (the citation's
origin-bearing event must be reachable through the parent_id chain).
ALCE measures a semantic-side axis: does the cited passage actually
*entail* the claim? These are two different dimensions of "did the
citation do the work it claimed to do" — RAB asks "is the chain
intact?", ALCE asks "is the chain meaningful?". A future paper v1.4
revision could mention ALCE as the natural outer validation of PC, but
that mention requires the D-alce ALCE-grade cycle to complete first.
This smoke is **not** that evidence yet; it is the plumbing under it.

## Re-verification

```bash
git clone https://github.com/Hashevolution/James-RAG-Evol
cd James-RAG-Evol && git checkout <merge-commit>
python scripts/research/download_alce.py --variant asqa
python scripts/research/alce_smoke_run.py --n 20 --model gemma4:e4b
# → reports/external/alce/asqa-smoke-<ts>.{result.json,bench.jsonl}
```

The smoke is deterministic given the same model checkpoint; the only
source of run-to-run variance is the LLM's sampling (we use the model
defaults; future cycles can pin temperature in the pre-reg).

## Closes / opens

**Closes**:
- Cycle γ Phase C.3 pre-registration §4 steps 1–4 (download → adapter
  + tests → smoke run with locked scope).
- The cycle γ design memo's 4-bench promise is now 3/4 measured: RGB
  ✓, MuSiQue ✓, ALCE ⭐ smoke ✓ (publication-tier deferred to D-alce);
  2WikiMultiHopQA remains the open 1/4.

**Opens**:
- **D-alce cycle**: T5-XXL TRUE NLI Mixture (or smaller RoBERTa-NLI
  alternative) integration, full N=300+, qampari + eli5 variants,
  publication-tier honest-tier ladder. Separate pre-reg.
- **2WikiMultiHopQA smoke** (the other remaining cycle γ 4-bench
  promise leg) — separate pre-reg, similar smoke-then-D pattern.
- **Paper v1.4 §"ALCE outer validation"**: gated on D-alce, not on
  this smoke.

## Pointers

- Pre-registration (locked): `docs/research/cycle-gamma-c3-alce-smoke-preregistration-2026-06-11.md`
- Cycle γ design memo: `docs/design/v0.4-cycle-gamma-external-benchmark-integration.md`
- Phase C.2 MuSiQue handover (closure of the multi-hop arc): `docs/handovers/v0.4-cycle-gamma-phase-c2-musique-retrieval-bottleneck-2026-06-10.md`
- ALCE upstream: https://github.com/princeton-nlp/ALCE (Gao et al., EMNLP 2023)
- Re-verification artifacts: `reports/external/alce/asqa-smoke-20260611T051943Z.{result.json,bench.jsonl}`

## Related rules

- [[feedback_self_evaluation_trap]] — ALCE is an external standard with
  an external official scorer; closed-corpus producer rules out
  retrieval self-eval. The smoke uses the published gtr retrieval as-is.
- [[feedback_finding_size_honest_framing]] — magnitude framing is
  ⭐ infrastructure-only, NOT publication.
- [[feedback_n1_verdict_inflation_n3_caught]] — N=20 smoke is below the
  publication-tier n=3-paired / n=20-confirm rule by design; that rule
  re-applies at the D-alce tier.
- [[mechanism_layer_intent_axis_alignment]] — ALCE axis = citation;
  JAMES adapter's intent axis is answer-with-context; the LLM's prompt
  compliance bridges the two.
- [[feedback_eval_cycle_vs_collab_arc_separation]] — this smoke is the
  RAB/JAMES internal cycle, NOT joint piece evidence.
