# Cycle γ Phase A closure + Phase B launch handover

**Date**: 2026-06-08
**Session author**: Claude Opus 4.7 (1M context)
**Status**: Phase A complete (10 PRs, #724-#733). Phase B opened
with 3 PRs (#734-#736) + one operator-grade finding.

---

## 1. What this session accomplished

### Phase A closure (cycle γ scaffolding) — 10 PRs

All 4 external benchmarks (RGB / ALCE / MuSiQue / 2WikiMultiHopQA)
now have loader, scorer, and a unified runner. 83/83 cycle γ tests
green at A.5 close; 128/128 after Phase B #3.

| Sub-phase | PR | Component |
|---|---|---|
| A.0 | #724 | `eval/external/__init__` + base + ExternalQuery schema |
| A.1 | #725 | RGB loader (auto-download from GitHub raw) |
| A.2 | #726 | ALCE loader (ASQA + QAMPARI + ELI5 dispatch) |
| A.3 | #727 | MuSiQue + 2WikiMultiHopQA loaders |
| A.4.0 | #728 | ExternalScorer abstract base + ScoreAxis |
| A.4.1 | #729 | RGB scorer (noise_robustness + negative_rejection_f1) |
| A.4.2 | #730 | MuSiQue scorer (SQuAD-style EM/F1 + support_idx_recall) |
| A.4.3 | #731 | 2Wiki scorer (EM/F1/support_fact_f1/f1_by_type) |
| A.4.4 | #732 | ALCE scorer (pluggable NLI + citation prec/recall) |
| A.5 | #733 | Unified `scripts/external_bench_run.py` + runner engine |

Public surface: `python scripts/external_bench_run.py --bench
{rgb,alce,musique,2wiki} --mode {closed-corpus,james} --model
<m> --out <path>` runs any of the four benches end-to-end through
either a Gemma closed-corpus baseline or the full JAMES stack.
3 producers ship: `StubProducer` (tests), `ClosedCorpusGemmaProducer`,
`JamesEngineProducer`.

### Phase B opened (smoke + bug catches) — 3 PRs

The first live LLM run uncovered 2 production wiring bugs and 1
scope gap. Each landed as its own PR (`feedback_measurement_smoke_
caught_wiring_bugs` rule applied 3 times).

| # | type | PR | Issue |
|---|---|---|---|
| #1 | fix | #734 | RGB loader assumed JSON array; published `en.json` is JSONL |
| #2 | fix | #735 | GemmaClient hardcoded `MAX_PROMPT_LEN=4000` silently chopped 35k-char RGB rows → env-configurable |
| #3 | feat | #736 | RGB loader emits dual-axis (noise + negrej) per row — abstention axis was structurally unmeasurable before |

### Live measurement on RGB-en n=50 (dual-axis, uncapped)

| Axis | Score | n | Notes |
|---|---|---|---|
| `noise_robustness_accuracy` | **0.600** | 25 | gemma4:e4b closed-corpus, 15/25 hits, lowercase substring match |
| `negative_rejection_f1` | **0.000** | 25 | TP=0, FP=1, FN=25, TN=24 |

Run: `reports/cycle_gamma/rgb-en-dual-axis-20260608.json`.
Elapsed 91.3s for 50 queries (≈1.8s/query), 0 errors.

---

## 2. ⭐⭐ candidate finding — operator review needed

**raw gemma4:e4b on RGB-en negative-rejection setting = 0% abstain
= 100% hallucinate on no-evidence.**

Per-query confusion matrix from the dual-axis run:
- 25 negrej queries: every single one produced a confident answer
  ("The film premieres on Jan. 2 at 9:00 p.m. ET via CNN.", "LSU
  defeated the Iowa State Cyclones in the 2022 Citrus Bowl.", etc.)
  using ONLY distractor docs as context.
- 25 noise queries: 1 over-abstained (FP), 24 produced answers, 15
  matched gold.

Honest framing (per `feedback_finding_size_honest_framing`):
- ⭐ stable Phase B operational baseline: gemma4:e4b raw,
  closed-corpus, single model, no cross-stack comparison.
- ⭐⭐ candidate elevation pending: same 25 negrej queries through
  JAMES engine (full stack with abstention layer) to produce the
  before/after delta. If JAMES F1 lifts from 0.000 to a measurable
  positive value, that is direct empirical evidence for cycle γ
  design memo §2's core claim (JAMES abstention strength).
- ⭐⭐⭐ candidate: deferred to cross-model + cross-bench
  confirmation (still requires Phase C).

This finding is the **highest-information-density single artifact**
from cycle γ so far. The follow-up measurement is what cycle β + α
cycles avoided as a "forced discovery hunt" (per
`feedback_alpha_cycle_discovery_loop_end`); the difference here is
the measurement is **emergent + falsifiable + maps onto a
specific architectural claim**, not a manufactured search for
publishable signals.

---

## 3. State of Path 1 sequence (cycle γ track)

```
Cycle β #1 / #1.5 / #2 / #6 / #7   ✅ closed (prior session)
Cycle β #3 NATURAL oracle           🔴 skipped (self-eval trap)
v0.4.2 T5 Replayable Audit Graph    ✅ closed (prior session)
Cycle γ Phase A scaffolding         ✅ closed (10 PRs this session)
Cycle γ Phase B operator-gated      🟡 3 PRs landed + baseline measured
  ├ Phase B smoke #1 (JSONL)        ✅ #734
  ├ Phase B smoke #2 (prompt cap)   ✅ #735
  ├ Phase B #3 (dual-axis)          ✅ #736
  ├ RGB-en n=50 baseline measured   ✅
  └ JAMES-engine same-25 negrej     ⏳ operator (needs workspace)
Cycle γ Phase B (other benches)     ⏳ operator (ALCE/MuSiQue/2Wiki
                                          downloads required)
Cycle γ Phase C full measurement    ⏳ gated by B
Cycle γ Phase D cross-bench report  ⏳ gated by C
v0.5 entry gate (D2 evidence)       ⏳ gated by D
```

---

## 4. What the next session should do (decision tree)

The recommended sequence — pick **one** as the entry point:

### Option A — JAMES-engine RGB measurement (highest-value next step)

Build a JAMES workspace whose corpus mirrors the RGB-en docs (all
300 rows' positive+negative passages, ingested as separate documents
so the JAMES retriever can pick them). Then re-run:

```bash
JAMES_WORKSPACE=./workspaces/cycle_gamma_rgb_eval \
python scripts/external_bench_run.py \
    --bench rgb --variant en \
    --mode james --model gemma4:e4b \
    --n-samples 50 \
    --out reports/cycle_gamma/rgb-en-james-engine-20260609.json
```

Compare `negative_rejection_f1` against the 0.000 baseline. If
JAMES lifts it to >0.3 the abstention-layer claim has empirical
backing from an external benchmark. This is the cycle γ design
memo §2 core deliverable.

**Operator setup cost**: ~30-60 min (corpus ingestion script needed —
the loader's negative docs are per-query, so the workspace builder
needs to dedupe + flatten 300 × ~30 docs into the JAMES vector store).

### Option B — Cross-bench evidence (downloads required)

Run the same closed-corpus baseline on ALCE-ASQA + MuSiQue-ans +
2Wiki-dev to widen the evidence base. Each loader expects a
pre-downloaded fixture cache:

```bash
# ALCE (ASQA / QAMPARI / ELI5)
git clone https://github.com/princeton-nlp/ALCE
bash ALCE/download_data.sh
cp ALCE/data/*.json eval/external/_fixtures/alce/

# MuSiQue
# https://github.com/StonyBrookNLP/musique → download musique_v1.0.zip
# Unpack → eval/external/_fixtures/musique/musique_ans_v1.0_dev.jsonl

# 2WikiMultiHopQA
# https://github.com/Alab-NII/2wikimultihop → download data.zip
# Unpack → eval/external/_fixtures/2wiki/dev.json
```

Then `python scripts/external_bench_run.py --bench <name> ...` per
the help text in `scripts/external_bench_run.py`.

### Option C — handover-only (this session's actual exit)

Stop here. Memory + this doc capture state; next session picks A
or B.

---

## 5. Production-safety notes (read before any new PR)

1. **PR convention**: every cycle γ change goes through
   `feat|fix|chore/<topic> → gh pr create → gh pr merge --squash
   --auto --delete-branch`. This session accidentally pushed one
   `chore: .gitignore` directly to main (commit `ff74243`) — that
   is the only break in the pattern. Do not repeat.

2. **`feedback_measurement_smoke_caught_wiring_bugs` rule fires
   again**: any new live-LLM smoke against a fresh bench/wiring
   path is expected to surface 1-3 production bugs that unit tests
   miss. Budget time for the catch + fix + re-run loop. The 3 catches
   this session were all real wiring bugs, not measurement noise.

3. **`feedback_self_evaluation_trap` rule**: JAMES does not write
   fixtures or oracles for these benches. The loaders pull
   published rows verbatim; the scorers implement the published
   formulas. The ALCE default verifier flags itself as NOT
   ALCE-grade so a future cycle γ report cannot accidentally claim
   ALCE-grade scores from the fallback.

4. **`feedback_finding_size_honest_framing`**: the 0% negrej
   finding is ⭐⭐ candidate, not "JAMES proved its abstention
   value" — that elevation requires Option A above to land first.
   Do NOT promote the framing in a public artifact (PR description,
   release note, joint piece) until the paired JAMES-engine
   measurement lands.

5. **GemmaClient prompt cap default stays 4000**: production
   callers see byte-identical behaviour. Only callers that
   explicitly set `JAMES_GEMMA_MAX_PROMPT_CHARS` (or construct
   `ClosedCorpusGemmaProducer`) get the lifted cap.

---

## 6. Files of interest

| Purpose | Path |
|---|---|
| Runner engine | `eval/external/runner.py` |
| CLI shim | `scripts/external_bench_run.py` |
| RGB loader (dual-axis) | `eval/external/rgb_loader.py` |
| RGB scorer | `eval/external/rgb_scorer.py` |
| GemmaClient cap helper | `core/gemma_client.py` (`_resolve_max_prompt_len`) |
| Phase B smoke result | `reports/cycle_gamma/rgb-en-dual-axis-20260608.json` |
| Phase B smoke log | `reports/cycle_gamma/_rgb-en-dual-axis.log` |
| Cycle γ design memo | `docs/design/v0.4-cycle-gamma-external-benchmark-integration.md` |
| Phase A closure memory | `memory/project_cycle_gamma_phase_a_closed.md` |
| Smoke-bugs rule | `memory/feedback_measurement_smoke_caught_wiring_bugs.md` |
| Self-eval trap rule | `memory/feedback_self_evaluation_trap.md` |
| Honest framing rule | `memory/feedback_finding_size_honest_framing.md` |
