# Cycle γ Phase B Option A results — JAMES vs raw on RGB-en

**Date**: 2026-06-08 (same session as Phase A closure + Phase B
launch handover)
**Status**: Option A complete. ⭐⭐ confirmed (paired delta on
both axes); ⭐⭐⭐ candidate gated on Phase C cross-model + cross-
bench.

---

## 1. Headline

| axis | raw closed-corpus | JAMES full-stack (workspace-isolated) | Δ |
|---|---|---|---|
| `noise_robustness_accuracy` | 0.600 (15/25) | **0.840 (21/25)** | **+0.240** |
| `negative_rejection_f1` | 0.000 (TP=0, FN=25) | **0.276 (TP=4, FN=21)** | **+0.276** |

The JAMES stack improved both axes against the same 25 paired
fixture rows. This is the **first external-bench paired evidence**
for the cycle γ design memo §2 abstention claim.

---

## 2. Experiment design (what the user authorised at the handover §4)

The Phase B baseline measurement showed `negative_rejection_f1 =
0.000` for raw `gemma4:e4b` on RGB-en closed-corpus. The natural
follow-up was a paired JAMES measurement. The handover §4 decision
tree presented three options; the operator chose the 2-workspace
design ("제대로 측정"):

| Workspace | Content | Use |
|---|---|---|
| #1 `cycle_gamma_rgb_full`   | 11,641 chunks (3828 positive + 7813 negative passages from all 300 RGB-en rows) | noise_robustness measurement (JAMES has access to both gold and distractors) |
| #2 `cycle_gamma_rgb_negrej` | 7,813 chunks (negative passages only, all 300 rows stripped of positives) | negative_rejection measurement (no gold anywhere; JAMES abstention must drive the result) |

Total ingestion time: 90 min (workspace #1) + 60 min (workspace #2)
= 150 min, on BAAI/bge-m3 embedder. Per-row rate ~17 s/row dominated
by chunk count (~40 chunks per row × 300 rows = ~12k embeddings).

This design + new `setting_filter` kwarg on the RGB loader (PR #738)
lets the runner pair each workspace with the matching axis slice via
`scripts/external_bench_run.py --bench rgb --setting-filter <axis>
--mode james ...`. No core/ code changed; the comparison is a
pure-integration extension.

---

## 3. Per-axis detail

### noise_robustness_accuracy

```
raw closed-corpus    : 15/25 hits = 0.600
JAMES full-stack     : 21/25 hits = 0.840
Δ                    : +0.240 (+40% relative)
```

JAMES has to find the gold passage via retrieval from a corpus of
11,641 chunks — every other row's passages are distractors. Despite
the added retrieval risk, the full stack (query_rewrite +
orchestrated retrieval + reranker + verifier) beats the raw model
fed the gold context directly. Result: JAMES's reasoning + reranking
adds 6 hits to a baseline that was already given the answer.

### negative_rejection_f1

```
raw closed-corpus    : TP=0,  FP=1,  FN=25, TN=24 → F1 = 0.000
JAMES full-stack     : TP=4,  FP=0,  FN=21, TN=0  → F1 = 0.276
Δ                    : +0.276 (absolute)
```

The TN=24 → TN=0 difference is because workspace #2 only contains
negrej queries (setting_filter), so there are no truth-present rows
in the JAMES measurement. The headline number is the F1 itself.

Per-query inspection:
- Row 3: `"ANSWER: insufficient information. The provided context
  confirms that Jason Semore spent time at Valdosta State, but it
  does not specify what position he held there."` — clean
  abstention.
- Rows 0/1/2/4 (and 20 others): still hallucinated (e.g. row 0
  still says "premieres Jan 2 at 9:00 PM ET" with no actual
  evidence) — the abstention layer fires only on a minority of
  negrej queries.

Raw model abstained on 0/25; JAMES abstained on 4/25. The lift is
real but the absolute level is low — **JAMES "starts" abstaining,
it does not "abstain well" yet**. This is the honest summary;
elevation to "JAMES solves abstention" requires (a) a much higher
absolute F1 and (b) cross-bench / cross-model confirmation.

---

## 4. Honest framing (per `feedback_finding_size_honest_framing.md`)

| Tier | Status | Reason |
|---|---|---|
| ⭐ stable | ✅ confirmed | Paired baseline + treatment on same fixture rows, both axes report positive delta, n=25 each. Operational evidence. |
| ⭐⭐ confirmed | ✅ confirmed | First external-bench paired evidence for cycle γ §2's JAMES abstention value claim. Both axes show JAMES lift on the published RGB benchmark (Chen et al. 2024 EMNLP). |
| ⭐⭐⭐ candidate | ⏳ pending | Single model (gemma4:e4b), single bench (RGB-en), single domain (2022 news). Phase C (cross-model + cross-bench) required before any universal-law-class framing. |

Caveats that any further analysis must cite:
- **n=25 per axis** — small sample, no 95% CI calculated. The
  +0.240 / +0.276 deltas could shift by ±0.10 with the next 25.
- **JAMES absolute negrej F1 = 0.276** — JAMES still hallucinates
  on 21/25 negrej queries. The lift is real; the floor is low.
- **Workspace #2 cross-row contamination**: the runner uses one
  workspace with all 300 rows' negatives ingested, so JAMES
  retrieval can pull CEO-keyword distractors from other rows for
  a "next CEO of nonexistent company" query. The paper-faithful
  alternative (per-query workspace) would be ~300× more expensive;
  this is a documented trade-off, not a bug.
- **JAMES is 18× slower than raw closed-corpus** (~36 s/query vs
  ~2 s/query). Production latency trade-off matters when
  generalising the result to operational claims.

---

## 5. Path 1 sequence update

```
Cycle β #1 / #1.5 / #2 / #6 / #7   ✅ closed (prior session)
Cycle β #3 NATURAL oracle           🔴 skipped (self-eval trap)
v0.4.2 T5 Replayable Audit Graph    ✅ closed (prior session)
Cycle γ Phase A scaffolding         ✅ closed (10 PRs)
Cycle γ Phase B
  ├ Phase B smoke #1 (JSONL)        ✅ #734
  ├ Phase B smoke #2 (prompt cap)   ✅ #735
  ├ Phase B #3 (dual-axis)          ✅ #736
  ├ RGB-en raw baseline measured    ✅
  ├ Phase B Option A setting_filter
    + 2-workspace builder           ✅ #738
  └ Option A JAMES paired RGB-en    ✅ (this doc)
Cycle γ Phase B (other benches)     ⏳ operator (ALCE/MuSiQue/2Wiki
                                          downloads required)
Cycle γ Phase C full measurement    ⏳ gated by B
Cycle γ Phase D cross-bench report  ⏳ gated by C
v0.5 entry gate (D2 evidence)       ⏳ gated by D
```

---

## 6. What the next session should do

The natural extensions:

### Option A' (continuation of A) — strengthen JAMES negrej

JAMES negrej F1 = 0.276 is a clear improvement over 0.000 but is
far from "solved". Investigate WHY 21/25 negrej queries still
hallucinate — likely candidates:
- The cross-row distractors look strong enough that the verifier
  passes them (retriever returns high-similarity-but-wrong
  passages).
- The abstention layer fires only on certain answer shapes
  ("insufficient information") and misses others (model says
  "ANSWER: Drama" with no caveat).
- The temperature / sampling regime allows confident-sounding
  fabrications.

A 1-2 PR investigation that lifts JAMES negrej F1 from 0.276 to
0.4+ would meaningfully strengthen the cycle γ §2 abstention claim
on a single bench. Lower priority than Phase C breadth.

### Option B (cross-bench breadth) — ALCE / MuSiQue / 2Wiki

Each requires operator-side fixture download (handover Phase A/B
launch doc §4 Option B). After downloads:

```bash
python scripts/external_bench_run.py --bench alce --variant asqa --mode closed-corpus ...
python scripts/external_bench_run.py --bench musique --variant ans --mode closed-corpus ...
python scripts/external_bench_run.py --bench 2wiki --mode closed-corpus ...
```

Then run the same paired JAMES vs raw comparison per bench.

### Phase C (multi-model breadth) — gemma3:12b, mixtral, llama

Re-run RGB-en n=25 noise + n=25 negrej with each of:
- gemma3:12b (model capability ceiling)
- mixtral:8x7b (different family)
- llama3.1:8b (size match to gemma)

If JAMES lift holds across families → ⭐⭐⭐ candidate
strengthens. If it disappears (e.g. mixtral's raw negrej already
high), the cycle γ §2 framing needs to specify which model class
benefits from JAMES.

### Recommended order

1. **Phase C cross-model on RGB-en** (cheapest — workspaces already
   built, just change `--model`). ~50 min per model × 3 = ~2.5h.
   This is the fastest path to ⭐⭐⭐ confirmation.
2. **Option B cross-bench** after operator download. Higher
   prep cost but widens evidence base.
3. **Option A' negrej deep-dive** last — diagnostic / engineering
   work that doesn't add to publication evidence directly.

---

## 7. Files of interest

| Purpose | Path |
|---|---|
| Raw baseline measurement | `reports/cycle_gamma/rgb-en-dual-axis-20260608.json` (gitignored) |
| JAMES noise measurement | `reports/cycle_gamma/rgb-en-james-noise-20260608.json` (gitignored) |
| JAMES negrej measurement | `reports/cycle_gamma/rgb-en-james-negrej-20260608.json` (gitignored) |
| Comparison reports | `reports/cycle_gamma/rgb-en-compare-{noise,negrej}-20260608.md` (gitignored) |
| Workspace #1 (full) | `workspaces/cycle_gamma_rgb_full/chroma_db_bge_m3/` (gitignored) |
| Workspace #2 (negrej) | `workspaces/cycle_gamma_rgb_negrej/chroma_db_bge_m3/` (gitignored) |
| Corpus builder | `scripts/research/cycle_gamma_rgb_corpus_build.py` |
| Comparison tool | `scripts/research/cycle_gamma_rgb_compare.py` |
| Loader setting_filter + 2-workspace API (PR #738) | `eval/external/rgb_loader.py` + `eval/external/runner.py` + `scripts/external_bench_run.py` |
| Prior handover (Phase A + Phase B launch) | `docs/handovers/v0.4-cycle-gamma-phase-a-closure-and-phase-b-launch-2026-06-08.md` |
| Updated memory entry | `memory/project_cycle_gamma_phase_b_rgb_baseline.md` |
| Honest framing rule | `memory/feedback_finding_size_honest_framing.md` |
| Self-eval trap rule | `memory/feedback_self_evaluation_trap.md` |
