# Cycle γ Phase E-min — cross-model results + hypothesis-2 (tradeoff) measurement plan

**Date**: 2026-06-09 (continuation of Phase E-min mxtral session)
**Status**: Phase E-min **cross-model COMPLETE** (mxtral + gemma4 + llama).
Result is a **split**, not a clean 3/3. Next session = **hypothesis-2
(tradeoff) measurement on MuSiQue** — the decisive missing half.

---

## 1. Headline

Phase E-min ran 3 JAMES components (`rerank` / `typed_filter` /
`cognitive_stages`) disabled, one at a time, on RGB-en across 3 models
(n=25 per cell, 0 errors, ~3.5h total live LLM).

**Result: 2 of 3 components cross-model confirmed net-positive-when-disabled
on RGB-en; `typed_filter` is model-dependent (llama reversal).**

| Component | mxtral (Δnoise/Δnegrej) | gemma4 | llama | Cross-model verdict |
|---|---|---|---|---|
| `rerank` | +0.040 / +0.050 | +0.040 / +0.209 | +0.040 / +0.098 | 🟢 **3/3 net+** |
| `cognitive_stages` | +0.160 / +0.050 | +0.040 / +0.111 | +0.120 / +0.050 | 🟢 **3/3 net+** |
| `typed_filter` | +0.080 / +0.050 | 0.000 / +0.162 | **−0.080** / 0.000 | 🟡 **2/3 — llama noise reversal** |

R0 references (full-JAMES, nothing disabled), from Phase C 4-model baseline:

| Model | noise_robustness | negative_rejection_f1 |
|---|---|---|
| mxtral:8x7b | 0.760 | 0.3871 |
| gemma4:e4b | 0.840 | 0.2759 |
| llama3.1:8b | 0.800 | 0.3871 |

---

## 2. The critical caveat — this is only the "loss half" of a tradeoff

**RGB-en `noise_robustness` + `negative_rejection` are BOTH abstention-family
axes on a single-hop factoid task. They are NOT the design-intent (home-turf)
axes of these 3 components.** Per the 6th catch
(`feedback_single_axis_ablation_misframing`) and the layer-intent↔axis rule
(`mechanism_layer_intent_axis_alignment`), a "remove / default-off" verdict is
**forbidden** until each component is also scored on its own home-turf axis.

| Component | Home-turf (design-intent) axis | Exercised by RGB-en? |
|---|---|---|
| `rerank` (cross-encoder) | near-miss semantic re-ranking → graded answer / path recall | ❌ (RGB is single-hop) |
| `typed_filter` (α-8 ontology) | evidence-of-absence preservation, poison/catalog robustness | partial (negrej overlaps); **home turf = adversarial, OUTSIDE cycle γ's 4 benches** |
| `cognitive_stages` | multi-step reasoning → multi-hop graded answer | ❌ (RGB is not multi-hop) |

So the Phase E-min cross-model result establishes:
> "On RGB-en-shaped tasks (single-hop factoid + noise/null), `rerank` and
> `cognitive_stages` are net-negative across all 3 models."

It does **NOT** establish "remove them." It is the **LOSS HALF** of a
possible tradeoff.

### The three hypotheses (user framing, 2026-06-09)

1. **Remove entirely** — components are deadweight. ❌ Cannot conclude from
   current data (home-turf axes never measured).
2. **Tradeoff** — components contribute on their home turf (multi-hop /
   near-miss / poison), only net-negative on RGB abstention. 🟡 Plausible,
   mechanism known (RAG "cheat-sheet effect" → over-answering), **UNPROVEN**.
   **← user-selected priority: if proven, keep components + design harmony
   (e.g. query-type routing: ON for multi-hop, OFF for abstention-prone).**
3. **Redesign the combination** — the 3 interact badly. 🟡 Partially testable
   from existing per-query overlap + 2-way/3-way disabled cells (not yet run).

---

## 3. What the next session must do — prove/refute hypothesis 2 on MuSiQue

The decisive measurement = the **GAIN HALF**. Run the **same paired ablation**
(R0 vs each component OFF) on **MuSiQue** (multi-hop, the home turf of
`cognitive_stages` and a strong retrieval-precision exercise for `rerank`),
measuring the **primary axes** `em` / `f1` / `support_idx_recall`.

```
Already have (Phase E-min):  component ON → RGB abstention/noise LOSS  (cross-model locked)
Need next (Phase C.2):       component ON → MuSiQue multi-hop GAIN
─────────────────────────────────────────────────────────────────────
Combined per component = the tradeoff (hypothesis 2) proof/refutation:
  • component OFF DROPS MuSiQue em/f1 + (known) RAISES RGB abstention  ⇒ HYP 2 PROVEN (tradeoff → keep + harmonize)
  • component OFF flat/up on MuSiQue                                   ⇒ HYP 1 (deadweight even on home turf → default-off candidate)
```

This is **not scope creep** — it is the genuine completion of the 9th catch's
"multi-axis measurement is duty" (Phase E-min's noise+negrej are both
abstention-family; the component's design-intent axis is still unmeasured).
MuSiQue is an **external standard benchmark with an official scorer**, so it
passes the self-eval-trap bar (`feedback_self_evaluation_trap`) =
publication-grade. This IS cycle γ Phase C.2 (MuSiQue + 2Wiki) from the design
memo.

### 3.1 Readiness (honest)

| Layer | Status |
|---|---|
| Loaders / scorers / runner / dispatch (rgb/alce/musique/2wiki) | ✅ BUILT (Phase A, 83 tests) |
| MuSiQue scorer axes = `em`, `f1`, `support_idx_recall` | ✅ confirmed (graded/multi-hop primary) |
| MuSiQue data downloaded | ❌ not present (`eval/external/_fixtures/musique/` empty) |
| JAMES workspace (ingested MuSiQue corpus) | ❌ not built |

### 3.2 Prep steps (operator, ~the data-prep gate)

1. **Download MuSiQue** (official `download_data.sh` from
   StonyBrookNLP/musique) → place `musique_ans_v1.0_dev.jsonl` at
   `eval/external/_fixtures/musique/` (the loader's `_default_cache_dir()`),
   or pass `--cache-dir`.
2. **Build a JAMES workspace** mirroring RGB's pattern. Adapt
   `scripts/research/cycle_gamma_rgb_corpus_build.py` →
   `cycle_gamma_musique_corpus_build.py`: ingest every query's paragraphs
   (supporting + distractor) under deterministic source ids
   (`musique-ans-<row_id>-p<idx>`). Subsample to match Phase E-min n (~25–50
   queries) for paired comparability. Workspace e.g.
   `workspaces/cycle_gamma_musique_ans`.
3. **Paired ablation run** (mirror `scripts/research/_phase_e_cross_model.sh`):
   R0 + `DISABLE_RERANK` + `DISABLE_COGNITIVE_STAGES` (skip `typed_filter` here
   — see §4), per model, on MuSiQue:
   ```bash
   env "JAMES_WORKSPACE=$WS" ["JAMES_DISABLE_<KNOB>=1"] \
     python scripts/external_bench_run.py --bench musique --variant ans \
     --mode james --model <model> --n-samples 25 \
     --out reports/cycle_gamma/phase-c2/musique-ans-<model>-<knob|R0>.json
   ```
4. **Combine** MuSiQue Δem/Δf1 (gain) with Phase E-min RGB Δ (loss) per
   component per model → tradeoff verdict.

Estimated compute: multi-hop synthesis is slower than RGB; budget ~15–20
min/run. R0 + 2 knobs × 3 models = 9 runs ≈ 2.5–3h, plus one-time data prep +
ingest.

---

## 4. typed_filter is the odd one out — split it off

`typed_filter` failed cross-model (llama noise −0.080 when disabled = it HELPS
llama noise robustness). Its home turf (poison / evidence-of-absence) is
**outside** cycle γ's 4 benches. **Do NOT** fold it into the MuSiQue run.
Measure it separately on the adversarial sweep (`scripts/adversarial_sweep.py`,
Track 2c poison set) where R1–R5 evidence-of-absence preservation is the
design intent. Until then: `typed_filter` stays at ⭐ stable / cross-model
**unconfirmed**, default unchanged.

---

## 5. Honest framing tier (current, end of this session)

| | rerank | cognitive_stages | typed_filter |
|---|---|---|---|
| ⭐ stable (multi-axis paired, n=25) | ✅ | ✅ | ✅ |
| ⭐⭐ cross-model (RGB-side net-negative) | ✅ 3/3 | ✅ 3/3 | ❌ 2/3 (llama reversal) |
| remove (hyp1) vs tradeoff (hyp2) | ⏳ MuSiQue | ⏳ MuSiQue | ⏳ adversarial |
| default change licensed? | ❌ not yet | ❌ not yet | ❌ not yet |

**No cycle β default revision PR is licensed yet.** The RGB-side loss half is
locked for 2/3 components; the home-turf gain half is the gate.

---

## 6. Files of interest

| Purpose | Path |
|---|---|
| Phase E-min cross-model JSONs (gemma4 + llama, 12 files) | `reports/cycle_gamma/phase-e/phase-e-{gemma4,llama}-{noise,negrej}-*.json` (gitignored) |
| Phase E-min mxtral JSONs (noise) | `reports/cycle_gamma/phase-e/phase-e-mxtral-noise-*.json` (gitignored) |
| Cross-model run logs | `reports/cycle_gamma/phase-e/_run-{gemma4,llama}.log` (gitignored) |
| Cross-model runner (reusable) | `scripts/research/_phase_e_cross_model.sh` |
| R0 baselines (Phase C, 4-model) | `reports/cycle_gamma/rgb-en-james-{noise,negrej}[-<model>]-20260608.json` |
| RGB corpus-build (template for MuSiQue ingest) | `scripts/research/cycle_gamma_rgb_corpus_build.py` |
| MuSiQue loader / scorer | `eval/external/musique_loader.py` / `eval/external/musique_scorer.py` |
| External bench runner | `scripts/external_bench_run.py` |
| Cycle γ design memo (Phase C.2) | `docs/design/v0.4-cycle-gamma-external-benchmark-integration.md` |
| Prior session handover (mxtral-only) | `docs/handovers/v0.4-cycle-gamma-phase-e-min-mxtral-results-2026-06-09.md` |
| 6th catch (single-axis → defect framing ban) | `memory/feedback_single_axis_ablation_misframing.md` |
| 9th catch (measurement-driven identity) | `memory/feedback_james_identity_measurement_driven.md` |
| Self-eval trap rule | `memory/feedback_self_evaluation_trap.md` |

---

## 7. Path 1 sequence update

```
Cycle γ Phase E-min mxtral ✅ (3 components net+ when disabled, single model)
Cycle γ Phase E-min cross-model ✅ (THIS doc)
   ├ rerank + cognitive_stages: 3/3 net+ (RGB-side LOSS half locked)
   ├ typed_filter: 2/3 (llama reversal) → split to adversarial track
   └ NO default revision licensed — loss half only

NEXT SESSION (natural entry):
- Hypothesis-2 measurement = Phase C.2 MuSiQue paired ablation (GAIN half)
   ├ data prep: download MuSiQue + build cycle_gamma_musique_ans workspace
   ├ paired ablation: R0 + DISABLE_RERANK + DISABLE_COGNITIVE_STAGES × 3 models
   └ combine with RGB loss half → tradeoff (keep+harmonize) vs deadweight (default-off)
- typed_filter: separate adversarial_sweep measurement (its home turf)
- (later) 2Wiki cross-bench confirm; D1 retrieval; D2 V3 directive softener

If hyp2 PROVEN (component helps multi-hop, hurts abstention):
- design harmony, NOT removal — e.g. IntentClassifier / query-type routing:
  components ON for multi-hop/answerable, OFF for abstention-prone queries.
```

---

## 8. Session statistics

- **Phase E-min cross-model runs**: 12 (gemma4 6 + llama 6), n=25 each, 0 errors
- **Total Phase E-min**: mxtral 3 + cross-model 12 = 15 cells, ~3.5h live LLM
- **JAMES core code changed**: 0 lines (measurement-only)
- **Cross-model verdict**: split (2/3 confirmed, 1/3 model-dependent) — honest
  framing held, no over-claim to "3/3 / remove"
