# Cycle γ Phase D — Mechanistic ablation design

**Status**: design + tooling. Execution operator-gated (~1.5-2h
live LLM compute per model).
**Date**: 2026-06-08

---

## 1. Question

JAMES Phase B+C found a per-query JAMES abstention set on RGB-en
negrej (mxtral / llama both reach {3, 9, 14, 15, 17, 18}, gemma4
absorbs partial). **Which JAMES pipeline component is the proximate
cause of those specific abstention triggers?**

Candidate components (each exposes a `JAMES_DISABLE_*=1` env-var
ablation knob from α-6 cycle):

| Knob | Component | What it skips |
|---|---|---|
| `JAMES_DISABLE_RAG_RETRIEVAL` | retrieval | vector_store.search call; pipeline_loops.py:53 |
| `JAMES_DISABLE_RERANK` | reranker | rerank() call; pipeline_loops.py:87 |
| `JAMES_DISABLE_GRAPH` | graph traversal | pipeline_loops.py:175 |
| `JAMES_DISABLE_TYPED_FILTER` | α-8 typed-filter | graph_typed_filter R1-R5 |
| `JAMES_DISABLE_COGNITIVE_STAGES` | planner/reflect/verify | planner.py:90, reflect.py:341, verify.py:102 |
| `JAMES_DISABLE_VERIFY` | verify only | verify.py:104 (separate from cog-stages) |
| `JAMES_DISABLE_ABSTENTION` | abstention softener | pipeline_synth.py:261 |

## 2. Ablation matrix

Run on mxtral:8x7b (n=25 negrej, workspace #2 = negrej-only
ingested). One run per ablation. Reference = full-JAMES mxtral
abstention set {3, 9, 14, 15, 17, 18} from Phase C.

| Run | Env | Expected diagnostic |
|---|---|---|
| R0 | full-JAMES (baseline reference) | {3, 9, 14, 15, 17, 18}, F1 0.387 |
| R1 | `JAMES_DISABLE_RAG_RETRIEVAL=1` | If set collapses to raw → retrieval is the trigger (but retrieval is the workspace search itself — likely empties context) |
| R2 | `JAMES_DISABLE_RERANK=1` | If set unchanged → rerank not the trigger |
| R3 | `JAMES_DISABLE_GRAPH=1` | If set unchanged → graph not the trigger |
| R4 | `JAMES_DISABLE_TYPED_FILTER=1` | If set unchanged → α-8 typed-filter not the trigger |
| R5 | `JAMES_DISABLE_COGNITIVE_STAGES=1` | If set collapses → planner/reflect/verify family is the trigger |
| R6 | `JAMES_DISABLE_VERIFY=1` (only) | Distinguishes verify-specific from cog-stages broader |
| R7 | `JAMES_DISABLE_ABSTENTION=1` | If set shrinks dramatically → abstention softener is the explicit trigger |

## 3. Interpretation rules

| Pattern | Conclusion |
|---|---|
| Only R7 (`DISABLE_ABSTENTION`) collapses the set toward raw {9, 15} | Abstention softener IS the "true signal" emitter. ⭐⭐ confirmed at component level. |
| R5 / R6 also collapse | Cognitive stages (verify in particular) contribute. Multi-component signal. |
| R3 / R4 also collapse | Graph / typed-filter contributes — unlikely but check. |
| R1 collapses but R7 unchanged | Retrieval-context-to-abstention coupling is the mechanism, not the softener itself. |
| Multiple ablations all reduce, none zeroes | Distributed signal (no single component is the source). |
| All ablations unchanged from R0 | Either ablation knobs are no-ops on this code path (regression) OR signal is in some other layer not yet exposed. Investigate. |

## 4. Honest framing constraints (apply before claim-finalization)

Per `memory/feedback_finding_size_honest_framing.md` +
`memory/feedback_eval_cycle_vs_collab_arc_separation.md`:

1. **n=25 on single model on single bench** — ⭐ stable at best.
   ⭐⭐ requires same pattern on a second model.
2. **Not a joint-piece evidence** — this is JAMES-internal Phase D
   mechanistic work, separate from collab arc.
3. **Component-level finding requires LITERATURE CHECK** — if the
   collapse points to a specific component, prior-art search for
   that component's mechanism (e.g., "abstention softener prompt
   pattern in RAG") must precede any universal-mechanism claim.
4. **Per-query overlap method**: report which queries get lost
   under each ablation, not just F1 deltas. Aggregate F1 hides
   per-query mechanism.

## 5. Operator launch (when ready)

Pre-conditions:
- Workspace #2 (`workspaces/cycle_gamma_rgb_negrej/`) exists from
  Phase A (preserved). 7813 chunks negative-only RGB-en.
- mxtral:8x7b model already pulled in Ollama.
- `scripts/research/cycle_gamma_phase_d_ablate.py` (this PR)
  available.

Launch (sequential, ~30-50 min total for 7 ablations × n=25 on
mxtral; gemma4 alternative is faster ~10 min/run):

```bash
# Use mxtral for strongest absorption baseline (full 6/6 set)
MODEL=mixtral:8x7b
for KNOB in DISABLE_RAG_RETRIEVAL DISABLE_RERANK DISABLE_GRAPH \
            DISABLE_TYPED_FILTER DISABLE_COGNITIVE_STAGES \
            DISABLE_VERIFY DISABLE_ABSTENTION; do
  JAMES_WORKSPACE=$(pwd)/workspaces/cycle_gamma_rgb_negrej \
  JAMES_$KNOB=1 \
  python scripts/external_bench_run.py --bench rgb --variant en \
    --mode james --model $MODEL \
    --setting-filter negative_rejection --n-samples 25 \
    --out reports/cycle_gamma/phase-d-mxtral-$KNOB.json
done
```

Compare with the cycle γ existing artifact
`reports/cycle_gamma/rgb-en-james-negrej-mixtral-20260608.json`
(R0 reference).

## 6. Comparison output

After all 7 runs, the comparison should produce:

| Ablation | abst. set | Δ from R0 set | F1 | Δ F1 from R0 |
|---|---|---|---|---|
| R0 reference | {3, 9, 14, 15, 17, 18} | — | 0.387 | — |
| R1 | TBD | TBD | TBD | TBD |
| ... | ... | ... | ... | ... |

Use `scripts/research/cycle_gamma_rgb_compare.py` (existing) for
each pair (R0 vs Rk). Aggregate manually into the table for the
Phase D handover.

## 7. What this PR ships

- This design doc (`docs/design/v0.4-cycle-gamma-phase-d-
  mechanistic-ablation.md`)
- `scripts/research/cycle_gamma_phase_d_ablate.py` — convenience
  wrapper that runs the full 7-knob matrix sequentially with
  progress logging and per-run JSON output
- Tests for the wrapper (unit-level only; live LLM smoke is
  operator-gated)

## 8. What this PR does NOT ship

- Live LLM ablation results (operator-gated; ~30-90 min compute)
- Component-level mechanism claim (gated on results + prior art
  check)
- Joint-piece-bound output (Phase D is JAMES-internal evaluation
  arc per `memory/feedback_eval_cycle_vs_collab_arc_separation.md`)
