# Cycle γ Phase D — Mechanistic ablation results (mxtral n=25, 7 knobs)

**Date**: 2026-06-08 (continuation of Phase A+B+C +
prior-art-positioning + scope-correction session)
**Status**: Phase D matrix executed (mxtral:8x7b, n=25 per
ablation, 7 ablation knobs, total 5607s ≈ 93 min compute, 0
errors). Findings are **narrower than the design hypothesis
anticipated** — original "abstention softener emits true signal"
hypothesis **partially falsified**; sharper finding documented
below.

**Critical 6th honest-framing catch**: user
`자메스 설계 구조상 잘못된것이 있다는 근거가 있는건가 분석해봐`
caught an over-claim before it could ship. See
`memory/feedback_single_axis_ablation_misframing.md`.

---

## 1. Raw matrix

| Ablation | F1 | ΔF1 vs R0 | TP / FP / FN / TN | Set |
|---|---|---|---|---|
| **R0 reference (full-JAMES)** | 0.387 | — | 6/0/19/0 | {3, 9, 14, 15, 17, 18} |
| `DISABLE_GRAPH` | 0.387 | **0** | 6/0/19/0 | {3, 9, 14, 15, 17, 18} ← identical |
| `DISABLE_VERIFY` | 0.387 | **0** | 6/0/19/0 | {3, 9, 14, 15, 17, 18} ← identical |
| `DISABLE_RAG_RETRIEVAL` | 0.214 | **-0.173** | 3/0/22/0 | {3, 7, 24} (5 lost, 2 gained) |
| `DISABLE_ABSTENTION` | 0.333 | **-0.054** | 5/0/20/0 | {9, 14, 15, 17, 18} (1 lost: {3}) |
| `DISABLE_RERANK` | 0.438 | **+0.050** | 7/0/18/0 | {3, 7, 9, 15, 17, 23, 24} |
| `DISABLE_TYPED_FILTER` | 0.438 | **+0.050** | 7/0/18/0 | {3, 9, 14, 15, 16, 17, 18} |
| `DISABLE_COGNITIVE_STAGES` | 0.438 | **+0.050** | 7/0/18/0 | {3, 7, 9, 14, 15, 17, 18} |

Raw artifacts in `reports/cycle_gamma/phase-d/` (gitignored).

---

## 2. Layer-intent alignment audit (the 6th catch)

The design doc's hypothesis ("`DISABLE_ABSTENTION` collapses set
→ softener is the trigger") was measured cleanly. The unexpected
**+0.050 ΔF1 on rerank / typed_filter / cognitive_stages** ablations
tempted an over-claim "JAMES has self-suppressing components / default
revision candidate / +0.15 lift potential". The user catch reset the
analysis to align with the α-5 cycle's layer-intent-axis matrix rule
(`memory/mechanism_layer_intent_axis_alignment.md`).

### Per-knob primary-axis alignment

| Ablation | Component primary axis (from layer-intent matrix) | Measured axis (Phase D) | Verdict shape |
|---|---|---|---|
| `DISABLE_ABSTENTION` | abst_f1 | abst_f1 | ✅ aligned — valid verdict |
| `DISABLE_RAG_RETRIEVAL` | all (fundamental) | abst_f1 | ✅ aligned (fundamental grounding) |
| `DISABLE_GRAPH` | path + graded | abst_f1 only | ⚠️ misaligned (ΔF1=0 expected = regression-check pass) |
| `DISABLE_VERIFY` | graded | abst_f1 only | ⚠️ misaligned (ΔF1=0 = regression-check pass) |
| `DISABLE_RERANK` | noise_robustness + graded | abst_f1 only | ❌ misaligned — verdict requires noise + graded measurement |
| `DISABLE_TYPED_FILTER` | abst_f1 + path | abst_f1 measured, path not | ⚠️ partial alignment — path measurement needed |
| `DISABLE_COGNITIVE_STAGES` | graded | abst_f1 only | ❌ misaligned — verdict requires graded measurement |

Per the α-5 rule (memory `mechanism_layer_intent_axis_alignment`):
**single-axis measurement on a misaligned axis cannot support
"defect" / "harmful" / "default revision" framing**. To make that
class of claim, the component's primary axis must be measured
simultaneously and a Pareto verdict computed.

The user catch identified this exact pattern as α-5
wrong-fix-averted #7 (ADAPTIVE_BUDGET) being re-instantiated.

---

## 3. What Phase D actually shows (honest framing)

### ✅ Valid findings (axis-aligned)

1. **Abstention softener has measurable design-intent contribution.**
   `DISABLE_ABSTENTION` → ΔF1 -0.054 (1 query, {3}, lost).
   Softener accounts for **1/6 of the JAMES abstention set on this
   slice**. Real but small.

2. **Retrieval is the dominant abstention grounding source.**
   `DISABLE_RAG_RETRIEVAL` → ΔF1 -0.173, 5 of 6 abstentions lost
   + 2 spurious gained. Without retrieval, JAMES has no context to
   ground in, abstention naturally degrades — expected behavior,
   not a defect. Consistent with RAG-as-scaffolding literature
   (see `docs/research/cycle-gamma-prior-art-positioning.md`).

3. **Graph / Verify zero-op on abst_f1 = expected regression-check
   pass.** Both components have primary axes other than abstention
   (graph → path + graded; verify → graded). ΔF1 = 0 is what the
   layer-intent matrix predicts.

### ⚠️ Speculative findings (axis-misaligned, NOT defect evidence)

4. **`DISABLE_RERANK` / `DISABLE_TYPED_FILTER` /
   `DISABLE_COGNITIVE_STAGES` each show abst_f1 +0.050.**
   These are valid measurements but:
   - Each component's primary axis (noise / graded / path) was
     NOT measured in this Phase D matrix.
   - +0.050 may be entirely offset by losses on their primary
     axes (e.g., rerank could -0.20 noise_robustness).
   - n=25 noise band: +0.050 = 1-3 query swings, possibly within
     noise.
   - α-5 cycle #7 pattern: cost-axis layer measured on quality
     axis → false "harmful" verdict. Same trap here.
   - **DOES NOT support** "structural defect" / "default revision
     candidate" / "+0.15 lift potential" framing without
     multi-axis Pareto measurement.

---

## 4. Tier reassessment (post-Phase D)

| Tier | Status | Reason |
|---|---|---|
| ⭐ stable | ✅ | 7 ablation × n=25 paired matrix, 0 errors, 93 min compute. Measurement quality preserved. |
| ⭐⭐ confirmed | ⚠️ scoped narrowly | (a) Softener contributes 1 query to abstention set (axis-aligned). (b) Retrieval dominates abstention grounding (axis-aligned). (c) Graph/verify zero-op = expected. Other findings are speculative pending multi-axis follow-up. |
| ⭐⭐⭐ universal-law | ❌ still REJECT | Per `docs/research/cycle-gamma-prior-art-positioning.md`. Phase D adds component-level granularity but does not change literature subsumption. |

### What would change ⭐⭐ → ⭐⭐⭐?

Same gate as the broader cycle γ finding: multi-bench (ALCE /
MuSiQue / 2Wiki) × multi-model (gemma4 / llama / mxtral)
per-query overlap with consistent ablation matrix. Phase D is
single-bench / single-model component analysis — not novelty-grade.

---

## 5. What would be needed to claim "JAMES structural defect"?

For the speculative "rerank / typed_filter / cog_stages are
abstention-harmful" finding to become a genuine defect claim:

### Required additional measurements

1. **Multi-axis Pareto per disabled component**:
   - `DISABLE_RERANK` measured on noise_robustness + graded
     + latency simultaneously
   - `DISABLE_TYPED_FILTER` measured on path + abst_f1
     simultaneously
   - `DISABLE_COGNITIVE_STAGES` measured on graded + latency
     simultaneously
2. **Cross-model confirmation**: same ablation matrix on gemma4 +
   llama. If the +0.050 lift only shows on mxtral, the finding is
   model-specific.
3. **Larger n**: n=50 or n=100 to reduce noise band — +0.050
   represents 1-3 queries on n=25, may not survive noise.
4. **Net axis-weighted delta computation**: per layer-intent matrix,
   compute Pareto-dominance or trade-off classification.

### Scope estimate

Multi-axis ablation = each component's ablation run on workspace
#1 (full noise + negrej, 50 queries). 7 components × 50 queries ×
~15 min/run ≈ 9-10 hours compute per model. × 3 models = ~30
hours. Separate cycle (Phase E or independent research project).

Not a cycle γ continuation — different scope.

---

## 6. Open questions explicitly NOT closed by Phase D

- Does the softener-contribution {3} persist on gemma4 / llama? (n=1 from this set)
- Do the +0.050 ablation lifts persist across models?
- What's the net trade-off when 3 components are simultaneously disabled? (composability not measured)
- Why does retrieval-only mode (DISABLE_RAG_RETRIEVAL → set {3, 7, 24}) include query 3? (Query 3 is the {3} that softener was responsible for in R0. Without retrieval AND with softener still on, somehow still abstains — needs raw-output inspection.)

These are followups, not Phase D claims.

---

## 7. Files of interest

| Purpose | Path |
|---|---|
| Phase D results JSONs (7 ablations) | `reports/cycle_gamma/phase-d/*.json` (gitignored) |
| Phase D summary | `reports/cycle_gamma/phase-d/_summary.md` (auto-generated, gitignored) |
| Matrix runner log | `reports/cycle_gamma/phase-d/_matrix-mxtral.log` (gitignored) |
| Phase D design doc | `docs/design/v0.4-cycle-gamma-phase-d-mechanistic-ablation.md` |
| Matrix runner script | `scripts/research/cycle_gamma_phase_d_ablate.py` |
| Comparison tool | `scripts/research/cycle_gamma_rgb_compare.py` |
| Prior art positioning (cycle γ) | `docs/research/cycle-gamma-prior-art-positioning.md` |
| 6th catch rule (this session) | `memory/feedback_single_axis_ablation_misframing.md` |
| α-5 layer-intent matrix (mother rule) | `memory/mechanism_layer_intent_axis_alignment.md` |
| Phase A+B+C context | `docs/handovers/v0.4-cycle-gamma-phase-c-4model-true-signal-2026-06-08.md` |

---

## 8. Path 1 sequence update

```
Cycle β ✅
v0.4.2 T5 ✅
Cycle γ Phase A ✅ (10 PRs scaffolding)
Cycle γ Phase B Option A ✅ (4-model RGB-en, true-signal narrowed)
Cycle γ Phase C ✅ (4-model paired)
Cycle γ prior-art positioning ✅ (⭐⭐⭐ REJECT)
Cycle γ scope correction ✅ (cycle γ ≠ joint piece, rule #5)
Cycle γ Phase D mechanistic ablation ✅ (this doc; narrow valid findings + rule #6)
   ├─ softener contributes 1 query (axis-aligned, valid)
   ├─ retrieval dominates abstention (axis-aligned, valid)
   ├─ graph/verify zero-op (axis-aligned, expected)
   └─ rerank/typed_filter/cog_stages +0.050 on abst_f1 (axis-misaligned, NOT defect evidence)

Future arcs (separate cycles, NOT cycle γ continuation):
- Phase E multi-axis ablation (~30h compute) — only if defect claim becomes a priority
- Cross-bench (ALCE/MuSiQue/2Wiki) — operator gated, see Option B guide
- Joint piece M9 prep — collab arc, separate progression
```

---

## 9. Lessons embedded

| Catch # | What | Memory rule |
|---|---|---|
| 1 | post-hoc fit framed as prediction | `feedback_finding_size_honest_framing` |
| 2 | 0.000 negrej audit | (existing; audit script preserved) |
| 3 | family vs size separation | (drove Phase C mxtral measurement) |
| 4 | prior art search recommendation | `docs/research/cycle-gamma-prior-art-positioning.md` |
| 5 | cycle γ ↔ joint piece scope separation | `feedback_eval_cycle_vs_collab_arc_separation` |
| **6** | **single-axis ablation → "defect" framing violates layer-intent rule** | **`feedback_single_axis_ablation_misframing` (this PR)** |

Each catch narrowed framing without invalidating measurement data.
Measurement integrity = 200+ LLM calls (Phase B+C) + 175 LLM calls
(Phase D) preserved; framing claims serially reduced as evidence
boundaries clarified.

This session's measurement-to-framing ratio is the model for future
cycles: don't claim more than the axis-aligned evidence supports.
