# Design Memo — α-6 Sector × LLM Ablation Matrix

> **Status**: design. Successor to α-5 (T0 closed 2026-05-31, Branch B verdict).
> α-5 measured 5 routing flags at 1 model tier. α-6 reshapes to:
> (a) **sector-level** ablation (10 sectors of JAMES infrastructure),
> (b) **multi-LLM** comparison (gemma family + claude + openai + others),
> (c) **B+C from α-5 prior recommendation folded as α-6 Phase 0** —
> answer the user's actual question: *"Does the JAMES reasoning stack
> outperform plain gemma4 / vanilla RAG?"*
>
> **작성일**: 2026-05-31 PM (post α-5 T0 closure)
> **Trigger**: 2026-05-31 user clarification:
> *"기본 자메스 와 슈퍼 자메스의 차이가 정확히 뭐야"* +
> *"권장대로하되, 자메스 인프라별로 섹터를 스펙별로 묶어서 어떤게 추가되면 좋고 나쁜지가 나오도록 하고, 나중에는 젬마4 뿐아니라 다른 llm, 모델별로도 각각 측정할수 있게"*

---

## 0. TL;DR

α-5 toggled 5 routing flags at gemma4:e4b. The 3 cells that ran
(L1, L5, sanity) told us routing layers are inert at the production
tier, but they did NOT compare JAMES vs vanilla gemma4 or vanilla
RAG — because the L0 / L1 baselines already had every other JAMES
sector (vector / graph / citation / abstention / audit / lifecycle /
policy) silently on.

α-6 fixes both gaps:

1. **Sector ablation** — decompose JAMES into 10 sectors, each
   toggleable as a unit. Cells are sector combinations, not flag
   combinations.
2. **Multi-LLM extension** — measure each sector combination across
   multiple local Ollama models first (gemma3:1b/4b/12b/27b, gemma2:2b,
   gemma4:e4b, qwen2.5:7b, llama3.1:8b, deepseek-v2:16b — 9 inventoried
   locally as of 2026-05-31). API providers (claude / gpt / gemini)
   deferred to Phase 4+ per user clarification *"일단 자메스에 깔린
   로컬 모델들 부터 해보면될듯함"*.

The matrix is `sectors × LLMs`. Each cell answers
*"adding sector S to LLM L moves quality by Δ at cost Δ."*

---

## 1. Sector taxonomy (10 sectors)

| ID | Sector | Concrete code surface | Ablation difficulty |
|---|---|---|---|
| **S1** | Retrieval foundation (vector RAG) | `core/retrieval_engine.py`, `core/vector_store.py`, chroma + bge-m3 top-k | EASY — already a flag (`JAMES_RAG_DISABLED` or similar) |
| **S2** | Knowledge graph layer | `core/graph_engine.py`, `core/graph_rag_engine.py`, wiki entity traversal | MEDIUM — needs new flag `JAMES_DISABLE_GRAPH` |
| **S3** | Query preprocessing | `core/query_expander.py`, `entity_anchor`, `query_rewrite` | DONE — `JAMES_ENABLE_ENTITY_ANCHOR` / `JAMES_ENABLE_QUERY_REWRITE` |
| **S4** | Citation / Sources layer | `core/reasoning/pipeline.py:343` `sources: [d["source"] for d in docs[:3]]` | MEDIUM — needs flag `JAMES_DISABLE_SOURCES_FIELD` |
| **S5** | Abstention / Refusal | `core/reasoning/pipeline.py` abstention phrasings + empty-result short-circuit | MEDIUM — needs `JAMES_DISABLE_ABSTENTION` |
| **S6** | Cognitive stages | `core/reasoning/` planner / reflect / verify (separate from synth) | MEDIUM — per-stage flags exist; need bundle flag |
| **S7** | Routing layers (α-5 measured) | AUTO_ROUTER + ADAPTIVE_BUDGET + SCOPE_ROUTING | DONE — these are the α-5 flags |
| **S8** | Lifecycle (T1-T7) | `core/lifecycle/` validity / derivation / supersede | LOW-VALUE for benchmark — MultiHop-RAG has no time axis. Measure later. |
| **S9** | Policy + Trust | `core/policy_engine.py`, `core/memory/trust.py` | LOW-VALUE for benchmark — policy gates security, not answer quality. Measure later. |
| **S10** | Audit / Observability | `core/audit_*`, `core/observability.py` | NOT MEASURABLE on quality axes — replay capability is a different axis. Carry as note, not cell. |

**Measured sectors for α-6 quality matrix**: S1, S2, S3, S4, S5, S6, S7.
**Carried as notes** (not in cells): S8, S9, S10 — they're real
JAMES differentiators but don't move the 5-axis oracle on a
non-temporal benchmark like MultiHop-RAG.

---

## 2. The α-5 → α-6 mental shift

### α-5 cell schema (what we just did)

```
each cell = (routing_layer_combo, single_model)

L1 baseline = {ENTITY_ANCHOR=1, QUERY_REWRITE=1,
               AUTO_ROUTER=0, ADAPTIVE_BUDGET=0, SCOPE_ROUTING=0,
               model="gemma4:e4b"}

L5 full     = {ENTITY_ANCHOR=1, QUERY_REWRITE=1,
               AUTO_ROUTER=1, ADAPTIVE_BUDGET=1, SCOPE_ROUTING=1,
               model="gemma4:e4b"}
```

L1 ↔ L5 measures "adding 3 routing flags helps?" while every other
JAMES sector stays silently on.

### α-6 cell schema (proposal)

```
each cell = (sector_bitmap, LLM_provider)

C_minus     = {S1=0, S2=0, S3=0, S4=0, S5=0, S6=0, S7=0, model=M}
            = pure model, no JAMES at all (just /chat/ to the model)

C_rag_basic = {S1=1, S2=0, S3=0, S4=0, S5=0, S6=0, S7=0, model=M}
            = model + vanilla RAG (chroma top-k) only

C_rag_cited = {S1=1, S2=0, S3=0, S4=1, S5=0, S6=0, S7=0, model=M}
            = + citation/sources layer (S4)

C_rag_graph = {S1=1, S2=1, S3=1, S4=1, S5=0, S6=0, S7=0, model=M}
            = + graph + query preprocessing
            (closest to "JAMES production stack without routing/cognitive")

C_rag_full  = {S1=1, S2=1, S3=1, S4=1, S5=1, S6=1, S7=0, model=M}
            = JAMES full stack EXCEPT routing layers
            (this is α-5's L1!)

C_rag_routed = {S1=1, S2=1, S3=1, S4=1, S5=1, S6=1, S7=1, model=M}
            = JAMES full stack WITH routing layers
            (this is α-5's L5!)
```

α-6 = `{C_minus, C_rag_basic, C_rag_cited, C_rag_graph, C_rag_full, C_rag_routed}` × `{M = gemma4:e4b, gemma3:4b, gemma3:12b, claude-haiku-4-5, claude-sonnet-4-6, gpt-5-mini, ...}`.

α-5's L1 / L5 / sanity become exactly 2 cells of α-6
(`C_rag_full` and `C_rag_routed` at gemma4:e4b). They carry forward.

---

## 3. Cell costs (compute estimate)

At MultiHop-RAG balanced-100, 1 cell ≈ 110 min on local gemma4:e4b.
Estimate per provider class:

| Provider class | Per-cell cost (100 queries) | Constraints |
|---|---|---|
| Local Ollama (gemma3/4 family) | 90-120 min wall-clock | local compute, free |
| Local Ollama (larger models 12b+) | 200-400 min | local compute, slow |
| API (Anthropic claude-haiku-4-5) | 8-15 min, ~$0.10-0.30 | rate limit, $-cost |
| API (claude-sonnet-4-6) | 8-15 min, ~$1-3 | rate limit, $-cost |
| API (openai gpt-5-mini) | 8-15 min, ~$0.20-0.50 | rate limit, $-cost |
| API (gemini-2.5-flash) | 8-15 min, ~$0.10-0.30 | rate limit, $-cost |

**Full matrix size**: 6 cells × 6 models = 36 cells. At local-tier
average 100 min/cell × 30 local cells + ~10 min average × 6 API
cells = **52 hours wall-clock + ~$20-50 API cost** for the full
matrix.

**Phasing** (next section) cuts this drastically by running cells
adaptively.

---

## 4. Phasing — adaptive matrix

Don't run 36 cells in one go. Each phase gates the next.

### Phase 0 — α-5 closure carry-over (FREE, already have data)

Treat α-5's L1 + L5 cells as α-6's `C_rag_full(gemma4:e4b)` and
`C_rag_routed(gemma4:e4b)`. No new measurement; just relabel.

**Yields**: 2 cells of the matrix already filled.

### Phase 1 — The core comparison the user actually wanted (4 cells)

```
C_minus(gemma4:e4b)      — pure gemma4
C_rag_basic(gemma4:e4b)  — gemma4 + vanilla chroma RAG
C_rag_graph(gemma4:e4b)  — + graph + preprocessing
C_rag_full(gemma4:e4b)   — full JAMES stack (already have from α-5)
```

3 new cells × 110 min = ~5.5 hours, single tier. Answers the user's
question: *"does each JAMES sector add value?"*

Pre-req: implement flags for S1, S2, S4, S5, S6 (currently only S3
+ S7 have flags). **This is 2-3 days of engineering**.

### Phase 2 — Tier-gated check (3 cells, optional)

Same 4 Phase 1 sectors at M_S (gemma3:4b) — tests whether small
models benefit MORE from JAMES sectors (the tier-gating hypothesis
α-5 couldn't answer).

If `Δ(rag_full - minus)` at M_S is larger than at M_M → JAMES
helps small models more → tier-gated default-on.

### Phase 3 — Local-LLM family extension (user-clarified 2026-05-31: local first)

User clarification post-design:
*"위에서 llm 모델별이라는 것은 일단 자메스에 깔린 로컬 모델들 부터
해보면될듯함"*
→ Phase 3 = cross-family + cross-scale **local Ollama models** first.
API providers deferred to Phase 4+.

Inventoried locally (per `ollama list` 2026-05-31):

| Model | Size | Family | Role in α-6 Phase 3 |
|---|---|---|---|
| `gemma3:1b` | 815 MB | Google gemma3 | extreme-small tier (below current M_S) |
| `gemma3:4b` | 3.3 GB | Google gemma3 | M_S (already α-5 tier definition) |
| `gemma4:e4b` | 9.6 GB | Google gemma4 | M_M production (already α-5 measured) |
| `gemma3:12b` | 8.1 GB | Google gemma3 | M_L (already α-5 tier definition) |
| `gemma3:27b` | 17 GB | Google gemma3 | XL tier (above current M_L) |
| `gemma2:2b` | 1.6 GB | Google gemma2 | older-family small |
| `qwen2.5:7b` | 4.7 GB | Alibaba qwen | cross-family small/mid |
| `llama3.1:8b` | 4.9 GB | Meta llama | cross-family mid |
| `deepseek-v2:16b` | 8.9 GB | DeepSeek MoE | cross-family MoE |

**Phase 3 recommended sub-phasing** (run cheapest cells first):

| Sub-phase | Cells | Models | Rationale | Time |
|---|---|---|---|---|
| 3.a | `C_rag-full + C_rag-routed` × {gemma3:1b, gemma3:4b, gemma3:12b, gemma3:27b} | gemma3 family scale ladder | tests "does the α-5 inert verdict hold across gemma3 scales?" | ~12-18 h |
| 3.b | `C_rag-full + C_rag-routed` × {gemma2:2b, qwen2.5:7b, llama3.1:8b, deepseek-v2:16b} | cross-family at comparable size to gemma4:e4b | tests "is JAMES family-universal or gemma-specific?" | ~8-12 h |
| 3.c | `C_minus + C_rag-basic + C_rag-graph + C_rag-full` × {best-discriminating model from 3.a/b} | full sector ablation on the model where Δ is largest | answers "which JAMES sectors transfer to other families?" | ~6-9 h |

Skip:
- `qwen2.5-coder:*` (code-specialized; off-task for multi-hop QA)
- `llava:13b` (multimodal; no image queries in MultiHop-RAG)

**Cost: zero** ($ + auth) — all local compute. Wall-clock total
~26-40 hours across 3.a + 3.b + 3.c, runnable as background tasks
spread over multiple days.

### Phase 4 — API-backed LLM comparison (deferred until Phase 3 verdict)

Same 4 Phase 1 sectors at one API model
(`claude-haiku-4-5` recommended — cheapest, fast). Tells whether
JAMES sectors help an already-strong API model or are local-model
crutches. Gated on Phase 3 — if Phase 3 already shows
family-universality, Phase 4 mostly confirms; if Phase 3 shows
gemma-specific lift, Phase 4 disambiguates "local vs API" from
"family vs family."

Cost: ~1 hour wall-clock + ~$5 API per LLM.

### Phase 5 — Cross-provider broadening (optional, deferred)

Repeat Phase 4 on gpt-5-mini, gemini-2.5-flash for cross-provider
external evidence. Cost: ~$10-15.

### Phase 6 — Routing on top (defer)

Add S7 (routing) cells across the same matrix. α-5 already showed
S7 inert at gemma4:e4b; Phase 6 tests whether it's inert at smaller
models / cross-family models / API models too.

---

## 5. Engineering preconditions

### 5.1 New env flags needed (Phase 1 blocker)

| Sector | Flag | Default | Effect when 0 |
|---|---|---|---|
| S1 | `JAMES_DISABLE_RAG_RETRIEVAL` | 0 | bypass chroma top-k, send empty context to LLM |
| S2 | `JAMES_DISABLE_GRAPH` | 0 | skip graph traversal, `graph_paths` stays empty |
| S4 | `JAMES_DISABLE_SOURCES_FIELD` | 0 | `response.sources` not emitted (forces path axis to score 0 via_sources) |
| S5 | `JAMES_DISABLE_ABSTENTION` | 0 | always answer, never refuse |
| S6 | `JAMES_DISABLE_COGNITIVE_STAGES` | 0 | planner / reflect / verify stages skipped |

Implementation cost per flag: ~30-90 min including a contract test.
5 flags = ~1 day of work.

### 5.2 LLM provider abstraction (Phase 4 blocker, NOT Phase 3)

`core/llm_catalog.py` + `core/model_resolver.py` already support
Ollama (per `core/gemma_client.py`). **Phase 3 is local-only**
(per user clarification 2026-05-31) so no provider abstraction
extension needed for Phase 3.

For Phase 4+ API providers: check whether the existing modules
support claude / openai / gemini, or just Ollama. If just Ollama,
add provider adapters (~1-2 days per provider including auth +
cost tracking). **Defer until Phase 3 verdict.**

### 5.3 Cost tracking (Phase 4+ blocker, NOT Phase 3)

Local-only Phase 3 uses the existing chars-based `token_cost` axis
+ seconds-based `latency_cost` unchanged. Phase 4+ needs USD-based
cost axis for API providers; deferred.

### 5.4 Rate limiting (Phase 4+ blocker, NOT Phase 3)

Local Ollama has no rate limit (constrained by GPU only). Phase 4+
API providers need per-provider rate-limit-aware bench runner;
deferred.

### 5.5 AUTO_ROUTER multi-tier backend registration (Phase 1+ blocker)

Post-α-5 self-audit (2026-05-31 PM) surfaced that AUTO_ROUTER's
policy in `core/reasoning/router.py` requires multiple backends
registered with distinct `tier` capabilities (`small`/`medium`/`large`)
to be a meaningful routing layer. α-5 ran with ONLY the always-on
`ollama_local` backend (tier="small"); the policy's
`_first_in_tier("medium" / "large")` lookups returned None on every
branch, so AUTO_ROUTER collapsed to legacy = `JAMES_LLM_MODEL` for
every call.

**Implication**: any α-6 cell that intends to test AUTO_ROUTER must
register a multi-tier backend set first. Two viable patterns:

- **Pattern A** — register multiple ollama backends (one per model
  size) with distinct tier capabilities. New code: a multi-instance
  `OllamaTieredBackend` (`core/reasoning/backends/ollama_tiered.py`)
  that registers `ollama_small` / `ollama_medium` / `ollama_large`,
  each bound to a specific model tag. AUTO_ROUTER then routes among
  them; the called backend honors its bound model tag rather than
  the env-default. Implementation: ~1 day. Test seam: small extension
  to existing `test_router_*` tests.

- **Pattern B** — register an API provider (e.g. `claude_code_cli`
  at tier="large") alongside ollama (tier="small"). Smaller scope
  but only tests one ollama-vs-API axis; AUTO_ROUTER then routes
  across providers, not within ollama. This is the path Phase 4+
  already needs.

Recommended sequence: Pattern A first (still $0, exposes AUTO_ROUTER's
intra-family behavior cleanly), Pattern B as a bonus when Phase 4
lands.

### 5.6 Per-layer per-axis intent matrix (NEW design contract)

Post-α-5 self-audit also surfaced that not every routing layer
should be judged by the same 5-axis verdict. Each layer has a
**design intent** specifying which axes it should move. Judging a
cost-optimization layer by quality axes (or vice versa) miscategorises
its verdict.

| Layer | Design intent (per code docstrings + closure memories) | Axes the matrix should judge it on |
|---|---|---|
| **AUTO_ROUTER** (D5) | Right-size the model per query for both quality (large for hard) and cost (small for easy) | path / graded / abst_f1 (quality) + token / latency (cost). All 5. |
| **ADAPTIVE_BUDGET** (D1) | Reduce wasted tokens / latency. **Quality-neutral by design** ("zero quality regression" per `direction_1_adaptive_budget_closure`) | token / latency primary. Quality axes as **regression check** (within noise) only. |
| **SCOPE_ROUTING** (LEO) | Route based on retrieval evidence scope — narrow → small / cheap, wide → large / thorough | path + token + latency primary. Graded / abst_f1 as secondary quality-neutral check. |
| **ENTITY_ANCHOR + QUERY_REWRITE** (S3 in α-6 taxonomy) | Improve retrieval recall / fluency on query side | path / graded primary. Cost axes secondary. |
| **Citation layer** (S4) | Surface source documents in answer | path primary. Others secondary. |
| **Graph layer** (S2) | Concept-centric multi-hop traversal | path + graded primary. Others secondary. |
| **Abstention layer** (S5) | Refuse when grounding insufficient | abst_f1 primary. Quality protection on truth-present queries (FP guard). |
| **Cognitive stages** (S6) | Multi-step planning / reflection / verification | graded primary. Latency cost as expected trade-off. |

α-6 cell verdicts use the per-layer intent column rather than a
uniform 5-axis Pareto. A layer that hits its intent axes and stays
within noise on its non-intent axes is "adopt"; a layer that misses
its intent axes is "reject" regardless of stable non-intent
behavior.

**This is the bucket-(a) layer-intent-axis-mismatch sub-category** of
the 4-step rule generalisation, locked in memory
`feedback_oracle_phrase_artifacts` §"확장 3".

### 5.6.1 `verdict_per_layer` JSON schema (cell-level)

Each α-6 cell JSON gains a `verdict_per_layer` block alongside the
existing `aggregate` block. The new block is canonical for cells
where one or more sector flags differ from the L1 baseline; cells
that don't toggle any sector emit an empty `verdict_per_layer: []`
and fall back to the legacy 5-axis Pareto verdict.

```json
{
  "verdict_per_layer": [
    {
      "layer_id": "ADAPTIVE_BUDGET",
      "flag": "JAMES_ADAPTIVE_BUDGET",
      "flag_value": "1",
      "design_intent": "토큰 효율 (quality-neutral)",
      "primary_axes": ["token_cost", "latency_cost"],
      "regression_check_axes": ["path_coverage", "graded_answer", "abstention_f1"],
      "prerequisites_met": true,
      "prerequisite_notes": null,
      "per_axis_delta": {
        "token_cost": {"delta": -19.13, "noise_band": 50.0, "verdict_contribution": "flat"},
        "latency_cost": {"delta": +2.38, "noise_band": 5.0, "verdict_contribution": "flat"},
        "path_coverage": {"delta": -0.007, "noise_band": 0.02, "verdict_contribution": "within-noise"},
        "graded_answer": {"delta": -0.010, "noise_band": 0.02, "verdict_contribution": "within-noise"},
        "abstention_f1": {"delta": -0.091, "noise_band": 0.05, "verdict_contribution": "regression"}
      },
      "verdict": "reject",
      "reason": "regression on regression-check axis (abstention_f1) without compensating primary-axis improvement"
    }
  ]
}
```

Field semantics:

| Field | Meaning | Required |
|---|---|---|
| `layer_id` | canonical short name (matches the per-layer intent table) | yes |
| `flag` | env flag controlling the layer | yes |
| `flag_value` | value in this cell (`"0"` / `"1"`) | yes |
| `design_intent` | one-line statement (from §5.6 table) | yes |
| `primary_axes` | axes the layer is designed to move | yes |
| `regression_check_axes` | axes the layer should not move past noise | yes |
| `prerequisites_met` | bool — were the layer's measurement preconditions met? (e.g., multi-tier backend for AUTO_ROUTER) | yes |
| `prerequisite_notes` | string or null — explains "not in evidence" if prerequisites missed | conditional |
| `per_axis_delta` | per-axis Δ + noise_band + verdict_contribution | yes |
| `verdict_contribution` per axis | one of `"primary-improvement"` / `"primary-flat"` / `"primary-regression"` / `"within-noise"` / `"regression"` (for regression-check axes) | yes |
| `verdict` | one of `"strong-adopt"` / `"adopt"` / `"efficiency-adopt"` / `"tier-gated"` / `"reject"` / `"zero"` / `"not in evidence"` | yes |
| `reason` | one-sentence justification | yes |

### 5.6.2 Verdict rule (per-layer)

```
if not prerequisites_met:
    verdict = "not in evidence"
else if any regression_check_axes has verdict_contribution == "regression":
    verdict = "reject"
else if all primary_axes within noise:
    verdict = "zero"
else if primary_axes positive AND cost axes improved:
    verdict = "strong-adopt"
else if primary_axes positive AND cost axes flat:
    verdict = "adopt"
else if primary_axes flat AND cost axes improved:
    verdict = "efficiency-adopt"
else if primary_axes positive AND cost axes regressed:
    verdict = "tier-gated"  # explain per tier in `reason`
```

The legacy `_classify_five_axis_delta` (in
`scripts/qvt_ablation_matrix.py`) stays for α-5 cells; α-6 cells
add the per-layer block alongside it. Matrix render aggregates the
per-layer verdicts up; the uniform 5-axis verdict serves as the
last-resort summary when sector flags don't apply.

### 5.6.3 Application example — the α-5 L5 cell, re-judged

If we apply the new schema to α-5's L5 cell (which had
ADAPTIVE_BUDGET + SCOPE_ROUTING + AUTO_ROUTER on), the per-layer
verdicts come out as:

| Layer | prerequisites_met | primary_axes Δ | regression_check Δ | verdict |
|---|---|---|---|---|
| AUTO_ROUTER | **false** (single-backend) | n/a | n/a | **"not in evidence"** |
| ADAPTIVE_BUDGET | true | token -1.5%, latency +3.6% | abst_f1 -0.091 | **"reject"** (regression on quality protection axis) |
| SCOPE_ROUTING | true | path +0%, token -1.5%, latency +3.6% | graded -1%, abst_f1 -9% | **"reject"** (regression on abst_f1 + graded) |

The honest cell-level summary: routing-stack inert OR mildly
regressive; one layer cannot be evaluated.

This matches the corrected α-5 closure narrative (§6.2.5 of
`alpha-5-publishable-narrative-DRAFT.md`).

---

## 6. External baseline comparison (the "C" from α-5 prior recommendation)

Independently of the in-house matrix, cross-reference against
published baselines for the same benchmark. MultiHop-RAG paper
(Tang & Yang, EMNLP 2024) reports baselines for various RAG
configurations.

Procedure:
1. Read their evaluation methodology (`metrics.py` in their repo).
2. Map their metrics → our 5-axis where mappable.
3. Reproduce one of their published baseline configurations
   locally (e.g., vanilla LangChain RAG with bge-m3 + a small
   model) to confirm our test bed reproduces their numbers
   ±noise.
4. Pin the reproduced numbers as an external anchor in the matrix.

This deliverable lands as a separate doc:
`reports/research-runs/multihop-rag-paper-baseline-comparison.md`.

Effort: 1-2 days of engineering, no compute beyond rerunning one
existing configuration.

---

## 7. Naming convention

Cell name format: `C_<sector-spec>_<model-tag>`

- `C_minus_gemma4-e4b` — pure model
- `C_rag-basic_gemma4-e4b` — + S1
- `C_rag-cited_gemma4-e4b` — + S1 + S4
- `C_rag-graph_gemma4-e4b` — + S1 + S2 + S3 + S4
- `C_rag-full_gemma4-e4b` — JAMES no-routing (α-5 L1)
- `C_rag-routed_gemma4-e4b` — JAMES + routing (α-5 L5)
- `C_rag-full_claude-haiku-4-5` — JAMES no-routing on API
- ...

Cell JSON filename follows: `qvt-ablation-cell-C_<spec>_<model>.json`.

---

## 8. Deliverables per α-6 phase

| Phase | Deliverable | Trigger | $ | Time |
|---|---|---|---|---|
| 0 | Relabel α-5 cells into α-6 namespace | free, immediate | $0 | 0 |
| Engineering pre | 5 new env flags (S1, S2, S4, S5, S6) | engineering decision | $0 | ~1 day |
| 1 | 3 new cells answering "does each JAMES sector add value at gemma4:e4b?" | Phase 0 + Engineering pre | $0 | ~5.5h |
| 2 | 3 cells at M_S (gemma3:4b) — tier-gated check | Phase 1 verdict has signal | $0 | ~3h |
| **3a** | gemma3 scale ladder: 1b / 4b / 12b / 27b × {C_rag-full, C_rag-routed} | Phase 1 | $0 | ~12-18h |
| **3b** | cross-family: gemma2:2b / qwen2.5:7b / llama3.1:8b / deepseek-v2:16b × {C_rag-full, C_rag-routed} | Phase 1 | $0 | ~8-12h |
| **3c** | full sector ablation on best-discriminating model from 3a/b | 3a + 3b | $0 | ~6-9h |
| 4 | claude-haiku — multi-LLM check, API | Phase 3 + LLM provider abstraction | ~$5 | ~1h |
| 5 | gpt + gemini cross-family broadening | Phase 4 confirms | ~$10-15 | ~1h |
| 6 | Routing-layer interaction with smaller / cross-family / API models | independent, low-priority | TBD | TBD |

Each phase outputs:
- Matrix render with new column / new rows
- Per-cell JSON in workspace
- Cycle summary doc update
- Findings entry per surprising Δ

---

## 9. Open methodology questions

### 9.1 Is `C_minus` (pure LLM) reasonable to call "RAG-free"?

`C_minus` sends just the user's question to the model — no context.
For a true "RAG-free" comparison, the model gets no documents at
all. This is the right baseline for *"does retrieval help at all?"*
but pure LLMs on MultiHop-RAG will likely score very low on
path/graded (no docs cited, no facts). The point is the comparison,
not the absolute number.

### 9.2 Provider-cost axis vs local-compute axis

API providers don't have a meaningful "tokens" cost (they bill by
input+output tokens which doesn't map onto our chars-based
proxy). Cost axis for API cells becomes USD; cost axis for local
cells stays seconds/chars. The matrix needs **two cost columns**
when mixed.

### 9.3 What if Phase 1 shows a sector hurts (negative Δ)?

E.g., adding S5 (abstention) might lower graded_answer because it
short-circuits some borderline-answerable queries. If so, the
finding is bucket-(a) (architecture: refusal threshold too eager)
or bucket-(b) (model-specific: abstention pattern model-dependent).
The 4-step rule from
`memory/feedback_oracle_phrase_artifacts.md` applies before
proposing a fix.

### 9.4 Should S8 / S9 / S10 (lifecycle / policy / audit) get measured at all?

These three sectors are JAMES's load-bearing differentiators
(Replayable RAG ⊂ S10 + S8) but don't move quality axes on a
non-temporal benchmark. Three options:
- (a) Skip — note as "not measurable on this benchmark"
- (b) Build a temporal benchmark for S8 / S10 — separate research direction
- (c) Use existing audit-replay invariants as deterministic
  contract tests (already done in v0.4.0 test suite)

Recommendation: (c) — already covered by `test_t7_release_gating_invariants.py`;
no need to add to α-6.

### 9.5 Multi-LLM identity invariance

If the matrix shows JAMES helps claude AND gemma4 AND gpt
equally, that's a strong universality claim. If only gemma4
benefits, that's "JAMES is a small-model crutch" which is a real
position (not bad, just specific).

---

## 10. Position guard (per user clarification 2026-05-31)

The user explicitly flagged
*"Replayable RAG 도 자메스의 특성 중 하나라 봐야할것같고 아예 전체를 거기에
끼워맞추려하면 원래 목적을 잃을수도 있으니"*
— don't reframe the platform identity around one differentiator.

Applied here:
- α-6 measures **JAMES sectors' marginal contributions**, not
  "Replayable RAG validation."
- Outputs frame as *"sector S adds Δ at LLM L"*, not as *"more
  evidence for Replayable RAG."*
- Sector S8 / S10 (lifecycle / audit — the Replayable RAG bits)
  stay out of cells; they're acknowledged as orthogonal axes the
  benchmark can't measure.
- Mother-platform identity (local-first auditable knowledge
  reasoning platform, 6-dim readiness toward v1.0) stays primary
  in any α-6 publication.

---

## 11. Cycle naming + relation to α-5

α-5 cycle stays closed. T0 evidence published as
"production-tier routing-layer verdict." α-6 cycle is a separate
ship.

| Cycle | Question | Status |
|---|---|---|
| α-5 (closed) | "Do routing layers help at production tier?" | No (Branch B) |
| α-6 Phase 1 | "Does each JAMES sector add value at gemma4:e4b?" | scheduled |
| α-6 Phase 2 | "Tier-gating — do smaller models benefit more?" | gated |
| α-6 Phase 3 | "Are JAMES sectors LLM-universal?" | gated |
| α-6 Phase 4 | "Cross-family broadening" | optional |

Each phase ends with publishable closure. The full α-6 doesn't
need to run end-to-end before the next phase becomes meaningful.

---

## 12. Out of scope for this design memo

- Implementation of the 5 new sector flags — separate PRs, one per flag
- LLM provider adapter implementations — separate PRs per provider
- API auth / billing integration — operator action
- Specific MultiHop-RAG paper baseline reproduction — separate
  measurement deliverable
- Cycle-closure narrative integration — wait for Phase 1 results
- Cost projection for the full matrix — refined per phase

---

## 13. References

- α-5 cycle closure: ROADMAP §v0.4.x α-5 cycle "T0 verdict + post-closure"
- α-5 publishable narrative (Branch B): `reports/research-runs/alpha-5-publishable-narrative-DRAFT.md` §6.2
- α-5 backlog reconciliation §7.10: `docs/handovers/v0.4.x-backlog-reconciliation-2026-05-30.md`
- Matrix runner: `scripts/qvt_ablation_matrix.py`
- 4-step rule: `memory/feedback_oracle_phrase_artifacts.md`
- Position guard: `memory/feedback_jameses_positioning_replayable_rag.md` §"정정 2026-05-31"
- LLM provider modules to extend: `core/llm_catalog.py`, `core/model_resolver.py`, `core/gemma_client.py`
