# α-8 Cloud Tier Extension — Design Memo

> Status: **DESIGN / pre-implementation** (2026-06-04). Extends the
> existing α-8 measurement matrix (`scripts/qvt_ablation_matrix.py`)
> with a new model tier `M_CLOUD` so Direction α (cloud reasoning)
> quality differentiation is measured *inside* the existing 6-cell ×
> 5-tier × 5-axis infrastructure, not as a separate ad-hoc script.
>
> Per design memo §4.1 sequencing — "build cloud tier first; run
> local-vs-cloud quality differentiation as an extension of the α-8
> test line, **integrated and progressive, alongside the build**" —
> the canonical realization of "integrated and progressive" is **a
> tier inside the existing matrix**, not a standalone paired script.
> This memo formalizes that integration.

## 0. Why this memo exists (and why a tier, not a script)

This session previously shipped `scripts/research/local_vs_cloud_paired.py`
(S4, PR #701) as a paired n=3 LLM-judge harness — useful as a
reasoning-isolated supplement but **not the canonical Direction α
measurement**, because:

1. The α-8 matrix runner already exists with full server-spawn-per-cell
   isolation, 5-axis non-saturating oracle, paired N=3 + variance
   bands, paired-diff renderer (`compare_paired_cells.py`), and cell
   JSONs that the entire α-cycle uses as the verdict substrate.
2. The 5-axis oracle (path / graded / abstention_f1 / token_cost /
   latency_cost) is **stricter** than the LLM-judge in `paired.py` —
   it does not have the `lenient_judge` caveat (`memory/
   project_direction_alpha_local_vs_cloud_quality_thread §Caveat #1`).
3. The matrix runs against the **full retrieval pipeline** (JAMES
   server fresh-spawned per cell), not the gold-evidence reasoning-
   isolated fixture, addressing the `gold-evidence ≠ real noisy
   retrieval` caveat (#2 of the same memo).
4. Cloud tier inserted into the matrix produces immediate **marginal
   contribution per cell × per question_type** by reuse of the
   paired-cell diff infrastructure that already drove the α-5/6/7/8
   verdicts.

This memo binds the cloud measurement work to (1)-(4) instead of
recreating the infrastructure as a paired-only side script.

## 1. Stage 0 decisions (operator-confirmed 2026-06-04)

| # | Decision | Value | Rationale |
|---|---|---|---|
| 0.1 | Cloud tier id | `M_CLOUD` (single tier) | One Claude default model first; Opus/Sonnet split deferred to Stage 6 result |
| 0.2 | Measurement fixture priority | `step7` (16Q) first, then `multihop_rag` (100Q) | Fast + broad + regression-safe. multihop_rag added at sub-stage 4c if Stage 4 verdict warrants the cost |
| 0.3 | Measurement scope escalation | smoke (1 cell × N=1 × 5Q) → 4a (1 cell × N=3 × step7) → 4b (2 cells × N=3 × step7) → 4c (multihop) → 4d (6-cell full) | Cost dial — escalate only on positive prior-stage signal |
| 0.4 | Judge protocol | 5-axis oracle only (no LLM-judge supplement) | α-8 matrix standard; LLM-judge belongs to `paired.py` reasoning-isolated supplement, not the main verdict |

## 2. Matrix runner changes — exact code shape

### 2.1 `_TIER_MODELS` extension (the `JAMES_LLM_MODEL` semantic gap)

Today `_TIER_MODELS[tier]` maps to an **Ollama model tag** that
`_cell_env()` assigns directly to `JAMES_LLM_MODEL`:

```python
# core/reasoning/.../router resolves a backend; JAMES_LLM_MODEL is the
# model tag the backend asks Ollama for. For Ollama backends the tag IS
# the routing key. For cloud backends the tag is meaningless — the
# backend (claude_code_cli) ignores it.
_TIER_MODELS: Dict[str, str] = {
    "M_XS":  "gemma3:1b",
    "M_S":   "gemma3:4b",
    "M_M":   "gemma4:e4b",
    "M_L":   "gemma3:12b",
    "M_XL":  "gemma3:27b",
}
```

Cloud tier breaks this 1:1 mapping. Two design options:

| Option | Shape | Trade-off |
|---|---|---|
| A. Extend `_TIER_MODELS` value to a struct | `{"M_CLOUD": {"model": "", "backend_id": "claude_code_cli", "extra_env": {...}}}` | Breaking change to existing 5 entries' value type; touches `_cell_env`, `_run_cell` payload, `_render_report` simultaneously |
| **B. Keep `_TIER_MODELS` as-is + add sibling `_TIER_BACKEND_OVERRIDE`** | `_TIER_MODELS["M_CLOUD"] = ""` (sentinel) and `_TIER_BACKEND_OVERRIDE["M_CLOUD"] = {"JAMES_REASONING_BACKEND": "claude_code_cli", "JAMES_ENABLE_CLAUDE_BACKEND": "1", "JAMES_FORCE_CLOUD": "1"}` | Local tiers stay byte-identical; M_CLOUD-only divergence isolated to two new dict entries + 4 conditional branches |

**Decision: Option B.** Minimizes regression surface; existing 5 tiers
are not touched at the data-structure level. Conditional logic for the
M_CLOUD branch is ≤ 4 lines per call site.

### 2.2 `_cell_env()` divergence (line 412)

Current:

```python
env["JAMES_LLM_MODEL"] = _TIER_MODELS[tier]
```

New (additive, default branch unchanged for local tiers):

```python
if tier in _TIER_BACKEND_OVERRIDE:
    # Cloud tier — JAMES_LLM_MODEL meaningless (backend ignores it);
    # apply the cloud routing env triplet so server-spawn boots into
    # the cloud-routed path. Pre-α-8-cloud-tier 5 local tiers unaffected.
    env.update(_TIER_BACKEND_OVERRIDE[tier])
    # Leave JAMES_LLM_MODEL unset — claude_code_cli ignores it.
else:
    env["JAMES_LLM_MODEL"] = _TIER_MODELS[tier]
```

### 2.3 `_run_cell()` payload (line 664)

Current payload includes `"model": _TIER_MODELS[tier]`. For M_CLOUD
this would write `""` which is uninformative. Add a `"backend_id"`
field — populated from `_TIER_BACKEND_OVERRIDE[tier]["JAMES_REASONING_BACKEND"]`
when present, else `"ollama_local"` (the default backend for the
existing 5 tiers):

```python
backend_id = (
    _TIER_BACKEND_OVERRIDE[tier].get("JAMES_REASONING_BACKEND",
                                      "ollama_local")
    if tier in _TIER_BACKEND_OVERRIDE
    else "ollama_local"
)
payload = {
    ...
    "model": _TIER_MODELS[tier],          # "" for M_CLOUD, model tag otherwise
    "backend_id": backend_id,             # NEW — cloud cells show
                                          #       "claude_code_cli";
                                          #       local cells stay
                                          #       "ollama_local"
    ...
}
```

Bump cell schema: `qvt-ablation-cell-v3` → `qvt-ablation-cell-v4`
when `backend_id` field is present. Older renderers read v2/v3 cells
fine (missing `backend_id` → defaults to `"ollama_local"` in renderer
fallback). Forward-compat is one-way.

### 2.4 `_render_report()` row format (line 860)

Current row:

```
| {row} ({row_label}) | {tier} | `{model}` | {deltas[...]} | ... |
```

For M_CLOUD `{model}` is empty. New format that's compatible with
both old (model only) and new (model + backend_id):

```python
model_display = c.get("model") or ""
backend_display = c.get("backend_id", "ollama_local")
if model_display:
    tag = f"`{model_display}`"          # local: `gemma4:e4b`
else:
    tag = f"`{backend_display}`"        # cloud: `claude_code_cli`
```

### 2.5 Where M_CLOUD lives in the iteration

`_render_report()` iterates `_TIER_MODELS` keys (line 817 / 828). Adding
`M_CLOUD` to `_TIER_MODELS` auto-includes it in renderer iteration —
no separate render path needed.

`--tiers M_CLOUD` CLI selector works unchanged (existing arg parser
filters against `_TIER_MODELS` keys).

## 3. Caveat boundary — what this measurement does and does NOT close

The 5 v2 caveats from `memory/project_direction_alpha_local_vs_cloud_
quality_thread §Caveat` mapped against what M_CLOUD-in-matrix actually
addresses:

| # | Caveat | This measurement | Status |
|---|---|---|---|
| 1 | Lenient judge (contradictory both CORRECT) | 5-axis oracle is **deterministic** + strict (gold_signal matching, abstention_truth ground truth); no LLM-judge ambiguity | **CLOSED** by oracle protocol |
| 2 | Gold-evidence ≠ real noisy retrieval | matrix runs against fresh-spawned server with the real retrieval pipeline (hybrid + graph + rerank + abstention) | **CLOSED** by infrastructure |
| 3 | Easy first-N cases | step7 (16Q) is the v0.4 regression suite — known mixed difficulty (negative / dedup / lang-mix / security / meta categories). multihop_rag (100Q at sub-stage 4c) adds hop-count 2-4 + null-query | **PARTIALLY CLOSED**; closer at 4c with multihop_rag |
| 4 | Small n | matrix N=3 paired per cell | **PARTIALLY CLOSED**; same n as α-8 closure (see `feedback_n1_verdict_inflation_n3_caught` for the n=3 confirm rule) |
| 5 | gemma3:4b ≠ gemma4:e4b (production tier) | M_CLOUD is compared against **all** local tiers, including M_M = gemma4:e4b production tier | **CLOSED** by tier sweep |

→ Stage 4 (step7 4a/4b) closes caveats 1, 2, 5 fully and 4 to the same
level as α-8 closure. Stage 4c (multihop_rag) is the gating sub-stage
for caveat 3 — only run if Stage 4 step7 result shows directional
signal worth the multihop cost.

This is **what** the measurement actually proves vs. doesn't prove —
must be cited verbatim in the Stage 5 report's caveat block.

## 4. Cost model — calls per stage (hard upper bound)

`claude -p` Max-plan call estimate per scope (single-query ≈ one claude
call per stage 'reason:egress', no judge call):

| Stage | Scope | Bench calls per cell-run | Cell-runs | **Total cloud calls** | Wall time est. |
|---|---|---:|---:|---:|---:|
| 3 (smoke) | 1 cell × M_CLOUD × N=1 × 5Q | 5 | 1 | **5** | ~3-5 min |
| 4a | C_rag-ontology × M_CLOUD × N=3 × step7 (16Q) | 16 | 3 | **48** | ~15 min |
| 4b | + C_rag-full × M_CLOUD × N=3 × step7 | 16 | 3 | **+48 (96 cumulative)** | ~+15 min |
| 4c | + multihop_rag (100Q × N=3 × 2 cells) | 100 | 6 | **+600 (696 cumulative)** | ~+3-5h |
| 4d | 6-cell coverage × M_CLOUD × N=3 × step7 | 16 | 18 | **+96 cumulative on top of 4b** (= 192 total step7) | ~+30 min |

**Default plan: smoke → 4a → 4b, then verdict gate for 4c/4d.** Total
~100 cloud calls if stopping after 4b — comfortably inside Max-plan
daily budget per `memory/feedback_direction_alpha_max_plan_research_cloud`.

**Pre-flight cost guard**: matrix runner prints estimated cloud-call
count when M_CLOUD is in `--tiers` AND the operator has not set
`JAMES_CLOUD_CALL_BUDGET=<n>`. Above the budget the runner refuses
to start until the operator either raises the budget or trims scope.
Implementation in Stage 2 PR; budget env name reserved here.

## 5. Schema migration — cell JSON v3 → v4

Forward-compatible additive change. Existing v3 cells (already on
disk from α-8) read unchanged in the v4 renderer (missing
`backend_id` defaults to `"ollama_local"`). New v4 cells include
`"backend_id"` so a cloud cell is unambiguously identifiable.

Cell JSON paths unchanged (`qvt-ablation-cell-<cell-or-row>-<tier>.json`).
M_CLOUD cells land at `qvt-ablation-cell-C_rag-ontology-M_CLOUD.json`
etc. — naming consistency with the existing 30+ local-tier cells.

`compare_paired_cells.py` operates on cell JSON pairs and treats all
tiers identically (the diff is value-based, not tier-aware), so no
change needed for paired-diff usage.

## 6. Renderer changes for the local-vs-cloud comparison

The existing `_render_report()` produces a row-per-cell verdict table
(per `_classify_five_axis_delta` rule). With M_CLOUD added, the table
gains M_CLOUD rows for each enabled cell. Two follow-up rendering
additions (in Stage 5, not Stage 2):

- **Local-best vs cloud diff column** — for each cell, compute
  `Δ(M_CLOUD − best_local_tier)` per axis. This is the operator-
  facing answer to "does cloud beat the best local tier on this cell?".
- **By question_type cross-tab** — Stage 5 report includes a
  per-question-type table showing where cloud wins (currently only
  inference_query / comparison_query / temporal_query if multihop_rag
  is included).

These are Stage 5 report writer additions, not Stage 2 runner
changes. The cell JSONs already carry `aggregate_by_question_type`
(α-5 cross-tab field).

## 7. Server-side wiring (already in place)

Cloud-routing wire from S5 (PR #702) + Windows env / cwd fix from S5c
(PR #704) are already on `main`. The matrix runner's `_spawn_server`
boots `server_llmwiki:app` under the cell env, so setting the cloud
routing triplet via `_TIER_BACKEND_OVERRIDE["M_CLOUD"]` reaches the
synth call path through `trace_synth_call`:

1. Server boots with `JAMES_FORCE_CLOUD=1 + JAMES_REASONING_BACKEND=claude_code_cli + JAMES_ENABLE_CLAUDE_BACKEND=1`
2. `bench.py` issues `/query/` calls against the server
3. Each `/query/` enters the synth stage → `trace_synth_call` → router resolves `claude_code_cli` backend → `force_cloud_enabled()` returns True → `run_cloud_egress` wraps the call → real `claude -p` returns answer → unmask → audit
4. Bench output JSON captures the answer + latency + token count
5. Cell-runner aggregates 5-axis oracle scores from the bench JSON

No new server-side code. This entire wiring path is the Stage 1-3 +
S5 build's pay-off.

## 8. Out of scope (deliberately deferred)

- **S6 difficulty router design** — gated on this measurement's
  Stage 5 report. Specific signals (hop count / question_type / first-
  pass low-confidence) confirmed by data, not speculation.
- **S7 full egress masking policy** — gated on a fixture with real
  entity-sensitive PII (current step7 / multihop_rag have no PII; mask
  is no-op). Cycle to construct such a fixture is a separate design
  memo.
- **`local_vs_cloud_paired.py` deprecation** — kept as reasoning-
  isolated supplement (gold evidence) measuring a different axis than
  the matrix. Deprecation only if Stage 5 finds it provides no
  unique signal vs the matrix.
- **Cloud tier UI picker (S5b)** — env-flag pattern continues; UI
  surface gated on operator demand.

## 9. Acceptance — Stage 5 report verdict tree

Once Stage 4 produces cell JSONs, Stage 5 report applies the existing
α-8 verdict tree from `memory/project_alpha_8_closure_state §verdict`
+ `feedback_finding_size_honest_framing`:

| `M_CLOUD − M_M` median graded Δ | Verdict | Implication |
|---|---|---|
| `≥ +0.030` outside noise band | ⭐⭐ **adopt** | Cloud tier proven valuable; S6 difficulty-router design entered with data |
| `+0.010 to +0.030` | tier-gated | Conditional on cell × question_type pattern; S6 design entered with narrower scope |
| `< +0.010` | ⭐ operational only | Direction α premise still unproven; v0.5 domain pilot prioritized over cloud-tier extension |
| `< 0` | reject | Cloud tier underperforms production local on this fixture; revisit fixture construction (caveat 3 sub-stage 4c) before any further cloud cycle |

**Honest framing rule** (per `feedback_finding_size_honest_framing` +
`feedback_alpha_cycle_discovery_loop_end`): any positive verdict
labeled with the **⭐ tier explicitly**, prior-art check (frontier-
cloud-beats-local-7B-on-multihop is the literature default
expectation — Pinecone 2024 / Semantic Invariance 2026), and **n=3
paired confirm rule** before any closure / collaborator share.

## 10. 한국어 요약

α-8 측정 매트릭스 (`scripts/qvt_ablation_matrix.py`)에 `M_CLOUD`
티어 하나 추가. design memo §4.1의 "build alongside, integrated"
sequencing 결정의 정확한 실현 = **별도 paired script가 아니라 기존
6-cell × 5-tier 매트릭스 안에 cloud 티어 추가**. 5-axis 결정론적
oracle + N=3 paired + 실 retrieval pipeline = v2 9Q paired script의
5 caveat 중 #1/#2/#5 완전 closure, #3/#4 부분 closure (4c multihop
sub-stage에서 #3 완료). Option B 채택 — `_TIER_MODELS`는 그대로
두고 `_TIER_BACKEND_OVERRIDE` sibling dict 추가, M_CLOUD-only
divergence 4 line 안 분기. cell JSON schema v3→v4 forward-compat
(missing `backend_id`은 `"ollama_local"` default). 비용 가드 =
smoke → 4a → 4b 단계적 escalate, 100 calls 안에서 verdict 결정.
S6/S7 design은 Stage 5 report 결과 후. 측정 자체가 oracle 게이트.
