# V3' Protocol v1 — A reproducible measurement protocol for LLM cost behavior

> **Status**: v1, frozen 2026-05-24
> **Schema policy**: additive only (see §10 Versioning)
> **Reference driver**: `scripts/research/v3prime_e_mode_split.py` (`gemma4:e4b`, T=0.2, N=20/cell)
> **Reference data**: `reports/research-runs/v3prime-e-mode-split-20260523T151159.json`
> **Reference report**: `reports/promo-assets/v3prime-e-substitution-synthesis-result.md`
> **External adopter** (first lab to mirror the protocol): Robin Converse — [triavalabs/gemma4-26b-mode-split](https://github.com/triavalabs/gemma4-26b-mode-split), 2026-05-23
> **Audience**: any lab measuring LLM cost behavior (token efficiency, output determinism, mode entry costs) across stacks / models / task workloads

---

## 1. Purpose

V3' Protocol v1 specifies a **reproducible measurement procedure** for three independent axes of LLM cost behavior:

| Axis | What it measures |
|---|---|
| **1. Mode split** | substitution (verbatim retrieval) vs synthesis (new reasoning), and whether sampling layer is bypassed in either mode |
| **2. Workload gradient** | how synthesis cost scales with task weight (heavy / light / no synthesis) |
| **3. Model-scale efficiency** | how synthesis cost (token count + answer convergence) scales inversely with parameter count |

The protocol is designed so two independent labs can produce **directly comparable result JSONs** by following this spec without reading each other's code. Robin Converse's 2026-05-23 26b sweep is the first external adoption — her `sweep.py` is independent code that produces a JSON conforming to this schema, and her analysis pulled the JAMES JSON as the cross-stack template.

This document is **versioned and citable**. If you cite a measurement made under this protocol, cite this file at its commit hash for unambiguous reference.

---

## 2. Protocol overview

A V3' measurement is a **single-variable sweep** over `num_predict` (token cap budget), with optional second-dimension extension over prompt arms. The sweep is held to four invariants:

| Invariant | Why |
|---|---|
| Single model per sweep | mixing models creates a second variable; separate sweeps must be run per model |
| Constant temperature | sampling-noise floor stays constant; T=0.2 reference (rationale §6) |
| Verbatim prompt templates | prompt rotation introduces unmeasured wording variance |
| Byte-identical context fixture | fixture rotation introduces unmeasured retrieval signal |

The swept variable is `num_predict`. Minimum 2 caps recommended — a "default" cap (e.g. 200, 400) representing typical operator settings, and a "lifted" cap (e.g. 4096) representing budget-uncapped behavior. The cap pair lets the spec detect cap-floor effects.

Optional second-dimension arm extension: `arm ∈ {substitution, synthesis}` measures both retrieval-shaped and reasoning-shaped prompts on the same fixture. This is the V3'.e shape — the reference driver. Single-arm variants (V3'.a / .b / .c / .d) collapse arm dimension to one (typically synthesis-only).

---

## 3. Protocol specification

### 3.1 Held constants (per sweep)

A sweep MUST hold the following constant across all calls:

```
- model               (string, e.g. "gemma4:e4b")
- temperature         (float, e.g. 0.2)
- prompt template     (one string per arm, verbatim — no paraphrase rotation)
- context fixture     (one string, byte-identical bytes across calls)
```

These appear in the `metadata` object of the result JSON. Verifying that two sweeps are comparable requires reading their `metadata` and matching on (model, temperature, prompt template, context fixture).

### 3.2 Swept variables

The sweep varies:

```
- num_predict ∈ {default_cap, lifted_cap, ...}    (at least 2 caps recommended)
- run_idx ∈ {1, ..., N}                            (statistical floor, §6)
```

If the driver runs multiple arms, the arm dimension is additionally swept:

```
- arm ∈ {substitution, synthesis, ...}             (optional, single-arm sweeps are valid)
```

Total trials = `len(arms) × len(caps) × N`. For the reference V3'.e shape: 2 arms × 2 caps × 20 runs = **80 calls per sweep**.

### 3.3 Per-call telemetry (REQUIRED fields)

Each call MUST record at minimum:

| Field | Type | Source |
|---|---|---|
| `elapsed_s` | float | wall-clock seconds between request send and response receive |
| `ollama_done_reason` | string | "stop" / "length" / "load" / etc. (Ollama field, normalize for non-Ollama backends) |
| `ollama_eval_count` | int | tokens generated by the model |
| `raw_response_sha256` | string (16-char hex prefix) | content hash for unique-set computation |
| `raw_response_text` | string | full response text (REQUIRED in v1 — enables downstream re-analysis) |
| `non_empty` | bool | true if the response is not blank-only |
| `run_idx` | int | iteration index within (arm, cap), 1-based |
| `num_predict` | int | the swept cap value for this call |
| `arm` | string | the arm name for this call |

Additional fields (`response_bytes`, `response_chars`, `ollama_total_duration_ms`, arm-specific domain predicates) are optional but recommended for downstream summary computation.

### 3.4 Computed summary (per cell)

A "cell" is one (arm, cap) combination. The driver SHOULD emit a per-cell summary in addition to per-call records, computed as:

| Summary field | Computation |
|---|---|
| `n` | total calls in the cell |
| `non_empty` | count of calls where `non_empty == true` |
| `avg_lat` | `mean(elapsed_s)` over all calls in the cell |
| `unique_outputs` | `len({raw_response_sha256 for call in cell if call.non_empty})` |
| `domain_hit` | count of calls matching the arm-specific domain predicate (optional) |

The `unique_outputs` field is what surfaces axis 1 (mode split) determinism. A cell with `unique_outputs == 1` across N ≥ 20 calls is strong evidence that the mode bypasses the sampling layer at the given temperature (see §8.1 worked example).

---

## 4. JSON schema (v1)

### 4.1 Top-level structure

```json
{
  "metadata":  { ... see §4.2 ... },
  "fixtures":  { ... see §4.3 ... },
  "runs":      { ... see §4.4 ... }
}
```

### 4.2 `metadata` object

| Field | Type | REQUIRED? | Notes |
|---|---|---|---|
| `started_utc` | ISO 8601 string | yes | sweep start, UTC |
| `completed_utc` | ISO 8601 string | yes | sweep end, UTC |
| `driver` | string | yes | driver script identifier (e.g. `"v3prime_e_mode_split.py"`) |
| `stage` | string | yes | sweep type identifier (e.g. `"mode_split"`, `"single_stage"`) |
| `hypothesis` | string | yes | one-sentence statement of what's being tested + attribution |
| `model` | string | yes | model tag verbatim (e.g. `"gemma4:e4b"`) |
| `temperature` | float | yes | sampling T |
| `n_per_cap_per_arm` | int | yes | N — calls per cell |
| `context_chars` | int | yes | character count of the context fixture |
| `caps_tested` | array[int] | yes | swept `num_predict` values |
| `arms` | array[string] | yes | swept arm names (`["substitution", "synthesis"]` or `["single"]`) |
| `endpoint` | string | recommended | model server URL (Ollama, vendor API, sovereign proxy) |
| `prompt_templates` | object | recommended | brief description per arm (one line each) |
| `context_fixture_role` | string | optional | one-sentence rationale for the chosen fixture |

The reference driver uses key name `ollama_url` instead of `endpoint`. Both are valid in v1; non-Ollama drivers SHOULD use `endpoint`.

### 4.3 `fixtures` object

```json
{
  "context": "<verbatim context string>",
  "substitution_prompt": "<verbatim prompt for substitution arm, with context placeholder substituted>",
  "synthesis_prompt": "<verbatim prompt for synthesis arm, ...>"
}
```

Single-arm sweeps include only the relevant arm's prompt. The fixtures object MUST contain the post-substitution prompt strings (i.e. the exact bytes sent to the model after template + context interpolation), not the template string.

### 4.4 `runs` object

```
runs[arm_name][cap_str] = list[CallRecord]
```

Where `cap_str` is the cap value as a string (JSON object keys must be strings) and `CallRecord` follows §3.3.

For the reference V3'.e sweep:
```json
{
  "runs": {
    "substitution": {
      "400":  [ /* 20 CallRecords */ ],
      "4096": [ /* 20 CallRecords */ ]
    },
    "synthesis": {
      "400":  [ /* 20 CallRecords */ ],
      "4096": [ /* 20 CallRecords */ ]
    }
  }
}
```

### 4.5 Schema validation

A minimal Python validator (≤ 30 lines, stdlib only):

```python
import json
from pathlib import Path

REQUIRED_META = {"started_utc", "completed_utc", "driver", "stage",
                 "hypothesis", "model", "temperature", "n_per_cap_per_arm",
                 "context_chars", "caps_tested", "arms"}
REQUIRED_CALL = {"elapsed_s", "ollama_done_reason", "ollama_eval_count",
                 "raw_response_sha256", "raw_response_text", "non_empty",
                 "run_idx", "num_predict", "arm"}

def validate(path):
    d = json.loads(Path(path).read_text(encoding="utf-8"))
    assert REQUIRED_META.issubset(d["metadata"]), \
        f"metadata missing: {REQUIRED_META - d['metadata'].keys()}"
    for arm in d["metadata"]["arms"]:
        for cap in d["metadata"]["caps_tested"]:
            cell = d["runs"][arm][str(cap)]
            assert len(cell) == d["metadata"]["n_per_cap_per_arm"]
            for call in cell:
                assert REQUIRED_CALL.issubset(call), \
                    f"call missing: {REQUIRED_CALL - call.keys()}"
    return True
```

If this validator passes, the JSON conforms to v1 and is comparable to other v1 JSONs at the cell-by-cell level.

---

## 5. Fixture design guideline

### 5.1 Context fixture

- **Length**: < 1,000 characters recommended. Longer fixtures push into context-window edge cases that vary by model; shorter fixtures keep the measurement focused on the cap-floor / mode-split signal rather than long-context behavior.
- **Self-contained**: the fixture must contain enough information that both substitution and synthesis prompts can succeed without external knowledge. Use a single domain (e.g. policy document, product spec, fact sheet).
- **One language**: language-mixing creates a second variable. Korean + English in the same fixture has been observed to shift output behavior on `gemma4:e4b` (separate hypothesis C in the original eval). Stick to one language per sweep.
- **Byte-identical**: the exact same bytes — no whitespace normalization, no NFKC, no lowercasing — across all calls. The fixture appears in `fixtures.context` verbatim.

### 5.2 Substitution-arm prompt

Asks for verbatim retrieval of a named section. Shape:

```
"Return verbatim the section titled '<section_name>' from the context below.
Do not paraphrase. Do not add commentary. Do not include other sections.

Context:
{context}

<section_name> section (verbatim):"
```

Three properties:
- **Imperative voice** ("Return verbatim", "Do not paraphrase") — removes reasoning ambiguity
- **Named target** (the section title) — gives the model a deterministic retrieval target
- **No example output** — example outputs leak format hints that bias the response

### 5.3 Synthesis-arm prompt

Asks for a new recommendation or decision over context clauses. Shape:

```
"Based on the policy context below, advise whether <specific case>. Justify
the decision in 2-3 sentences citing the specific clause that determines the
outcome.

Context:
{context}

Recommendation:"
```

Three properties:
- **Single-decision target** ("advise whether X") — limits scope to one synthesis judgment
- **Citation requirement** ("citing the specific clause") — forces the model to ground in context, surfacing whether reasoning + grounding both happened
- **Output length constraint** ("2-3 sentences") — bounds the response so a single cap value can capture both default and natural-stop behavior

### 5.4 Fixture for cross-stack comparison

If your sweep is intended for cross-stack comparison with another lab's sweep, the context fixture + prompts MUST be **byte-identical** between the two sweeps. The reference V3'.e fixture is in `scripts/research/v3prime_e_mode_split.py` (CONTEXT_FIXTURE constant) — copy that string verbatim if you want directly comparable cells. Robin's `sweep.py` does exactly this.

---

## 6. Statistical floor recommendations

| Recommendation | Value | Rationale |
|---|---|---|
| N per cell | ≥ 20 | N=10 is acceptable for first-pass exploration but N=20 is the load-bearing read. Below N=20, single-call noise dominates the `unique_outputs` and `non_empty` ratio signal |
| Temperature | T = 0.2 reference | T=0.0 collapses some samplers to greedy mode, hiding mode-split distinction. T=0.2 allows minor sampling without dominating the `unique_outputs` signal. T > 0.5 introduces too much variance for cap-floor detection |
| Cap pair | (default_cap, 4096) | default_cap is the operator's typical setting; 4096 is far above the ~500-token reasoning floor observed on `gemma4:e4b`. If the model's natural-stop range is unknown, run a single-call probe first to size the lifted cap |
| Per-cap N | consistent across cells | rotating N across cells (e.g. N=10 on one cell, N=20 on another) confounds the statistical comparison. Pick one N for the sweep |

### 6.1 Reproducibility metadata (REQUIRED)

The following MUST be recorded somewhere in `metadata` or in the result report:

- Ollama version (or vendor SDK version) at sweep time
- Model tag verbatim (e.g. `gemma4:e4b`, not `gemma4-e4b` or `gemma-4-e4b`)
- Endpoint class — `local-loopback` / `local-network` / `vendor-managed` / `sovereign-proxy`
- Hardware class — at minimum, GPU vs CPU; ideally GPU model + VRAM

Two sweeps that match on (model, temperature, prompt template, context fixture, Ollama version) but differ on endpoint class produce directly comparable **within-endpoint** signals (e.g. sub vs synth latency ratio) but NOT directly comparable **wall-clock latencies** (see §8.3 latency caveat).

---

## 7. Reproducibility checklist

Before publishing a sweep result, verify:

- [ ] Driver script committed to a public git repository
- [ ] Context fixture is verbatim in the driver source (not loaded from external file at runtime)
- [ ] Prompt templates are verbatim in the driver source
- [ ] Per-call `raw_response_sha256` saved (enables unique-set computation)
- [ ] Per-call `raw_response_text` saved (v1 REQUIRED; enables downstream re-analysis without re-running the sweep)
- [ ] `metadata.model` matches the actual model tag (`ollama list` output)
- [ ] N is consistent across all cells (or per-cell N recorded explicitly)
- [ ] Sweep completed without retry-on-error masking (any retries should be visible in `metadata`)
- [ ] Result JSON validates against the §4.5 minimal validator

If all checked, the sweep is v1-compliant and citable.

---

## 8. Worked examples

### 8.1 Axis 1 (mode split) — substitution bypasses sampling

**JAMES V3'.e on `gemma4:e4b`**, 2026-05-24:
- substitution arm @ cap=400: `unique_outputs = 1/20` (sha256 set size 1 across 20 non-empty calls)
- substitution arm @ cap=4096: `unique_outputs = 1/20`
- 40 calls total → 1 unique 290-char canonical response

**Robin Converse 26b** on `gemma4:26b` MoE, 2026-05-23:
- substitution arm @ cap=400: `unique_outputs = 1/20`
- substitution arm @ cap=4096: `unique_outputs = 1/20`
- 40 calls total → 1 unique canonical response (different token count: eval_count 38 vs JAMES's 62)

Cross-stack interpretation: substitution-mode is bit-for-bit deterministic at T=0.2 on both `gemma4:e4b` (8B params) and `gemma4:26b` MoE (25.8B params). The output text differs between the two models (different bytes), but within each model the output set has cardinality 1. This is what we call **"sampling-layer bypass"** — the model behaves as if T=0 for substitution prompts regardless of the nominal T setting.

### 8.2 Axis 3 (model-scale efficiency) — synthesis convergence

Same two sweeps as §8.1, synthesis arm:

| Stack | Synthesis `unique_outputs` @ cap=400 | @ cap=4096 |
|---|---|---|
| e4b (8B params) | 15/15 (100% of success) | 20/20 (100%) |
| 26b MoE (25.8B params) | 6/20 (30%) | 9/20 (45%) |

Cross-stack interpretation: synthesis is variable on both stacks (unique > 1), but **the variability scales inversely with parameter count**. Larger model → answers cluster more tightly around a small set of decision paths. Smaller model → answers fan out across the response space.

This is axis 3's "answer convergence" layer, complementary to its "token efficiency" layer (Robin's 2026-05-23 original sub-finding: 26b uses ~9× fewer tokens per successful synthesis call than e4b on the same fixture).

### 8.3 Latency caveat (cross-stack)

Robin's 26b substitution latency is 3.8s (Hetzner CCX33 sovereign Ollama + Caddy reverse proxy). JAMES e4b substitution latency is 0.8s (local Ollama loopback). **These are not directly comparable as wall-clock measurements** — Robin's number includes public HTTPS round-trip + reverse proxy.

What IS comparable across stacks at the wall-clock level:
- **Within-endpoint ratios** — e.g. Robin's sub 3.8s vs synth 5.0s ratio, or JAMES's sub 0.8s vs synth 4.0s ratio. Both show synth taking longer than sub at comparable token counts, confirming the mode-split finding in physical time.
- **`ollama_eval_count`** — tokens-per-call, independent of network/proxy time. Use this for the token-efficiency comparison (axis 3 layer a).

---

## 9. Extension points (backward-compatible)

V1 schema is designed for additive growth. The following extensions are **backward-compatible** — v1 validators continue to pass:

| Extension | How |
|---|---|
| Additional per-call fields | Add to CallRecord (validators check `issuperset`, not `==`) |
| Additional arms | Append to `metadata.arms` + add `runs[new_arm]` |
| Additional caps | Append to `metadata.caps_tested` + add `runs[arm][new_cap]` |
| Additional per-cell summary metrics | Compute post-hoc from `raw_response_text` (no schema change required) |
| New downstream analysis fields | Compute from existing `raw_response_text` + sha256 fields; emit alongside the sweep JSON in a separate analysis JSON |

**Examples of v2-candidate extensions** (not yet specified):
- `raw_response_logprobs` per call (model-side logprobs if exposed by the endpoint)
- `prompt_token_count` (input tokens; currently only output is tracked via `eval_count`)
- `system_prompt` field in `metadata` for non-base-model sweeps

These would require a v2 if they become mandatory; until then they ride as optional additive fields.

---

## 10. Versioning policy

| Version | Status | Date | Changes |
|---|---|---|---|
| v1 | current | 2026-05-24 | Initial spec. `raw_response_text` per call is REQUIRED in v1 (allows unique-set + canonical-text inspection without re-running). |

**Breaking change policy**: a breaking change (REQUIRED field removal, REQUIRED field type change, or runs structure reorganization) requires a v2 with:

1. A migration guide showing how v1 JSONs map to v2 (or how to round-trip)
2. A grace window of **at least 12 months** where v1 validators are still maintained — matching the project-wide deprecation policy in `docs/VERSIONING.md` so external adopters only need to remember one grace rule across the whole codebase
3. Driver code in `scripts/research/` keeps v1 emission for at least one release cycle after v2 lands

This policy aligns with the broader project versioning approach (SemVer + 12-month deprecation). The unified 12-month window favors citation stability over iteration speed — V3' spec is a research artifact intended for cross-lab adoption, so adopters' analysis pipelines benefit more from predictability than from a faster v2 turnaround.

---

## 11. Citation

When citing measurements made under this protocol, cite this spec at its commit hash for unambiguous reference. Suggested format:

> *"Measured under V3' Protocol v1 (Hashevolution, [commit-hash], `docs/research/v3prime-protocol-v1.md`)."*

For BibTeX use:

```bibtex
@misc{v3prime_protocol_v1,
  author       = {Hashevolution},
  title        = {V3' Protocol v1 — A reproducible measurement protocol for LLM cost behavior},
  year         = {2026},
  howpublished = {\url{https://github.com/Hashevolution/James-RAG-Evol/blob/main/docs/research/v3prime-protocol-v1.md}},
  note         = {Commit hash specifies the exact spec version. Adopters and measurement contributions are listed in the External adopter table within §11; spec authorship is solely Hashevolution. Joint research outputs that use this protocol (e.g. cross-stack analyses, the planned Track 5 joint paper) carry their own multi-author bylines and are cited separately.}
}
```

**Spec author vs adopter — important distinction**: the BibTeX `author` field above (`Hashevolution`) reflects who authored the spec text itself. Labs that follow the spec to produce measurements (Robin Converse / Triava Labs, future contributors) are **adopters**, not spec authors — their attribution lives in the table below + in any joint research output (Track 5 joint paper, etc.) where their measurements appear. This separation keeps the citation graph clean: cite the spec by author=Hashevolution for the protocol itself, cite the joint paper by its multi-author byline for any cross-stack finding produced under the protocol.

External lab adoptions and result publications using this protocol are welcome to be listed below — open a PR adding a row:

| Adopter | Stack | Result location | Date |
|---|---|---|---|
| Hashevolution (JAMES) | local Ollama, `gemma4:e4b` | `reports/research-runs/v3prime-*.json` + `reports/promo-assets/v3prime-e-substitution-synthesis-result.md` | 2026-05-22 — 2026-05-24 |
| Robin Converse (Triava Labs) | sovereign Ollama (Hetzner CCX33 + Caddy), `gemma4:26b` MoE | [triavalabs/gemma4-26b-mode-split](https://github.com/triavalabs/gemma4-26b-mode-split) | 2026-05-23 |

---

## 12. Related documents

- **Result reports** using this protocol:
  - `reports/promo-assets/v3prime-e-substitution-synthesis-result.md` — JAMES V3'.e + cross-stack replication with Robin 26b + Direction 4 sub-findings
  - `reports/promo-assets/gemma4-e4b-cognitive-stages-eval.md` — V3'.a/.b/.c/.d 4-stage cap-budget validation + Reader contributions

- **Driver scripts** (reference implementations):
  - `scripts/research/v3prime_e_mode_split.py` — V3'.e (mode_split shape, 2-arm × 2-cap)
  - `scripts/research/v3prime_query_rewriter.py` — V3'.a (single-stage)
  - `scripts/research/v3prime_planner.py` — V3'.b
  - `scripts/research/v3prime_reflect.py` — V3'.c
  - `scripts/research/v3prime_verify.py` — V3'.d

- **Track / cycle handovers**:
  - `docs/handovers/v0.3.x-measurement-framework-track.md` — 6-direction follow-up plan
  - `docs/handovers/v0.3.x-ali-collaboration-track.md` §Track 5 — joint-paper narrative
  - `docs/handovers/v0.3.x-gemma4-feedback-track.md` — internal evidence pile

- **External adopter artefacts**:
  - https://github.com/triavalabs/gemma4-26b-mode-split — Robin Converse's 26b companion repo (sweep.py + analysis.md + raw JSON)
  - https://github.com/Hashevolution/James-RAG-Evol/issues/448 — Robin's cross-stack result submission

---

## 한국어 요약

V3' Protocol v1은 LLM의 cost behavior를 세 가지 독립 축(mode split / workload gradient / model-scale efficiency)에서 측정하는 **재현 가능한 측정 protocol** 명세입니다. 단일 변수 sweep (token cap 변동) + optional arm 차원 (substitution / synthesis)을 핵심으로, 두 lab이 같은 spec을 따라 작성한 결과 JSON이 cell 단위로 직접 비교 가능하도록 schema를 frozen 형식으로 명시.

JSON schema는 `metadata` / `fixtures` / `runs` 세 top-level key로 구성되며, `runs[arm][cap]` 단위의 per-call record가 `raw_response_text`를 v1 REQUIRED 필드로 포함합니다. 이 필드 덕분에 downstream 분석에서 unique-output count, canonical text 검증 등을 sweep을 다시 돌리지 않고 수행 가능 — 첫 외부 adopter Robin Converse(Triava Labs)가 2026-05-23 자메스 schema를 그녀의 26b sweep template으로 채택한 것이 이 spec의 핵심 사용 사례.

Versioning은 additive-only로 운용 (v1 → v2 breaking change는 6개월 grace + migration guide 필수). 인용은 이 문서의 commit hash로 지정. 외부 lab의 adoption row 추가는 PR로 환영.
