# v0.3.x — Gemma 4 Variant 3×3 Evaluation Plan (cross-author pre-registration)

> Status: locked in writing 2026-05-18, results not yet collected
> Type: pre-experiment design document; falsification criteria committed before any cell runs
> Authors of record: Jiwon (PROJECT JAMES), Ali Afana (Provia)
> Trigger: 4-turn LinkedIn DM exchange + dev.to comment thread, 2026-05-16 → 2026-05-18
> Companion documents:
>   - `docs/handovers/v0.3.x-ali-collaboration-track.md` — coding tracks behind this plan
>   - `reports/promo-assets/gemma4-e4b-cognitive-stages-eval.md` — variant baseline (read this before reading the matrix data)
>   - `reports/promo-assets/devto-gemma4-write-track.md` — published Write-track piece, four-hypothesis framing
>   - `eval/regression/step7_queries.json` — the 13-query corpus used as ground truth
>   - `core/reasoning/backends/__init__.py` — Backend Protocol type that the Provider contract formalizes

## Why this document exists before the experiment runs

A common failure mode in cross-author empirical work is **post-hoc framing drift** — the authors interpret the result through whichever lens makes the data look strongest, after the data is in hand. The result is a single article with hidden degrees of freedom that the reader cannot audit.

This document pre-registers:

- The exact 3 × 3 cell matrix
- Which layer of each system gets measured per cell
- The four hypotheses that the cell pattern can falsify
- The four publication-format decisions, one per result combination, locked **before** results land

If the experiment kicks off, this page is the canonical reference. If the matrix changes later, the changes happen here as labeled diff entries (not silent edits), and Ali signs off in the LinkedIn DM thread.

## Matrix

3 variants × 3 temperatures × 1 prompt structure per side. **9 cells per side**, 18 cells total. **1 run per cell** (Ali's choice; matches signal/noise budget for exploratory cross-author work — see "Single run rationale" below).

### Variants (rows)

| Variant | Model id | Active params | Architecture role |
|---|---|---|---|
| **E4B** | `gemma4:e4b` | 4 B effective | Edge variant — known to ship `synth.rag` reliably on Korean prose (eval report's "where E4B wins" section) |
| **26B MoE** | `gemma-4-26b-a4b-it` | 4 B active / 26 B total | Mixture-of-experts — Ali's controlled experiment found this routed sequencing correctly on the white-shirt Scenario 2 |
| **31B Dense** | `gemma-4-31b-it` | 31 B (all active) | Dense — Ali's controlled experiment found this collapsed sequencing into single behavior under his temperature-0.3 cap |

### Temperatures (columns)

| Temperature | Rationale |
|---|---|
| **0.3** | Ali's original Round 2 cap. Production-realism setting for chat reply work |
| **0.7** | Mid-range. Standard "natural conversation" default for most chat-LLM APIs |
| **default** | Whatever the model ships as its untuned default (Gemma 4 default = 1.0 per Google AI Studio's published config as of mid-May 2026) |

### Prompt structure (one per side)

**Jiwon side** — typed graph paths in retrieval context:

```
[retrieved chunk 1]  (entity A, sensitivity=public)
[graph_path]         A --[CAUSES]--> X --[REQUIRES]--> Y --[BLOCKED_BY]--> B
[ontology]           relation 'CAUSES' directed=true weight=0.85
[query]              <natural language question>
```

**Ali side** — natural-language refuse instruction in catalog reply context:

```
You are a sales rep at <store>. Reply with one short message in Palestinian
Arabic. If the customer asks for something not in the catalog, say "we
don't have that" honestly and offer an alternative from what's available.

Catalog:
<JSON of available SKUs>

Customer:
<test message>
```

(The Ali-side prompt is the exact same one as his published controlled experiment's Round 2 — see `reports/promo-assets/gemma4-e4b-cognitive-stages-eval.md` references.)

### What gets measured per cell

**Single measurement site per side**, deliberately:

- **Jiwon side** — synthesis layer output (`core/reasoning/pipeline_synth.py` LLM call sites, lines 136 / 210 / 255 — the three branches of `generate_answer`). The conditioning layer (typed-schema retrieval input) is upstream and is **not** what gets measured; it is held constant across all 9 Jiwon-side cells.
- **Ali side** — final reply over retrieved catalog, same single call site he measured in his published controlled experiment.

This single-layer choice is the key methodological commitment: same measurement site across the 9 cells per side means **any per-cell difference is attributable to the variant or the temperature, not to a measurement-site confound**.

### Single run rationale

1 run per cell is exploratory, not statistical. We choose this because:

- 9 cells × 13 queries × 1 run = 117 query-runs per side. Already large enough that single-run noise on individual queries doesn't dominate the per-cell pattern.
- Ali's published controlled experiment used 1 run per cell at 4 cells. The result was robust enough at that scale that scaling up the design (4 → 9 cells) is more useful than scaling up the run-count (1 → N runs per cell).
- The four interactions (prompt-structure × arch, prompt-structure × temp, arch × temp, triple) are read off the *pattern* of cells, not the precision of any single cell. Pattern-level conclusions survive single-run noise as long as the design fills the matrix.
- If the data is ambiguous at single runs, that itself is a finding — we publish it that way rather than fish for clean numbers via re-runs.

If a future replication wants statistical power, it can run N runs per cell against this committed baseline. The bench runner (`scripts/bench.py --suite=step7`) already supports that.

## Held-constant variables

These are pinned across all 18 cells; any drift invalidates the measurement.

| Variable | Held-constant value |
|---|---|
| Wiki corpus (Jiwon side) | Snapshot at `wiki/` HEAD at experiment kickoff. Hash recorded in the result JSON |
| Catalog data (Ali side) | The same 34-product test boutique from the published controlled experiment |
| STEP 7 13-query suite (Jiwon side) | `eval/regression/step7_queries.json` as committed — no edit until experiment closes |
| Query payload (Ali side) | Same 6 customer scenarios as the published controlled experiment, expanded with 7 more queries Ali drafts to match the 13-query count |
| Sampling parameters not under test | `top_p`, `top_k`, `repetition_penalty` — fixed at each model's published defaults, recorded in the result JSON |
| Hardware / OS | Same machine per side, same OS, same Ollama (Jiwon) / same Gemini API endpoint (Ali) version. Versions recorded in the result JSON |
| Time of day | Same UTC-day window per side. Network and rate-limit conditions noted |
| Random seed | Recorded if the backend supports it. Otherwise noted "not pinned" in the result JSON |

## Hypotheses and falsification criteria

The cell pattern lands the result in one of four named cells of this table. The cell determines the writeup format (next section).

| # | Hypothesis | Confirming cell pattern | Falsifying cell pattern |
|---|---|---|---|
| **H1** | The MoE/Dense divergence Ali documented at temperature 0.3 narrows when temperature rises | Ali side: Dense flat/monotonic across 0.3/0.7/default; MoE shows a non-monotonic curve | Both flat or both curved → divergence is **not** thermostat-driven |
| **H2** | The MoE/Dense divergence collapses for typed-graph-path conditioning (Jiwon side) | Jiwon side: no per-cell architecture-pattern divergence above noise (all 9 cells produce similar `synth.rag` quality) | Same divergence pattern appears on the Jiwon side → ambiguity lives upstream of synthesis layer, framing needs to walk back |
| **H3** | E4B is the meta-reasoning-floor outlier (eval report's Hypothesis A) — and the floor is *layer-specific* | E4B cells fail more on Ali-side refuse-instruction than on Jiwon-side typed-graph synthesis | E4B fails uniformly across both sides → meta-reasoning floor is task-shape-independent |
| **H4** | The temperature×architecture interaction is non-additive (triple interaction is non-zero) | Cells at (MoE, 0.3) ≠ cells at (Dense, 0.3) AND (MoE, default) ≈ (Dense, default) → architecture×temperature non-additive | The triple interaction is small or zero → arch and temp matter separately, no architectural temperature-coupling |

We commit to publish each H column's outcome **whether it confirms or falsifies the framing in the published Build-track and Write-track pieces**. If H2 falsifies, Jiwon-side framing in the Build-track piece walks back publicly. If H1 falsifies, Ali-side framing in his controlled experiment walks back publicly.

## Decision matrix — writeup format by cell

The matrix below was first drafted in `docs/handovers/v0.3.x-ali-collaboration-track.md` § Track 5. Reproduced here with the added falsification handles from H1–H4.

| H1 (temp) | H2 (Graph-RAG collapse) | Writeup | First author | Walk-back? |
|---|---|---|---|---|
| confirms | confirms | **Joint Part 2**, single article — predictions both sharpen, the story is one finding | Ali | none |
| confirms | falsifies | Two linked articles — Ali sharpens his framing to "synthesis layer specifically," Jiwon writes "where the divergence shows up on Graph-RAG too" follow-up | each their own | partial (Ali's framing narrows) |
| falsifies | confirms | Joint article framed as "Graph-RAG result aligns with Ali's prediction, but the temperature axis dissolves the architecture claim" | Ali | partial (Ali's temp framing) |
| falsifies | falsifies | Walk-back required for both Ali and Jiwon-side framings. Joint piece framed as "the original framings were both narrower than the data needs" — credibility from being willing to | Ali (his original framing got most public attention) | yes, both |

The matrix is committed before results land. That is the entire point of Track 4's pre-experiment scope-lock note.

## Replication script (Jiwon side)

Once the Provider contract lands (Track 1 of the Ali Collaboration Track), the 9-cell sweep on Jiwon side runs as:

```bash
# Hold-constant: wiki corpus snapshot, STEP 7 query suite, JAMES rev
git rev-parse HEAD > reports/swap_eval/experiment-jiwon-side-<date>/git-rev.txt
sha256sum wiki/*.md | sort > reports/swap_eval/experiment-jiwon-side-<date>/wiki-hash.txt

for variant in e4b 26b-moe 31b-dense; do
  for temp in 0.3 0.7 1.0; do
    JAMES_LLM_MODEL=gemma4:$variant \
    JAMES_LLM_TEMPERATURE=$temp \
    python scripts/bench.py --suite=step7 \
      > reports/swap_eval/experiment-jiwon-side-<date>/$variant-temp$temp.json
  done
done

python scripts/swap_eval_report.py \
  reports/swap_eval/experiment-jiwon-side-<date>/ \
  > reports/swap_eval/experiment-jiwon-side-<date>/matrix.md
```

`scripts/swap_eval_report.py` is a Track 3 deliverable — it does not yet exist. The plan above is what the script needs to produce.

`JAMES_LLM_TEMPERATURE` is currently inferred from per-call kwargs; if the env-var form does not exist, Track 1's Provider contract adds it (a Provider must accept `temperature` as part of `complete(...)`).

## Open methodological questions (pre-locked: out of scope for this matrix, list for future replication)

These are explicitly **not in this 3×3** but would be valuable for follow-up runs by either side:

- Adding **multi-run variance** (3+ runs per cell) for any cells that pattern-suggest a different story than the single-run signal
- Adding **instruction-order ablation** (positive-first vs negative-first instructions on Ali side, per the eval report's Track-5 follow-up note)
- Adding **all-Korean JSON schema** prompt variant on Jiwon side (eval report's Hypothesis C — Korean keys instead of English JSON keys, to test KO/EN mix confound)
- Adding **smaller dense Gemma** (gemma4:e2b, if Google ships one) to test whether the failure pattern scales with density or with parameter count
- Adding **non-Gemma controls** (Qwen 3 dense, Llama 3 dense, Mixtral MoE) to test whether the MoE/Dense pattern generalizes beyond the Gemma 4 family

The 3×3 in this document is the **minimum viable cross-author result**, not the full research program.

## What this document does *not* commit to

- A specific date for the swap run. That waits on Track 1 (Provider contract) landing.
- A specific writing schedule. After the matrix runs and the falsification cells settle, both sides draft within their own pace and synchronize at the joint review step.
- An exclusivity arrangement. Either side is free to publish independently in parallel on adjacent topics. The matrix and its result are the only joint deliverable.
- A guarantee that the joint piece publishes at all. If both H1 and H2 falsify and the writeup would be confusing rather than illuminating, we agree (via the Track 4 scope-lock note) to publish each side's revision separately rather than force a joint piece.

## How this document gets updated

- **Edits before kickoff**: any change is a labeled diff entry at the bottom of the document, with the author who initiated and the LinkedIn DM thread date of the agreement. No silent edits.
- **Edits after kickoff**: matrix is frozen. The only changes allowed are the "open methodological questions" section (recording new questions raised by the data) and the "Results" section (added once data lands).
- **Sign-off**: Track 4's pre-experiment scope-lock note will be the formal artifact both authors reference. This document is the technical companion to it.

---

## Diff log

(Add entries below as the matrix evolves.)

| Date | Author | Change | LinkedIn DM ref |
|---|---|---|---|
| 2026-05-18 | Jiwon | Initial pre-registration | First three turns of the DM thread |
