# v0.3.x — Gemma 4 Feedback Track

> Status: open — **V3'.a/.b/.c/.d in-house cross-validation closed 2026-05-22** (all four stages 0/10 → 10/10 at default → 4096 cap, **stage-independent** ~500-token reasoning floor). **V3'.e refined Pattern S landed 2026-05-23 (PR #440)** — substitution arm 20/20 @ cap=400 (floor-immune) vs synthesis arm 14/20 @ cap=400 (partial floor). Three-workload-level signature spans heavy / light / no synthesis. **Robin Converse 26b 2×2 matrix RESOLVED 2026-05-23 UTC 12:39** (issue [#448](https://github.com/Hashevolution/James-RAG-Evol/issues/448) + repo [triavalabs/gemma4-26b-mode-split](https://github.com/triavalabs/gemma4-26b-mode-split)) — **both reference signatures shifted** (substitution 62→38; synthesis 400-450→49-54); pre-registered "next research thread" arm triggered, but signatures shifted systematically revealing **3rd axis: model-scale efficiency** (26b synthesis ~9× more token-efficient than e4b on same fixture). Mode-split framing held; gradient quantification needs sub-clause "scales inversely with parameter count".
> Created: 2026-05-18
> Companion document: `reports/promo-assets/gemma4-e4b-cognitive-stages-eval.md`
> Original challenge submission: `reports/promo-assets/devto-gemma4-challenge.md`
> Maintainer of this track: project owner (Hashevolution)

## Purpose

`reports/promo-assets/gemma4-e4b-cognitive-stages-eval.md` is a public fair-witness report on `gemma4:e4b` failure modes on JAMES v0.3's cognitive stages. The report explicitly **asks for external feedback** (X, dev.to comments, Reddit, GitHub issues). This handover tells the next session how to **route, evaluate, and act on** that incoming feedback without re-deriving the whole context.

Any session continuing this track should be able to read this single page and know:

- where the data lives
- which channels feedback is expected from
- how to interpret each kind of incoming reply
- what code/doc change a given reply justifies (and which ones it does not)

## What ships in the report (state at 2026-05-18)

| Section in the report | Claim |
|---|---|
| TL;DR | gemma4:e4b empty-response on 5 meta-stages; gemma3:12b control passes all 9 |
| Setup | reproducible (env vars, model tags, query string, ollama versions) |
| Observed behavior (gemma4:e4b) | 6 LLM calls tabulated, 5 of which returned 0 chars in 2–4 s |
| Control (gemma3:12b) | full 9-step trace, ~ 39 s wall-clock |
| Where Gemma 4 succeeds | synth.rag long-form + entity-extract JSON — **public credit kept** |
| Failure pattern | 4 traits common to the 5 empty stages |
| Hypotheses A / B / C / D | meta-reasoning floor / early EOS / KO+EN mix / prompt-truncation |
| What I'd love feedback on | 5 specific questions |
| Reproduction | full PowerShell block |
| 한국어 요약 | KO summary, same hypotheses |

The report intentionally **does not** advocate for a single hypothesis — that's the work this feedback round funds.

## Feedback channels and routing

Expected inbound channels, in order of likelihood:

| Channel | Probable form | Where to record |
|---|---|---|
| **dev.to comments** on the Gemma 4 Challenge submission (`devto-gemma4-challenge.md` → linked URL) | Free-form replies from local-LLM operators | Quote into the eval report's "Reader contributions" section (see Section template below) |
| **X / Twitter** (post linking the eval report) | Short replies with screenshots, model names, single-issue prompts | Same — paste the X permalink + verbatim text |
| **Reddit r/LocalLLaMA** (if the report is cross-posted from `reports/promo-assets/reddit-locallama.md`) | Detailed technical comments, often with numbers | Same |
| **GitHub issues** on `Hashevolution/James-RAG-Evol` | Most actionable form: tagged + traceable + can attach traces | Reference the issue from the eval report and close-the-loop |
| **HN Show-HN replies** (low odds — already past Show-HN window for the project) | Short tech opinions | Same |
| **Geek News (KO)** | Korean-language opinions | Same — KO replies go in a separate KO subsection of the report |

The `launch-tracker.md` in `reports/promo-assets/` is the running log; any feedback citation should also land there for continuity.

## Decision matrix — how a piece of feedback changes the project

The four hypotheses in the report map to four kinds of actionable feedback:

### Hypothesis A — meta-reasoning capacity at 4 B is the floor

**Confirming feedback** (look for): "We tried gemma4:e4b on a critique-of-a-draft task and got the same empty-response pattern. Bumping to 7B+ variants fixed it." Or: published evals (LMSys, Helm) showing the gap is structural.

**Project response**:
- Keep `gemma3:12b` as the documented default for the cognitive-stage layer.
- Open a follow-up PR for **option A2** (route only the meta stages through a separate `JAMES_LLM_MODEL_META` — let `synth.rag` stay on e4b for its long-context win).
- Do not pursue prompt-side workarounds — they will be marginal.

### Hypothesis B — early stop-token emission

**Confirming feedback** (look for): "We set `num_predict=512` and the same prompts on e4b now return non-empty." Or: Ollama / Gemma 4 chat template bug reports.

**Project response**:
- Patch `core/gemma_client.py` to log the raw Ollama response (token count, stop reason) for empty-response cases — gives us better diagnostics.
- Open a small PR adding `num_predict` and `temperature` overrides per stage if a single setting fixes it.
- File an Ollama issue with the trace.

### Hypothesis C — Korean instruction + English JSON schema confusion

**Confirming feedback** (look for): "We changed our JSON keys to all-Korean (e.g. `재작성된_질의`) and e4b stopped returning empty." Or: anyone reproducing with an all-English version of the same prompt.

**Project response**:
- Try **one experimental prompt** in `core/retrieval/query_rewriter.py` with Korean-key JSON, behind an env flag, to verify.
- If it works: that's a 2 KB diff on each of the 4 affected stage files (`query_rewriter.py`, `planner.py`, `reflect.py`, `verify.py`). Quick win.
- If it doesn't: hypothesis A is more likely.

### Hypothesis D — JAMES side prompt-truncation artifact

**Confirming feedback** (look for): only really verifiable internally — examine the truncated prompt sent to `verify.fact_check` for that one stage and check whether the truncation chopped a JSON example mid-brace.

**Project response**:
- This is a JAMES bug, not a Gemma 4 bug. Even if true, it only explains 1 of the 5 empty responses. So it's a side-issue worth fixing on its own, but **does not invalidate the report's main claim**.
- File this as a separate GitHub issue, fix in a small PR, and footnote the eval report (don't let it muddy the headline).

## When a piece of feedback is *not* actionable here

These categories should be acknowledged in the report (Reader contributions section) but **do not change the project**:

- Replies that swap to a completely different model family (Llama, Qwen, DeepSeek) and report success — interesting but doesn't tell us why Gemma 4 e4b fails.
- "Just use Claude / GPT-4o" — already on the roadmap as Option A2 + the `JAMES_REASONING_BACKEND=claude_code_cli` switch shipped in #305.
- "Your prompts are bad" without a specific change to test — note it, ask the commenter for the specific change, do not refactor speculatively.

## Section template for the eval report — "Reader contributions"

When the first substantive piece of feedback arrives, append a new section to `reports/promo-assets/gemma4-e4b-cognitive-stages-eval.md` titled `## Reader contributions`. Format each entry as:

```markdown
### 2026-MM-DD — <source: X / dev.to / Reddit / GitHub issue #N>

**Reporter**: <handle or "anonymous">
**Permalink**: <URL>
**Hypothesis they support / refute**: A / B / C / D / new
**Verbatim quote** (or paraphrase + link):
> ...

**Project response**: <link to PR or issue, or "noted, no change">
```

Keep this section append-only — never edit old entries, even if a later report contradicts them. The chronology is part of the data.

## Files this track touches

```
reports/promo-assets/gemma4-e4b-cognitive-stages-eval.md   ← primary deliverable
reports/promo-assets/devto-gemma4-challenge.md             ← upstream challenge entry
reports/promo-assets/launch-tracker.md                     ← chronological log of all promo activity
docs/handovers/v0.3.x-gemma4-feedback-track.md             ← this file
```

Possible code touches once feedback arrives (each its own PR):

| If hypothesis confirmed | Likely PR scope |
|---|---|
| A | `core/reasoning/backends/__init__.py` — extend `get_default_backend_id()` so per-stage `JAMES_LLM_MODEL_{stage}` overrides land cleanly (this builds directly on #305) |
| B | `core/gemma_client.py` — log raw Ollama response on empty; optional `num_predict` per-stage env |
| C | `core/retrieval/query_rewriter.py` (+ planner.py, reflect.py, verify.py) — Korean-key JSON prompt variant behind an opt-in env, with old prompt as fallback for byte-compat |
| D | `core/gemma_client.py` or whichever module owns the 4 000-char truncation — replace hard cap with prompt-aware truncation (preserve closing `}` / `]`) |

## CLAUDE.md gate alignment

| Rule | This track |
|---|---|
| #1 — no domain features before v1.0 | Track is platform / observability, no vertical coupling |
| #2 — bench numbers on `core/retrieval` / `core/graph` / `core/reasoning` PRs | Each follow-up PR above must paste STEP 7 numbers; the report PR itself touches no core code |
| #3 — self-evolution is opt-in only | Unrelated |
| #4 — architecture changes via `architecture` label PR | None of the follow-ups change `docs/ARCHITECTURE.md §5.7` schema — they extend env conventions only |
| #5 — no file in `core/` exceeds 20 KB | All listed follow-ups are small per-file additions; size budget remains comfortable |

## Suggested first action for the next session

If a real reply has already arrived by the time the next session opens this file:

1. Read the reply verbatim.
2. Map it to one of A / B / C / D using the decision matrix above.
3. Append a "Reader contributions" entry to the eval report.
4. Either:
   - File the suggested follow-up PR (if the matrix says "code change"),
   - Or reply on the channel with a follow-up question (if the comment is interesting but underspecified).

If no reply has arrived yet:

1. Don't poke the network — let the post run.
2. Use the same session to start one of the orthogonal next-step queues (E spot-check, F STEP 7 bench, or A2/B cognitive-backend wiring).

## 한국어 요약

`reports/promo-assets/gemma4-e4b-cognitive-stages-eval.md` 는 `gemma4:e4b` 의 cognitive-stage 실패 패턴을 정직하게 기록 + 외부 피드백 환영 보고서. 이 핸드오버는 **피드백이 들어왔을 때 다음 세션이 무엇을 해야 하는지** 의 정리:

1. **채널별 인입 형태** — dev.to / X / Reddit / GitHub issue / Geek News (KO) 각각 어디로 들어올지.
2. **가설 A/B/C/D 별 결정 매트릭스** — 어떤 피드백이 어떤 코드 변경을 정당화하는지.
3. **"Reader contributions" 섹션 템플릿** — 피드백 첫 도착 시 보고서에 어떻게 누적할지.
4. **CLAUDE.md rule 확인** — 후속 PR 모두 rule 위반 없음.
5. **다음 세션 첫 액션 가이드** — 피드백 도착 vs 미도착 각각 무엇을 할지.

핵심 원칙: 보고서는 fair-witness — 특정 가설을 미리 옹호하지 않음. 피드백 누적이 가설 검증의 데이터. 추측 기반 prompt refactor 금지 — 구체적 change 가 있는 피드백만 행동으로 옮긴다.

---

## Internal evidence pile (alongside external feedback)

Independent of the external-feedback cycle described above, every
in-house observation of `gemma4:e4b` failure modes feeds the same
hypothesis-A/B/C/D matrix. Two artefacts on the internal track:

| Source | Date | Finding | Hypothesis support |
|---|---|---|---|
| `reports/promo-assets/gemma4-e4b-cognitive-stages-eval.md` | 2026-05-18 | 5 of 6 cognitive-stage prompts return empty in 2–4 s; gemma3:12b passes all 9 | A + B |
| `docs/research/gemma4-event-emit-experiment-2026-05-21.md` | 2026-05-21 | 10-iteration repeat on PR-11b extraction prompt — 1 event-emit / 3 no-event / 6 empty-or-truncated. **Same failure shape on a non-cognitive (extraction) task.** | A + B (strong) ; D weakened |
| `reports/research-runs/v3prime-query-rewriter-20260522T021221.json` | 2026-05-22 | V3'.a single-variable sweep on query_rewrite: 0/10 at 200-cap (all `done_reason=length`, `eval_count=200`, 0 bytes) → 10/10 at 4096-cap (`done_reason=stop`, ~520 `eval_count`, ~100 bytes). **Mechanism**: ~500 hidden reasoning tokens consumed before first visible output token; any cap below this floor is deterministic empty. | **B confirmed (with mechanism)** ; A practically refuted for query_rewrite |
| `reports/research-runs/v3prime-planner-20260522T063918.json` (workstation push pending) | 2026-05-22 | V3'.b single-variable sweep on plan.decompose: 0/10 at 400-cap (avg 4.3s) → 10/10 at 4096-cap (avg 7.1s). **Cross-stage consistency**: latency scales linearly with cap (V3'.a 200-cap → 2.1s, V3'.b 400-cap → 4.3s = exactly 2×), confirming ~500-token reasoning floor is **stage-independent** model-level property of gemma4:e4b. | **B confirmed at 2 stages with stage-independent mechanism** ; A practically refuted for planner ; E weakened (no stage-specific behavior observed) |
| `reports/research-runs/v3prime-e-mode-split-20260523T051{654,851}.json` + `reports/promo-assets/v3prime-e-substitution-synthesis-result.md` | 2026-05-23 | V3'.e cap × prompt-type sweep (N=20) on e-commerce refund fixture: **substitution arm 20/20 @ cap=400** (0.8s latency, verbatim retrieval) vs **synthesis arm 14/20 @ cap=400** (4.0s latency, refund recommendation). Substitution arm bypasses the ~500-token reasoning floor entirely; synthesis arm hits a *partial* floor — neither V3'.a~d's all-or-nothing 0/10 nor substitution's clean 20/20. **Three workload levels span three cap-behaviour signatures**: heavy synthesis (4-stage cognitive) 0/10 → light synthesis (refund) 14/20 → no synthesis (verbatim) 20/20. | **B refined into gradient form** — synthesis-mode entry has a token cost that scales with task weight, not a constant floor. A practically refuted across all 3 layers. Robin Converse's "two operating modes" framing confirmed on gemma4:e4b. |
| Robin Converse 26b 2×2 matrix companion repo + issue #448 (`https://github.com/triavalabs/gemma4-26b-mode-split` + `Hashevolution/James-RAG-Evol#448`) | 2026-05-23 UTC 12:39 | **RESOLVED same-day** — Robin shipped the 26b matrix within hours of committing. 80 calls / 6 min / zero failures. **Substitution**: 20/20 @ both caps, `ollama_eval_count=38 flat`, **1 unique response across 40 calls** (bit-for-bit deterministic at T=0.2, sharper than our V3'.e flatline finding). **Synthesis**: 20/20 @ both caps (vs our 14/20 @ cap=400), mean=50.7 @ cap=400 / 54.5 @ cap=4096 — **~9× more token-efficient than e4b** on same fixture. Both reference signatures shifted (62→38; 400-450→49-54), triggering pre-registered "next research thread" arm. | **Architecture-invariance test resolved with 3rd-axis discovery** — strict pre-reg says "next thread"; Robin's substantive reading "framing held, data sharpened" is also true (mode-split preserved, gradient scales with parameter count). New axis: **model-scale efficiency** = synthesis cost ∝ 1/param-count; substitution invariant. Joint piece now has 3 independent axes (mode split / workload gradient / model-scale efficiency). Headline 10-word phrase still 3-author locked; "and inversely to parameter count" proposed as draft sub-clause (not yet 3-author confirmed). |
| **Action item from Robin's Finding 1 — extend V3'.e to record unique-output count per cell** | 2026-05-23 (open) | Robin observed `1 unique response across 40 substitution calls` on 26b — "the mode genuinely bypasses sampling". Our V3'.e JSON records `eval_count` flat at 62 but did NOT compute unique-output count. If e4b also produces bit-for-bit identical output, **"bypasses sampling layer"** becomes a publishable mechanism on its own (axis 1 strengthened). Trivial driver patch: add `unique_response_count` per cell to `v3prime_e_mode_split.py`. | Pending operator decision. NOT urgent (existing 62-flatline data already supports the mode-split finding); but **publishable-mechanism-level** evidence if confirmed. |

The 2026-05-21 experiment is interesting because it extends the
failure inventory from cognitive-stage prompts (the 2026-05-18 eval's
domain) to extraction prompts. Same model + same short-structured-
JSON shape → same ~60% empty-response rate. This pushes the
working hypothesis from "meta-reasoning floor" toward "short
structured-JSON output floor" — a slightly different framing of A
that future reader contributions can refine.

If a piece of inbound feedback addresses extraction-style prompts
(not just reasoning), cite the 2026-05-21 experiment in the
"Reader contributions" entry — it's the closest in-house comparison
point.

---

*Related: dev.to Gemma 4 Challenge entry (`reports/promo-assets/devto-gemma4-challenge.md`), JAMES_REASONING_BACKEND env wiring (PR #305), 2026-05-21 event-emit experiment (`docs/research/gemma4-event-emit-experiment-2026-05-21.md`).*
