# A2 Default-Flip — Quality Delta Card Pre-Template

> **Status**: pre-template. Use this scaffold when the three gates
> below are satisfied. Operator fills the placeholders, opens the
> PR. Do not flip the default before gates clear.
>
> **What gets flipped**: `JAMES_GEMMA4_E4B_THINK_OFF` env default
> from "0" (unset == ON, current production) to "1" (OFF, A2's
> opt-in plumbing made default).
>
> **Why this is a pre-template, not a PR**: per backlog §7.2 (2026-05-30
> A3 v2 LLM-judge result reframing), the default-flip is NOT scheduled.
> Synthetic fixtures + deterministic grader + local LLM-judge were
> ruled insufficient as an oracle for a global behaviour change.
> The new gate is real-query evidence. This file captures the form
> that evidence has to take so the operator does not have to
> re-derive the structure under time pressure.
>
> **Pretemplate id**: D12 (`docs/handovers/v0.4.x-backlog-reconciliation-2026-05-30.md` §7.8)

---

## 0. Gate checklist (all three must clear before fill-in)

- [ ] **G1 — LLM-judge agreement**: gemma3:12b paired-judge run over
      ≥ 3 real-query Δ samples (think=ON vs OFF) returns no
      "ON wins" verdict. (A3 v2 already cleared this on synthetic
      cells per #611; G1 re-runs the test on real-query data.)
- [ ] **G2 — Real-query dogfood window**: ≥ 1 week of production
      query traffic served with `JAMES_GEMMA4_E4B_THINK_OFF=1`
      enabled. No quality complaint, no operator regression noted.
- [ ] **G3 — Real-query Quality Delta Card** (this card filled
      in with real numbers, not synthetic): 5-axis paired
      comparison on a real-query baseline JSON.

If any gate fails, file a finding under `reports/research-runs/qvt-ablation-findings.md`
with `bucket: (b)` (model behaviour — the flag was supposed to be
safe and isn't on real queries) and **do not flip**.

---

## 1. Operator fill-in — PR body skeleton

Below is the PR body the operator pastes. Sections marked `<FILL>`
require the real data.

```markdown
## Summary

A2 (#609) plumbed `JAMES_GEMMA4_E4B_THINK_OFF` as opt-in. This PR
flips the default from "0" to "1" — gemma4:e4b runs with
think=False by default in production.

Gate evidence:
- G1 (LLM-judge agreement on real-query Δ): see §<n> below
- G2 (≥ 1 week dogfood at `<window_start>` → `<window_end>`): see §<n>
- G3 (5-axis Quality Delta Card vs real-query baseline): see §<n>

## Verification

- Existing A2 contract tests still green (no behaviour change inside
  `core/reasoning/think_policy.py` — only the default of the env
  flag changes).
- Real-query baseline JSON: `<path_or_sha>`
- LLM-judge transcript: `reports/research-runs/<judge_run>.json`
- Dogfood window operator log: `<pointer_to_audit_log_range>`

## Quality Delta vs baseline_<sha> (real-query, 5-axis)

| axis              | baseline (ON) | this PR (OFF default) |  Δ  | noise band | sign |
|-------------------|---------------|-----------------------|-----|------------|------|
| Path Recall       | <FILL>        | <FILL>                | <FILL> | ±<FILL> | <FILL> |
| Graded Answer     | <FILL>        | <FILL>                | <FILL> | ±<FILL> | <FILL> |
| Abstention F1     | <FILL>        | <FILL>                | <FILL> | ±<FILL> | <FILL> |
| Token Cost (p50)  | <FILL>        | <FILL>                | <FILL> | ±<FILL> | <FILL> |
| Latency Cost (p50)| <FILL>        | <FILL>                | <FILL> | ±<FILL> | <FILL> |

**Pareto verdict** (per `eval/qvt/oracle.py::_classify_five_axis_delta`):
<FILL — strong-adopt / adopt / efficiency-adopt / tier-gated / reject / zero>

**Method note**: real-query baseline JSON captured at
`<workspace_or_prod>/eval/qvt/baseline_<sha>.json` over <N> real queries
from the dogfood window (`<window_start>` → `<window_end>`), de-duplicated
to remove operator self-tests. Cost axes use p50; p95 reported in §<n>
for tail sensitivity.

## Why this is not over-claim

The α-3 cycle's published narrative documents the threshold for global
behaviour change: synthetic-fixture-only evidence is insufficient. This
PR ships the flip only because real-query evidence agrees with the
synthetic prior. If they ever disagree on future model versions, the
flip rolls back and the new flag default is re-decided per gate.

## Out of scope

- Other thinking-capable models (gemma4 family beyond e4b) — opt-in
  only until each model gets its own gate sequence (per A2 model
  whitelist policy).
- Removal of the `JAMES_GEMMA4_E4B_THINK_OFF` env flag itself — the
  flag stays so operators can re-enable `think=ON` for diagnostic
  comparison runs without code change.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
```

---

## 2. Real-query baseline capture procedure (G3 input)

Use the existing tooling:

```bash
# 1. Capture real-query baseline against the current ON-by-default JAMES
JAMES_WORKSPACE=<production_workspace> \
  python scripts/qvt_capture_baseline.py \
    --suite real_query_<window_id> \
    --runs 1 \
    --baseline-out baseline_<sha>_real_query_ON.json

# 2. Flip the flag locally (without committing) and capture again
JAMES_WORKSPACE=<production_workspace> \
JAMES_GEMMA4_E4B_THINK_OFF=1 \
  python scripts/qvt_capture_baseline.py \
    --suite real_query_<window_id> \
    --runs 1 \
    --baseline-out baseline_<sha>_real_query_OFF.json

# 3. Re-score both with the canonical oracle
python scripts/qvt_rescore_baseline.py \
    --bench baseline_<sha>_real_query_ON.json \
    --bench baseline_<sha>_real_query_OFF.json \
    --emit-delta-card delta_a2_default_flip.md
```

The third command emits a paste-ready Markdown table for the PR body.
If the re-score script does not yet support a 2-bench delta render
at flip time, fall back to manual table fill — the per-axis
aggregates are in each JSON's `aggregate.five_axis` block.

---

## 3. Reading-list for the operator (before fill-in)

- `docs/handovers/v0.4.x-backlog-reconciliation-2026-05-30.md` §7.2
  — the original A3 v2 reframing that produced the real-query gate.
- `memory/feedback_oracle_phrase_artifacts.md` — 4-step verification
  rule. Run before interpreting Δ values that look saturated /
  zero / extreme.
- `docs/design/v0.4-qvt-alpha-5-ablation-matrix.md` §3.1 — Pareto
  verdict walk-through, applies verbatim to the 5-axis above.
- `scripts/research/v3prime_a3_v2_llm_judge.py` — paired-judge
  harness reused for G1.

---

## 4. Why "pretemplate" and not "land the flip now"

The α-5 cycle delivered three measurement-debt corrections (#618,
#619, #623) that retroactively changed how aggregates are computed
on the canonical baselines. **Any flip evaluation pre-dating those
corrections is now untrusted**. A fresh real-query baseline post
α-5 closure is the cheapest reset; this pre-template is the
operator-facing entry point.

Sequencing rule: this card never ships before the α-5 cycle
closure PR (`reports/research-runs/alpha-5-cycle-summary-DRAFT.md`)
lands, because the closure PR is what re-blesses the canonical
oracle that G3 measures against.

---

## 5. Out of scope for this pretemplate

- Schema for the LLM-judge G1 transcript (covered by #611's harness)
- Schema for the dogfood G2 operator log (free-form, capture window
  + reasonable "no complaints" attestation)
- Filing the PR — operator does that with real numbers
- Other model defaults — each model with thinking capability gets
  its own gate sequence; do not generalise this card

---

## 6. Cross-references

- A2 plumbing PR: #609
- A3 quality boundary: #608
- A3 v2 LLM-judge: #611
- Reframing decision (real-query gate): `docs/handovers/v0.4.x-backlog-reconciliation-2026-05-30.md` §7.2
- 5-axis oracle: `eval/qvt/oracle.py` (post α-5 corrections #618/#619/#623)
- Pareto classifier: `eval/qvt/oracle.py::_classify_five_axis_delta`
- Backlog Lane D row: D12
