# 2026-05-07 · M3.9 Phase 1A v3 KILL CRITERIA FIRED — Mixed-disable hypothesis FALSIFIED

## Priority & ROI

**Priority**: this entry is the kill-criteria record. Action item:
**revert or escalate** — current Phase 1A v3 commit (`d083b2e`)
causes a 132% TTFT regression, opposite of the expected -30%.

**ROI of the experiment**: Phase 0 instrumentation +
implementation + bench validation cost ~3 hours total codex+claude.
Saved us from shipping the regression unverified. Per
`feedback_docs_priority_roi_evidence.md` rule: **kill criteria
gates Phase 1 acceptance**. Validated.

**Negative case revealed**: my entire M3.9 architecture hypothesis
(F4-Small precondition was the cause of 10× tax) is
**incorrect**. The 10× tax comes from somewhere else entirely;
removing the precondition just routes the same workload through
Mixed instead of Split, and **Mixed turns out to be SLOWER at
long-ctx**.

## Kill criteria evaluation (per M3.9 plan + Phase 0 wins entry)

Kill criteria threshold:
> "if Phase 1A v3 lands and longctx 4k/c=4 TTFT reduction is < 30%,
> a deeper issue exists (the per-step cost itself) — abandon
> Phase 1A v3, escalate to source archaeology"

Actual measurement:

| Metric | F4-Small (`c63c31c`) | Phase 1A v3 (`d083b2e`) | Δ |
|---|---:|---:|---|
| out tok/s mean | 122.5 | **69.9** | **-43%** |
| TTFT mdn | 3403 ms | **7905 ms** | **+132%** |
| ITL mdn | 20.1 ms | 29.4 ms | +46% |
| TPOT mdn | (n/a) | 60.2 ms | n/a |
| E2E mdn | 8.53 s | 15.41 s | +81% |

**TTFT increased by +132%** vs target -30%. Kill criteria fires.

## What's now known to be true

Mixed path IS being used (Phase 1A v3 successfully removed the
precondition):
- Plan distribution: 21 plan=mixed, 0 plan=split, 1 plan=prefill
- Metric: ok_true=38, ok_false=0 (Mixed always succeeds when invoked)

Mixed step durations from server log:
- Range 521 ms - 1424 ms per step
- Average ~700 ms over 21 steps = ~14.7 s total prefill work
- vs Split's prior 7 × 1.2 s = 8.4 s

So **Mixed produces MORE total prefill work for the same logical
workload**. That's the regression.

## Hypotheses for why Mixed is slower at long-ctx

(Not yet verified — needs source archaeology Phase 1A v3 cost
analysis):

### H_M1 — Multi-slot ring overhead (least likely)

Phase 1A v3 added 4-slot async readback ring. Each launch now
does an extra D2D + D2H copy (snapshot to ring slot). Per-launch
overhead.

Magnitude est: D2D copy of B×4 bytes is ~µs scale. 21 mixed
steps × 36 layers × 4 µs = ~3 ms extra. Far below the ~7-second
TTFT regression. Unlikely sole cause.

### H_M2 — Mixed path runs more total steps than Split (likely)

Split path bundles a chunk into one launch + a decode batch into
another launch (2 launches per "logical step"). Mixed path
combines them into one launch. If the sum of work is the same
but Mixed has finer-grained step partitioning, it could result
in MORE iterations of the scheduler loop per request → more
small steps each paying admission/plan/launch overhead.

Could check: total step count Phase 0 vs Phase 1A v3 at same
bench duration.

### H_M3 — Mixed kernel actually slower per token at this shape

Mixed kernel handles varlen Q across decode rows (qlen=1) and
prefill rows (qlen=N) in same launch. Could be inefficient at
the specific workload (4 rows × 2k chunk + maybe some decode
rows).

### H_M4 — Per-step KV cache page allocation in Mixed differs from Split

Mixed path's `paged_kv_pool.alloc_tokens` happens for all prefill
rows together in one launch. Split path interleaves alloc_tokens
between prefill kernel and decode launches. Could be subtle
allocator pathology.

## Recommended action

**Option A** (safest): **revert Phase 1A v3**(`d083b2e`)to
restore Split path. Long-ctx regression too severe to ship.

**Option B** (investigate): keep Phase 1A v3 in main BUT add a
config flag (`--scheduler-mixed-or-split-policy`) defaulting to
SPLIT, with mixed-only opt-in for testing. This preserves the
substrate without regressing production.

**Option C** (root cause): defer the revert decision; immediately
do source archaeology on Mixed path per-step cost. If a fixable
issue is found in 1-2 hours, fix in place. If not within that
budget, revert (Option A).

I recommend **Option B** + immediate Option C investigation.
Option B preserves the multi-slot ring (good substrate for future)
but defaults to Split (no regression). Option C runs in parallel
to find what makes Mixed slower; if found, flip default to Mixed
later.

## Next steps

1. Notify codex via tmux: kill criteria fired, recommend Option B
2. Codex / Claude joint: source archaeology on Mixed step cost
   - Compare step breakdown: what does plan=mixed do that
     plan=split doesn't?
   - Trace: nsys re-trace with Phase 1A v3 (now traceable since
     Mixed runs visible in steps)
3. If H_M2 or H_M3 confirmed: implement targeted fix
4. If no fixable issue: ship Option B and treat Mixed as future
   enabling for some-other-shape

## Implications for M-final roadmap

- **Phase 1A v3 NET ZERO at long-ctx** (or NEGATIVE). Don't claim
  long-ctx victory yet.
- **F4-Small still the only validated win** so far at high-conc.
- M_pf P1 (prefetch wiring) was queued; still valid but lower
  priority since long-ctx isn't getting the expected bump from
  Phase 1A v3 anyway.
- "World-first" trajectory may need to revisit kernel-axis
  (M_b.2 FP8 prefill) or other approach.

## Bench artifacts

- Phase 1A v3 bench: `bench-output/2026-05-07-m3.9-1av3-longctx-4k/`
- Server log with step breakdowns: `/tmp/m39-1av3-server.log`
- Compare to F4-Small: `bench-output/2026-05-07-m3.6-f4small-longctx-8k/`
  (note: 8k there, 4k here; comparable shape modulo size)

## Rule

- **Kill criteria are not theatrical** — they fire for a reason.
  Phase 1A v3 looked great in source-survey + math + Phase 0
  metric distribution, but actual bench TTFT regressed 132%.
  Without the kill criteria gate (per memory rule
  `feedback_docs_priority_roi_evidence.md`), this would have
  shipped to main and someone would have to revert it later.
- **The "obvious cause" can be wrong**. F4-Small precondition
  WAS the structural reason Mixed didn't run at long-ctx. But
  removing it didn't help because Mixed itself is slow at this
  shape. The precondition was protecting from the regression,
  not just the silent token loss.
- This is the second F4-Small-related cliff (first: M_b.1 Phase B
  no-delta in `2e60844`; second: this). Pattern: optimizations
  driven by trace observation can fail to materialize because
  the trace's "obvious cause" isn't what limits the workload.
