# M3.9 Phase 0 — Split-Plan Tax Instrumentation

## Context

Long-context evidence showed ARLE pure prefill at `15.4 us/tok` but split
mixed prefill+decode at `154 us/tok`. Source survey found Qwen3 already has a
unified mixed-batch path, so Phase 0 adds counters to classify whether the
runtime is actually executing that path or falling back to separate launches.

## What Worked

- Replaced the mixed-batch boolean return with a backend-neutral
  `MixedBatchOutcome`.
- Added explicit fallback reasons at every Qwen3 `decode_batch_with_prefill`
  early return, including `prefill_seq_len_mismatch`.
- Counted scheduler-side mixed-plan fallbacks that retreat before model
  dispatch as `scheduler_pre_dispatch_fallback`.
- Recorded mixed path `ok_true` / `ok_false` counts in `ServerMetrics`.
- Exposed `engine_prefill_path_stats` through `/v1/stats?format=json`, plus a
  `prefill_path=...` suffix in text stats and Prometheus counters.

## Bench Status

Status: `pending-remote` / owner-run GuideLLM handoff.

This instrumentation is a runtime change under `infer/src`, but the M3.9
Phase 0 acceptance flow assigns the actual longctx 4k/c=4 GuideLLM run and
nsys trace to the owner after this patch lands. Codex did not start the bench
in this turn so the GPU stays available for the paired trace run.

Expected readout during the longctx 4k/c=4 run:

```text
prefill_path=ok_true:<N>,ok_false:<M>,prefill_seq_len_mismatch:<K>,scheduler_pre_dispatch_fallback:<S>
```

If `ok_false` dominates, M3.9 Phase 1A should fix the firing fallback
condition. If `scheduler_pre_dispatch_fallback` dominates, the next step is a
scheduler launch-candidate audit. If `ok_true` dominates, the next diagnosis
step is kernel-time breakdown for attention KV scaling or varlen inefficiency.

## Goal

Diagnosis: classify the split-plan 10x tax as mixed-path fallback (H1) versus
executed mixed path with attention/KV or varlen kernel cost (H2/H3).

## Hypothesis

If the 10x tax is H1, `engine_prefill_path_stats.ok_false_count` will dominate
and `prefill_seq_len_mismatch` or another fallback reason will explain the
ratio. If `ok_true_count` dominates, the fallback hypothesis is falsified.

## Command

Owner-run command shape:

```bash
target/release/infer --port 8000 --max-seq-len 8192 --num-slots 8

guidellm benchmark run --target http://localhost:8000 \
  --profile concurrent --rate 4 \
  --data 'prompt_tokens=4096,...,output_tokens=256,...' \
  --max-seconds 120 --warmup 10 --random-seed 20260416
```

Stats readout:

```bash
curl -s 'http://localhost:8000/v1/stats?format=json' | jq '.engine_prefill_path_stats'
curl -s 'http://localhost:8000/v1/stats' | tr ' ' '\n' | rg '^prefill_path='
```

## Environment

- **Backend:** CUDA
- **Model:** Qwen3-4B BF16 weights (`infer/models/Qwen3-4B`)
- **Hardware:** owner-run CUDA host; local validation used RTX-class sm_89
- **Feature set:** `cargo build --release -p infer --features cuda`
- **Env vars:** `NVCC_CCBIN=/usr/bin/g++-14`,
  `INFER_TILELANG_PYTHON=/home/ckl/projects/arle/.venv/bin/python`,
  `TORCH_CUDA_ARCH_LIST=8.9`
- **Commit:** pending at entry creation; fill with landed commit in the
  after-bench snapshot.

## Results

Pending owner-run GuideLLM and nsys results.

## Delta vs Baseline

- **Baseline:** split-plan confirmation at
  [`2026-05-07-split-plan-10x-tax-confirmed.md`](2026-05-07-split-plan-10x-tax-confirmed.md):
  pure prefill `15.4 us/tok`, split mixed `154 us/tok`, longctx 4k/c=4 TTFT
  `3403 ms`.
- **After:** pending owner-run snapshot.

## Artefacts

- Raw: pending `bench-output/2026-05-07-m3.9-phase0-longctx-4k-c4/`
- nsys: pending owner-run trace

## Verification

Passed locally before review:

- `cargo check --release -p infer --features cuda`
- `cargo check --release -p infer --no-default-features --features metal,no-cuda`
- `cargo check --release -p infer --no-default-features --features no-cuda`
- `cargo clippy --release -p infer --features cuda -- -D warnings`
- `cargo fmt --all --check`
- `cargo test --release -p infer --features cuda server_metrics --lib`
- `cargo test --release -p infer --no-default-features --features no-cuda server_metrics --lib`
- `cargo test --release -p infer --features cuda --test e2e -- --test-threads=1`
- `cargo test --release -p infer --features cuda --test greedy_consistency -- --test-threads=1`

Review:

- Round 1: P2 asked for an explicit pending GuideLLM bench stub; this entry now
  carries the owner-run command, environment, baseline, and artefact placeholders.
- Round 2: P2 found scheduler-side mixed-plan fallbacks before model dispatch
  were not counted; added `scheduler_pre_dispatch_fallback` without changing
  fallback behavior.
- Round 3: no actionable findings.

## Rule

Instrumentation should report the model's mixed-path decision without changing
that decision. The scheduler owns metrics recording; model code only returns
the backend-neutral outcome and reason.
