# 2026-04-15 · Tiered KV Cache M0.3 + M3a — remote CUDA acceptance

## Context

M0.3 lifted BF16 paged-KV from `page_size = 1` to `page_size = 16`
with per-format dispatch (INT8 / FP8 / TurboQuant intentionally stay
at 1). M3a landed the host-tier structural skeleton — `HostPinnedPool`,
`LocalCudaTransport`, `Coordinator`, and tier-aware `RadixNode`
metadata fields — without any runtime behaviour yet. Local win note:
[`2026-04-15-tiered-kv-m0.3-m3a-local.md`](2026-04-15-tiered-kv-m0.3-m3a-local.md).

This note records the remote acceptance run against
[`../../plans/tiered-kv-cache-m0.3-m3a-remote-acceptance.md`](../../plans/tiered-kv-cache-m0.3-m3a-remote-acceptance.md).

## Environment

- GPU: NVIDIA L4 24 GB (driver 580.82.07, CUDA 13.0, SM 89)
- Model: Qwen3-4B BF16, `Qwen/Qwen3-4B` HF Instruct variant
- Build: `cargo build -p infer --release` (default features, `cuda`),
  FlashInfer 0.6.3, Triton 3.5.1 AOT
- Commits evaluated: `bfbfd6f` (first run) → `85bc85b` (post-pull
  rebuild; M0.3 bench numbers below come from the `bfbfd6f` run since
  the 85bc85b changes do not touch the paged-pool hot path)
- Server: `target/release/infer --model-path models/Qwen3-4B --num-slots 4`
  (defaults; `cuda_graph=true`, `kv_cache_dtype=bf16`)
- Cargo env: `CARGO_HOME=/tmp/cargo-home-local`

## Static sanity (§2)

- `rg default_page_size|page_size = 16|max_total_pages|free_pages|page_indices|seq_lens` in
  `crates/cuda-kernels/src/{kv_types,paged_kv,flashinfer}.rs` →
  `KVFormat::BF16 => 16` at `kv_types.rs:23`, INT8/FP8/TurboQuant stay
  at 1; `TokenKVPool` is page-aware (`free_pages`, `page_indices`,
  `seq_lens`, `max_total_pages`); FlashInfer metadata references
  runtime `max_total_pages`. ✅
- `rg kv_cache_to_paged_range_hnd_cuda|kv_cache_to_paged_range_hnd_kernel` →
  FFI decl at `crates/cuda-kernels/src/ffi/kv.rs:35`, kernel +
  launcher at `crates/cuda-kernels/csrc/kv/kv_cache_to_paged.cu:87`
  and `:301`. ✅
- `rg Coordinator|HostPinnedPool|LocalCudaTransport|hit_count|tier_location|soft_pin_until|fingerprint`
  in `infer/src/{kv_tier.rs,prefix_cache.rs,types.rs,kv_tier/*}` →
  `Coordinator`, `HostPinnedPool`, `LocalCudaTransport` all exported
  from `kv_tier.rs`; `prefix_cache::Node` has `hit_count`,
  `tier_location`, `soft_pin_until`, `fingerprint` fields at
  `prefix_cache.rs:73,76,82,88`; `BlockFingerprint` type in
  `types.rs:27`. ✅

## Build / test gates (§3)

Same shared gate as M2b:

```
cargo build -p infer --release                             # cold 4m 11s, incremental ~12s
cargo test --workspace --exclude mlx-sys --release --lib   # 348 passing
cargo test -p infer --release --test e2e                   # Phase 1-4 pass after replay-drift fix
cargo fmt --all -- --check                                 # clean
```

Pre-existing failures (same list as M2b) are not in scope: e2e_qwen35
baseline drift, greedy_consistency B=3 decode, clippy unused import in
tools.

The e2e Phase 3 regression was NOT introduced by M0.3/M3a but surfaced
here for the first time because M0.3 + the c3f65f7 offload retirement
hardened the Phase 1 baseline and the stream/non-stream consistency
check stopped being short-circuited by an earlier panic. See
[`../errors/2026-04-15-e2e-phase3-replay-drift.md`](../errors/2026-04-15-e2e-phase3-replay-drift.md)
for the root cause + fix committed to `server_engine.rs`.

## BF16 page-size gate (§4)

```
python3 scripts/bench_throughput_sweep.py \
    --url http://localhost:8000 \
    --label page16 \
    --save docs/experience/wins/2026-04-15-bench-page16.json
```

Comparing against the same-host 2026-04-13 **page1** baseline
([`2026-04-13-bench-page1.md`](2026-04-13-bench-page1.md),
commit `c531315`, same `--num-slots 4`):

```
                            page1 (2026-04-13)    page16 (2026-04-15)   delta
──────────────────────────────────────────────────────────────────────────────
In | Out  | C=1
128 | 64            30.0 t/s             29.9 t/s           -0.1  (noise)
128 | 128           30.1 t/s             29.8 t/s           -0.3  (noise)
128 | 256           29.9 t/s             29.7 t/s           -0.2  (noise)
128 | 512           29.7 t/s             29.6 t/s           -0.1
512 | 128           29.3 t/s             29.1 t/s           -0.2
512 | 256           29.3 t/s             29.3 t/s            0
512 | 512           29.1 t/s             29.4 t/s           +0.3
1024| 128           28.5 t/s             28.7 t/s           +0.2
1024| 256           28.5 t/s             29.1 t/s           +0.6   ← long-context win
1024| 512           28.3 t/s             29.2 t/s           +0.9   ← long-context win
2048| 256           27.1 t/s             28.5 t/s           +1.4   ← long-context win

In | Out  | C=2..4
512 | 256 | 2       57.0 t/s             56.7 t/s           -0.3  (noise)
512 | 256 | 4      111.4 t/s            109.3 t/s           -2.1  (noise)
128 | 128 | 2       57.4 t/s             57.1 t/s           -0.3
128 | 128 | 4      108.3 t/s            111.3 t/s           +3.0

C ≥ 8 rows
128 | 256 | 8        0.0 t/s            113.2 t/s          (recovered)
512 | 256 | 8        0.0 t/s            109.6 t/s          (recovered)
128 | 256 | 16       0.0 t/s            113.2 t/s          (recovered)
512 | 256 | 16       0.0 t/s            109.6 t/s          (recovered)
128 | 256 | 32       0.0 t/s            113.2 t/s          (recovered)
512 | 256 | 32       0.0 t/s            109.6 t/s          (recovered)
128 | 256 | 64       0.0 t/s            113.1 t/s, 16 err  (recovered)
```

**C=1..4 summary**: throughput flat within noise on short contexts,
small wins (+0.2 to +1.4 tok/s) on long contexts (1024/2048), which is
consistent with page16 reducing per-token metadata overhead on
sequences where the metadata cost was a larger share of the decode
budget. No regression on any C≤4 row.

**C≥8 summary**: In the 2026-04-13 page1 sweep these rows all reported
`0 tok/s` because the batched-decode CUDA error killed the server's
CUDA context on the first C≥8 config and every subsequent request
returned an empty 200 stream
(`docs/experience/errors/2026-04-13-batched-decode-high-concurrency.md`).
On 2026-04-15 with M2a+M2b+M0.3+M3a+M3b landed, the sweep runs all
C≥8 configs to completion with non-zero throughput. C=64 is the only
row with explicit errors (16/? at the tail under queueing), all
others report `Err = 0`. The earlier "sticky CUDA error" class of
regressions is contained.

Peak C=1: 30.0 → 29.9 tok/s (flat)
Peak C=4: 111.4 → 111.3 tok/s (flat)
ITL p50 range, C=1..4: 33.0-36.6 ms → 33.3-35.1 ms (tighter, matching
the L4 HBM-bound floor)

§4 acceptance criteria:

- [x] Sweep completes without CUDA faults on all 22 configs.
- [x] BF16 decode / prefill throughput does not regress vs page1.
- [x] No short-context crash or metadata-shape error in server logs.

## M3a structural smoke (§5)

```
cargo test -p infer --release kv_tier          # 20 pass (0 failed)
cargo test -p infer --release prefix_cache     # 33 pass (0 failed)
```

All `kv_tier` tests (including `LocalCudaTransport` structural tests,
`HostPinnedPool` capacity, `Coordinator` command queueing) and all
`prefix_cache` tests (including the new M3b `lookup_or_stage_*` and
`evict_with_policy_*` classifications) pass on the L4 host.

§5 optional stub grep:

```
rg "LocalCudaTransport requires at least one transfer op|GPU required: LocalCudaTransport poll is a structural stub" \
   infer/src/kv_tier/transport/local_cuda.rs
```

Both error messages still present — real `cudaMemcpyAsync` behaviour
is correctly deferred to later M3 runtime work. ✅

## Sign-off

- [x] Static sanity checks passed.
- [x] Build/test gate passed on CUDA after the replay-drift fix.
- [x] BF16 page-size sweep completed, compared against same-host
      `page1` baseline, **no regression on C≤4 and a recovery on
      C≥8**.
- [x] `kv_tier` + `prefix_cache` structural smoke passed on CUDA.
- [x] This win note exists with raw output and the page1 vs page16
      comparison.

**M0.3 + M3a accepted on the 2026-04-15 L4 host.**

## Rule

Holding the paged-KV hot path at `page_size = 16` for BF16 while
leaving quantized formats at `page_size = 1` is the right
asymmetry: **don't change the per-row format of quantized pages just
to chase per-token bookkeeping overhead**, because the per-format
pool bytes per head are already tight and the metadata savings on
quantized formats is small relative to the scale / dequant cost.

**Corollary**: the `page1 → page16` delta on 2026-04-15 is net-neutral
at C≤4 and slightly positive at long contexts. The real wins from
page16 come on workloads with much deeper contexts (>4k) where the
metadata cost is a meaningful fraction of the decode cycle. On short
L4 traces, expect the lift to read as "no regression," not "clear
speedup." That is the gate that actually matters here — page size is
a structural prereq for M3/M4, not a performance knob.
