# 2026-05-07 · M3.6 vLLM long-context 8k control — ARLE wins decode, vLLM wins prefill

## Goal

- Fill the comparison gap left by `c63c31c` (F4-Small long-context
  validation): is ARLE F4-Small actually ahead of vLLM at the
  long-context 8k/256/c=4 shape?
- Goal type: **diagnostic** — without a same-shape vLLM control,
  ARLE's 23.9 ms ITL is unanchored.

## Hypothesis

Going in, two possibilities:
1. F4-Small advantage carries over: ARLE ≤ vLLM ITL → confirms
   sync-removal is universal win
2. ARLE's 23.9 ms is just a small-batch artifact and vLLM is also
   ≤ 25 ms or better → F4-Small is shape-specific

## Command

vLLM via `scripts/vllm_serve_control.sh`:
```bash
scripts/vllm_serve_control.sh --max-num-seqs 8 --max-model-len 12288
```

Bench replays the F4-Small longctx shape (`c63c31c` shape):
```bash
guidellm benchmark run --target http://localhost:8000 \
  --profile concurrent --rate 4 \
  --data 'prompt_tokens=8192,...,output_tokens=256,...' \
  --max-seconds 60 --warmup 10 --random-seed 20260416
```

## Environment

- Hardware: NVIDIA RTX 4080 SUPER, CUDA 13.2.1, SM 8.9
- vLLM 0.20.1, `--max-num-seqs 8`, `--max-model-len 12288`,
  `--kv-cache-dtype fp8`, `--attention-backend TRITON_ATTN`
- ARLE side cited from `c63c31c` (commit 8f83c80 build, F4-Small)
  with same 12288 max-seq-len + num-slots 8
- Bench shape: identical (8192/256, c=4, 60s + 10s warmup)

## Results

| Metric | ARLE F4-Small (`c63c31c`) | vLLM s8 long-ctx | Δ (vLLM vs ARLE) |
|---|---:|---:|---:|
| out tok/s mean | 92.2 | **105.6** | **vLLM +14.5%** |
| in tok/s mean | 3,314 | 3,924 | vLLM +18.4% |
| **TTFT mdn** | **4,961 ms** | **2,367 ms** | **vLLM 2.10× faster TTFT** |
| TTFT p95 | 8,112 ms | 4,693 ms | vLLM 1.73× faster |
| **ITL mdn** | **23.9 ms** | 26.7 ms | **ARLE 1.12× faster ITL** |
| ITL p95 | 42.1 ms | 30.3 ms | vLLM 1.39× faster (tail) |
| TPOT mdn | 43.8 ms | 36.0 ms | vLLM 1.22× faster |
| E2E mdn | 11.2 s | 9.2 s | vLLM 1.22× faster |

## Headline finding

**Mixed picture — kernel-level lessons clearly:**

- **ARLE decode (steady-state per-token) wins by 10.5%**
  (23.9 ms vs 26.7 ms ITL). F4-Small's gain on decode IS real and
  generalizes to long-context decode.
- **vLLM prefill wins by 52%** (TTFT 2367 ms vs 4961 ms for the
  same 8192-token prompt). This is a HUGE gap.
- **Net throughput: vLLM wins by 14.5%** because at c=4 with 8k
  prompts, prefill cost dominates request lifetime; ARLE's slow
  prefill drags the per-second request rate.

Per-token decode breakdown (ITL):
- ARLE F4-Small: 23.9 ms / 4 rows = 5.97 ms/row
- vLLM TRITON_ATTN: 26.7 ms / 4 rows = 6.68 ms/row
- ARLE wins per-row by 11% on decode.

Per-token prefill breakdown (TTFT - kv_cache_load assuming cold):
- ARLE: 4961 ms / 8192 tokens = **0.61 ms/prefill_token**
- vLLM: 2367 ms / 8192 tokens = **0.29 ms/prefill_token**
- ARLE prefill is **2.1× slower per-token** than vLLM.

## What this implies for next-tier work

The TTFT gap reframes the M_b vs M3.7 priority decision:

### M_b.2 (FP8 TileLang HD128) is now justified — for prefill, not decode

The M_b.2 plan was originally written assuming FP8 attention was
the high-conc bottleneck. Phase 1 + F4-Small data already showed
ARLE wins high-conc decode without M_b. **But long-ctx 8k prefill
is ARLE's weakest shape** — and the Phase 1 trace showed
`prefill_attention_paged_qk_norm_rope_hd128_kernel` (200 ms / 4176
calls = 47 µs avg, single-call latency-dominated) as a measurable
GPU consumer.

Per-prefill-token math:
- 8192 prompts × 36 layers × 47 µs/call = ~13.9 s GPU compute
- ARLE actual TTFT 4.96 s — much better than 13.9 s, so prefill is
  parallelized across requests
- vLLM at 2.37 s suggests **vLLM's prefill kernel is ~2.1× faster
  per token AND/OR vLLM batches prefill chunks more efficiently**

### M3.7 §B.1.2 (async chunk completion in prefill) is now urgent

`grep -rn ctx.sync infer/src/model/qwen3/prefill.rs` shows three
sync sites at lines 121, 380, 391 — these are per-chunk completion
checks. Chunked prefill at 8k would hit all three per chunk × 8
chunks (assuming 1k chunk size) × 36 layers worth of work between
syncs.

Removing those syncs (M3.7 B.1.2) should give a measurable TTFT
improvement at long-context. **Magnitude estimate**: if prefill
sync overhead is ~100 ms per chunk × 8 chunks = 800 ms, removing
gets ARLE TTFT from 4961 → ~4161 ms. Still slower than vLLM 2367
ms, but a large step.

The remaining ~1800 ms gap is likely kernel-level (M_b.2 territory
— TileLang FP8 prefill replacing the hand-CUDA `prefill_attention_paged_qk_norm_rope_hd128_kernel`).

### Strategy update

| Track | Pre-this-bench | Post-this-bench |
|---|---|---|
| F4-Small extension (B.1.1, B.1.3 — non-greedy + legacy sample sync) | nice-to-have | nice-to-have (uncovered cases not bench-validated) |
| **B.1.2 — async prefill chunk sync** | low priority | **HIGH — biggest TTFT lever measured** |
| **M_b.2 — FP8 TileLang prefill (not decode)** | speculative | **MEDIUM — addresses kernel-level prefill gap** |
| F4-Big (in-graph sampling) | optional | optional, no new evidence either way |
| Tier A (multi-stream pipeline) | future | future |

## Problems

- vLLM TTFT 2367 ms includes vLLM's own warmup/cudagraph effects;
  the comparison is "warmup-amortized" since both ran 10s warmup +
  60s steady-state. Should be apples-to-apples.
- ARLE numbers come from c63c31c (separate run). vLLM bench used
  identical seed (20260416). Rerunning ARLE this iteration would
  cost ~1 min; deferred for now to save tick time.
- No nsys trace from this bench (vLLM-side trace deferred per
  earlier wins entry).

## Bench Status

- vLLM bench artifact: `bench-output/2026-05-07-m3.6-vllm-longctx-8k/`
  (gitignored).
- ARLE comparison numbers from prior wins entry `c63c31c`.

## Next actions

1. **Push M3.7 B.1.2 (async prefill chunk sync) as next codex
   delegation** — biggest measured TTFT lever for long-context.
   Update M3.7 plan to bump B.1.2 priority above B.1.1 (non-greedy)
   and B.1.3 (legacy).
2. **Workload sweep**: run F4-Small + vLLM at 4k/16k/32k input ×
   c=4/8/16 to confirm the per-row decode advantage stays and
   characterize the prefill gap across shapes.
3. **Update M_b.2 framing** from "FP8 decode" to "FP8 prefill"
   priority (the actual kernel under-performance is on prefill,
   per the new TTFT data).

## Rule

- **Headline metric (out tok/s) can hide the real story**. The
  high-conc bench showed ARLE +30% past vLLM (out tok/s) which
  framed the project as "won". The long-ctx control reveals ARLE
  still 14.5% BEHIND vLLM at that shape because prefill, not
  decode, dominates throughput when concurrency is low. Always
  decompose throughput into TTFT + ITL + concurrency before
  declaring victory.
- **F4-Small generalizes on decode but does NOT cover prefill**.
  This is consistent with the implementation (F4-Small only changed
  `sample_batch_greedy_readback`). Future plan revisions should
  consistently distinguish "decode-axis" from "prefill-axis"
  improvements.
