# 2026-05-07 · M3.6 Phase 2 vLLM s14 control — bench (trace deferred)

## Goal

- M3.6 Phase 2 control bench per
  [`docs/plans/M3.6-decode-throughput-no-retraction-tax.md`](../../plans/M3.6-decode-throughput-no-retraction-tax.md)
  Phase 2: same workload shape against vLLM `--max-num-seqs 14` to
  produce direct ARLE-vs-vLLM ITL/throughput numbers under identical
  test conditions.
- Goal type: **diagnostic** (independent verification of codex's
  Phase 1 step 3 vLLM s14 numbers + per-token efficiency analysis).

## Hypothesis

vLLM s14 numbers should match codex's
[`m6-highconc-phase1-vllm-s14`](../../plans/m6-cuda-vllm-gap-followups.md)
within scout noise (~5%). The interesting comparison is per-token
ITL: ARLE Phase 1 measured 66 ms ITL at decode_batch ~38; vLLM s14
should be much lower because it processes a smaller batch per step.

## Command

vLLM up via `scripts/vllm_serve_control.sh` (default `--max-num-seqs 14
--max-model-len 2048`). Bench replays codex's exact shape:

```bash
guidellm benchmark run --target http://localhost:8000 \
  --model Qwen/Qwen3-4B --processor infer/models/Qwen3-4B \
  --profile concurrent --rate 64 \
  --data 'prompt_tokens=1024,prompt_tokens_min=1024,prompt_tokens_max=1024,output_tokens=256,output_tokens_min=256,output_tokens_max=256' \
  --max-seconds 45 --warmup 5 --random-seed 20260416
```

## Environment

- Hardware: NVIDIA RTX 4080 SUPER, CUDA 13.2.1, SM 8.9
- vLLM: `0.20.1` from `/tmp/arle-vllm-venv`
- Driver: nsys 2025.6.3.541 (trace deferred — see Problems)
- Build: `998bfee` server script (no patches)

## Results

### Bench numbers (45s steady-state at c=64)

| Metric | This run | codex prior `m6-highconc-phase1-vllm-s14` | Δ |
|---|---:|---:|---:|
| out tok/s | **646.8** | 614.55 | +5.3% |
| in tok/s | 3927 | n/a | — |
| TTFT p50 | 17,616 ms | 17,609 ms | +0.04% |
| ITL p50 | **20.0 ms** | 20.24 ms | -1.2% |
| ITL p99 | 21.0 ms | n/a | — |
| TPOT p50 | 88.6 ms | n/a | — |
| E2E p50 | 22.7 s | n/a | — |
| Concurrency mdn | 64 | 51 | — |

vLLM Phase 2 numbers reproduce codex's prior s14 control to within
5% (scout noise). Confirms the gap finding is stable.

### Direct ARLE-vs-vLLM comparison (same 1024/256 c=64 shape)

| Metric | ARLE s48 (Phase 1) | vLLM s14 | vLLM advantage |
|---|---:|---:|---:|
| out tok/s | 462 | 647 | **1.40× vLLM** |
| **ITL p50** | **66 ms** | **20 ms** | **3.30× faster vLLM** |
| TTFT p50 | 3,729 ms | 17,616 ms | 4.7× faster ARLE |
| Effective decode batch | ~38 | ~14 | — |

## Headline finding

**vLLM is 3.3× faster per-token despite running a 2.7× smaller
batch.** The ITL gap is the ground-truth signal:

- ARLE: 66 ms / token × 38 rows = **2,508 ms of "token work" per
  step** (decoupled — one step delivers ~38 tokens)
- vLLM: 20 ms / token × 14 rows = **280 ms of "token work" per
  step**

Per-token-per-row efficiency (lower is better):
- ARLE: 66 ms / 38 = **1.74 ms / row / token**
- vLLM: 20 ms / 14 = **1.43 ms / row / token**

vLLM is **18% faster at the per-row level** AND benefits from
fewer rows in the step (less attention work, simpler GEMM shapes).

This rules out "batch is too big" as a fix direction — even if ARLE
matched vLLM's 14-row decode batch, the per-row overhead would still
leave a gap. The kernel + step pipeline gap is real.

### Cross-reference with Phase 1 NVTX evidence

From `2026-05-07-m3.6-phase1-nsys-arle-s48-highconc.md`:
- ARLE `step_total` avg = 82 ms
- ARLE `step_decode_kernel_launch` avg = 1.0 ms (CPU side)
- ARLE `cuStreamSynchronize` 48.6% of CPU API time, avg 65.7 ms

If vLLM's ITL = 20 ms and ARLE's step_total = 82 ms (= ARLE's ITL
≈ 66 ms after distribution effects), the ratio is 4.1×. ARLE's
per-step work is dominated by the GPU-completion sync wait that
vLLM does NOT pay (vLLM uses CUDA graphs + in-graph sampling +
in-graph token-id-to-embedding lookup, so the entire decode step
completes before any CPU sync is needed).

This is the **F4 candidate**: eliminate per-step sync in ARLE's
decode loop by:
1. Capturing argmax + token-id-to-embedding into the same CUDA
   graph as the model forward.
2. Only syncing once every K decode steps (K = streaming chunk size,
   not 1).

Estimated upper bound on ARLE improvement from F4 alone: if all 65
ms of cuStreamSynchronize per call goes away and ARLE per-step
shrinks to ~20 ms, ARLE would reach ~38 rows × 20 ms-equivalent =
new step ITL ~ 53 ms (still 1.3× slower than vLLM per-row), so out
tok/s would jump from 462 to ~700+. That is the "world-first
parity" target the project is chasing.

## Problems

1. **Phase 2 nsys trace not captured.**
   - Attempt 1: `nsys profile --delay 60 --duration 60 bash scripts/vllm_serve_control.sh`
     vLLM under nsys took 125+ s to be ready; capture window
     (60-120 s) ended before bench started.
   - Attempt 2: `--delay 30 --duration 60`. vLLM under nsys still
     took >60 s (instrumentation overhead 3-5× slows Python+CUDA
     imports). Capture window missed bench again.
   - Attempt 3: `nsys start --session-new=...` after vLLM was up
     normally — `nsys start` doesn't accept `--trace=...` (different
     syntax than `nsys profile`); session ended without recording
     anything.
   - **Decision**: ship Phase 2 as bench-only this tick. Trace
     capture deferred to next tick using the corrected pattern
     (spawn vLLM under `nsys profile --delay 90 --duration 60` —
     vLLM's normal startup is 25 s but under nsys is 65-90 s).
2. vLLM under nsys is **substantially slower at startup** than
   under direct invocation (25 s direct vs >60 s under nsys). Plan
   for it; don't reuse ARLE's `--delay 25` parameter.

## Learnings

- vLLM normal-startup time on this hardware: **~25 s**. nsys-wrapped:
  **~65-90 s** (instrumentation cost on Python+CUDA import).
- `nsys start` is not the equivalent of `nsys profile` — the
  argument set differs and `--trace` is profile-only. To use the
  session API for an already-running process, the target must be
  spawned via `nsys launch --session-new=...` first; only then can
  `nsys start --session=...` activate capture against it.
- Per-row ITL is a more honest cross-system metric than overall
  ITL — it normalizes for batch shape. ARLE 1.74 ms/row vs vLLM
  1.43 ms/row tells the same story even when batches differ 38 vs 14.

## Bench Status

- vLLM bench: 646.8 out tok/s, 20.0 ms ITL — within 5% of codex's
  prior baseline.
- Trace artifact: **NONE this run**. Bench artifact:
  `bench-output/2026-05-07-m3.6-vllm-s14-nsys/benchmarks.json` (no
  sha256 since it's not the trace; small JSON, gitignored as part
  of bench-output).
- ARLE Phase 1 trace remains:
  `bench-output/2026-05-07-m3.6-arle-s48-nsys/arle-s48.nsys-rep`
  sha256 `d0711d7a0333cdbc61c1f32c608aa3b9113fd443c389794a78b6faf73b5dfb4e`.

## Next actions (decided by Phase 1 + Phase 2 bench evidence)

The ARLE trace + vLLM bench are sufficient to make the next-step
decision without a vLLM trace:

1. **F4 candidate scope-out**: investigate whether ARLE's decode
   step can be captured as a single CUDA graph that includes argmax
   + token-id-to-embedding lookup. If yes, the per-step
   `cuStreamSynchronize` goes away. This is THE concrete code
   change the trace evidence licenses.
2. **Phase 2 trace re-attempt next tick** with corrected
   `nsys profile --delay 90 --duration 60` parameters. Useful for
   confirming vLLM's per-step structure (graph replay rate,
   sampling kernel co-residence) but NOT blocking the F4 work.
3. **Phase 4 ncu deep-dive on `decode_attention_varlen_quantized_partial_kernel`**
   becomes lower priority — even if the kernel is matched to vLLM's
   speed, the per-step sync cost from F1 trace findings is the
   bigger 4.1× ratio. Fix sync first, then revisit kernel.

## Rule

- vLLM benchmark numbers reproduce within 5% across runs at this
  shape on this hardware. Future control runs only need a single
  point unless results diverge — saves GPU time.
- nsys instrumentation cost on Python+CUDA workloads is **3-5×**
  the normal startup time. Always set `--delay` to at least
  `2× normal_startup` to avoid capturing only warmup.
- Per-row ITL (= ITL / decode_batch_avg) is the honest cross-system
  benchmark. Quote it alongside aggregate ITL in future wins.
