# M3.6 — close the high-conc decode-throughput gap (Nsight-led)

> Discovered 2026-05-07 from codex's evidence-only slot-width sweep
> ([`m6-cuda-vllm-gap-followups.md`](m6-cuda-vllm-gap-followups.md)
> §"Phase 1 verify"). Equal-width vLLM control rejected the
> "admission width alone explains the gap" hypothesis.
>
> **Methodology revision (2026-05-07, user direction)**: lead with
> Nsight Systems profiling, not bench-driven F1/F2/F3 sweeps. The
> industry-standard root-cause path for "scheduler-vs-kernel" gaps in
> LLM inference is `nsys` first, `ncu` second (see §References).
> Bench-and-fix without trace evidence is anti-pattern.

## P0 finding (evidence chain)

Codex's Phase-1 sweep at slots=48 exposed a gap that bench data
alone cannot localize:

| Metric | s48 sweep value |
|---|---:|
| `active` peak (slots holding requests) | **47** |
| `running_batch` peak (slots in scheduler running queue) | **38** |
| `decode` peak (slots advancing one token in launched step) | **38** |
| `kv_util` peak | 95.4% |
| out tok/s | 442 |
| vLLM s14 equal-width control out tok/s | **615** (+39%) |

The gap could plausibly live in any of three layers, and no
service-trace metric distinguishes between them:

| Layer | Hypothesis | Service-metric proxy | Distinguishable from metrics alone? |
|---|---|---|---|
| Scheduler CPU overhead | Mixed-plan retract loop + admission burns Python-equivalent CPU ms per tick | `Idle` plan share, `step_us` distribution | Partial — `Idle` share masks GPU-bound stalls |
| Scheduler policy (page accounting) | `step_mixed_launch` `retract_decode_to_fit(extra_pages>0)` causes decode churn | running_batch=38 vs active=47 gap | Suggestive but not conclusive |
| Decode kernel efficiency | TileLang BF16 attention or GEMV slower than vLLM's Triton at high effective batch | TPOT, ITL p50/p99 | No — same metric value can come from CPU bubble OR slow kernel |

`nsys` resolves all three in one trace by exposing:
- CPU thread time per scheduler function (catches WukLab-style
  "scheduling overhead dominates" pattern)
- GPU bubble periods between kernel launches (catches CPU-bound
  scheduling stalls)
- Per-kernel duration grouped by name (catches per-kernel inefficiency)
- CUDA Graph replay validation (catches static-shape regressions)

## Plan

### Phase 0 — install + dry-run

```bash
sudo pacman -S nsight-systems nsight-compute   # cachyos packages,
                                               # ~600 MB combined
nsys --version && ncu --version                # confirm in PATH
```

Tools needed but not currently on this box (verified
2026-05-07: `which nsys ncu` returns not-found; CUDA toolkit 13.2.1
ships only `compute-sanitizer`, not the GUI profilers). `sudo` is
required because both packages install under `/opt/`. CUPTI is
already present at `/opt/cuda/extras/CUPTI`.

Dry-run smoke test on a 5-token completion to confirm tooling works
end-to-end before the real high-conc capture. The smoke `.nsys-rep`
should show `cuda_kernel_*` events and at least one NVTX range — if
either is missing, the build is missing the linker flag for
`-lnvToolsExt` (NVTX), or `nsys` did not pick up CUPTI.

**Why this phase is now urgent (not just preferred)**: codex's
Phase-1 Step 3 evidence (above, F1 falsified) means scheduler-policy
levers have been exhausted as a bench-driven candidate set. The
next falsification point is decode kernel + CUDA graph + launcher
overhead, which only Nsight resolves.

### Phase 1 — `nsys` capture at the s48 high-conc shape

Use the existing canonical wrapper (per
[`docs/bench-and-trace-spec.md`](../bench-and-trace-spec.md) §"Trace
inputs" L56):

```bash
scripts/profile_nsys_guidellm.sh m3.6-arle-s48-highconc \
  --target http://localhost:8000 \
  --num-slots 48 \
  --max-seq-len 2048
```

The wrapper already produces `.nsys-rep` + `.sqlite` + stats summary
under `bench-output/<date>-<label>/`, gitignored, with the canonical
flag set (`--trace=cuda,nvtx,osrt`, `--cuda-graph-trace=node`).
Match codex's Phase-1 high-conc shape from
[`m6-highconc-phase1-arle-2048-s48`](m6-cuda-vllm-gap-followups.md)
(1024 in / 256 out, c=64, 45 s, 5 s warmup) on the guidellm side.

Required NVTX ranges (landed in `998bfee`; this is the only ARLE runtime
code change in Phase 1):
- `step_total` — full scheduler tick
- `step_admission` — admission (`assign_slots` in current CUDA runtime)
- `step_plan` — step plan selection (`plan_step`)
- `step_mixed_launch_retract` — the retract loop in `decode.rs:387-413`
- `step_decode_kernel_launch` — `launch_decode_batch_from_tokens`
- `step_mixed_kernel_launch` — direct `forward_mixed_batch` launch
- `step_prefill_kernel_launch` — `step_prefill_batch`
- `step_dispatch_emits` — `dispatch_decode_emits`

These are cheap (~ns per range) and let `nsys stats` produce a clean
per-phase breakdown. Implementation note: CUDA 13 ships NVTX v3 as a
header-only C ABI, so ARLE uses a tiny `infer/build.rs`-compiled C shim plus
`infer/src/scheduler/cuda/nvtx_scopes.rs` RAII guards. The shim is gated to
`cuda && !no-cuda`; Metal/no-cuda builds do not compile or link NVTX.

### Phase 2 — vLLM s14 equal-width control trace

Same `nsys` flags, vLLM with the canonical command from vLLM's
profiling docs:

```bash
VLLM_WORKER_MULTIPROC_METHOD=spawn \
nsys profile \
  --trace-fork-before-exec=true \
  --cuda-graph-trace=node \
  --output=/tmp/vllm-s14-highconc \
  vllm serve <args from codex's m6-highconc-phase1-vllm-s14>
```

This is the comparison baseline: where does vLLM spend its time at
the same effective decode width?

### Phase 3 — analyze, then fix

Open both traces in Nsight Systems UI. Categorise:

| Trace finding | Implied fix tier |
|---|---|
| ARLE has GPU bubbles > 50 µs between decode kernel launches that vLLM does not | F1 (scheduler CPU overhead — instrument what blocks the launch) |
| Mixed-step retract NVTX range is wide and recurrent | F2 (page accounting policy — see candidate fixes below) |
| ARLE decode kernel duration > vLLM decode kernel duration at same batch | M3.5 / M_b kernel-level work (separate plan; do not confuse with M3.6) |
| ARLE `step_admission` >> vLLM equivalent | scheduler structure (loop collapse — already covered by M3.5) |

Only after the trace identifies the layer, pick the matching fix
candidate from §Candidate fixes below. Do not stack fixes
speculatively.

### Phase 4 — ncu kernel deep-dive (only if Phase 3 points to kernel layer)

If and only if Phase 3 trace shows decode-kernel duration is the
dominant cost, profile the hot kernel with `ncu --set full` on a
single decode launch. Do not run `ncu` on the full server — it
serializes every kernel for full counter capture and is unusable
for end-to-end runs. Hand off to M_b / M3.5 with the ncu report.

## Candidate fixes (parked until trace evidence selects one)

Each is a hypothesis that a specific trace finding would license. None
should land before its evidence does.

### F1 — request-count cap on concurrent prefill ⚠ **FALSIFIED 2026-05-07**

**Status: rejected by evidence** — codex's Phase-1 Step 3 sweep
(`c3b0449`, see [`m6-cuda-vllm-gap-followups.md`](m6-cuda-vllm-gap-followups.md)
§"Phase 1 step 3: prefill admission controls") tested
`--prefill-max-requests=1|2|4` at the s48 high-conc shape:

| Config | out tok/s | Δ vs s48 default (402) | prefill_rows share |
|---|---:|---:|---:|
| default (unbounded) | 402 | — | 9.2% |
| `prefill_max_requests=1` | 374 | **-7.0%** | 2.5% |
| `prefill_max_requests=2` | 408 | +1.5% (noise) | 2.6% |
| `prefill_max_requests=4` | 408 | +1.5% (noise) | 2.6% |

The throttle mechanically works (prefill share collapses from 9.2%
to 2.5%) but throughput is unchanged or regresses, so prefill
admission rate is NOT the bottleneck. **Do not implement F1 as a
static cap.** A future scheduler policy change in this region must
be designed as decode-priority / mixed-step collapse with trace
evidence behind it, not as a request-count default flip. Originally
proposed formula `max(1, min(2, num_slots / 16))` is preserved here
only as a record of the falsified hypothesis.

- **Trace evidence required**: NVTX `step_admission` shows multiple
  prefill requests admitted per tick at high kv_util, AND
  `step_mixed_launch_retract` is wide.
- **Mechanism**: `PrefillBudget::from_scheduler_for_decode_slots`
  already wires `prefill_max_requests` into `StepTokenBudget`
  (`execution.rs:144-153`); just changing the default is sufficient.
- **Note**: codex's currently-running sweep at s=48 with
  `--prefill-max-requests=1|2|4` IS this validation, but as a
  bench-only check. Once the trace lands, codex's sweep results map
  cleanly to the fix decision.

### F2 — bias `select_mixed_launch_prefill_candidates` against decode-retraction

When `kv_util > 0.85`, refuse to admit a prefill candidate whose
`additional_pages_needed_for_slot` would force retraction.

- **Trace evidence required**: `step_mixed_launch_retract` NVTX
  range fires every tick at high kv_util; tracing GPU bubble inside
  the retract loop.
- **Mechanism**: in `step_mixed_launch` (`decode.rs:374`) before
  the retract loop, filter out candidates whose `extra_pages` would
  push pool past 100% — only when `kv_util > threshold`.
- **Risk**: starve prefill if kv_util stays above threshold
  permanently. Mitigation: max prefill defer count per slot
  (e.g. 4 ticks); after that fall back to old retract behavior.

### F3 — split prefill page budget from decode pool entirely

vLLM-style: reserve a fixed fraction (e.g. 5-10%) of the KV pool as
"prefill scratch" that decode never sees. Mixed steps allocate
prefill into the scratch pool; decode never gives up pages.

- **Trace evidence required**: even with F1+F2, GPU bubble around
  page-allocation paths is dominant; `additional_pages_needed_for_slot`
  shows up in NVTX as a hot range across many ticks.
- **Mechanism**: `PageBudget::from_scheduler` becomes two budgets:
  `decode_budget` (95%) and `prefill_scratch_budget` (5%).
- **Cost**: ~150 LOC; touches `PageBudget`, `PrefillBudget`,
  page-allocator hooks. New CLI: `--prefill-scratch-fraction 0.05`.
- **Risk**: 5% pool reserved means slightly fewer concurrent decode
  slots fit. At s48, ~10 tokens per slot of decode capacity loss.

## Test plan

For each fix that does land:

| Fix | Existing tests must pass | New test |
|---|---|---|
| F1 | `cargo test --release -p infer --features cuda --test e2e`, `--features cuda --test scheduler_kv_pressure_drains` (the `--features cuda` flag is required — both tests are `#![cfg(feature = "cuda")]` and silently report 0 tests run without it) | `infer/tests/scheduler_prefill_max_requests_default.rs` (also `#![cfg(feature = "cuda")]`) |
| F2 | same | `infer/tests/scheduler_decode_protected_kv_util.rs` |
| F3 | same | `infer/tests/scheduler_prefill_scratch_pool.rs` |

## Bench gate

After each landed fix:

1. `scripts/bench_guidellm.sh m3.6-<f1|f2|f3>-<label>` — high-conc
   shape (1024 in / 256 out, c=64), `--num-slots 48` and 64.
2. Compare against `m6-highconc-phase1-arle-2048-s48` (442 out tok/s)
   and `m6-highconc-phase1-vllm-s14` (615 out tok/s).
3. Wins entry MUST cite both baselines with Δ% rows AND link the
   nsys trace artifact `bench-output/<date>-<label>/*.nsys-rep` path
   with sha256 (per `bench-and-trace-spec.md` §§8-9 — raw captures
   stay gitignored, sha256 in the entry is the durable reference).
4. Regression on TTFT p99 > +20%, e2e correctness, greedy_consistency,
   or M4.5 drain → revert; trace must explain the regression.

## Acceptance (whole milestone)

- ARLE high-conc out tok/s at s48 ≥ 0.95 × vLLM s14 throughput
  (≥ 584 out tok/s).
- TTFT p99 at s48 ≤ baseline + 20%.
- All in-tree correctness tests still green.
- Three nsys captures **kept under `bench-output/<date>-<label>/`
  (gitignored) and cited by sha256 in the wins entry**, per
  [`docs/bench-and-trace-spec.md`](../bench-and-trace-spec.md)
  §§8-9 — never commit raw `.nsys-rep` files (hundreds of MB):
  ARLE before-fix, ARLE after-fix, and the vLLM s14 control.
  Annotated timeline PNGs (< 500 KB) MAY be committed alongside the
  wins entry.

## Out of scope

- Decode-kernel-level optimizations (TileLang attention, GEMV
  vectorization). M3.6 is scheduler policy + observability only.
  Kernel work routes through M3.5 / M_b after Phase 4 ncu report.
- Spec-decode interactions. `step_mixed_launch` runs only on the
  non-spec path; spec paths have their own `retract_decode_to_fit`
  callers (`spec_path.rs:95, 343`) and stay untouched.
- Multi-GPU / tensor-parallel scheduling.

## References

- vLLM official profiling guide:
  https://docs.vllm.ai/en/stable/contributing/profiling/ —
  canonical `nsys` command shape, `VLLM_WORKER_MULTIPROC_METHOD=spawn`
  requirement.
- WukLab MLSys "Can Scheduling Overhead Dominate LLM Inference
  Performance?": https://mlsys.wuklab.io/posts/scheduling_overhead/ —
  used Nsight Systems with injected timing to find vLLM scheduling
  overhead can exceed 50% of inference time; SGLang v0.4 zero-overhead
  scheduler keeps it under 2%.
- NVIDIA Nsight Compute Profiling Guide:
  https://docs.nvidia.com/nsight-compute/ProfilingGuide/index.html —
  per-kernel deep-dive use; serializes execution, do NOT use for
  end-to-end serving.
- TensorRT-LLM Performance Analysis:
  https://nvidia.github.io/TensorRT-LLM/performance/perf-analysis.html —
  built-in Nsight Systems hooks as the recommended starting point.
- Existing retract infrastructure: `decode.rs:12-150`
  (`retract_victim_score`, `retract_decode_to_fit`,
  `requeue_preempted_decode`).
- M4.5 narrow drain fix that established the retract callable
  surface on the regular decode path:
  [`M4.5-kv-preemption-on-regular-decode-path.md`](M4.5-kv-preemption-on-regular-decode-path.md).
- Discovery thread: codex's
  [`m6-cuda-vllm-gap-followups.md`](m6-cuda-vllm-gap-followups.md)
  §"Phase 1 verify" (commit `3a0189c`).
