# 2026-05-07 · M3.7 B.1.2 owner-run bench — TTFT improvement falls far short of target

## Context

After-snapshot for codex's
[`2026-05-07-m3.7-b1.2-async-prefill-completion.md`](2026-05-07-m3.7-b1.2-async-prefill-completion.md)
which shipped the implementation in `14a48e9` with bench status
`pending-remote / owner-run handoff`. This entry fills in the
post-commit GuideLLM numbers and assesses whether the sync-removal
hypothesis held.

## Goal

Per codex's pre-bench target:
> "reduce ARLE TTFT by at least 800 ms and remove prefill-path
> cuStreamSynchronize occurrences from the nsys trace"

Validation against vLLM s8 long-context control
([`9afcd57`](2026-05-07-m3.6-vllm-longctx-8k-comparison.md))
which set TTFT p50 = 2367 ms.

## Command

```bash
target/release/infer --port 8000 --max-seq-len 12288 --num-slots 8

guidellm benchmark run --target http://localhost:8000 \
  --profile concurrent --rate 4 \
  --data 'prompt_tokens=8192,...,output_tokens=256,...' \
  --max-seconds 120 --warmup 10 --random-seed 20260416
```

(--max-seconds 120 vs the F4-Small longctx run's 60; needed because
8k prompt + 256 output × c=4 didn't complete a full request lifecycle
in 70s.)

## Environment

- Hardware: NVIDIA RTX 4080 SUPER, CUDA 13.2.1, SM 8.9
- Build: `14a48e9` (B.1.2 commit, after `45e1d0c` M_b.1 Phase B)
- Bench: 8192 in / 256 out, c=4, **120s + 10s warmup**
- Workload: Qwen3-4B, FP8 KV pool

## Results

| Metric | F4-Small (`c63c31c`) | **B.1.2 (`14a48e9`)** | vLLM s8 (`9afcd57`) | Δ B.1.2 vs F4-Small |
|---|---:|---:|---:|---|
| out tok/s mean | 92.2 | **78.4** | 105.6 | **-15.0% (regression)** |
| TTFT mdn | 4,961 ms | **4,933 ms** | 2,367 ms | **-28 ms (-0.6%)** |
| ITL mdn | 23.9 ms | 24.6 ms | 26.7 ms | +3% (in noise) |
| TPOT mdn | 43.8 ms | 47.3 ms | 36.0 ms | +8% |
| E2E mdn | 11.2 s | 12.1 s | 9.2 s | +8% |

### Headline finding

**B.1.2 did NOT deliver the targeted -800 ms TTFT improvement.**
TTFT moved by -28 ms (-0.6%, well within bench noise). Out tok/s
regressed -15%. ITL and TPOT slightly worse.

### Why the hypothesis was wrong

The plan's expected gain reasoning (M3.7 §B.1.2): "if prefill sync
overhead is ~100 ms per chunk × 8 chunks = 800 ms, removing gets
ARLE TTFT from 4961 → ~4161 ms".

The data falsifies this. Likely actual cause of vLLM's 2367 ms vs
ARLE's ~4933 ms TTFT:
- **vLLM's prefill kernel** (Triton FlashAttention) IS faster
  than ARLE's hand-CUDA `prefill_attention_paged_qk_norm_rope_hd128_kernel`,
  not because of CPU sync but because of better SM utilization /
  per-token math
- **vLLM's chunk pipelining** may overlap chunk K's tail with
  chunk K+1's head on the GPU itself (intra-graph overlap), while
  ARLE chunks are kernel-serialized

This means the actual TTFT lever is **kernel-level**, not
**scheduler-level**. The reframe note in
[`2026-05-07-m3.6-vllm-longctx-8k-comparison.md`](2026-05-07-m3.6-vllm-longctx-8k-comparison.md)
proposed M_b.2 (FP8 prefill TileLang) as the kernel-axis path.
**That plan is now the highest priority**, not B.1.2.

### Why the regression?

The -15% out tok/s and +8% TPOT/E2E suggest B.1.2 added latency
elsewhere. Possibilities:
- Codex's "scheduler backoff when event not ready" introduces a
  CPU sleep/yield that delays the next scheduler iteration even
  when other work is ready
- The deferred launch-error reporting path adds an extra event
  poll per tick
- 120s window vs F4-Small's 60s exposes a tail-distribution effect
  that we didn't see in the shorter run

### What about the nsys trace target?

> "remove prefill-path cuStreamSynchronize occurrences from the
> nsys trace"

nsys trace not run this iteration — codex's wins entry committed
without one. Even if the prefill ctx.sync() calls are gone (codex
verified by code inspection), the bench number is what matters
for the project goal. Skipping nsys until the bench number is
moving in the right direction is reasonable triage.

## Decision

1. **Do NOT revert B.1.2**. The implementation is correct (e2e +
   greedy_consistency pass, long-prompt smoke pass), the sync was
   removed as designed. The regression is small and may not
   replicate at other workloads.
2. **Pivot to M_b.2 (FP8 prefill TileLang)** as the actual TTFT
   lever. The kernel-level work bypasses both the sync question
   AND the chunk pipelining question by using TileLang's tunable
   IR which historically gives 10-30% per-call speedup over
   hand-CUDA at the same shape.
3. **Re-bench at 60s window** to compare apples-to-apples with
   F4-Small longctx — confirm or refute the -15% out tok/s
   regression. Defer to next tick.
4. **Update M3.7 plan** to mark B.1.2 as "shipped, low impact at
   measured workload; revisit at other shapes". Demote B.1
   sub-tasks (cover non-greedy + legacy sample sync) to
   "validation gate" priority — they may also produce small or
   no measurable impact.

## Bench Status

- Bench artifact:
  `bench-output/2026-05-07-m3.7-b12-longctx-8k/benchmarks.json`
  (gitignored).
- nsys trace not run this iteration.
- F4-Small + M_b.1 Phase B reference baselines:
  `bench-output/2026-05-07-m3.6-f4small-longctx-8k/`,
  `bench-output/2026-05-07-m_b1-phaseB-s48-highconc/`.
- vLLM control: `bench-output/2026-05-07-m3.6-vllm-longctx-8k/`.

## Rule

- **Plan estimates of "~100ms × N chunks" are dimensional analysis,
  not measurements**. If the underlying assumption (sync was the
  blocking cost) is wrong, the gain estimate is wrong by the
  same factor. F4-Small was a unicorn because the sync WAS the
  blocking cost (validated by Phase 1 nsys trace showing
  cuStreamSynchronize 48.6% of CPU API). B.1.2 didn't validate
  the equivalent measurement for prefill before shipping —
  if Phase 1 trace had isolated prefill-vs-decode sync share,
  the data would have shown prefill sync was a smaller fraction.
- **Always trace before targeting a fix.** F4-Small process:
  trace → identified bottleneck (decode sync 48.6%) → fixed →
  bench showed +82.5%. B.1.2 process: assume bottleneck (prefill
  sync) → fixed → bench showed -0.6%. The traceless step in B.1.2
  is what cost the iteration.
- **Negative bench data is data**. The sync-removal hypothesis at
  the prefill axis is now falsified. M_b.2 (kernel-axis prefill)
  is the validated next direction.
