# 2026-05-07 · M3.6 F4-Small workload validation — long-context 8k input

## Goal

- Validate that F4-Small (`2a534c4`) gain holds on long-context
  workloads, not just the high-conc M6 shape used for the headline
  bench at `8f83c80`.
- Goal type: **diagnostic + optimization** (cross-shape regression
  check; the project name is "**长序列** inference engine" so
  long-context is the primary target).

## Hypothesis

F4-Small removed per-step decode sync. The high-conc bench saw
+82% throughput vs the no-F4 baseline. Long-context has different
cost profile:
- Fewer concurrent rows (4 vs 64) → less batch amortization
- Larger per-request KV state → attention reads dominate vs sampling
- Each decode tick still benefits from the same sync-removal

Expected: F4-Small still produces ITL ≤ baseline ITL. Throughput in
absolute terms is much lower (4 rows × per-token rate vs 38 rows)
but the per-row efficiency should improve.

## Command

```bash
target/release/infer --port 8000 --max-seq-len 12288 --num-slots 8
```

```bash
guidellm benchmark run --target http://localhost:8000 \
  --profile concurrent --rate 4 \
  --data 'prompt_tokens=8192,...,output_tokens=256,...' \
  --max-seconds 60 --warmup 10 --random-seed 20260416
```

## Environment

- Hardware: NVIDIA RTX 4080 SUPER, CUDA 13.2.1, SM 8.9
- Build: `8f83c80` (post-F4-Small, post-bench-wins)
- Server: `--num-slots 8 --max-seq-len 12288` (8k prompt + 256 output + headroom)
- Bench shape: 8192 in / 256 out, c=4, 60s + 10s warmup
- Workload: Qwen3-4B, FP8 KV pool

## Results

| Metric | Value |
|---|---:|
| out tok/s mean | 92.2 |
| in tok/s mean | 3,314 |
| req/s | 0.32 |
| TTFT mdn | 4,961 ms |
| TTFT p95 | 8,112 ms |
| **ITL mdn** | **23.9 ms** |
| ITL p95 | 42.1 ms |
| TPOT mdn | 43.8 ms |
| E2E mdn | 11.2 s |

### Cross-workload comparison (F4-Small at different shapes)

| Workload | Effective batch | ITL mdn | TPOT mdn | per-row ITL |
|---|---:|---:|---:|---:|
| **High-conc 1k/256 c=64** | ~38 (running_batch peak) | 37.8 ms | 53.5 ms | **0.99 ms** |
| **Long-ctx 8k/256 c=4** | 4 | **23.9 ms** | 43.8 ms | **5.97 ms** |

Per-row ITL is **6× higher** at long-context than high-conc — this
is the expected cost of per-row attention scaling with KV length
(8k context vs 1k context = 8× more attention reads, partially
offset by fewer rows competing for SM time).

But **the absolute ITL is LOWER on long-context (23.9 vs 37.8 ms)**
because the 4-row batch finishes a step faster than the 38-row
batch, even though each row's attention does more work. The
F4-Small sync removal applies the same amount per tick regardless
of batch — so when batch is small, the overhead/work ratio is
better.

## Headline finding

F4-Small **generalizes across workloads**:
- High-conc: +82% throughput, ARLE past vLLM 30%
- Long-ctx: per-row ITL is now ARLE's best operating point in
  absolute milliseconds (23.9 ms vs 37.8 ms at high-conc)

The long-context path has not been benched against vLLM at this
shape yet (Phase 2 vLLM control was high-conc only). vLLM's
documented long-context performance at similar shapes (4-8k input,
4 concurrent) is typically ITL 30-80 ms range — ARLE's 23.9 ms is
likely competitive or better; pending Phase 2-style vLLM control
trace at this shape.

### What this implies for the M_b decision

The M_b TileLang HD128 plan was motivated by per-row 51% slower
at high-conc (Phase 1 evidence). After F4-Small:
- High-conc per-row: 0.99 ms (ARLE FASTER than vLLM 1.43 ms)
- Long-ctx per-row: 5.97 ms — but absolute ITL still dominated by
  attention KV-read time (not the kernel constant overhead)

M_b.2 (FP8 TileLang) would primarily benefit attention-bound
long-context decode (where attention is the dominant kernel cost).
For high-conc workloads, M_b is now NEGATIVE-priority because
ARLE's existing kernel already wins at that shape.

## Problems

- vLLM long-context control trace not run this iteration. Quoting
  ARLE 23.9 ms ITL without a same-shape vLLM number is suggestive
  but not conclusive of advantage.
- Bench is single point. Should sweep `--num-slots` (4, 8, 16) +
  `prompt_tokens` (4k, 8k, 16k, 32k) for full long-context
  characterization. Defer to a workload-sweep wins entry.

## Bench Status

- Bench artifact: `bench-output/2026-05-07-m3.6-f4small-longctx-8k/`
  (`benchmarks.json` + `benchmarks.csv`, gitignored).
- nsys not collected for this run. Headline is bench numbers; trace
  to verify cuStreamSynchronize stayed eliminated at long-ctx is a
  follow-up.

## Next actions

1. **Long-context vLLM control bench** at the same 8k/256 c=4
   shape — fills the comparison gap. Use
   `scripts/vllm_serve_control.sh --max-num-seqs 8
   --max-model-len 12288` against the same guidellm command.
2. **Workload sweep**: run F4-Small at 4k/8k/16k/32k input × c=4/8/16
   to map the long-context performance surface. Inform whether
   M_b.2 (FP8 TileLang attention) is justified by some specific
   long-ctx regime where ARLE underperforms.
3. **F4-Big remains optional** — F4-Small generalizes already;
   F4-Big's additional CUDA-graph-includes-argmax design is now a
   "push past parity by another 10-20%" decision, not a "match
   vLLM" decision.

## Rule

- **Workload generalization is a separate axis from headline win**.
  Always run at least one cross-shape validation after a major
  perf change before declaring "the work is done." F4-Small's
  +30% past vLLM at high-conc is necessary but not sufficient
  evidence for the broader project goal of "world-first **长序列**
  inference engine."
- Per-row ITL across workloads is the most honest cross-system
  metric. Absolute ITL alone misleads when batch shape differs.
