# P1.2 Graph Capture Hoist — Hybrid W4 Decode Scratch, CUDA, 2026-05-09

## Goal

- **License / kill gate:** hoist Marlin W4/W4A8 decode scratch out of per-call linear dispatch so hybrid W4 decode can participate in CUDA Graph capture while preserving the P1.0 W4A8-prefill TTFT win.

## Hypothesis

- P1.0 kept hybrid prefill on W4A8 but left decode eager, producing a +11.5% ITL regression versus the W4A16 Marlin baseline.
- Preallocating decode-lifetime Marlin scratch should make the decode body graph-capturable and bring ITL back to W4A16 parity without losing the hybrid TTFT gain.

## Command

Server:

```bash
INFER_HYBRID_W4A8_PREFILL=1 \
CUDA_HOME=/opt/cuda \
NVCC_CCBIN=/usr/bin/g++-14 \
INFER_TILELANG_PYTHON=/home/ckl/projects/arle/.venv/bin/python \
TORCH_CUDA_ARCH_LIST=8.9 \
./target/release/infer \
  --model-path infer/models/Qwen3-4B-W4-hybrid-zpfix \
  --port 8000 --num-slots 8 --max-seq-len 5120
```

Benchmark:

```bash
PATH=/home/ckl/projects/arle/.venv/bin:$PATH \
scripts/bench_guidellm.sh p1.2-graph-hoist-hybrid-c4 \
  --model Qwen3-4B-W4-hybrid-zpfix \
  --processor infer/models/Qwen3-4B \
  --concurrencies 4 --max-seconds 120 --warmup 10 \
  --data 'prompt_tokens=4096,prompt_tokens_stdev=1,prompt_tokens_min=4096,prompt_tokens_max=4096,output_tokens=256,output_tokens_stdev=1,output_tokens_min=256,output_tokens_max=256'
```

## Environment

- **Backend:** CUDA
- **Model:** Qwen3-4B hybrid W4A16/W4A8 zero-point-fixed checkpoint
- **Hardware:** NVIDIA GeForce RTX 4070 Ti SUPER, 16 GiB
- **CUDA / driver:** CUDA 13.2.78, driver 595.71.05
- **Commit:** measured on `2e21da1` plus the P1.2 graph-hoist diff
- **Feature set:** `--features cuda`
- **Non-default flags / env vars:** `INFER_HYBRID_W4A8_PREFILL=1`, `NVCC_CCBIN=/usr/bin/g++-14`, `INFER_TILELANG_PYTHON=/home/ckl/projects/arle/.venv/bin/python`, `TORCH_CUDA_ARCH_LIST=8.9`
- **Server graph evidence:** startup captured CUDA Graphs for B=1..8 with the hybrid W4 checkpoint after Marlin scratch hoist.

## Implementation

- Added decode-lifetime `MarlinDecodeScratch` covering the W4A16 Marlin and W4A8 Marlin temporary buffers that previously allocated inside each linear dispatch.
- Threaded optional scratch through Qwen3 single-token and batched decode contexts only for Marlin-family models, so BF16 models do not pay the extra scratch memory.
- Kept the hot batched decode scratch on `BatchDecodeBuffers`; single-token `DecodeBuffers` allocates scratch lazily so `num_slots` does not multiply the Marlin scratch footprint at startup.
- Passed scratch through `CudaOpsBackend` with lifetime-bound `RefCell` access instead of an unsafe raw pointer.
- Routed decode-phase linear dispatch through scratch-aware Marlin helpers while preserving existing eager free functions and callsites.
- Removed the W4A8 CUDA Graph exclusion from `supports_cuda_graph_decode()`; LoRA remains excluded.

## Results — GuideLLM 4k/c=4

| run | checkpoint | gate | TTFT p50 | TTFT p99 | ITL p50 | ITL p99 | E2E mean | out tok/s | req/s |
|---|---|---|---:|---:|---:|---:|---:|---:|---:|
| W4A16 baseline | `Qwen3-4B-W4A16-sym-g128-marlin` | n/a | 2384.3 ms | 2571.5 ms | 11.76 ms | 11.91 ms | 5.46 s | 191.49 | 0.727 |
| P1.0 hybrid fixed | `Qwen3-4B-W4-hybrid-zpfix` | `INFER_HYBRID_W4A8_PREFILL=1` | 1632.1 ms | 1810.8 ms | 13.11 ms | 13.29 ms | 5.05 s | 205.64 | 0.800 |
| P1.2 graph hoist | `Qwen3-4B-W4-hybrid-zpfix` | `INFER_HYBRID_W4A8_PREFILL=1` | 1638.8 ms | 1802.8 ms | 11.41 ms | 11.58 ms | 4.62 s | 224.95 | 0.836 |

## Delta vs W4A16 Baseline

| metric | W4A16 baseline | P1.2 graph hoist | delta |
|---|---:|---:|---:|
| TTFT p50 | 2384.3 ms | 1638.8 ms | -31.3% |
| TTFT p99 | 2571.5 ms | 1802.8 ms | -29.9% |
| ITL p50 | 11.76 ms | 11.41 ms | -3.0% |
| ITL p99 | 11.91 ms | 11.58 ms | -2.8% |
| out tok/s | 191.49 | 224.95 | +17.5% |
| E2E mean | 5.46 s | 4.62 s | -15.4% |

## Delta vs P1.0 Hybrid Fixed

| metric | P1.0 hybrid fixed | P1.2 graph hoist | delta |
|---|---:|---:|---:|
| TTFT p50 | 1632.1 ms | 1638.8 ms | +0.4% |
| TTFT p99 | 1810.8 ms | 1802.8 ms | -0.4% |
| ITL p50 | 13.11 ms | 11.41 ms | -13.0% |
| ITL p99 | 13.29 ms | 11.58 ms | -12.9% |
| out tok/s | 205.64 | 224.95 | +9.4% |
| E2E mean | 5.05 s | 4.62 s | -8.5% |

## Service Metrics

| metric | value |
|---|---:|
| peak active | 4 |
| peak waiting | 0 |
| peak running_batch | 4 |
| peak prefill_queue | 3 |
| plan labels | `idle=12069`, `decode=6637`, `prefill=94`, `split=0`, `mixed=0` |
| peak kv_util | 84.4% |
| prefix hit rate | 0.0% |
| completed input tokens | 393312 |
| incomplete input tokens | 0 |
| completed output tokens | 24576 |
| incomplete output tokens | 0 |

## License-Or-Kill

| criterion | result | verdict |
|---|---|---|
| Preserve P1.0 TTFT win | +0.4% TTFT p50 vs P1.0, -31.3% vs W4A16 baseline | pass |
| Remove P1.0 ITL regression | 11.41 ms p50, -3.0% vs W4A16 baseline | pass |
| Whole-request throughput | +17.5% output tok/s vs W4A16, +9.4% vs P1.0 | pass |
| Production default flip | Still blocked by W4A8-only accuracy gate | keep env-gated |

P1.2 licenses the graph-hoist implementation. It does not license a default-on W4A8-only path because W4A8-vs-BF16 token parity remains a separate known blocker.

## Verification

- `cargo fmt --all --check` PASS
- `git diff --check` PASS
- `cargo check --release -p infer --features cuda` PASS
- `cargo clippy --release -p infer --features cuda --lib -- -D warnings` PASS
- `cargo test --release -p infer --features cuda weight_loader::tests::load_hybrid_w4_marlin_decode_accepts_preallocated_marlin_scratch -- --nocapture` PASS
- `cargo test --release -p infer --features cuda --test e2e -- --test-threads=1` PASS before the final review cleanup; post-cleanup rerun attempts were blocked at model load by concurrent Claude baseline servers holding ~12.5 GiB on the same GPU.
- `cargo check -p infer --no-default-features --features cuda,no-cuda` PASS
- `cargo check --release -p infer --no-default-features --features no-cuda` PASS
- `cargo check -p infer --no-default-features --features metal,no-cuda` PASS
- `cargo test --release -p infer --features cuda --test greedy_consistency test_greedy_w4a8_marlin_optional -- --test-threads=1 --nocapture` PASS with graph on
- `INFER_TEST_CUDA_GRAPH=0 cargo test --release -p infer --features cuda --test greedy_consistency test_greedy_w4a8_marlin_optional -- --test-threads=1 --nocapture` PASS with graph off
- `codex review --uncommitted -c sandbox.timeouts.exec_seconds=900` PASS, 0 actionable findings after the per-slot scratch and backend lifetime fixes
- Full `greedy_consistency` still fails `test_w4a8_vs_bf16_token_diff`; that is the pre-existing W4A8 accuracy gate, not a graph-hoist regression.

## Problems

- The benchmark was captured before the final review cleanup that replaced the raw scratch pointer with a lifetime-bound `RefCell` and made single-token scratch lazy. The cleanup reduces risk and memory footprint; it does not change the measured batched decode dispatch path.
- Fresh post-cleanup e2e reruns were attempted while Claude benchmark jobs were active. They failed before weights loaded with `CUDA_ERROR_OUT_OF_MEMORY` and `pre_model_load free=1.76-1.90 GB`, matching the external server footprint rather than a P1.2 assertion failure.
- Full W4A8-vs-BF16 token parity remains blocked. Hybrid production should stay env-gated until the W4A8 accuracy issue is fixed.

## Learnings

- The P1.0 ITL penalty was graph-capture overhead, not an inherent hybrid-W4 decode cost. Once Marlin temporaries are allocated at decode-context lifetime, hybrid W4 achieves better-than-W4A16 ITL while keeping the W4A8 prefill TTFT win.
- Scratch memory must be model-gated. Allocating Marlin scratch for BF16 models would silently inflate the normal serving footprint without benefit.
- Decode-state scratch should be allocated at the same lifetime as the hot execution plan, not at every slot, otherwise a graph-capture fix can quietly become a capacity regression.

## Artefacts

- Raw: `bench-output/2026-05-09-p1.2-graph-hoist-hybrid-c4/benchmarks.json`
- CSV: `bench-output/2026-05-09-p1.2-graph-hoist-hybrid-c4/benchmarks.csv`
- HTML: `bench-output/2026-05-09-p1.2-graph-hoist-hybrid-c4/benchmarks.html`
- Command: `bench-output/2026-05-09-p1.2-graph-hoist-hybrid-c4/command.txt`
- Service trace: `bench-output/2026-05-09-p1.2-graph-hoist-hybrid-c4/service_stats_trace_summary.md`
