# M3.6 F4-Small — async greedy readback without per-step stream sync

## Context

M3.6 Phase 1/2 traces showed the high-concurrency CUDA gap was scheduler-side
serialization, not mixed prefill policy: `cuStreamSynchronize` dominated the
decode tick CPU API time while greedy sampling read back token IDs before the
next launch could proceed.

## What Worked

- Kept Qwen3 greedy argmax on GPU and staged sampled token IDs back into the
  next decode input buffer with D2D copies.
- Moved CPU token/logprob visibility to copy-stream D2H guarded by a CUDA
  event; scheduler polls readiness and defers emit bookkeeping instead of
  synchronizing the compute stream.
- Added slot-owner tracking for sampled rows so row remaps use D2D when a slot
  moves and H2D only for rows without a sampled-token owner.
- Added a device snapshot buffer for async readback. Directly reading
  `argmax_out` from the copy stream raced with the next tick reusing that
  buffer; snapshotting fixed the repeated-token failure caught by `e2e`.
- Added a max-token boundary guard: if a deferred sampled token will complete a
  request, the scheduler does not launch one speculative extra decode tick for
  that slot.
- Kept the steady decode hot path at one bulk D2D handoff: per-row D2D remap is
  only used when row order changes or new rows lack sampled-token owners.
- Query the async readback CUDA event with error propagation, so real CUDA
  failures surface instead of spinning forever as "not ready".
- While a greedy readback is deferred, spec-decode routing is downgraded to the
  normal decode path; spec verifier paths still consume CPU-visible tokens and
  must not run ahead of the delayed append.

## Verification

- `cargo check --release -p infer --features cuda`
- `cargo fmt --all --check`
- `cargo check --release -p infer --no-default-features --features metal,no-cuda`
- `cargo clippy --release -p infer --features cuda -- -D warnings`
- `cargo test --release -p infer --features cuda --test e2e -- --test-threads=1`
- `cargo test --release -p infer --features cuda --test greedy_consistency -- --test-threads=1`

## Bench Status

Pending user bench/trace:

- `scripts/bench_guidellm.sh m3.6-f4small-s48-highconc`
- nsys trace target: `cuStreamSynchronize <= 25%` CPU API share.

## Rule

Async readback must snapshot reusable GPU outputs into a stable device buffer
before copy-stream D2H. Event readiness only protects copy completion; it does
not protect the source buffer from a later compute-stream reuse.
