# GPU/CPU crossover — the device path nearly catches the CPU at 1.5B (RX 7600)

> **The crossover is real and measured.** At 135M the CPU beat the GPU **7.2×** on Q8 decode
> (the device path is launch-bound: ~330 fixed-cost ops/token dwarf the tiny per-op compute).
> At **Qwen2.5-1.5B** — ~11× the per-token compute — the GPU's larger GEMMs amortize that fixed
> launch tax almost completely: the CPU's lead collapses from **7.2× → 1.16×**. This is direct
> evidence for the device-path thesis: the GPU wins as the model grows, and one more size step
> (3B+) would likely flip it to a GPU win. Unblocked by the `q8_matmul` input-tiling fix
> (`8c74fd9`) that lifted the old inDim≤2048 cap.

## The crossover

`figures/vulkan-q8-crossover.svg` (figure not committed — see the table below for the same data)

Q8 decode tok/s, same forward pass, GPU (Vulkan, RX 7600) vs CPU (pure-Go legacy), both Q8_0:

| model | params/FFN | CPU Q8 decode | GPU Q8 decode | **CPU / GPU** |
|---|---|---:|---:|---:|
| SmolLM2-135M | 135M / ffn 1536 | 176.9 | 24.6 | **7.2×** |
| **Qwen2.5-1.5B** | 1.5B / ffn 8960 | 18.4 | 15.9 | **1.16×** |

The ratio falling 7.2× → 1.16× as per-token compute grows ~11× is the launch-bound signature
inverting: at 135M the fixed ~330-op/token dispatch tax dominates (CPU wins big); at 1.5B the
per-op GEMM is large enough that the same fixed tax is a small fraction (GPU nearly at parity).

## Artifacts (committed, self-describing)

| arm | decode tok/s | prefill | artifact |
|---|---:|---|---|
| GPU Q8 (vulkan) | 15.9 (62.9 ms/tok) | P16 7.9, P64 14.1 | `crossover-qwen2.5-1.5b-gpu-q8-20260619.json` |
| CPU Q8 (legacy) | 18.4 (54.3 ms/tok) | P16 153, P64 186, P256 194 | `crossover-qwen2.5-1.5b-cpu-q8-20260619.json` |

`-decode-steps 32 -decode-reps 2`, lean GGUF quantize-at-load (Q8_0), `FAK_WORKERS=8`. The GPU
artifact records `selected=vulkan`, `precision=Q8_0`, `tier=discrete:AMD Radeon RX 7600`.

## Honest caveats

- **Prefill still favors the CPU heavily** (153→194 vs 7.9→14.1 tok/s): the device prefill loops
  single tokens (HAL prefill isn't batched), so it runs at decode speed while the CPU batches its
  prefill GEMM. Batched device prefill is the next throughput lever (the same one flagged for
  the 135M lane); it does not affect the decode crossover above.
- **A transient VRAM-allocation panic** can appear at larger prefill shapes (the buffer pool's
  drain-and-retry usually absorbs it; a smaller `-prefill-sizes` avoids it). The decode number
  here is stable across reps. Robust large-shape allocation on the 1.5B is tracked follow-up.
- Correctness: the Q8 device GEMM is bit-close to the CPU Q8 reference at this FFN width
  (`TestVulkanQ8MatMulWideInput`, in=8960, cosine ≥ 0.9999) and the HAL forward gate is
  argmax-exact; the decode tok/s is a faithful measurement of the same computation.

## Reproduce

```powershell
pwsh internal/compute/build_vulkan.ps1 lib   # rebuild the shim with the tiled q8_matmul
# GPU arm (smaller prefill avoids the transient large-shape alloc):
$g="$env:USERPROFILE\.cache\fak-models\gguf\Qwen2.5-1.5B-Instruct.Q8_0.gguf"
go run -tags vulkan ./cmd/modelbench -gguf $g -lean -backend vulkan -quant `
    -prefill-sizes "16,64" -decode-steps 32 -decode-reps 2 -out experiments/gpu/crossover-qwen2.5-1.5b-gpu-q8-20260619.json
```
