# Vulkan Q8 throughput — SmolLM2-135M on the AMD RX 7600 (fak in-kernel)

> **First committed Q8-on-GPU throughput numbers from the `modelbench` harness.** Until
> 2026-06-19 `modelbench -backend vulkan -quant` hard-refused ("compute HAL sessions are
> f32-only today"), even though the Q8 weight-upload + device-GEMM path was fully wired in
> `internal/model/hal.go` and correctness-gated. Commit `84c2e6c` lifts the refusal for any
> backend that advertises `Caps().UploadDtype` (the Vulkan backend when `haveQ8`), unblocking
> the measurement below. The CUDA path on the RTX 4070 ([`README.md`](README.md)) is the
> sibling GPU lane; this is its native-Windows Vulkan counterpart on a different box.

## What this measures

fak's own pure-Go forward pass (`internal/model`) dispatching every matmul to the AMD GPU
through the `compute.Backend` Vulkan path (`-tags vulkan`, `internal/compute/vulkan*` →
`libfakvulkan.a`), Q8_0 weights uploaded once and resident on-device. **No `llama.cpp` /
`libggml` anywhere in this path** — it is fak's engine end to end. Three arms over the *same*
Q8 forward pass triangulate the result:

- **gpu-q8** — `-backend vulkan -quant`: the path this campaign exists to measure.
- **gpu-f32** — `-backend vulkan`: the previous Vulkan regime (f32 weight traffic).
- **cpu-q8** — `-backend legacy -quant`: the CPU Q8 reference on the same box.

## Correctness — the gate, not a vibe

The Q8 GPU forward pass is bit-faithful to the CPU Q8 reference on the real RX 7600
(`go test -tags vulkan ./internal/model -run HALVulkan`):

```
TestHALVulkanForwardMatchesNative      — real-model decode argmax-exact over 10 tokens, prefill cosine=1.00000000
TestHALVulkanQ8ForwardMatchesComputeQ8 — Q8 model path prefill cosine=1.00000000 step cosine=1.00000000
```

All 22 op kernels (incl. `Q8MatMul2/3`) pass the Approx gate. The throughput win is reuse +
narrower weight traffic, not a numerics shortcut.

## Figures

Dependency-free SVG, regenerated from the committed JSONs by
`make_vulkan_q8_charts.py` (generator not committed — run `python make_vulkan_q8_charts.py`) so
the figures cannot drift from the data.

Decode tok/s across the three arms — `figures/vulkan-q8-decode.svg` (figure not committed).

Prefill ladder, GPU Q8 vs GPU f32 — `figures/vulkan-q8-prefill.svg` (figure not committed).

## Throughput (from the committed JSON artifacts)

All cells: SmolLM2-135M, 64 decode steps / 3 reps (median per-token), prefill ladder 3 reps
(median). `FAK_WORKERS=8`.

| arm | decode tok/s | prefill P=16 | P=64 | P=256 | P=512 | artifact |
|---|---:|---:|---:|---:|---:|---|
| **gpu-q8**  | **24.6** | 15.6 | 22.0 | 24.0 | 24.8 | `q8gpu-smollm2-135m-gpu-q8-20260619.json` |
| gpu-f32 | 16.5 | 12.6 | 17.3 | 18.9 | 18.7 | `q8gpu-smollm2-135m-gpu-f32-20260619.json` |
| cpu-q8  | 176.9 | 969.2 | 1618.6 | 1826.4 | 1519.2 | `q8gpu-smollm2-135m-cpu-q8-20260619.json` |

### Two honest findings

**1. Q8 weight-narrowing buys a real ~1.5× on the GPU.** Decode **24.6 vs 16.5 tok/s
(1.49×)** and prefill ~25–30% faster at every length, gpu-q8 over gpu-f32 — same forward
pass, same device, only the weight dtype changes. The RX 7600 decode path is memory-bound, so
cutting weight traffic ~4× (Q8 codes + per-block scales vs f32) directly raises throughput.
This is the payoff of the wired Q8 upload path, and the reason to keep `-quant` on for the
device backend.

**2. On 135M the CPU wins outright — the device path is launch-bound, not compute-bound.**
cpu-q8 decode **176.9 tok/s** is **7.2× the gpu-q8 24.6**, and cpu-q8 prefill (969–1826 tok/s,
batched GEMM) is **40–75× the GPU's single-token-looped prefill**. The GPU runs ~330 device
ops/token, each paying a fixed host→device dispatch tax that dwarfs 135M's tiny per-op
compute; the CPU's parallel matmul has no such tax and batches its prefill. This is the *same*
launch-bound regime the CUDA/RTX-4070 lane documents ([`README.md`](README.md) §"Throughput"),
now confirmed on a second vendor (Vulkan/AMD). **The device path is the architecture that
scales to models too big for CPU residency — not a win at 135M.** The lever to close it is
batched device prefill + a capture-replay graph (async/`GraphCompile` are both `false` in the
RX 7600 caps today), not a faster kernel.

### Cross-backend consistency

| GPU lane | box | model | decode tok/s | regime |
|---|---|---|---:|---|
| Vulkan Q8 (this) | RX 7600 (native Win) | SmolLM2-135M | 24.6 | launch-bound |
| CUDA Q8 ([README](README.md)) | RTX 4070 (WSL) | Qwen2.5-3B | 25.1 | launch-bound |

Both of fak's GPU backends land at ~25 tok/s decode and are launch-bound, across two vendors,
two OSes, and a 22× model-size gap — strong evidence the ceiling is the per-op dispatch model,
not any one kernel or driver.

## Conditions

- **Box:** AMD Radeon RX 7600 (discrete), Vulkan 1.4.349, AMD proprietary driver, native
  Windows (the GPU is only reachable through the Windows Vulkan loader — this lane does NOT
  run in WSL).
- **Model:** SmolLM2-135M (30 layers), fak-format export, Q8_0 weight path.
- **Build:** `pwsh internal/compute/build_vulkan.ps1 lib` (rebuilds `libfakvulkan.a` with the
  MinGW cgo toolchain), then `modelbench -tags vulkan`. `FAK_WORKERS=8` to leave host cores
  for the editing session.

## Reproduce

```powershell
pwsh internal/compute/build_vulkan.ps1 lib      # rebuild the shim lib with the cgo toolchain
pwsh fak/experiments/gpu/q8_gpu_campaign.ps1 `
    -dir internal/model/.cache/smollm2-135m -model smollm2-135m `
    -prefill "16,64,256,512" -decode 64 -decodeReps 3 -prefillReps 3
```
