---
version: 0.9.0
date: 2026-06-18
headline: "fak Metal prefill reaches 0.77x llama.cpp on M3 Pro — a cooperative-simdgroup attention kernel takes the GPU-resident forward from 0.33x to near-parity."
themes: ["fak", "metal", "gpu"]
highlights:
  - "Metal prefill 1331 tok/s vs llama.cpp 1727 (pp256, Qwen2.5-1.5B Q8) — 0.77x, up from 0.18x."
  - "Cooperative simdgroup attention kernel: 4.6x prefill gain over the shipped hybrid."
  - "Attention dropped ~280ms -> ~22ms by killing a 128-float register spill (P=256)."
  - "CUDA backend: single-stream + gated CUDA-graph capture; WSL launch wall measured."
  - "radixkv: live SmolLM2-135M Q8 wall-clock added to the SGLang RadixAttention benchmark."
---

**TL;DR** — fak's opt-in Metal prefill is now within ~1.3x of llama.cpp Metal on an M3 Pro
(0.77x at pp256, up from the 0.18x first hybrid), driven by a GPU-resident forward plus a
rewritten cooperative attention kernel. The release also captures the CUDA backend's
single-stream/graph-capture work and the radixkv SGLang benchmark's real-model run. Decode and
the pure-Go default build are unchanged; all GPU work stays behind `-tags fakmetal` / the CUDA tag.

## Metal prefill near-parity on M3 Pro (#fak #metal)

- **Cooperative simdgroup attention — the parity lever.** Rewrote the prefill attention kernel
  (`internal/metalgemm/forward.m` `attn_k`) from one-thread-per-(t,h) to one **simdgroup (32
  lanes) per (t,h)**, the lanes splitting the head dim and reducing the QK dot with `simd_sum`.
  - *Why:* the old kernel's 128-float thread-private `acc[]` spilled to device memory and gave
    terrible occupancy — profiling pinned ~280ms of a 432ms GPU forward (pp256, 1.5B) on it alone.
  - *How:* `acc` is now 4 floats/lane in registers; attention falls to ~22ms, full GPU forward
    432→175ms. argmax-identical to the CPU Q8 path (`max|Δlogit|≈0.29`).
- **Result:** pp256 prefill **290 (shipped hybrid) → 576 (GPU-resident) → 1331 tok/s**, now
  **0.77x** of llama.cpp Metal (1727) measured back-to-back.
- **Tier hygiene.** Declared `internal/radixkv` at tier 3 (composer) so the architest layered-DAG
  gate — and the whole suite — is green (`internal/architest/architest_test.go`).

## CUDA backend — single-stream + graph capture (#gpu)

- **Single-stream execution with gated CUDA-graph capture.** Reworked the CUDA path
  (`internal/compute/cuda.go`, `cuda_kernels.cu`, `cuda_backend.h`) onto one stream with optional
  graph capture, and added the HAL hook (`internal/model/hal.go`, `kv.go`).
  - *Why:* amortize per-launch overhead; the work also measured and documented the WSL launch
    wall (`fak/GPU.md`).

## radixkv — real-model SGLang benchmark (#fak)

- **Live SmolLM2-135M Q8 wall-clock** added to the RadixAttention comparison
  (`fak/RADIXATTENTION-RESULTS.md`, `experiments/radixattention/radixbench-smollm2-135m-q8.json`).
  - *Why:* move the SGLang prefix-reuse comparison from synthetic to a real model on the
    kernel-owned KV cache.
