---
version: 0.13.0
date: 2026-06-18
headline: "GPU weight quantization: f16 reaches Qwen2.5-1.5B at llama.cpp parity, Q8_0 fits a 2.5-3B model in 8 GB VRAM."
---

**TL;DR** — The CUDA backend can now narrow matmul weights on upload: f16 (half the
VRAM + Ada tensor cores, Qwen2.5-1.5B on the GPU at equal-precision parity with
llama.cpp) and Q8_0 (~1.125 bytes/param — the lever that fits a 2.5-3B model in this
laptop's ~6.2 GiB free VRAM, where f16 tops out near 1.5B). Both are held to the same
Approx gate: argmax-exact on a real model, logit cosine ≥ 0.999. The default pure-Go and
cpu-reference paths are byte-identical to before.

## CUDA f16 weight path — half the VRAM, Ada tensor cores (#fak #compute #cuda)

- **Matmul weights upload as f16** when `FAK_CUDA_F16=1`, via the `UploadDtype` cap +
  `PreferredWeightDtype` seam. Halves resident weight bytes — the lever that fits a
  multi-billion-param model on an 8 GB 4070.
  - *Why:* f32 weights cap the GPU at ~sub-1B on this card; f16 reaches Qwen2.5-1.5B.
  - *How:* `internal/compute/cuda.go` (`fcuda_matmul_f16` via `cublasGemmEx`), narrowed
    at `model/hal.go:matWeightHAL`. Norm weights/biases stay f32.
- **Witnessed at equal-precision parity with llama.cpp** on Qwen2.5-1.5B.

## CUDA Q8_0 weight path — fits a 2.5-3B model in 8 GB (#fak #compute #cuda #33)

- **Q8_0 weights** (int8 codes + a per-32-block f32 scale, ~1.125 bytes/param) upload to
  VRAM when `FAK_CUDA_Q8=1`, dequantized inline by a fused GEMV.
  - *Why:* ~3.6× denser than f16 — fits a 2.5-3B model where f16 (2 bytes/param) cannot.
  - *How:* `fcuda_matvec_q8` in `internal/compute/cuda_kernels.cu` (one output row per
    block, vectorized char4/float4 loads). Weight-only quant — the activation stays f32,
    so numerics match the scalar form and argmax-exact is preserved.
- **Same block format as llama.cpp's Q8_0 GGUF** and fak's CPU Q8 lane, so the head-to-head
  stays apples-to-apples.
- **Lean load** — Q8 weights are sourced from the lean store (`m.q8w`) so a quant-only
  model never needs the dropped f32. `gpucheck -lean` argmax-checks the Q8 GPU path against
  the CPU Q8 lane on a real model; `cmd/modelbench` now accepts `-quant` on a narrowing
  backend.
- **Gate:** `TestCUDAMatMulQ8MatchesRef` (cosine) + a `FAK_CUDA_Q8=1` rung in
  `build_cuda.sh`; all 5 CUDA configs green on the RTX 4070. cpu-ref lacks the seam → stays
  f32, Reference bit-identity witness untouched.

## sessionbench — streamed checkpoints + live heartbeats (#fak #bench)

- **Per-cell checkpoint streaming + live arm heartbeats** in `cmd/sessionbench`, with seeded
  prefill samples, so a long sweep reports progress as it runs instead of only at the end.
