---
version: 0.3.0
date: 2026-06-17
headline: "Q8_0 quantized inference lane: a register-blocked int8 prefill GEMM (plus a SwiGLU-bottleneck fix) halves the prefill gap to llama.cpp, alongside grounded turn-tax error-rate measurement."
themes:
  - q8-quantized-prefill
  - turn-tax-grounding
highlights:
  - "Q8_0 quantized forward path lands: decode near-parity with llama.cpp, prefill ~2x faster."
  - "Register-blocked int8 GEMM micro-kernel (AVX-512), bit-identical to its scalar reference."
  - "Found and fixed the real prefill bottleneck: the serial SwiGLU, not the GEMM."
  - "Prefill gap to llama.cpp Q8_0 narrowed ~3.8x to ~1.7-1.8x (native, all-core)."
  - "Turn-tax benchmark grounded: modeled error rate replaced by a measured distribution."
---

**TL;DR** — fak gains a Q8_0 quantized inference lane (the same block format llama.cpp's GGUF
uses). Decode reaches near-parity with llama.cpp Q8_0; this release's focus, prefill, is now
~2x faster end-to-end via a register-blocked int8 GEMM micro-kernel and a measurement-driven
fix to the activation path — closing the prefill gap to llama.cpp from ~3.8x to ~1.7-1.8x. The
f32 path is byte-for-byte untouched and every correctness rung stays green. Also ships the
grounded turn-tax benchmark (measured error-rate distribution).

## Q8_0 quantized inference + register-blocked prefill GEMM

- **Q8_0 quantized lane** — `Model.Quantize` + `Session.Quant` route the seven weight matmuls
  and the LM head through Q8_0 (llama.cpp's GGUF block format) with a hand-written AVX2/AVX-512
  int8 SIMD dot, bit-identical to a scalar reference. Decode ~8 ms/tok, near-parity with
  llama.cpp Q8_0. Opt-in; the proven f32 path is byte-for-byte unchanged.
  - *How:* `internal/model/quant.go`, `quant_amd64.{go,s}`, `quant_forward.go`.
- **Register-blocked int8 prefill GEMM** — replaces the per-element `qdot8` sweep with a 4x4
  tile micro-kernel (`qgemm8tile512`): 16 zmm accumulators held live across all blocks, one
  horizontal reduce per output instead of per output*block, each weight/activation block
  reused across the tile. GEMM phase ~2.6x faster, bit-identical to its scalar reference.
  - *How:* `internal/model/quant_gemm.go`, `qgemm8tile512` in `quant_amd64.s`; pinned by
    `TestQGemm8AsmMatchesScalar`.
- **The bottleneck that wasn't the GEMM** — profiling showed the GEMM was only ~1/3 of prefill;
  the largest single cost was the **serial SwiGLU** (`silu` calls `math.Exp` over ~11.8M
  elements/prefill, single-threaded). Parallelizing it (bit-identical, per-element independent)
  was the biggest individual win. The prior doc claim that the residual was "the GEMM
  micro-kernel" is corrected here, from measurement.
- **Result (this 16c/32t Zen5 box)** — Q8 prefill-256 **285 -> 138 ms native (2.07x)**,
  **467 -> 238 ms WSL-16t (1.96x)**; gap to llama.cpp Q8_0 (74.7-82.9 ms) narrowed from ~3.8x
  to ~1.7-1.8x (native, all-core). Parity approached, not reached — the residual is now spread
  across GEMM/quant/attention (the same hand-tuned-GGML boundary, smaller).
  - *Why native is the fair axis:* llama.cpp's prefill is thread-independent at this size; fak's
    scales with cores, so the all-core comparison is apples-to-apples.
- **Correctness, no tolerance spent** — the Q8 gate (argmax-exact vs the HF oracle, logit cosine
  >= 0.995 vs f32, teacher-forced agreement) holds; the f32 R2/R14/oracle rungs and the decode
  kernel's bit-identity stay green. Every load-bearing claim survived a 4-skeptic adversarial
  review; the two real issues it caught (a `q8round` divergence from `math.Round`; an AVX-512VL
  instruction-gating gap) are fixed, not papered over.
  - *Reproduce:* `cmd/modelbench -quant`, `cmd/q8bench`, `FAK_QPROFILE=1` (phase split),
    `FAK_QGEMM=legacy` (as-found A/B). Data under `experiments/model-baseline/`.

## Turn-tax benchmark — grounded error-rate measurement

- **Stochastic error-rate grounding** — the turn-tax benchmark (the extra error-code model turn
  the kernel deletes) moved from a single modeled point estimate to a measured distribution via
  a live error-injection harness, so the "+1 retry turn" cost is witnessed, not assumed.
  - *How:* `fak/turnbench` + the agent error-injection harness; see `TURN-TAX-RESULTS.md`.
