---
version: 0.6.0
date: 2026-06-17
headline: "Apple Silicon joins the fleet — an arm64 NEON Q8 lane (decode 2.4x, prefill 5-6x) + pure-Go HF loaders run Qwen2.5-1.5B on an M3, measured head-to-head vs llama.cpp — alongside a compute/HAL backend seam, a turn-agent benchmark, and a production-readiness audit."
themes:
  - apple-silicon-neon
  - model-loaders
  - in-kernel-model
  - compute-backend
  - turn-agent-bench
  - fleet-nodes
  - docs
highlights:
  - "arm64 NEON Q8_0 kernel: int8 decode was 1.24x slower than f32, now 1.9x faster."
  - "NEON prefill GEMM: prompt processing 5-6x faster on Apple Silicon."
  - "Pure-Go HF safetensors loaders run any Llama/Qwen2 checkpoint — no torch."
  - "Memory-lean quantize-at-load fits a 7B-class model on a 36 GB box."
  - "Measured M3 head-to-head vs llama.cpp on the same Qwen2.5-1.5B Q8_0."
  - "Compute/HAL backend seam, turn-agent benchmark, and production-readiness audit."
---

**TL;DR** — fak now runs on Apple Silicon as a first-class target: the missing arm64 NEON
Q8_0 lane (decode + prefill) plus pure-Go HuggingFace loaders take it from
*quantization-is-a-loss* to running **Qwen2.5-1.5B-Instruct** on an M3 Pro, benchmarked
honestly against llama.cpp on the identical Q8_0 weights. This release also lands the
`internal/compute` backend seam (the in-kernel model as a swappable engine), a turn-agent
benchmark with policy guards, and a production-readiness audit reconciled against HEAD.

## Apple Silicon — the arm64 NEON Q8 lane (#model)

- **NEON Q8_0 decode kernel.** The int8 dot was amd64-only (AVX2/AVX-512); on arm64 it fell
  back to scalar Go, so Q8_0 was a net loss — int8 decode ran **1.24× slower** than f32.
  `quant_arm64.{go,s}` adds the SDOT (FEAT_DotProd) kernel, **bit-identical** to the scalar
  reference (`TestQdot8NEONMatchesScalar`).
  - *Result:* int8 decode 17.68 → **7.30 ms/tok** (2.4×), now **1.9× faster than f32**.
  - *How:* `internal/model/quant_arm64.s` (SDOT via `WORD`; the float combine matches gc's
    arm64 FMA fusion via a single `FMADDS`); auxv HWCAP detect on linux, assume-true on darwin.
- **NEON prefill GEMM.** The batched GEMM still ran scalar on arm64; route each cell through
  the NEON dot. *Result:* prefill (P=256) 126 → **631 tok/s** on SmolLM2 (5.0×), ~10 → 55 tok/s
  on Qwen2.5-1.5B.
- **arm64 FMA-fusion tolerance.** Three f32 "bit-exact" rungs (reposition, profiler twin, and
  the new HAL-vs-legacy comparison) drift sub-ULP on arm64 because gc auto-fuses multiply-add.
  They now assert byte-exact on amd64 and ≤1e-4 elsewhere, with token-level checks kept exact.

## Model loaders — run the best model the box can hold (#model)

- **Pure-Go HuggingFace load.** `modelbench -hf <snapshot>` loads `config.json` +
  `model.safetensors` (bf16→f32) entirely in Go — no torch. The generic Llama/Qwen2 forward
  pass runs Qwen2.5-family checkpoints unchanged (head_dim derived, Qwen2 qkv-bias inferred).
- **Memory-lean quantize-at-load.** `model.LoadSafetensorsQuant` / `modelbench -lean` quantizes
  the big matmul weights at load and drops their f32 — ~1.1 B/param resident vs f32's ~5.1, the
  lever that fits a 7B-class model on 36 GB. Pinned bit-identical to load+quantize
  (`TestLoadSafetensorsQuantMatchesRegular`). Qwen2.5-1.5B peak RSS 18.8 → **11.1 GB**.
- **M3 head-to-head doc.** `fak/M3-LLAMACPP-RESULTS.md` records the apples-to-apples Q8_0
  comparison vs llama.cpp build 8200: fak decode 28.9 tok/s vs llama.cpp CPU 63.2 / Metal 67.5;
  prefill 55.5 vs 363 / 1747. Decode is bandwidth-bound (Metal ≈ CPU); the residual is a
  hand-tuned-kernel + register-blocked-GEMM + Metal boundary, not an architecture gap.

## Compute backend & in-kernel model (#compute)

- **`internal/compute` backend seam.** The in-kernel model becomes a swappable engine behind a
  reference-checked backend (`compute.Pick`/`RequireReference`); the HAL session is pinned to
  the legacy CPU reference token-for-token.
- **Clean decode profiling.** `MeasureCleanDecode` reports per-step session latency for an
  isolated, allocation-quiet decode measurement.

## Turn-agent benchmark (#bench)

- Turn-agent benchmark + policy guards and a realistic sweep runner, exercising the
  adjudication/turn loop under recorded workloads.

## Fleet nodes & docs (#fleet)

- Registered benchmark nodes for an x86/AVX laptop and the M3 Pro; recorded Q8 model/quant,
  fused-gate, ctx-admission, and Go micro results; node-bench now runs model benches from `fak/`
  and records the resolved toolchain.
- Production-readiness audit reconciled against HEAD; new hardware-portability and
  local-vs-frontier-parity explainers; the cross-agent shared-prefix finding demoted to an OPEN,
  settings-dependent work item.
