---
version: 0.7.0
date: 2026-06-17
headline: "The in-kernel model runs on the GPU — a modular CUDA backend behind compute.Backend, on-box witnessed on an RTX 4070, with a measured llama.cpp parity baseline — alongside a sharded lean loader that fits a 7B, served-path engine routing, and runtime controls (journal, rate limits, policy)."
themes: ["compute", "model", "engine", "kernel", "bench"]
highlights:
  - "GPU is no longer SIMULATED: a CUDA compute.Backend decodes the real model on an RTX 4070, argmax-exact vs the CPU reference."
  - "Default `go build` stays one pure-Go artifact; CUDA is an opt-in `-tags cuda` hardware seam."
  - "Measured llama.cpp CUDA baseline on the 4070: Qwen2.5-7B-Q4_K_M prefill 2256 t/s, decode 48 t/s."
  - "Sharded safetensors lean load fits a 7B on a 36 GB box; served path gains engine routing + rectangular prefill."
  - "Runtime controls land: durable journal, token-bucket rate limiting, richer policy."
---

**TL;DR** — The fusion crosses onto the GPU: a modular CUDA backend behind the frozen
`compute.Backend` seam runs the in-kernel model's decode on a real RTX 4070, witnessed
argmax-exact against the CPU reference, with the default build still pure-Go. The
`llama.cpp` GPU parity bar is now measured on-box. In parallel the model loader learned to
stream a 7B in shards, the served path gained engine routing + rectangular prefill, and the
kernel gained durable journaling, rate limiting, and policy controls.

## GPU — the in-kernel model runs on the GPU (#compute)

- **A CUDA backend behind `compute.Backend`** — cgo + cuBLAS SGEMM + custom kernels
  (RMSNorm, RoPE rotate-half, SwiGLU, residual/bias add, decode attention, first-max
  argmax) + a device-resident KV cache.
  - *Why:* the HAL's whole point — "adding a GPU backend is a new `Backend` registration,
    never an edit to the forward loop." This is the first such backend.
  - *How:* `internal/compute/cuda.go` (`//go:build cuda`), `cuda_kernels.cu`,
    `cuda_backend.h`; registered as an **Approx** peer `"cuda"`, with cpu-ref staying the
    Reference Default.
- **On-box witness, RTX 4070 (sm_89)** — device SGEMM cosine 1.0 / maxAbs 1.9e-6, a full
  Llama decode argmax-exact, and the real in-kernel model decoding on the GPU via the HAL.
  - *How:* `internal/compute/cuda_test.go`, `internal/model/hal_cuda_test.go`. The HAL
    adoption is proven **byte-identical** on cpu-ref (`hal_test.go`), so R2/R14 and the HF
    oracle cannot be perturbed; the device is held to the Approx gate by construction.
- **Default build stays pure-Go** — `go build ./...` produces one static artifact
  (`GoFiles=[compute.go cpuref.go]`); `cuda.go` is a CgoFile only under `-tags cuda`. CUDA
  is the sanctioned hardware seam of `DIRECTION.md`, off the request path.
- **Measured llama.cpp parity baseline** — Qwen2.5-7B-Instruct Q4_K_M on the 4070, full GPU
  offload: **prefill 2256 t/s, decode 48 t/s** (4.36 GiB resident, the `GPU-MODEL-PICK.md`
  fit confirmed). The honest asymmetry (a 7B fak-vs-llama tok/s is loader-gated) and the
  gap mapped to issues #32–#42 are recorded in `fak/GPU.md`.
- **No-sudo CUDA toolchain + build** — `internal/compute/setup_cuda_wsl.sh` (micromamba
  CUDA 12.6) + `build_cuda.sh` (nvcc → static lib → cgo). `test.sh` also now finds Go when
  it is not on the non-login PATH, so `test.ps1` works regardless of WSL distro.

## Run a 7B — sharded lean load (#model)

- **Sharded safetensors lean load** — stream a model shard-by-shard so a 7B fits a 36 GB
  box instead of materializing one giant blob.
  - *How:* `internal/model/safetensors_quant.go` (+ test), wired through `cmd/modelbench`.

## Served path — engine routing + rectangular prefill (#engine)

- **Engine routing and rectangular prefill** — route requests across engines and prefill
  ragged batches as rectangles.
  - *How:* `internal/engine/engine.go`, `internal/kernel/kernel.go`,
    `internal/model/batch.go` (+ tests).
- **Sensitive engine routes are gated** — guard the routes that must not be reached
  unauthenticated.
  - *How:* `internal/engine/engine.go` + `residency_test.go`.

## Runtime controls — journal, rate limits, policy (#kernel)

- **Durable journal** — an append-only record of kernel activity.
  - *How:* `internal/journal/journal.go` (+ test).
- **Token-bucket rate limiting** — bound request rate per lane.
  - *How:* `internal/ratelimit/ratelimit.go` (+ test).
- **Richer policy + IFC** — additional policy controls and an information-flow tweak.
  - *How:* `internal/policy/policy.go`, `internal/ifc/ifc.go` (+ tests),
    `examples/policy.example.json`.

## Tooling & onboarding (#bench)

- **Workload tooling + onboarding** — `cmd/sessionbench`, `cmd/fleetserve` workload
  generation, fetch-model scripts, and a `fak/GETTING-STARTED.md`.
  - *How:* `cmd/sessionbench/main.go`, `cmd/fleetserve/workload.go`,
    `scripts/fetch-model.{sh,ps1}`, `tools/transcript_workload.py`,
    `tools/workload_tune_sweep.py`, `tools/summarize_overnight_soak.py`.
- **Platform robustness** — `*.golden` pinned to `eol=lf` so the ABI golden-freeze test is
  platform-independent on Windows checkouts (`.gitattributes`).
