---
version: 0.10.0
date: 2026-06-18
headline: "A machine-wide GPU lease stops concurrent Metal benches from exhausting unified memory and panicking the box."
themes: ["fak", "metal", "gpu"]
highlights:
  - "New gpulease serializes GPU-heavy benches so concurrent -metal runs queue, not stack"
  - "Roots out the jetsam-cascade kernel watchdog panic from stacked unified-memory residency"
  - "metalgemm gains mg_reset to free the f16 weight table across model reloads"
  - "mg_upload now fails loudly on a full table or OOM instead of silently returning -1"
  - "CUDA backend reuses its captured exec graph across tokens (ExecUpdate)"
---

**TL;DR** — On 2026-06-18 several `-metal modelbench` processes launched across separate fleet sessions stacked their unified-memory residency on the same pool, overran RAM, and walked the box through a jetsam-kill cascade into a kernel watchdog panic. This release adds the coordination that was missing — a machine-wide GPU lease so heavy benches queue instead of stacking — plus a metalgemm weight-table teardown and a louder upload failure. Anyone running Metal benches on a shared Apple Silicon box wants this.

## GPU bench lease — no more stacked-residency panics (#fak #metal #gpu)

- **New `internal/gpulease`: a machine-wide advisory lease.** A GPU-heavy process takes an OS flock on one lockfile before loading its model and holds it for its lifetime, so concurrent launches QUEUE instead of stacking their residency.
  - *Why:* a `-metal` bench is heavy on unified memory (the CPU model plus its full f16 GPU copy, ~5 GB each); two or three at once exhausted RAM and panicked the kernel. Nothing coordinated those launches.
  - *How:* `gpulease.Acquire` blocks by default (`FAK_GPU_LEASE_NOWAIT=1` fails fast); the kernel drops the flock when the holder's fd closes, so the lease frees on any exit — including `os.Exit` or a crash.
- **`modelbench` acquires the lease before the model load**, gated on `metalgemm.Available()` so a CPU fallback never serializes behind the GPU lease.
  - *How:* `fak/cmd/modelbench/main.go` — acquire-before-load; `fak/internal/gpulease/lease.go`.
- **Tests cover busy/free, wait-then-win, timeout, and release-on-process-exit** (the last via a re-exec child — the invariant the whole fix rests on). `architest` declares `gpulease` at tier 1 (foundation).

## metalgemm — weight-table teardown + loud upload cap (#fak #metal)

- **`metalgemm.Reset()` (C `mg_reset` + `mg_fwd_reset`) frees the whole resident set.** It releases every uploaded f16 weight buffer and the matmul scratch and clears the GPU-resident forward's per-model topology, returning the weight table to empty.
  - *Why:* `mg_upload` retained every buffer forever (`CFBridgingRetain`, monotonic `gNW`) with no teardown, so a long-lived process reloading a model leaked the entire f16 weight set per load.
  - *How:* `fak/internal/metalgemm/metal.m`, `forward.m`; the device, queue, and compiled MSL pipelines stay live. A real-device test proves the table is reusable across reset cycles.
- **`mg_upload` now fails loudly.** It silently returned `-1` when the table hit `MG_MAX_W` (indistinguishable from OOM, and the caller turns `-1` into a panic); it now warns once on the cap and logs the byte size on a device-buffer alloc failure.

## CUDA backend — reuse the captured exec graph (#gpu)

- **Single captured CUDA graph is updated across tokens (`cuGraphExecUpdate`) instead of re-instantiated.**
  - *How:* `fak/internal/compute/cuda.go`, `cuda_kernels.cu`, `cuda_backend.h`; `model/hal.go`.

## radixkv — RadixAttention benchmark explainer (#fak)

- **From-first-principles explainer for the SGLang RadixAttention benchmark**, plus refreshed node benchmark outputs at HEAD.
  - *How:* `fak/RADIXATTENTION-EXPLAINER.md`, `fak/experiments/fleet-nodes/anthony/*`.
