---
version: 0.11.0
date: 2026-06-18
headline: "Sliding-window attention lands the compute-side long-context axis — O(N)→O(W) per token, byte-identical to the dense path when unset."
themes:
  - fak
  - model
  - long-context
highlights:
  - "Per-layer sliding-window attention as a read-time mask (#20), threaded into every CPU attention site."
  - "Unset window == full attention == byte-identical: R2/R3/R14, the HF oracle, batched + Q8 gates all unchanged."
  - "Window keyed off the absolute position, so it survives an Evict compaction (proven evict==never-saw)."
  - "6/6 new SWA witnesses green on the synthetic model; the dense rungs verified unchanged on real SmolLM2-135M."
  - "CUDA decode reaches llama.cpp Q8_0 parity (7.5 → 120 tok/s)."
---

**TL;DR** — fak gains the headline compute mechanism for long context: a per-layer
sliding window that bounds attention from O(N) to O(W) tokens, shipped as a read-time
mask that is a byte-for-byte no-op when unset. It is additive, fully witnessed on the
synthetic model, and leaves every proven bit-exact rung intact — the first step (and
the correctness precondition for the memory step) on the road to 1M-token context.

## Sliding-window attention — the long-context read-mask (#fak #model #long-context)

- **What changed** — a per-layer `Config.Window` and a `windowLo(W, qpos)` lower-bound
  thread into all six CPU attention sites (`forward.layer`, `kv.blockStep`,
  `prefillBatched`, `tokenHiddenQ`, `attnPrefillInto`, `batch.attnDecodeBatch` /
  `attnPrefillMultiInto`). A query attends only to the most-recent `W` key positions.
  - *Why:* dense O(N) attention is one of the three walls to 1M tokens; the window is
    the mechanism Mistral/Gemma2-3/Qwen-long/gpt-oss use to make a long cache tractable,
    and the precondition for later dropping out-of-window K/V to bound memory.
  - *How:* `internal/model/swa.go`; each site starts its score/V loops at
    `windowLo(W, qpos)` and softmaxes over `scores[wlo:]`.
- **Two invariants keep the proven core intact.**
  - *Unset == full == byte-identical:* a nil/`<=0` window makes `windowLo` return 0, so
    the dense path runs the identical instruction stream. `TestSlidingWindowDegeneratesToFull`
    proves a window wider than the prompt is `Float32bits`-equal to no window; R2
    (cached-decode==prefill, `max|Δ|=0`), R3 (evict==never-saw), R14, the HF oracle, and
    the batched/Q8 gates all stay green on real SmolLM2-135M (each verified in isolation).
  - *Keyed off the absolute position, not the loop index:* the window survives an
    `Evict` compaction (`KVCache.Evict` renumbers survivors), proven by
    `TestSlidingWindowSurvivesEvict` (windowed evict==never-saw, bit-exact).
- **Witnesses** — `go test ./internal/model -run Window` (6/6 green): the `windowLo`
  table, the no-op, the non-noop (last-pos `max|Δ|≈1.05` at window 2 over an 8-token
  prompt), the windowed R2 (Prefill == Forward == serial decode), windowed multi-user ==
  per-user serial, and the evict-survival proof. See `fak/SLIDING-WINDOW-RESULTS.md`.
- **Honest scope** — the *mechanism* is proven family-independently on the synthetic
  model; a real Mistral/Gemma SWA HF oracle (proving a family's window VALUE is wired
  from its checkpoint) needs the #19 rope-scaling axis + that checkpoint and is the
  follow-up. The read-mask bounds attention cost/scope, not cache memory yet.

## CUDA backend — decode parity with llama.cpp Q8_0 (#gpu)

- **What changed** — the CUDA decode path reaches parity with llama.cpp's Q8_0 decode,
  going from 7.5 to 120 tok/s.
  - *Why:* removes the standing "fak cannot be a peer of llama.cpp on GPU" caveat.
  - *How:* see `fak/GPU.md` and the `compute(cuda)` history since v0.10.0.

## Docs — drop the false "fak cannot be a peer of llama.cpp" framing (#fak)

- **What changed** — repo docs no longer claim fak cannot peer with llama.cpp on GPU;
  it reached GPU decode parity, so the framing was retired.
