---
version: 0.12.0
date: 2026-06-18
headline: "Bounded-memory sliding-window decode: a 1,000,000-token stream runs in 256 KV positions, not ~67 GB."
themes:
  - fak
  - model
  - long-context
highlights:
  - "Session.TrimToWindow evicts aged-out K/V via the proven Evict — O(window) cache for any stream length."
  - "Measured: 1,000,000 tokens decoded with the KV cache never exceeding 256 positions (window 128)."
  - "Bounded decode is argmax-identical to the full-cache windowed decode (RoPE is relative)."
  - "Renumbered survivors keep RoPE positions small — the high-position drift regime is never entered."
  - "Opt-in and a no-op when no window is set; the dense path stays byte-identical."
---

**TL;DR** — building on v0.11.0's sliding-window read-mask, fak now decodes
arbitrarily-long context in **bounded memory**. A position older than every layer's
window is provably never attended to again, so it can be dropped — and because the
proven `Evict` renumbers survivors and RoPE is relative, the bounded decode is
argmax-identical to keeping the full cache. This closes the **memory** and
**numerics** walls to 1M-token context; the compute wall fell in v0.11.0.

## Bounded-memory windowed decode — the memory + numerics lever (#fak #model #long-context)

- **What changed** — `Config.MaxWindow()` + `Session.TrimToWindow(slack)`: when the KV
  cache grows past `MaxWindow()+slack`, the oldest positions are evicted back down to
  `MaxWindow()` through the proven `KVCache.Evict`. A stream of any length decodes in
  **O(window)** cache instead of O(N).
  - *Why:* memory was the dominant wall to 1M tokens (~67 KB/token → ~67 GB for a 1M
    cache on SmolLM2-135M). A windowed model never reads beyond its window, so the rest
    is dead weight.
  - *How:* `internal/model/swa.go`; the drop reuses the verified eviction primitive, so
    not one bit of the proven KV core is re-derived.
- **Argmax-identical to the full cache.** `Evict` renumbers + re-RoPEs survivors, and
  RoPE is *relative* (a score depends only on `q-k`), so renumbering the whole surviving
  window preserves every within-window attention score. `TestBoundedWindowMatchesFullWindow`
  (teacher-forced) confirms step-for-step argmax agreement while the full cache grows
  with the stream and the bounded cache peaks at 2·window.
- **Two walls fall at once.**
  - *Memory:* **1,000,000 tokens decoded with the KV cache never exceeding 256 positions**
    (window 128) — `TestBoundedWindowMemoryIsBounded`, rerun via `FAK_LONGCTX_N=1000000`.
    O(window), independent of N.
  - *Numerics:* survivors renumber into a small contiguous range, so the RoPE positions
    the model sees never reach the high-position angle-drift regime.
- **Safety** — `TrimToWindow` is opt-in (no production decode path calls it yet) and a
  no-op when no window is configured, so the byte-identical no-window default from
  v0.11.0 is unchanged. This is the standard rolling-buffer KV cache (Mistral) made
  bit-faithful via the proven Evict. Witness: `go test ./internal/model -run BoundedWindow`
  (2/2 green); `fak/SLIDING-WINDOW-RESULTS.md`; `CLAIMS.md` updated.

## Road to 1M — status

With v0.11.0 (compute) and v0.12.0 (memory + numerics), all three mapped walls to
1M-token context are addressed and the engine decodes arbitrarily-long context in
bounded memory. The remaining item is positional **quality** for *non-windowed* long
context (rope-scaling, #19/#23) — which a windowed model does not require.
