---
version: 0.14.0
date: 2026-06-18
headline: "Llama3 RoPE-scaling: a model trained at 8K attends coherently at 128K — the positional-quality lever for non-windowed long context (#19)."
themes:
  - fak
  - model
  - long-context
highlights:
  - "Config.RopeScaling carries the NTK-by-parts inv_freq rescale Llama-3.1/3.2/3.3 use to reach 128K without retraining."
  - "One transform on the single invFreq builder — the whole CPU prefill+decode+batch path inherits it."
  - "Matches a literal port of HF's _compute_llama3_parameters (rel <= 1e-12) on the real Llama-3.1 values."
  - "Unset == default == byte-identical: SmolLM2/Qwen2.5 take the unchanged path; R2/R14/oracle/Q8 untouched."
  - "Static in position, so RoPE stays relative — Evict's single-rotation reposition is still bit-exact under scaling."
---

**TL;DR** — sliding-window attention (v0.11.0) and the bounded-memory decode
(v0.12.0) made *windowed* long context tractable in compute and memory. This release
addresses the remaining wall for *non-windowed* (full-attention) long context:
**positional quality**. `Config.RopeScaling` carries the NTK-by-parts RoPE
`inv_freq` rescale Llama-3.1/3.2/3.3 use to extend a model trained at 8K to **128K
without retraining** — so a position far past the trained range lands inside the
trained rotational range instead of aliasing. It is opt-in and a no-op when unset, so
the proven bit-exact core is untouched.

## Llama3 RoPE-scaling — the positional-quality lever (#fak #model #long-context)

- **What changed** — an optional `Config.RopeScaling` (loaded straight from HF
  `config.json`'s `rope_scaling` object) and one transform, `scaleInvFreq`, applied at
  the single inverse-frequency builder `invFreq`. Because both `newRope` (the
  from-scratch forward reference) and `cachedInvFreq` (decode + batch) route through
  `invFreq`, the entire CPU prefill+decode+batch path inherits the rescale from one
  site.
  - *Why:* sliding-window attention only helps a *windowed* model — it never sees a
    position beyond its window. A *full-attention* model at 128K needs its RoPE
    frequencies rescaled, or the far-apart attention scores alias. This is the
    mechanism that closes that gap.
  - *How:* `internal/model/rope_scaling.go` (the rescale) + `internal/model/kv.go`
    (`invFreq` applies it; the inv-freq cache key is fingerprinted by the scaling).
- **The NTK-by-parts rescale.** For each RoPE band with wavelength `2π/inv_freq`:
  high-frequency (local) bands are left untouched, low-frequency (global) bands are
  divided by `factor` (wavelength stretched ×`factor`), and a medium band is smoothly
  interpolated between — so `inv_freq` stays continuous. For the real Llama-3.1 values
  (factor 8 / low 1 / high 4 / orig 8192) this stretches the long-wavelength bands 8×.
- **Proven against HF's own reference.** `TestLlama3RopeScalingMatchesHFReference`
  compares the production rescale to a *literal port* of HF transformers'
  `_compute_llama3_parameters` (the two-step `torch.where` form, structurally different
  from the production switch) and agrees to rel ≤ 1e-12. `TestLlama3RopeScalingRegimes`
  proves all three frequency regimes fire and the long-context stretch happens.
- **Two invariants keep the proven core intact.**
  - *Unset == default == byte-identical.* A nil `RopeScaling` (SmolLM2/Qwen2.5)
    returns the base `1/θ^(2j/hd)` table on the same backing slice — so R2
    (cached-decode==prefill, `max|Δ|=0`), R14, the HF oracle, and the Q8 gate run the
    identical instruction stream. `TestRopeScalingUnsetIsByteIdentical`; the full
    `./internal/model` suite stays green. An unrecognized `rope_type` (yarn, longrope
    — not wired here) is the same no-op pass-through, so an unknown tag can never
    silently corrupt the table.
  - *Static in position ⇒ re-rotation-safe.* The rescale is applied once at table
    build, independent of sequence length, so RoPE stays *relative* and
    `KVCache.Evict`'s single-rotation reposition (the primitive the bounded-memory
    decode depends on) is still bit-exact under scaling.
    `TestLlama3RopeIsPositionPureUnderEvict` evicts a middle span from a scaled cache
    and gets a `math.Float32bits`-identical result to a cache that never saw the span.
- **Safety / scope** — opt-in; no production path sets `RopeScaling` yet, and the
  unset default is byte-identical. The **mechanism** is proven against HF's reference +
  the bit-exact rungs on the SmolLM2 path; a re-exported real Llama-3.1-8B HF oracle
  (which would additionally prove a checkpoint's `rope_scaling` flows through `Load`
  end-to-end) needs the 8B weights and is the separable follow-up — exactly as the SWA
  family-window-value oracle is for #20. Witness: `go test ./internal/model -run Rope`
  (green); `fak/ROPE-SCALING-RESULTS.md`; `CLAIMS.md` updated.

## Road to long context — status

| wall (non-windowed full attention) | addressed by |
|---|---|
| compute (per-token attention) | sliding-window read-mask — v0.11.0 (windowed) |
| memory | bounded-memory windowed decode — v0.12.0 (windowed) |
| **positional quality** | **llama3 inv_freq rescale — v0.14.0 (8K→128K)** |

longrope (Phi, #23) and the remaining #19 mechanical bits (qk-norm, attn/logit
soft-caps, per-projection bias, embed/logit scale) are the next separable rungs.
