---
version: 0.18.0
date: 2026-06-18
headline: "longrope RoPE-scaling (Part A): the Phi-3/3.5/4 inv_freq rescale extends a 4K-trained model to 128K, with the short/long regime pinned at session start."
themes:
  - fak
  - model
  - long-context
highlights:
  - "Config.RopeScaling.{Short,Long}Factor + longropeInvFreq wire HF's per-dim Phi rescale: inv_freq'[j] = inv_freq[j] / ext[j]."
  - "The #23 guardrail is implemented: the short/long regime is pinned to the model's STATIC max-context, never the live cache length."
  - "So RoPE stays relative — KVCache.Evict's single-rotation reposition is bit-exact under longrope (proven)."
  - "Routes through the single invFreq builder, so every CPU lane inherits it and the no-scaling default stays byte-identical."
  - "Proven against a from-scratch port of HF's 1/(ext·theta^(2j/dim)); malformed configs fail loud at Load."
---

**TL;DR** — building on the llama3 rescale (v0.14.0, #19), this wires the **second**
long-context positional family: **longrope** (Phi-3/3.5/4, #23). It is the per-dimension
`inv_freq` rescale Phi uses to extend a model trained at 4K to 128K without retraining.
This release ships **Part A — the `inv_freq` half**; the `attention_factor` (mscale) is
the next sub-rung. Opt-in and byte-identical when unset, so the proven core is untouched.

## longrope inv_freq rescale + the static-regime guardrail (#fak #model #long-context)

- **What changed** — `Config.RopeScaling` gains `ShortFactor`/`LongFactor` (Phi's
  per-dimension multiplier arrays, length `head_dim/2`), and `scaleInvFreq` gains a
  `"longrope"` case: `inv_freq'[j] = inv_freq[j] / ext[j]`, where `ext` is `long_factor`
  when the model's max context exceeds `original_max_position_embeddings`, else
  `short_factor` (HF `_compute_longrope_parameters`' inv_freq output — the pure per-dim
  divide, not a yarn-style blend).
  - *Why:* llama3 (#19) covers the Llama family's NTK-by-parts stretch; Phi uses a
    different, learned per-dimension rescale. This wires it.
  - *How:* `internal/model/rope_scaling.go` (`longropeInvFreq`); it routes through the
    single `invFreq` builder, so decode, batched prefill, the Q8 path, and `Evict` all
    inherit it for free — and a model without `rope_scaling` is byte-for-byte unchanged.
- **The #23 guardrail — implemented.** The short-vs-long regime is chosen ONCE from the
  **static** config (`MaxPositionEmbeddings > original`), never from the live cache
  length. If it read the cache length, a mid-session `Evict`/`TrimToWindow` shrink below
  `original` would flip the regime, rebuild `inv_freq` with `short_factor`, and make
  `Evict`'s single-rotation reposition rotate survivors with a different table than
  encoded them — breaking bit-exactness. Because the choice derives from immutable
  `Config` (which `Evict` and every forward path share), RoPE stays relative:
  `TestLongropeIsPositionPureUnderEvict` evicts a middle span under a long-regime config
  and gets a `Float32bits`-identical result to a cache that never saw the span.
- **Proven against HF.** `TestLongropeInvFreqMatchesHFReference` compares
  `longropeInvFreq` to a *from-scratch* port of HF's `inv_freq = 1/(ext·θ^(2j/dim))` (a
  different op order than the production `base[j]/ext[j]`) to rel ≤ 1e-12, and asserts
  the table changed. `TestLongropeRegimePinnedByStaticContext`,
  `TestLongropeCacheKeyDistinct` (the resolved regime + an FNV hash of the factor slices
  fingerprint the inv-freq cache key), and `TestLongropeValidateAndLoad` (absent/mis-sized
  arrays or non-positive context fail loud at `Load`) complete the set. 5/5 green; full
  `./...` green, so the bit-exact rungs are intact.
- **Honest scope** — this is the **inv_freq half**. longrope's `attention_factor` (mscale,
  ≈1.42× on Phi-128k scores) folds into the model-package cos/sin builders (gated `≠1.0`,
  byte-identical when unset) and is the next sub-rung; the device (HAL/Metal) RoPE lanes
  stay unwired as for llama3; a real Phi-128k HF oracle is the checkpoint-bound follow-up.
  Witness: `go test ./internal/model -run Longrope`; `fak/ROPE-SCALING-RESULTS.md`;
  `CLAIMS.md` updated.

## Long-context positional axis — status

| family | mechanism | state |
|---|---|---|
| Llama-3.x | NTK-by-parts (#19) | v0.14.0 + v0.15.1 (hardened) |
| Phi-3/3.5/4 | longrope per-dim rescale (#23) | **v0.18.0 (inv_freq); attention_factor next** |

Both families' `inv_freq` rescales are now wired, byte-identical when unset, and
position-pure under `Evict`. Remaining: longrope's `attention_factor`, the #19 mechanical
bits (qk-norm, soft-caps, per-projection bias), the device-lane wiring, and the
checkpoint-bound HF oracles.
