# Token-estimation calibration

Divergence of the dependency-free heuristic (`HeuristicEstimator`, `heuristic/v2`) and the `tiktoken` default from real tokenizers, across fixed corpus shapes. Generated by `benchmarks/token_calibration.py` (issue #493).

- Generated: 2026-06-11
- tiktoken model: `cl100k_base`
- ⚠️ **tiktoken encoding unavailable when generated** (offline / no warmed `TIKTOKEN_CACHE_DIR`). The `tiktoken` and relative-error columns read `n/a`; regenerate in an environment with a warmed cache to populate them.
- Provider tokenizers (Anthropic / Gemini): not run (opt-in via `CW_TOKEN_CALIBRATION_PROVIDERS` + a registered counter; never run in CI).

| Shape | Chars | Heuristic | tiktoken | Heuristic vs tiktoken |
|---|---|---|---|---|
| `prose_en` | 132 | 32 | n/a | n/a |
| `prose_cjk` | 96 | 88 | n/a | n/a |
| `json` | 144 | 36 | n/a | n/a |
| `code` | 123 | 30 | n/a | n/a |
| `logs` | 147 | 36 | n/a | n/a |

Reading the table: a large negative *Heuristic vs tiktoken* error on `prose_cjk` is the failure mode issue #525 fixes — the script-aware heuristic keeps CJK within roughly ±30% of tiktoken instead of under-counting ~4×. For Latin prose, JSON, code, and logs the heuristic tracks `len // 4`, which is adequate for budget headroom decisions.
