TurboQuant

TurboQuant Pro

Consumer-aware compression for embedding indexes and LLM KV caches. TurboQuant Pro compresses each vector by the metric its downstream consumer actually uses — retrieval recall for indexes, attention/generation quality for KV caches — not reconstruction cosine alone, which is repeatedly shown here to be blind, or even anti-correlated, with quality.

pip install turboquant-pro
tqp replay embedding_glove_recall --small   # reproduce the headline retrieval claim — CI-gated, runs in seconds

• Embedding retrieval: 32× compression at recall@10 ≈ 0.999 after identical reranking — a statistical tie with OPQ, clearly above RaBitQ at matched bytes, and 4–20× cheaper to build than OPQ (RaBitQ itself builds faster). Measured on a private 199k LaBSE sample and corroborated at 1M on Gutenberg; ledger status reported, i.e. a real single run you cannot replay from this repo. The CI-gated, public number is ~9.6× at recall@10 ≈ 0.999 on GloVe-100 (tqp replay embedding_glove_recall).
• KV caches: architecture-aware key quantization avoids a failure that is invisible to reconstruction metrics — PolarQuant keys read 0.995 cosine yet blow perplexity to ≈10⁴; per-channel keys keep it near fp16.
• At scale & in production: compressed-domain search, persisted / larger-than-RAM sharded + memory-mapped indexes, distribution-free rank certificates, one-command replay, and drift monitoring.

Every headline number — with its reproduction status, dataset, one-click notebook, and hardware — is a row in CLAIMS.md. The acceptance signal everywhere is rank fidelity / a certificate / the consumer's metric — never reconstruction cosine.

The latest published pre-release is 2.0.0a2 (production semantics for the vLLM KV connector, the Postgres track, self-metering anatomy instruments); master carries 2.0.0a3, unreleased (empirical false-clear rate, the reconciled claims ledger, the HF cache mask-size fix — see the Unreleased section of the changelog). The last stable line is 1.9.x (larger-than-RAM search + index format v3), and the tqp CLI and certification platform shipped in 1.8.0. APIs under connectors/ may still move before 2.0.0. Full notes: CHANGELOG.md.

Feature & stability matrix

The full table is in docs/api-stability.md (the source of truth); component reference in docs/API.md.

Scope & honesty: results are strongest on text embeddings and LLM workloads; multimodal APIs/presets exist but are less validated. "Beats RaBitQ" means under our matched-byte public protocol; "robust across every architecture" means every architecture tested. A recorded long-generation degradation of asym-NF4 KV quant did not survive re-validation (erratum 2026-08-15: measured nf4a gap −0.31 vs the recorded 13.7 on gov_report-512, n=40; a larger real collapse, 26.64, exists under symmetric NF4 only — see benchmarks/kvquant_matrix/REVAL-2026-08-08.md and the CHANGELOG Errata); readscope's C-11c measured operator drift along the sequence — a key compressed against an early operator is later read by a different one, with a 225%-of-uniform mispricing cost, sixteen head-cells against a paired null — and it has now been run against the real degradation curve and refuted as its mechanism (amended C-12, 2026-08-15, on the symmetric-NF4 collapse that does reproduce): teacher forcing removes the consistent growth (sign test p = 0.42) and the error's orientation does no work against a rotated null, so the collapse is autoregressive compounding of a large constant error, not drift. C-11c's drift stands as a measurement; its claim to explain this degradation is dead, recorded in calibration/records/c12-longgen-drift-sym.json. Negative results and caveats are kept first-class in docs/claims.md and the soundness audit.

Not to be confused with the similarly-named turboquant (the HuggingFace KV-cache implementation of the original ICLR TurboQuant algorithm). TurboQuant Pro is a broader, retrieval-first platform that uses that quantizer as one component.