TurboQuant

Installation

```bash pip install turboquant-pro

Build any component

tq = cfg.build_quantizer() # TurboQuantKV cache = cfg.build_cache() # TurboQuantKVCache rq = cfg.build_rope_quantizer() # RoPEAwareQuantizer mgr = cfg.build_manager() # TurboQuantKVManager (all layers)

Quick Start

```python from turboquant_pro import TurboQuantKV

Fit PCA on a sample of embeddings (5-10K vectors is sufficient)

pca = PCAMatryoshka(input_dim=1024, output_dim=384) result = pca.fit(sample_embeddings) print(f"Variance explained: {result.total_variance_explained:.1%}")

Auto-configure from model name — picks optimal K/V bits, RoPE-awareness

tq = TurboQuantKV.from_model("llama-3-8b") # balanced (K4/V3) tq = TurboQuantKV.from_model("gemma-2-27b", target="compression") # K3/V2

compressed_k = tq.compress(kv_key_tensor, packed=True, kind="key") # 4-bit keys compressed_v = tq.compress(kv_val_tensor, packed=True, kind="value") # 3-bit values key_approx = tq.decompress(compressed_k) # cos_sim > 0.995 (keys) val_approx = tq.decompress(compressed_v) # cos_sim > 0.978 (values)

Or manually:

Auto-Config API

Auto-detect model architecture and select optimal compression:

```python from turboquant_pro import AutoConfig

Works from a HuggingFace config dict too

cfg = AutoConfig.from_dict(model.config.to_dict(), target="compression") ```

Target presets:

Supported models: LLaMA 3 (8B, 70B), Gemma 2 (9B, 27B), Gemma 4 27B-A4B (262K context MoE), Qwen 2.5 (7B, 72B), Mistral 7B. Any HuggingFace model works via transformers.AutoConfig.

Eigenvalue-Weighted Mixed Precision (v0.9.0)

Theory: After PCA, early dimensions explain most variance. Spending 4 bits on high-eigenvalue dimensions and 2 bits on the tail gives better quality than uniform 3-bit at the same average storage.

How it works: pca.with_weighted_quantizer(avg_bits=3.0) auto-computes the bit schedule from cumulative variance thresholds (top 60% variance -> 4-bit, next 30% -> 3-bit, bottom 10% -> 2-bit). Each segment gets its own quantizer with the appropriate codebook.

Result: At 2.8 avg bits, beats uniform 3-bit (0.962 vs 0.958) in 7% less storage.

Unified Auto-Config API (v0.9.1)

How it works: TurboQuantKV.from_model("llama-3-8b") reads head_dim, n_kv_heads, rope_theta, and max_position_embeddings from a built-in model registry (or HuggingFace Hub), then selects optimal key_bits, value_bits, and RoPE-aware settings based on a target preset (quality/balanced/compression/extreme).

Config Ratio Cosine Recall Var% Time

PCA-128 + TQ2 113.8x 0.9237 78.7% 79.9% 2.2s PCA-256 + TQ3 41.0x 0.9700 92.0% 92.3% 0.7s PCA-384 + TQ4 20.9x 0.9906 96.0% 97.3% 0.6s PCA-512 + TQ4 15.8x 0.9949 96.3% 99.0% 0.6s

Recommendation (min recall >= 95%): PCA-384 + TQ4: 20.9x compression, 96.0% recall@10 ```

Integration Options

Feature Reference

A complete guide to every feature in TurboQuant Pro, the theory behind it, and when to use it.

Component map

Additional Components

• Streaming KV Cache (v0.3.0): Two-tier L1 hot / L2 cold cache for autoregressive generation.
• NATS Transport Codec (v0.3.0): Compressed wire format for NATS JetStream events (392 bytes vs 4096 bytes per embedding).
• vLLM Plugin (v0.5.0): TurboQuantKVManager multi-layer cache for vLLM integration.
• FAISS Integration (v0.5.0): TurboQuantFAISS wraps FAISS with auto PCA compression.
• Rust pgext (v0.5.0): Native PostgreSQL extension with tqvector type and <=> operator.
• Autotune CLI (v0.5.0): turboquant-pro autotune finds optimal compression in ~10 seconds.
• Model Weight Compression (v0.6.0-v0.7.0): SVD and activation-space PCA for LLM weight pruning.

---

Create the full pipeline: PCA-384 + TurboQuant 3-bit

pipeline = pca.with_quantizer(bits=3) # ~27x compression

FAISS Integration

Wrap FAISS indices with automatic PCA compression:

from turboquant_pro import PCAMatryoshka
from turboquant_pro.faiss_index import TurboQuantFAISS

pca = PCAMatryoshka(input_dim=1024, output_dim=384)
pca.fit(sample_embeddings)

index = TurboQuantFAISS(pca, index_type="ivf", n_lists=100)
index.add(corpus)  # Auto PCA-compressed
distances, ids = index.search(query, k=10)  # Auto PCA-rotated
print(index.stats())  # 2.7x smaller index

Supports Flat, IVF, and HNSW. Save/load indices to disk.

Native PostgreSQL Extension (Rust + CUDA)

The pgext/ directory contains a native PostgreSQL extension written in Rust (pgrx) that adds the tqvector data type directly to PostgreSQL — no Python needed.

-- Compress your entire table in one command
CREATE TABLE embeddings_tq AS
SELECT id, tq_compress(embedding::float4[], 3) AS tqv
FROM embeddings;

-- Search with cosine distance operator
SELECT id, tqv <=> tq_compress(query::float4[], 3) AS dist
FROM embeddings_tq ORDER BY dist LIMIT 10;

-- Check compression
SELECT tq_dim(tqv), tq_bits(tqv), tq_ratio(tqv) FROM embeddings_tq LIMIT 1;
-- 1024, 3, 10.6

Production benchmark (194K BGE-M3 1024-dim vectors on Atlas):

Build and install:

cd pgext
cargo install cargo-pgrx && cargo pgrx init --pg16 $(which pg_config)
cargo pgrx install --release
psql -c "CREATE EXTENSION tqvector;"

Optional GPU acceleration: cargo build --features gpu (requires CUDA 12.0+, cudarc).

See pgext/README.md for full API documentation.

PostgreSQL integration

tq.create_compressed_table(conn, "embeddings_compressed") tq.insert_compressed(conn, "embeddings_compressed", ids, embeddings) results = tq.search_compressed(conn, "embeddings_compressed", query, top_k=10) ```

Storage savings (1024-dim BGE-M3, 3-bit, no PCA truncation):

TurboQuant 3-bit alone compresses each vector from 4,096 to ~388 bytes (10.5x):

Components

vLLM KV Cache Plugin

Multi-layer KV cache manager with hot/cold tiering:

```python from turboquant_pro.vllm_plugin import TurboQuantKVManager

mgr = TurboQuantKVManager( n_layers=32, n_kv_heads=8, head_dim=128, bits=3, hot_window=512 )

Benchmarks vs SOTA (real data, all methods reranked identically)

At 32× compression, recall@10 on real LaBSE / multilingual-Gutenberg embeddings (RESULTS_labse_199k.md, RESULTS_gutenberg_1m.md):

turboquant-pro beats the 2024 binary-quantization SOTA (RaBitQ) at both operating points and ties OPQ, at 4–20× lower index build cost — and this holds at 1M scale (tq-pro 0.989 +rerank, tying OPQ). Fast search: the AVX2 ADC kernel (turboquant_pro/_adc/) reproduces this recall (0.9995 +rerank) at 3802 qps — 7.9× faster than naive flat-reconstruct and competitive with ScaNN — at 96 bytes, training-free (see docs/DESIGN_fast_adc.md). Full honest evaluation of every feature: COMPREHENSIVE_ANALYSIS.md.