TurboQuant-MLX

Requirements

• macOS with Apple Silicon (M1/M2/M3/M4)
• Python 3.10+
• 64 GB unified memory recommended for 20B+ models

The Metal kernels are JIT-compiled by MLX at first use, so no Xcode / CMake toolchain is required to install the package.

Install

pip install turboquant-mlx-full

The package is published as turboquant-mlx-full on PyPI, but importable as turboquant_mlx (without the -full suffix) — this matches the original project name and the examples in the Medium articles.

import turboquant_mlx
from turboquant_mlx.layers import TurboQuantKVCache, convert_cache_to_turboquant

Install from source (for development)

git clone https://github.com/manjunathshiva/turboquant-mlx.git
cd turboquant-mlx
pip install -e .

For evaluation utilities (perplexity benchmarking), also install the optional dependencies:

pip install "turboquant-mlx-full[eval]"

1. Build per-layer cache (correct types for hybrid models)

cache = make_prompt_cache(model)

Quick Start

Programmatic usage

```python from turboquant_mlx.layers import convert_cache_to_turboquant from mlx_lm.models.cache import make_prompt_cache

Generate (runs mlx-vlm's sampler — block-diffusion denoising for DiffusionGemma)

python -m turboquant_mlx.generate_vlm \ --model ./diffusiongemma-26B-A4B-it-tq3-g32 \ --prompt "Write a short paragraph about the ocean." --max-tokens 256 ```

CLI Options

python -m turboquant_mlx.convert --help

Options:
  --hf-path TEXT       HuggingFace model path or local path (required)
  --mlx-path TEXT      Output directory (default: mlx_model)
  --bits {2,3,4}       Quantization bit-width (default: 3)
  --group-size {32,64,128}  Elements per quantization group (default: 64)
  --rotation TEXT      Rotation method: hadamard, blockwise_hadamard, none
  --use-qjl           Enable 1-bit QJL residual correction (+1 bit overhead)
  --dtype TEXT         Model dtype before quantization: float16, bfloat16

5. Serve a TurboQuant model over an OpenAI-compatible API

turboquant-serve wraps mlx_lm.server and patches its loader so any TurboQuant model (quantization.mode = "turboquant" in config.json) loads through the PolarQuant path. Non-TurboQuant models pass through unchanged, so this is a drop-in replacement for mlx_lm.server.

```bash

CLI flags

turboquant-generate exposes the same controls:

turboquant-generate --model ./model-tq3 --prompt "..." \
    --kv-k-bits 8 --kv-v-bits 3 \
    --kv-min-tokens 128 \
    --kv-group-size 64

Side-by-side comparison harness (4 configs in one run)

python -m turboquant_mlx.benchmarks.demo_kv_v02 \ --model ./gpt-oss-120b-tq3 \ --prompt "Why is the sky blue?" \ --max-tokens 1024 --temp 0.7 --top-p 0.9 --repetition-penalty 1.1 ```