MLX-VLM

With trust remote code enabled (required for some models)

mlx_vlm.server --trust-remote-code

Require bearer authentication for API endpoints

mlx_vlm.server --api-key <secret-token>

Installation

The easiest way to get started is to install the mlx-vlm package using pip:

pip install -U mlx-vlm

The Gradio chat UI needs an extra dependency that is not part of the base install:

pip install -U 'mlx-vlm[ui]'

Quote the package name so that shells which expand square brackets, such as zsh, do not treat [ui] as a glob pattern.

Usage

Configure thinking defaults at startup

mlx_vlm.server --model Qwen/Qwen3.5-4B \ --enable-thinking \ --thinking-budget 512 \ --thinking-start-token "<think>" \ --thinking-end-token "</think>"

Optional diagnostics: inspect the automatically inferred cache groups.

print(apc.coordinator(model).plan.describe())

document = Path("long_document.txt").read_text()

try: # First request computes the full prefix and stores reusable K/V blocks. prompt1 = apply_chat_template( processor, model.config, prompt=f"{document}\n\nSummarize the key decisions.", num_images=0, ) for _ in stream_generate( model, processor, prompt1, max_tokens=128, temperature=0.0, apc_manager=apc ): pass

# Second request shares the same document prefix and only prefills the suffix. prompt2 = apply_chat_template( processor, model.config, prompt=f"{document}\n\nList the open engineering risks.", num_images=0, ) for chunk in stream_generate( model, processor, prompt2, max_tokens=128, temperature=0.0, apc_manager=apc ): print(chunk.text, end="", flush=True)

print(apc.stats_snapshot()) finally: apc.close()

To compare cold, warm-memory, warm-disk, and disk-eviction behavior with a
model, use the same direct API path:

from mlx_vlm import load, stream_generate from mlx_vlm.apc import APCManager, DiskBlockStore from mlx_vlm.prompt_utils import apply_chat_template

model_id = "Qwen/Qwen3-VL-4B-Instruct" contexts = [8000, 20000, 50000, 100000] disk_cap_gb = 0 # 0 means uncapped shard_max_blocks = 256 context_sweep_max_tokens = 1 # one token is enough to measure prefill reuse

test_prompt_tokens = 8000 fill_prompts = 80 eviction_disk_cap_gb = 3.0

os.environ["APC_DISK_SHARD_MAX_BLOCKS"] = str(shard_max_blocks)

model, processor = load(model_id) tokenizer = processor.tokenizer if hasattr(processor, "tokenizer") else processor

def disk_cap_bytes(gb: float): return None if gb <= 0 else int(gb * (1 << 30))

def make_context(target_tokens: int, seed: int = 0) -> str: line = ( f"Document {seed}: APC benchmark content with deterministic facts, " "dates, identifiers, and repeated technical notes.\n" ) line_tokens = max(1, len(tokenizer.encode(line, add_special_tokens=False))) text = line * max(1, target_tokens // line_tokens) while len(tokenizer.encode(text, add_special_tokens=False)) < target_tokens: text += line return text

def make_prompt(context: str, question: str) -> str: return apply_chat_template( processor, model.config, prompt=f"{context}\n\n{question}", num_images=0, )

def run_once(apc: APCManager, context: str, question: str, max_tokens: int = 32): prompt = make_prompt(context, question) apc.reset_stats()

last = None output = [] start = time.perf_counter() for chunk in stream_generate( model, processor, prompt, max_tokens=max_tokens, temperature=0.0, apc_manager=apc, ): output.append(chunk.text) last = chunk

if last is None: raise RuntimeError("generation returned no chunks")

return { "wall_s": time.perf_counter() - start, "prompt_tokens": last.prompt_tokens, "prompt_tps": last.prompt_tps, "generation_tps": last.generation_tps, "apc": apc.stats_snapshot(), "text": "".join(output).strip(), }

def print_result(label: str, result: dict) -> None: stats = result["apc"] print( f"{label:<12} " f"prompt_tokens={result['prompt_tokens']:>7} " f"prompt_tps={result['prompt_tps']:>8.1f} " f"gen_tps={result['generation_tps']:>7.1f} " f"matched={stats.get('matched_tokens', 0):>7} " f"disk_hits={stats.get('disk_hits', 0):>5} " f"disk_evictions={stats.get('disk_evictions', 0):>5}" )

def open_apc(cache_root: Path, namespace: str, disk_gb: float) -> APCManager: disk = DiskBlockStore( cache_root, namespace=namespace, max_bytes=disk_cap_bytes(disk_gb), ) return APCManager(num_blocks=4096, block_size=16, disk=disk)

def run_context_sweep() -> None: print("cold / warm-memory / warm-disk") with tempfile.TemporaryDirectory() as tmp: cache_root = Path(tmp) for target_tokens in contexts: context = make_context(target_tokens) namespace = f"{model_id}-context-{target_tokens}" apc = open_apc(cache_root, namespace, disk_cap_gb) try: print(f"\ncontext ~= {target_tokens} text tokens") print_result( "cold", run_once( apc, context, "Summarize the key decisions.", max_tokens=context_sweep_max_tokens, ), ) print_result( "warm-memory", run_once( apc, context, "List the open engineering risks.", max_tokens=context_sweep_max_tokens, ), ) finally: # Closing waits for queued disk writes before reopening the disk tier. apc.close()

apc = open_apc(cache_root, namespace, disk_cap_gb) try: print_result( "warm-disk", run_once( apc, context, "Extract the implementation timeline.", max_tokens=context_sweep_max_tokens, ), ) finally: apc.close()

def run_disk_eviction_workload() -> None: print("\ndisk eviction workload") with tempfile.TemporaryDirectory() as tmp: cache_root = Path(tmp) namespace = f"{model_id}-eviction" test_context = make_context(test_prompt_tokens, seed=0)

apc = open_apc(cache_root, namespace, eviction_disk_cap_gb) try: print_result( "seed", run_once(apc, test_context, "Summarize the retained test prefix."), ) finally: apc.close()

apc = open_apc(cache_root, namespace, eviction_disk_cap_gb) try: for i in range(fill_prompts): fill_context = make_context(test_prompt_tokens, seed=i + 1) run_once( apc, fill_context, f"Summarize filler document {i + 1}.", max_tokens=1, ) if (i + 1) % 10 == 0: stats = apc.stats_snapshot() print( f"filled={i + 1:>3} " f"disk_gb={stats.get('disk_bytes', 0) / (1 << 30):.2f} " f"disk_evictions={stats.get('disk_evictions', 0)}" ) finally: apc.close()

apc = open_apc(cache_root, namespace, eviction_disk_cap_gb) try: print_result( "post-fill", run_once( apc, test_context, "Check whether the retained test prefix still restores.", ), ) finally: apc.close()

run_context_sweep() run_disk_eviction_workload()

#### Server

Enable in-memory APC for the server with environment variables:

APC works with KV-cache quantization (`--kv-bits`):

APC persists caches to disk by default when enabled, under
`$MLX_VLM_CACHE_HOME/apc` (or `~/.cache/mlx-vlm/apc`), with a 20 GiB cap per
model namespace. Customize the location and cap:

Repeated requests with the same long prefix will hit APC automatically:

Use the same `X-APC-Tenant` value for requests that may share cached prefixes. Use different tenant values to isolate cache entries between users or workspaces.

Inspect and reset APC state:

`restored_tokens` counts prefix tokens successfully materialized for generation
across block and exact caches, in memory or on disk. A lookup alone does not
increase it, nor does a failed batch restore. A successful restore remains counted
if the request later fails or is cancelled.
The previous `served_tokens` block-storage count is now `stored_tokens`;
`token_hit_rate` retains its existing calculation,
`matched_tokens / (matched_tokens + stored_tokens)`.

Configure APC on a running server with `PATCH /v1/settings`. Use
`GET /v1/settings` to discover supported settings and read their current values:

curl -X PATCH http://localhost:8080/v1/settings \ -H 'Content-Type: application/json' \ -d '{ "apc_enabled": true, "apc_disk_enabled": true, "apc_memory_max_gb": 2, "apc_disk_max_gb": 20, "apc_checkpoint_interval_tokens": 1024 }'

The endpoint also accepts `apc_memory_reserve_gb`, `apc_disk_queue_max_gb`,
`apc_disk_path`, `apc_disk_shard_max_blocks`, `apc_block_size`, `apc_num_blocks`,
`apc_checkpoint_entries`, and `apc_checkpoint_guard_tokens`. When
`MLX_VLM_SERVER_API_KEY` is configured, include `Authorization: Bearer <key>`.

Settings apply on the next text-generation request through the existing model
reload path; the server process stays running. Reloading clears resident caches
and retires the previous disk writer after its generation worker finishes.
Existing disk files remain available. Updating APC options while APC is disabled
stages them for the next enable. The response reports `applied`, `rejected`,
`reload_kinds`, and the resulting settings; invalid sizes are rejected without
changing those values. Unchanged settings do not trigger a reload.

Use `null` for automatic memory budgets or the default disk path/cap. Zero has
specific meanings: `apc_memory_max_gb: 0` retains caches only on disk,
`apc_disk_max_gb: 0` removes the disk cap, and `apc_disk_queue_max_gb: 0` writes
synchronously. An empty `apc_disk_path` or `apc_disk_enabled: false` disables
persistence. PATCH merges with current settings; the optional
`{"op": "replace", "values": {...}}` form first restores startup environment
defaults. `/v1/cache/stats` shows the active manager after the next request.

Common APC environment variables:

| Variable | Default | Description |
|----------|---------|-------------|
| `APC_ENABLED` | `0` | Set to `1` to enable APC |
| `APC_NUM_BLOCKS` | `2048` | Number of in-memory APC blocks |
| `APC_BLOCK_SIZE` | `16` | Tokens per APC block |
| `APC_CHECKPOINT_ENTRIES` | `2` | In-memory checkpoint entries; also bounds snapshots captured per hybrid prompt (disk-only mode captures two) |
| `APC_CHECKPOINT_GUARD_TOKENS` | `1` | Tokens retained after a reusable hybrid checkpoint boundary; the default preserves the normal final-token prefill boundary |
| `APC_CHECKPOINT_INTERVAL_TOKENS` | `2048` | Spacing of intermediate hybrid checkpoints, rounded up to a multiple of `APC_BLOCK_SIZE`; `0` keeps only the final checkpoint |
| `APC_MEMORY_MAX_GB` | auto | Resident block and checkpoint budget in GiB: 10% of Metal's recommended working set, capped at 8 GiB; `0` retains caches only on disk |
| `APC_MEMORY_RESERVE_GB` | auto | Additional memory headroom in GiB: 10% of Metal's recommended working set, at least 1 GiB |
| `APC_DISK_ENABLED` | `1` | Set to `0` to disable disk persistence |
| `MLX_VLM_CACHE_HOME` | `~/.cache/mlx-vlm` | Base cache directory; APC uses its `apc` subdirectory unless `APC_DISK_PATH` is set |
| `APC_DISK_PATH` | cache directory above | Directory for persistent disk shards; an empty value disables persistence |
| `APC_DISK_MAX_GB` | `20` | Disk cap per model namespace in GiB; `0` means uncapped |
| `APC_DISK_QUEUE_MAX_GB` | `1` | Maximum tensor bytes held by queued disk writes in GiB; larger writes run synchronously; `0` makes all writes synchronous |
| `APC_DISK_SHARD_MAX_BLOCKS` | `256` | Max blocks per disk segment shard |
| `APC_MAX_POOL_TENSORS` | `450000` | Stops adding memory blocks before the Metal resource limit; disk writes continue |
| `APC_LAYER_MAJOR_MEMORY_MIN_TOKENS` | `50000` | Store long warm-memory prefixes as compact layer-major snapshots instead of per-block tensors |
| `APC_HASH` | `fast` | Set to `sha256` for a stable cryptographic hash |
| `APC_TRACE` | unset | Set to `1` for greppable store/reject/self-check log lines |

Custom cache layouts can opt in without APC model-name checks by implementing `prefix_cache_snapshot()` and `prefix_cache_restore(snapshot)`. In-tree dense, sliding-window, recurrent, composite, VLM, and Omni cache layouts are detected automatically. APC works with `--kv-bits` (including TurboQuant): the live KV cache stays quantized; pageable APC K/V blocks are stored as dequantized float K/V, so block-pool size does not shrink with quant.
When APC is enabled on the server, a non-fatal layout self-check runs at model load.

Requests with a shared document and different questions can reuse their common
prefix. Dense K/V caches, including compact layer-major memory snapshots, match
complete blocks before the first differing token. Hybrid, recurrent and sliding
window caches must restore a state captured before that divergence: their final
state cannot be rolled back by slicing K/V tensors.

Hybrid prefill now captures intermediate checkpoints as well as the final guard
checkpoint, in streaming, continuous batching and DiffusionGemma generation.
With the defaults, each prompt stores its latest 2,048-token boundary before the
final checkpoint and the final checkpoint itself. For example, a 30,000-token
document followed by 20 instruction tokens can reuse 28,672 tokens when the
instructions change. The same checkpoints persist across server restarts when
the disk tier is enabled.

Captures are bounded by `APC_CHECKPOINT_ENTRIES` to avoid copying the growing
hybrid cache at every prefill chunk. More entries retain more earlier boundaries;
a smaller interval gives finer reuse near the end. Reuse still requires a
retained boundary before the divergence. Short prompts below the interval,
divergence before the earliest retained checkpoint, and evicted checkpoints can
miss. Media checkpoints include all media tokens so the remaining suffix is text.
Changing checkpoint boundaries can change floating-point execution shapes, as
with other chunked or cached prefill paths.

The resident byte budget includes exact snapshots as well as pageable blocks.
Before embeddings and prefill, APC drains pending disk writes and evicts idle
checkpoints and blocks in LRU order within each tier. Admission uses current
Metal allocations, available system RAM, and the incoming prompt's estimated
cache growth, based on observed bytes per token. Leased blocks remain valid for
active requests. Snapshots that exceed the budget are written directly to disk
without making another resident copy, and disk restores stay out of the memory
LRU when promotion would exceed the budget. Disk pressure may therefore trade
latency for lower memory use.

These controls limit APC's memory overhead; model weights and an individual
request must still fit in memory. Tune the reserve for models with larger
prefill or vision temporaries. `/v1/cache/stats` reports resident bytes,
the memory budget, prefill reserve, memory evictions, and pending disk bytes.

#### KV Cache Quantization

Reduce KV cache memory during continuous batching with `--kv-bits`. Both uniform quantization and TurboQuant are supported. Compatible with Automatic Prefix Caching (`APC_ENABLED=1`).

Command Line Interface (CLI)

Generate output from a model using the CLI:

```sh

Server (FastAPI)

Start the server: ```sh mlx_vlm.server --port 8080