Features Performance Demo Install Downloads Docs Full Docs
⚡ Pure Rust · Production Ready

# Blazing-fast LLM Inference in Pure Rust

No PyTorch. No Python runtime. Just fast, portable, production-ready inference.

terminal

    $ curl -sSL https://guoqingbao.github.io/xinfer/install.sh | bash

`npm install -g xinfer-ai`
Get Started ▶ Watch Demo View on GitHub →

Features

## Built for Speed & Simplicity

Everything you need for production LLM inference — without the Python baggage.

🦀

### Zero Dependencies

Pure Rust backend — no PyTorch, no CUDA Python bindings, no Python runtime.

⚡

### Blazing Fast

Flash Attention, FlashInfer, CUDA Graphs, continuous batching. Up to 197+ tok/s.

🪶

### Tiny Footprint

Core scheduling & attention logic in under 5,000 lines of Rust.

🌍

### Cross-Platform

CUDA on Linux, Metal on macOS. Same binary, same API everywhere.

🏭

### Production Ready

OpenAI & Anthropic APIs, built-in Web UI, MCP tool calling, structured outputs.

🗜️

### KV Compression

TurboQuant 2–4 bit KV cache extends context up to 4.3× with minimal quality loss.

Performance

## Real-World Benchmarks

Tested on V100, A100, Hopper H800, and RTX 5090.

Model | Format | Size | Hardware | Speed | Note
--- | --- | --- | --- | --- | ---
Qwen3-30B-A3B | NVFP4 | 30B MoE | RTX 5090 (SM120) | 0 tok/s | HW NVFP4
Gemma4-26B-A4B | NVFP4 | 26B MoE | RTX 5090 (SM120) | 0 tok/s | HW NVFP4
Qwen3.6-35B-A3B | FP8 | 35B MoE | H800 (SM90) | 0 tok/s | HW FP8
DeepSeek-R1-Qwen3-8B | Q4_K_M | 8B | A100 (SM80) | 0 tok/s | GGUF
Llama-3.1-8B | ISQ Q4K | 8B | A100 (SM80) | 0 tok/s | SW quant
MiniMax-M2.5 | NVFP4 | 229B MoE | H800 ×2 (SM90) | 0 tok/s | SW NVFP4 (no HW)
Qwen3-30B-A3B | NVFP4 | 30B MoE | V100 (SM70) | 0 tok/s | SW FP4
Qwen3.6-27B | FP8 | 27B Dense | H800 (SM90) | 0 tok/s | HW FP8

* HW = hardware-accelerated. Hopper (SM90) supports HW FP8 but not HW NVFP4. Blackwell SM100 (B100/B200); SM120 (RTX 5090/RTX PRO 6000). Both support HW FP8 and NVFP4.

Models

## Supported Models & Formats

From 3B to 397B — dense, MoE, and multimodal architectures.

LLaMA 4 Qwen3 Qwen3.5 Qwen3.6 DeepSeek V3 DeepSeek R1 Gemma 3 Gemma 4 GLM4 GLM4.7 Flash Phi4 Mistral 3 VL MiniMax M2.5

Supported Formats

Safetensors GGUF FP8 NVFP4 MXFP4 GPTQ AWQ ISQ

KV Cache

## TurboQuant Compression

Extend context length up to 4.3× with --kvcache-dtype.

Mode (--kvcache-dtype) | Compression | Quality | GPU
--- | --- | --- | ---
default (BF16) | 1× | Baseline | All
fp8 | 2× | Near-lossless | SM70+ / M1+
turbo8 | 2.6× | High quality | SM70+ / M1+
turbo4 | 3.7× | Best balance | SM70+ / M1+
turbo3 | 4.7× | Max compression | SM70+ / M1+

Demo

## See It in Action

Watch xInfer serve a 35B model with real-time inference and tool calling.

Quick Start

## Install & Run

Get up and running in minutes.

bash

    curl -sSL https://guoqingbao.github.io/xinfer/install.sh | bash

bash

    npm install -g xinfer-ai

bash

    # Prerequisites: Rust, CUDA Toolkit (or Metal Xcode CLI)
    curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
    sudo apt-get install -y git build-essential

    export XINFER_REPO="https://github.com/guoqingbao/xinfer"
    # macOS/Metal: replace features with `metal`
    # SM70/SM75 (V100): remove `flashinfer` and `cutlass`
    cargo install --git $XINFER_REPO xinfer --features cuda,nccl,flashinfer,cutlass

bash

    # Build Python wheel from source
    pip install maturin maturin[patchelf]

    # FlashInfer backend (SM80+)
    ./build.sh --release --features cuda,nccl,flashinfer,cutlass,python

    # macOS Metal
    maturin build --release --features metal,python

    # Install the wheel
    pip install target/wheels/xinfer*.whl --force-reinstall

bash

    # SM70/SM75: remove `flashinfer` and `cutlass`
    ./build_docker.sh "cuda,nccl,flashinfer,cutlass"

### Run Examples

bash

    # HuggingFace model with Web UI
    xinfer --m Qwen/Qwen3.6-27B-FP8 --kvcache-dtype turbo4 --ui-server

    # Multi-GPU with local model
    xinfer --w /path/to/model --d 0,1 --ui-server

    # Python mode
    python3 -m xinfer.server --m Qwen/Qwen3.6-27B-FP8 --ui-server

Downloads

## Pre-built Packages

Pre-compiled binaries and pip wheels for every GPU architecture.

### Binary Downloads

### Pre-built Python Wheels (pip)

Documentation

## Learn More

Guides, API references, and integration tutorials.

Get Started Docker Performance Prefix Cache Multimodal Embedding MCP Integration Guided Decoding Rust Crate Add a Model Tokenizer API Tool Parsing