Mistral-RS

Install

Linux/macOS:

curl --proto '=https' --tlsv1.2 -sSf https://raw.githubusercontent.com/EricLBuehler/mistral.rs/master/install.sh | sh

Windows (PowerShell):

irm https://raw.githubusercontent.com/EricLBuehler/mistral.rs/master/install.ps1 | iex

Manual installation & other platforms

Docker

For quick containerized deployment:

docker pull ghcr.io/ericlbuehler/mistral.rs:latest
docker run --gpus all -p 1234:1234 ghcr.io/ericlbuehler/mistral.rs:latest \
  serve -m Qwen/Qwen3-4B

Docker images

For production use, we recommend installing the CLI directly for maximum flexibility.

Quick Start

Auto-tune for your hardware and emit a config file

mistralrs tune -m Qwen/Qwen3-4B --emit-config config.toml

Run using the generated config

mistralrs from-config -f config.toml

Start an API server with the built-in web UI

mistralrs serve -m google/gemma-4-E4B-it ```

For the server command, visit http://localhost:1234/ui for the web chat interface.

Python SDK

pip install mistralrs  # or mistralrs-cuda, mistralrs-metal, mistralrs-mkl, mistralrs-accelerate

from mistralrs import Runner, Which, ChatCompletionRequest

runner = Runner(
    which=Which.Plain(model_id="Qwen/Qwen3-4B"),
    in_situ_quant="4",
)

res = runner.send_chat_completion_request(
    ChatCompletionRequest(
        model="default",
        messages=[{"role": "user", "content": "Hello!"}],
        max_tokens=256,
    )
)
print(res.choices[0].message.content)

Python SDK | Installation | Examples | Cookbook

Rust SDK

cargo add mistralrs

use anyhow::Result;
use mistralrs::{IsqType, TextMessageRole, TextMessages, MultimodalModelBuilder};

#[tokio::main]
async fn main() -> Result<()> {
    let model = MultimodalModelBuilder::new("google/gemma-4-E4B-it")
        .with_isq(IsqType::Q4K)
        .with_logging()
        .build()
        .await?;

    let messages = TextMessages::new().add_message(
        TextMessageRole::User,
        "Hello!",
    );

    let response = model.send_chat_request(messages).await?;

    println!("{:?}", response.choices[0].message.content);

    Ok(())
}

API Docs | Crate | Examples

Supported Models

<details> <summary><b>Text Models</b></summary>

- Granite 4.0 - SmolLM 3 - DeepSeek V3 - GPT-OSS - DeepSeek V2 - Qwen 3 Next - Qwen 3 MoE - Phi 3.5 MoE - Qwen 3 - GLM 4 - GLM-4.7-Flash - GLM-4.7 (MoE) - Gemma 2 - Qwen 2 - Starcoder 2 - Phi 3 - Mixtral - Phi 2 - Gemma - Llama - Mistral </details>

<details> <summary><b>Multimodal Models</b></summary>

- Qwen 3.5 - Qwen 3.5 MoE - Qwen 3-VL - Qwen 3-VL MoE - Gemma 3n - Llama 4 - Gemma 3 - Mistral 3 - Phi 4 multimodal - Qwen 2.5-VL - MiniCPM-O - Llama 3.2 Vision - Qwen 2-VL - Idefics 3 - Idefics 2 - LLaVA Next - LLaVA - Phi 3V </details>

<details> <summary><b>Speech Models</b></summary>

- Voxtral (ASR/speech-to-text) - Dia </details>

<details> <summary><b>Image Generation Models</b></summary>

- FLUX </details>

<details> <summary><b>Embedding Models</b></summary>

- Embedding Gemma - Qwen 3 Embedding </details>

Request a new model | Full compatibility tables