开源推理引擎

Build from source

Use the Rust toolchain pinned in rust-toolchain.toml, a CUDA Toolkit with nvcc and cuBLAS, and a compatible NVIDIA driver. The default Qwen3 build needs no Python. Its driver floor is R545 / CUDA 12.3; newer toolkits and model-specific kernels can require a newer driver.

From the repository root, with the checkpoint downloaded:

export CUDA_HOME=/usr/local/cuda
cargo run --release -- --model-path models/Qwen3-4B

Always use --release for GPU builds. The server entrypoint is pegainfer-server; model crates contain the model implementation and diagnostics.

<details> <summary><strong>Feature builds and environment options</strong></summary>

Qwen3.5 uses Triton AOT kernels, requiring Python and Triton at build time:

uv venv
uv pip install triton
export PEGAINFER_TRITON_PYTHON=.venv/bin/python
cargo run --release --features qwen35 -- --model-path models/Qwen3.5-4B

Other model lines have their own hardware and build requirements; follow the model guides below. Run cargo run --release -- --help for the compiled-in CLI.

</details>

<details> <summary><strong>Windows source builds</strong></summary>

```powershell $env:CUDA_PATH = "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.x" cargo run --release -p pegainfer-server -- --model-path models/Qwen3-4B

Qwen3.5 additionally needs Triton at build time

uv venv .venv --python 3.12 uv pip install "triton-windows<3.7" $env:PEGAINFER_TRITON_PYTHON = ".venv\Scripts\python.exe" cargo run --release --features qwen35 -- --model-path models/Qwen3.5-4B ```

</details>

Quickstart

API

Point an OpenAI-compatible client at http://localhost:8000/v1. Both /v1/completions and /v1/chat/completions support streaming.

curl -s http://localhost:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"models/Qwen3-4B","prompt":"The capital of France is","max_tokens":32}'

curl -N http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"models/Qwen3-4B","messages":[{"role":"user","content":"Write a haiku about Rust."}],"max_tokens":64,"stream":true}'

The model guides document supported sampling fields and model-specific limits. Documentation covers the serving interface; metrics and dashboards cover observability.

Supported Models

Only qwen3 is enabled by default, including in the prebuilt binary. Build other lines with --features <feature>. At launch, --model-path selects a checkpoint and its config.json identifies the model family.

Capabilities and maturity differ by model. Quantized formats are model-specific; the Qwen paths listed here use BF16. DeepSeek-V2-Lite's retained correctness and benchmark gates are documented separately from production readiness.