Prerequisites

• Rust (2024 edition), CUDA Toolkit (nvcc, cuBLAS), CUDA-capable GPU
• NVIDIA driver R535 (CUDA 12.2) or newer; driver symbols resolve lazily at call time, so the cuda-12090 cudarc feature does not raise the driver floor
• The default build (Qwen3-4B / 8B) is pure Rust + CUDA — no Python at all
• Python 3 + Triton for qwen35-4b feature builds (build-time only — no Python at runtime)
• TileLang for deepseek-v4 feature builds (build-time only)
• deepseek-v4 / kimi-k2 EP paths additionally need NCCL ≥ 2.27 at runtime (ncclAlltoAll)

Qwen3.5 requires the feature-gated Triton AOT kernels (Python + Triton at build time)

uv venv && uv pip install triton export OPENINFER_TRITON_PYTHON=.venv/bin/python cargo run --release --features qwen35-4b -- --model-path models/Qwen3.5-4B

DeepSeek V4 Flash requires the feature-gated MP8 path and TileLang at build time

uv pip install "tilelang==0.1.9" export OPENINFER_TILELANG_PYTHON=.venv/bin/python cargo run --release --features deepseek-v4 -- --model-path models/DeepSeek-V4-Flash

Build & Run

```bash

Build & start server on port 8000 — no Python needed for the default Qwen3 build

export CUDA_HOME=/usr/local/cuda cargo run --release

> **Note**: The server CLI is in `openinfer-server`. Model crates such as `openinfer-qwen3`, `openinfer-qwen35-4b`, and `openinfer-deepseek-v4` contain model logic and diagnostics but are not server entrypoints. Use `cargo run --release` from the workspace root, or `cargo run --release -p openinfer-server -- --model-path <path>`.

Default Qwen3 build needs no Python

cargo build --release cargo run --release -p openinfer-server -- --model-path models/Qwen3-4B

Fresh-box dev setup

scripts/setup_dev.sh bootstraps a build environment on any fresh NVIDIA Ubuntu host: apt build deps + protobuf-compiler, uv, the rustup nightly pinned by rust-toolchain.toml, the vendored flashinfer/3rdparty/cccl submodule, then cargo build --release. CUDA is a prerequisite — it detects nvcc and fails loudly rather than installing a toolkit, so boot a CUDA image.

```bash bash scripts/setup_dev.sh

on a GPU whose arch the kernels don't target (e.g. V100 sm_70), compile for another:

OPENINFER_CUDA_SM=90 bash scripts/setup_dev.sh ```

To get the box itself, scripts/prime_devbox.sh provisions the cheapest match on Prime Intellect, has the box git-clone this repo over HTTPS, and runs setup_dev.sh — see the script header for one-time setup.

Quickstart

API

OpenAI-compatible /v1/completions endpoint.

Sampling and logprob support is model-dependent. Qwen models support the sampling controls above; the initial DeepSeek V4 path accepts greedy requests only and reports unsupported parameters through stop_reason.

Supported Models

Model type is auto-detected from config.json — just point --model-path at any supported model directory. Every model line is controlled by a cargo feature; only qwen3 is on by default, so the stock build serves Qwen3 with zero Python. Other lines require rebuilding openinfer-server with the matching --features ... flag before launch.

DeepSeek V4 support is intentionally narrower than the Qwen paths in the initial PR: it requires --features deepseek-v4, uses CUDA devices 0..7, serves greedy requests only, terminates unsupported logprobs and non-greedy sampling requests with an explicit stop_reason, and does not use CUDA Graph yet.

Accuracy and integration tests (need GPU + model weights)

OPENINFER_TEST_MODEL_PATH=models/Qwen3-4B cargo test --release -p openinfer-qwen3 --test hf_golden_gate OPENINFER_TEST_MODEL_PATH=models/Qwen3.5-4B cargo test --release -p openinfer-qwen35-4b --features qwen35-4b --test hf_golden_gate OPENINFER_TEST_MODEL_PATH=models/Qwen3.5-4B cargo test --release -p openinfer-qwen35-4b --features qwen35-4b --test e2e_scheduler OPENINFER_TEST_MODEL_PATH=models/DeepSeek-V4-Flash cargo test --release -p openinfer-deepseek-v4 --features deepseek-v4 --test mp8_manifest ```

Qwen3.5-4B vs current vLLM

Single RTX 5090 (32 GB), Qwen3.5-4B, BF16, TP1 — openinfer with the Qwen3.5 decode-tuning change, vLLM 0.23.0, both driven by vllm bench serve 0.23.0. Fixed random prompts, 64 measured requests, 2 warmups, text-only serving with prefix cache off on both engines. Full flags and caveats are in the Qwen3.5 benchmark report.

The decode-tuning change improves openinfer's own direct Qwen3.5 decode TPOT by about 2-3%. Against vLLM, prompt-len-1 decode is close, but vLLM still leads the 1024/256 decode and high-concurrency HTTP rows. TTFT rows are fixed-client timings because reported prompt-token totals differ on the longer prompts.