Pure Rust + CUDA LLM 是 AI Skill Hub 本期精选AI工具之一。综合评分 7.5 分，整体质量较高。我们推荐使用将其纳入你的 AI 工具库，帮助提升工作效率。
Pure Rust + CUDA LLM 是一款基于 Rust 开发的开源工具，专注于 Rust 等核心功能。作为 GitHub 开源项目，它拥有活跃的社区支持和持续的版本迭代，代码完全透明可审计，支持本地部署以保护数据隐私。无论是个人使用还是集成到企业工作流，都能提供稳定可靠的解决方案。
Pure Rust + CUDA LLM 是一款基于 Rust 开发的开源工具，专注于 Rust 等核心功能。作为 GitHub 开源项目，它拥有活跃的社区支持和持续的版本迭代，代码完全透明可审计，支持本地部署以保护数据隐私。无论是个人使用还是集成到企业工作流，都能提供稳定可靠的解决方案。
# 方式一：cargo install（推荐） cargo install pegainfer # 方式二：从源码编译 git clone https://github.com/xiaguan/pegainfer cd pegainfer cargo build --release # 二进制在 ./target/release/pegainfer
# 查看帮助 pegainfer --help # 基本运行 pegainfer [options] <input> # 详细使用说明请查阅文档 # https://github.com/xiaguan/pegainfer
# pegainfer 配置说明 # 查看配置选项 pegainfer --config-example > config.yml # 常见配置项 # output_dir: ./output # log_level: info # workers: 4 # 环境变量（覆盖配置文件） export PEGAINFER_CONFIG="/path/to/config.yml"
<p align="center"> <img src="logo.png" width="200" alt="pegainfer logo"> </p>
<p align="center"> Pure Rust + CUDA LLM inference engine. No PyTorch. No model framework runtime. </p>
<p align="center"> <a href="#performance">Performance</a> · <a href="#quickstart">Quickstart</a> · <a href="#supported-models">Models</a> · <a href="#api">API</a> · <a href="#architecture">Architecture</a> </p>
---
pegainfer is a from-scratch LLM inference engine written in ~9.6K lines of Rust, ~2.6K lines of CUDA, and ~1.4K lines of Triton GPU kernels. No PyTorch, no ONNX, no model framework runtime — just Rust plus CUDA, Triton AOT, and generated compatibility kernels.
The goal is to understand every layer of the inference stack by building it from the ground up, and to explore what a Rust-native inference engine can look like.
deepseek-v4 feature builds (build-time only)uv pip install "tilelang==0.1.9" export PEGAINFER_TILELANG_PYTHON=.venv/bin/python cargo run --release --features deepseek-v4 -- --model-path models/DeepSeek-V4-Flash
```bash
uv venv && source .venv/bin/activate uv pip install torch --index-url https://download.pytorch.org/whl/cu128
export CUDA_HOME=/usr/local/cuda export PEGAINFER_TRITON_PYTHON=.venv/bin/python cargo run --release
> **Note**: The server CLI is in `pegainfer-server`. Model crates such as `pegainfer-qwen3-4b`, `pegainfer-qwen35-4b`, and `pegainfer-deepseek-v4` contain model logic and diagnostics but are not server entrypoints. Use `cargo run --release` from the workspace root, or `cargo run --release -p pegainfer-server -- --model-path <path>`.
OpenAI-compatible /v1/completions endpoint.
| Field | Type | Default | Description |
|---|---|---|---|
prompt | string | (required) | Input text |
max_tokens | int | 128 | Maximum tokens to generate |
temperature | float | 0.0 | Sampling temperature (0 = greedy) |
top_k | int | 50 | Top-k sampling |
top_p | float | 1.0 | Nucleus sampling threshold |
stream | bool | false | Enable SSE streaming |
Sampling and logprob support is model-dependent. Qwen models support the sampling controls above; the initial DeepSeek V4 path accepts greedy requests only and reports unsupported parameters through stop_reason.
| Model | Architecture | Params | Status |
|---|---|---|---|
| [Qwen3-4B](https://huggingface.co/Qwen/Qwen3-4B) | Full attention (GQA) | 4B | Greedy + sampling |
| [Qwen3-8B](https://huggingface.co/Qwen/Qwen3-8B) | Full attention (GQA) | 8B | Greedy + sampling |
| [Qwen3.5-4B](https://huggingface.co/Qwen/Qwen3.5-4B) | Hybrid (24 linear + 8 full attention) | 4B | Greedy + sampling |
| [DeepSeek-V4-Flash](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash) | MoE + sparse attention, MP8 checkpoint | 671B total / 37B active | Initial greedy, feature-gated, 8-GPU MP8 |
Model type is auto-detected from config.json — just point --model-path at any supported model directory.
DeepSeek V4 support is intentionally narrower than the Qwen paths in the initial PR: it requires --features deepseek-v4, uses CUDA devices 0..7, serves greedy requests only, terminates unsupported logprobs and non-greedy sampling requests with an explicit stop_reason, and does not use CUDA Graph yet.
pegainfer是一个高质量的开源AI工具，提供了Pure Rust + CUDA LLM inference engine，支持高效、安全、可扩展的LLM inference engine开发和部署。然而，pegainfer的文档和社区支持还需要进一步改善。
该工具未明确声明开源协议，商业使用前请联系原作者确认授权范围，避免侵权风险。
AI Skill Hub 为第三方内容聚合平台，本页面信息基于公开数据整理，不对工具功能和质量作任何法律背书。
建议在沙箱或测试环境中充分验证后，再部署至生产环境，并做好必要的安全评估。
经综合评估，Pure Rust + CUDA LLM 在AI工具赛道中表现稳健，质量良好。如果你已有明确的使用需求，可以直接上手体验；如果还在评估阶段，建议对比同类工具后再做决策。
| 原始名称 | pegainfer |
| 原始描述 | 开源AI工具：Pure Rust + CUDA LLM inference engine。⭐336 · Rust |
| Topics | Rust |
| GitHub | https://github.com/xiaguan/pegainfer |
| 语言 | Rust |
收录时间：2026-05-22 · 更新时间：2026-05-22 · License：未公布 · AI Skill Hub 不对第三方内容的准确性作法律背书。