Mooncake AI技能包
经 AI Skill Hub 精选评估，Mooncake AI技能包 获评「强烈推荐」。已获得 5.3k 颗 GitHub Star，这款AI工具在功能完整性、社区活跃度和易用性方面表现出色，AI 评分 8.2 分，适合有一定技术背景的用户使用。
📚 深度解析
**为什么要使用开源工具而非商业 SaaS？**
对于个人开发者和有隐私需求的用户，本地部署的开源工具意味着数据不离本机，不受第三方服务商的数据政策约束。同时，开源工具通常没有使用次数限制和月度费用，一次安装即可长期使用，对于高频使用场景的总拥有成本（TCO）远低于订阅制商业工具。
**安装与环境准备**
Mooncake AI技能包 依赖 C++ 运行环境。建议通过 pyenv（Python）或 nvm（Node.js）管理 C++ 版本，避免全局环境污染。对于新手用户，推荐先创建虚拟环境（python -m venv venv && source venv/bin/activate），再安装依赖，这样即使出现问题也可以随时删除虚拟环境重新开始，不影响系统稳定性。
**社区与维护**
GitHub Issue 和 Discussion 是获取帮助的最快渠道。在提问前建议先检查 Closed Issues（已关闭的问题），大多数常见问题都已有解答。遇到 Bug 时，提供 pip list 的输出、完整错误堆栈和最小可复现示例，能显著提高开发者响应速度。AI Skill Hub 将持续追踪 Mooncake AI技能包 的版本更新，及时通知重要功能变化。
📋 工具概览
Kimi的开源推理服务平台，采用C++实现高性能LLM推理。支持KV缓存优化、RDMA通信和推理服务分解架构，适合部署和优化大型语言模型推理，面向开发者和企业级用户。
Mooncake AI技能包 是一款基于 C++ 开发的开源工具，专注于 LLM推理、高性能服务、分布式推理 等核心功能。作为 GitHub 开源项目，它拥有活跃的社区支持和持续的版本迭代，代码完全透明可审计，支持本地部署以保护数据隐私。无论是个人使用还是集成到企业工作流，都能提供稳定可靠的解决方案。
📖 中文文档
Kimi的开源推理服务平台，采用C++实现高性能LLM推理。支持KV缓存优化、RDMA通信和推理服务分解架构，适合部署和优化大型语言模型推理，面向开发者和企业级用户。
Mooncake AI技能包 是一款基于 C++ 开发的开源工具，专注于 LLM推理、高性能服务、分布式推理 等核心功能。作为 GitHub 开源项目，它拥有活跃的社区支持和持续的版本迭代，代码完全透明可审计，支持本地部署以保护数据隐私。无论是个人使用还是集成到企业工作流，都能提供稳定可靠的解决方案。
- 开源免费，支持本地部署，数据完全自主可控
- 活跃的 GitHub 开源社区，持续迭代更新
- 提供详细文档和使用示例，新手友好
- 支持自定义配置，灵活适配不同使用环境
- 可作为基础组件集成进现有技术栈或进行二次开发
- 本地部署运行，保护数据隐私，满足合规要求
- 自定义集成到现有系统，扩展技术栈能力
- 作为开源基础组件进行商业化二次开发
# 克隆仓库 git clone https://github.com/kvcache-ai/Mooncake cd Mooncake # 查看安装说明 cat README.md # 按 README 完成环境依赖安装后即可使用
- 访问 GitHub 仓库页面
- 按照 README 文档完成依赖安装
- 根据系统环境完成初始化配置
- 参考官方示例或文档开始使用
- 遇到问题可在 GitHub Issues 中查找解答
# 查看帮助 mooncake --help # 基本运行 mooncake [options] <input> # 详细使用说明请查阅文档 # https://github.com/kvcache-ai/Mooncake
# mooncake 配置说明 # 查看配置选项 mooncake --config-example > config.yml # 常见配置项 # output_dir: ./output # log_level: info # workers: 4 # 环境变量（覆盖配置文件） export MOONCAKE_CONFIG="/path/to/config.yml"
简介
A KVCache-centric Disaggregated Architecture for LLM Serving
Paper | Slides | Traces | Documentation | Blog | Slack
<br />
Mooncake is the serving platform for <a href="https://kimi.ai/"><img src="image/kimi.png" alt="icon" style="height: 16px; vertical-align: middle;"> Kimi</a>, a leading LLM service provided by <a href="https://www.moonshot.cn/"><img src="image/moonshot.jpg" alt="icon" style="height: 16px; vertical-align: middle;"> Moonshot AI</a>. Now both the Transfer Engine and Mooncake Store are open-sourced! This repository also hosts its technical report and the open-sourced traces.
🔄 Updates
- May 7, 2026: 🚀 vLLM officially features Mooncake Store — a deep dive into how Mooncake's distributed KVCache engine supercharges vLLM inference with high-throughput, memory-efficient, cross-instance KV cache sharing!
- Apr 29, 2026: SGLang introduces RDMA-based P2P weight transfer for large-scale distributed RL using Mooncake TransferEngine, achieving 7x faster weight updates for the 1T-parameter Kimi-K2 model (53s → 7.2s) with zero-copy RDMA transfer across thousands of GPUs.
- Mar 19, 2026: TorchSpec: Speculative Decoding Training at Scale is open sourced, using Mooncake to decouple inference and training via efficient hidden states management.
- Mar 5, 2026: LightX2V now supports disaggregated deployment based on Mooncake, enabling encoder/transformer service decoupling with Mooncake Transfer Engine for high-performance cross-device and cross-machine data transfer.
- Feb 25, 2026: SGLang merged Encoder Global Cache Manager, introducing a Mooncake-powered global multimodal embedding cache that enables cross-instance sharing of ViT embeddings to avoid redundant GPU computation.
<details> <summary>More</summary>
- Feb 24, 2026: vLLM-Omni introduces disaggregated inference connectors with support for both
MooncakeStoreConnectorandMooncakeTransferEngineConnectorfor multi-node omni-modality pipelines. - Feb 12, 2026: Mooncake Joins PyTorch Ecosystem We are thrilled to announce that Mooncake has officially joined the PyTorch Ecosystem!
- Jan 28, 2026: FlexKV, a distributed KV store and cache system from Tencent and NVIDIA in collaboration with the community, now supports distributed KVCache reuse with the Mooncake Transfer Engine.
- Dec 27, 2025: Collaboration with ROLL! Check out the paper here.
- Dec 23, 2025: SGLang introduces Encode-Prefill-Decode (EPD) Disaggregation with Mooncake as a transfer backend. This integration allows decoupling compute-intensive multimodal encoders (e.g., Vision Transformers) from language model nodes, utilizing Mooncake's RDMA engine for zero-copy transfer of large multimodal embeddings.
- Dec 19, 2025: Mooncake Transfer Engine has been integrated into TensorRT LLM for KVCache transfer in PD-disaggregated inference.
- Dec 19, 2025: Mooncake Transfer Engine has been directly integrated into vLLM v1 as a KV Connector in PD-disaggregated setups.
- Nov 07, 2025: RBG + SGLang HiCache + Mooncake, a role-based out-of-the-box solution for cloud native deployment, which is elastic, scalable, and high-performance.
- Sept 18, 2025: Mooncake Store empowers vLLM Ascend by serving as the distributed KV cache pool backend.
- Sept 10, 2025: SGLang officially supports Mooncake Store as a hierarchical KV caching storage backend. The integration extends RadixAttention with multi-tier KV cache storage across device, host, and remote storage layers.
- Sept 10, 2025: The official & high-performance version of Mooncake P2P Store is open-sourced as checkpoint-engine. It has been successfully applied in K1.5 and K2 production training, updating Kimi-K2 model (1T parameters) across thousands of GPUs in ~20s.
- Aug 23, 2025: xLLM high-performance inference engine builds hybrid KV cache management based on Mooncake, supporting global KV cache management with intelligent offloading and prefetching.
- Aug 18, 2025: vLLM-Ascend integrates Mooncake Transfer Engine for KV cache register and disaggregate prefill, enabling efficient distributed inference on Ascend NPUs.
- Jul 20, 2025: Mooncake powers the deployment of Kimi K2 on 128 H200 GPUs with PD disaggregation and large-scale expert parallelism, achieving 224k tokens/sec prefill throughput and 288k tokens/sec decode throughput.
- Jun 20, 2025: Mooncake becomes a PD disaggregation backend for LMDeploy.
- May 9, 2025: NIXL officially supports Mooncake Transfer Engine as a backend plugin.
- May 8, 2025: Mooncake x LMCache unite to pioneer KVCache-centric LLM serving system.
- May 5, 2025: Supported by Mooncake Team, SGLang release <a href="https://lmsys.org/blog/2025-05-05-large-scale-ep/" target="_blank">guidance</a> to deploy DeepSeek with PD Disaggregation on 96 H100 GPUs.
- Apr 22, 2025: LMCache officially supports Mooncake Store as a <a href="https://blog.lmcache.ai/2025-04-22-tencent/" target="_blank">remote connector</a>.
- Apr 10, 2025: SGLang officially supports Mooncake Transfer Engine for disaggregated prefilling and KV cache transfer.
- Mar 7, 2025: We open-sourced the Mooncake Store, a distributed KVCache based on Transfer Engine. vLLM's xPyD disaggregated prefilling & decoding based on Mooncake Store will be released soon.
- Feb 25, 2025: Mooncake receives the Best Paper Award at FAST 2025!
- Feb 21, 2025: The updated <a href="FAST25-release/traces" target="_blank">traces</a> used in our FAST'25 paper have been released.
- Dec 16, 2024: vLLM officially supports Mooncake Transfer Engine for disaggregated prefilling and KV cache transfer.
- Nov 28, 2024: We open-sourced the Transfer Engine, the central component of Mooncake. We also provide two demonstrations of Transfer Engine: a P2P Store and vLLM integration.
- July 9, 2024: We open-sourced the trace as a <a href="https://github.com/kvcache-ai/Mooncake/blob/main/FAST25-release/arxiv-trace/mooncake_trace.jsonl" target="_blank">JSONL file</a>.
- June 27, 2024: We present a series of Chinese blogs with more discussions on <a href="https://zhuanlan.zhihu.com/p/705754254">zhihu 1</a>, <a href="https://zhuanlan.zhihu.com/p/705910725">2</a>, <a href="https://zhuanlan.zhihu.com/p/706204757">3</a>, <a href="https://zhuanlan.zhihu.com/p/707997501">4</a>, <a href="https://zhuanlan.zhihu.com/p/9461861451">5</a>, <a href="https://zhuanlan.zhihu.com/p/1939988652114580803">6</a>, <a href="https://zhuanlan.zhihu.com/p/1959366095443064318">7</a>.
- June 26, 2024: Initial technical report release.
</details>
🎉 Overview
Mooncake features a KVCache-centric disaggregated architecture that separates the prefill and decoding clusters. It also leverages the underutilized CPU, DRAM, and SSD resources of the GPU cluster to implement a disaggregated KVCache pool.
The core of Mooncake is its KVCache-centric scheduler, which balances maximizing overall effective throughput while meeting latency-related Service Level Objectives (SLOs). Unlike traditional studies that assume all requests will be processed, Mooncake faces challenges in highly overloaded scenarios. To mitigate these, we developed a prediction-based early rejection policy. Experiments show that Mooncake excels in long-context scenarios. Compared to the baseline method, Mooncake can achieve up to a 525% increase in throughput in certain simulated scenarios while adhering to SLOs. Under real workloads, Mooncake’s innovative architecture enables <a href="https://kimi.ai/">Kimi</a> to handle 75% more requests.
🔥 Show Cases
Build From Source
For the default source build, use the automatic dependency script and standard CMake flow:
git clone https://github.com/kvcache-ai/Mooncake.git
cd Mooncake
sudo bash dependencies.sh
mkdir build
cd build
cmake ..
make -j
sudo make install # optional, make it ready to be used by vLLM/SGLangFor custom accelerator backends, Docker deployment, NVMe-oF, EFA, CXL, Redis / HTTP metadata, Rust bindings, or other advanced build options, see the Build Guide.
SGLang Integration ([Guide](https://kvcache-ai.github.io/Mooncake/getting_started/examples/sglang-integration/index.html))
Mooncake is deeply integrated into SGLang as a high-performance communication and storage backend. These integrations enable efficient KV cache transfer in PD-disaggregated serving, scalable multi-level KV caching through HiCache, fault-tolerant expert-parallel inference, high-performance multimodal pipeline data movement, and fast RDMA-based weight synchronization for large-scale RL training. Together, Mooncake and SGLang provide a production-oriented foundation for building elastic, high-throughput, and resource-efficient LLM and multimodal serving systems.
<details> <summary>Details</summary>
- PD Disaggregated Serving: SGLang officially supports Mooncake Transfer Engine as a backend for disaggregated serving and KV cache transfer, enabling prefill and decode workers to exchange KV cache data efficiently across devices and machines.
- Hierarchical KV Caching: Mooncake Store serves as an external storage backend in SGLang's HiCache system, extending RadixAttention with multi-level KV cache storage across device, host, and remote storage layers.
- Elastic Expert Parallel: Mooncake's collective communication backend and expert parallel kernels are integrated into SGLang to enable fault-tolerant expert parallel inference (Elastic EP).
- Cloud-Native SGLang HiCache Deployment with RBG: The RBG + SGLang HiCache + Mooncake integration provides a role-based, out-of-the-box cloud-native deployment solution that is elastic, scalable, and optimized for high-performance inference workloads.
- Encode-Prefill-Decode Disaggregation for Multimodal Serving: SGLang introduces Encode-Prefill-Decode disaggregation with Mooncake as a transfer backend. This enables compute-intensive multimodal encoders, such as Vision Transformers, to be decoupled from language model workers while transferring large embeddings efficiently through Mooncake’s RDMA-based engine.
- SGLang-Omni Multi-Stage Pipeline Data Transfer: SGLang-Omni integrates Mooncake as a relay backend for efficient cross-stage tensor and blob transfer in multimodal serving pipelines. This enables high-performance data movement between heterogeneous components such as thinker, talker, codec, and vocoder stages.
- RDMA-Based P2P Weight Transfer for Distributed RL: SGLang adopts Mooncake TransferEngine for RDMA-based peer-to-peer weight transfer in large-scale distributed reinforcement learning. This enables zero-copy weight updates across thousands of GPUs and significantly accelerates synchronization for trillion-parameter models.
</details>
vLLM Integration ([Guide](https://kvcache-ai.github.io/Mooncake/getting_started/examples/vllm-integration/index.html))
Mooncake integrates with vLLM to accelerate large language model serving through high-performance KV cache transfer and distributed KV cache storage. The integration supports both disaggregated prefill-decode serving and cross-instance KV cache sharing, helping vLLM deployments reduce TTFT, improve cache reuse, and scale more efficiently across multi-node inference clusters.
<details> <summary>Details</summary>
- Disaggregated prefill-decode serving: Mooncake enables vLLM to split prefill and decode workloads across different nodes. Through MooncakeConnector, vLLM transfers KV cache blocks from prefill workers to decode workers using Mooncake’s high-performance transfer engine, allowing prefill and decode resources to scale independently while keeping cross-node KV transfer overhead low.
- Distributed KV cache pooling and sharing: Mooncake Store extends vLLM from isolated per-instance KV caches to a shared, cluster-level KV cache pool. Through MooncakeStoreConnector, multiple vLLM instances can store, retrieve, and reuse KV cache blocks based on hash-based prefix caching, reducing redundant prefill computation and improving cache efficiency for workloads with repeated prefixes, especially agentic and multi-turn serving scenarios.
- vLLM-Omni stage communication: Mooncake also integrates with vLLM-Omni through
MooncakeTransferEngineConnectorandMooncakeStoreConnector, enabling efficient cross-node data exchange between vLLM-Omni stages.
</details>
🖥️ Supported Hardware
Mooncake supports hardware backends across accelerator vendors, cloud fabrics, and standard datacenter interconnects.
The following hardware partners and cloud platforms are supported by the Mooncake, covering GPUs, specialized AI accelerators, and cloud-native interconnects:
| <img src="image/partners/nvidia_logo.png" width="120" alt="NVIDIA"/> | <img src="image/partners/huawei_logo.png" width="120" alt="Huawei"/> | <img src="image/partners/amd_logo.png" width="120" alt="AMD"/> | <img src="image/hardwares/cambricon_logo.png" width="120" alt="Cambricon"/> | <img src="image/partners/moore_thread_logo.jpg" width="120" alt="Moore Threads"/> | <img src="image/partners/aws-logo.png" width="120" alt="AWS"/> |
|---|---|---|---|---|---|
| <img src="image/hardwares/MetaX_logo.png" width="120" alt="MetaX"/> | <img src="image/hardwares/T-Head_logo.png" width="120" alt="T-Head"/> | <img src="image/partners/aliyun_logo.png" width="120" alt="Alibaba Cloud"/> | <img src="image/partners/sunrise_logo.png" width="120" alt="Sunrise"/> | <img src="image/partners/hygon_logo.png" width="120" alt="Hygon"/> |
For complete protocol behavior, SDK requirements, and vendor-specific configuration, see the supported protocols, build guide, and Transfer Engine design docs.
🚀 Quick Start
Use Python package
The simplest way to use Mooncake Transfer Engine is using pip:
For CUDA-enabled systems:
- CUDA < 13.0
pip install mooncake-transfer-enginepip install mooncake-transfer-engine-cuda13For non-CUDA systems:
pip install mooncake-transfer-engine-non-cudaFor NPU systems:
pip install mooncake-transfer-engine-npu[!IMPORTANT] - The CUDA version (mooncake-transfer-engine) includes Mooncake-EP and GPU topology detection, requiring CUDA 12.1+. - The non-CUDA version (mooncake-transfer-engine-non-cuda) is for environments without CUDA dependencies, but it still needs system runtime libraries such aslibcurl4,libibverbs1,rdma-core,librdmacm1,libnuma1, andliburing2on Ubuntu. In a fresh environment, runsudo apt-get updatebefore installing them. - MLU support is currently available through source builds with-DUSE_MLU=ON; there is no dedicated prebuilt MLU wheel yet. - If users encounter problems such as missinglib*.so, first install the corresponding system runtime libraries. If the issue persists, uninstall the package and build the binaries manually.
高性能LLM推理平台，采用业界先进优化技术。代码质量优秀、更新活跃，是探索推理加速的优选方案。
- 构建企业知识库 / RAG 检索应用的团队
- 生产部署优先使用 Docker Compose 隔离依赖，并挂载 volume 持久化数据
- API key 直接提交到 git 仓库（请用 .env 并加入 .gitignore）
- 容器内无法访问宿主机 localhost — 使用 host.docker.internal
- Docker：Mooncake 提供官方镜像，docker compose up 一键启动
- 云端托管：可放在 Vercel / Railway / Fly.io 等 PaaS 平台
⚡ 核心功能
- 开源免费，支持本地部署，数据完全自主可控
- 活跃的 GitHub 开源社区，持续迭代更新
- 提供详细文档和使用示例，新手友好
- 支持自定义配置，灵活适配不同使用环境
- 可作为基础组件集成进现有技术栈或进行二次开发
- 构建企业知识库 / RAG 检索应用的团队
- 生产部署优先使用 Docker Compose 隔离依赖，并挂载 volume 持久化数据
- API key 直接提交到 git 仓库（请用 .env 并加入 .gitignore）
- 容器内无法访问宿主机 localhost — 使用 host.docker.internal
👥 适合人群
🎯 使用场景
- 本地部署运行，保护数据隐私，满足合规要求
- 自定义集成到现有系统，扩展技术栈能力
- 作为开源基础组件进行商业化二次开发
⚖️ 优点与不足
- +GitHub 5.3k Star，社区高度认可
- +Apache-2.0 协议，可免费商用
- +完全开源免费，无授权费用
- +本地部署，数据完全自主可控
- +开发者社区支持，遇问题可查可问
- −安装和初始配置可能需要一定技术基础
- −功能完整性通常不如成熟商业产品
- −技术支持主要依赖开源社区，响应速度不稳定
AI Skill Hub 为第三方内容聚合平台，本页面信息基于公开数据整理，不对工具功能和质量作任何法律背书。
建议在沙箱或测试环境中充分验证后，再部署至生产环境，并做好必要的安全评估。
✅ Apache 2.0 — 宽松开源协议，可商用，需保留版权声明和 NOTICE 文件，含专利授权条款。
🔗 相关工具推荐
❓ 常见问题 FAQ
AI Skill Hub 点评：Mooncake AI技能包 的核心功能完整，质量优秀。对于AI 技术爱好者来说，这是一个值得纳入个人工具库的选择。建议先在非生产环境试用，再逐步推广。
| 原始名称 | Mooncake |
| 原始描述 | 开源AI工具：Mooncake is the serving platform for Kimi, a leading LLM service provided by Moo。⭐5.3k · C++ |
| Topics | LLM推理高性能服务分布式推理KV缓存优化RDMA网络 |
| GitHub | https://github.com/kvcache-ai/Mooncake |
| License | Apache-2.0 |
| 语言 | C++ |
收录时间：2026-05-16 · 更新时间：2026-05-19 · License：Apache-2.0 · AI Skill Hub 不对第三方内容的准确性作法律背书。