Mooncake AI技能包
经 AI Skill Hub 精选评估，Mooncake AI技能包 获评「强烈推荐」。已获得 5.3k 颗 GitHub Star，这款AI工具在功能完整性、社区活跃度和易用性方面表现出色，AI 评分 8.2 分，适合有一定技术背景的用户使用。
📚 深度解析
**为什么要使用开源工具而非商业 SaaS？**
对于个人开发者和有隐私需求的用户，本地部署的开源工具意味着数据不离本机，不受第三方服务商的数据政策约束。同时，开源工具通常没有使用次数限制和月度费用，一次安装即可长期使用，对于高频使用场景的总拥有成本（TCO）远低于订阅制商业工具。
**安装与环境准备**
Mooncake AI技能包 依赖 C++ 运行环境。建议通过 pyenv（Python）或 nvm（Node.js）管理 C++ 版本，避免全局环境污染。对于新手用户，推荐先创建虚拟环境（python -m venv venv && source venv/bin/activate），再安装依赖，这样即使出现问题也可以随时删除虚拟环境重新开始，不影响系统稳定性。
**社区与维护**
GitHub Issue 和 Discussion 是获取帮助的最快渠道。在提问前建议先检查 Closed Issues（已关闭的问题），大多数常见问题都已有解答。遇到 Bug 时，提供 pip list 的输出、完整错误堆栈和最小可复现示例，能显著提高开发者响应速度。AI Skill Hub 将持续追踪 Mooncake AI技能包 的版本更新，及时通知重要功能变化。
📋 工具概览
Kimi的开源推理服务平台，采用C++实现高性能LLM推理。支持KV缓存优化、RDMA通信和推理服务分解架构，适合部署和优化大型语言模型推理，面向开发者和企业级用户。
Mooncake AI技能包 是一款基于 C++ 开发的开源工具，专注于 LLM推理、高性能服务、分布式推理 等核心功能。作为 GitHub 开源项目，它拥有活跃的社区支持和持续的版本迭代，代码完全透明可审计，支持本地部署以保护数据隐私。无论是个人使用还是集成到企业工作流，都能提供稳定可靠的解决方案。
📖 中文文档
Kimi的开源推理服务平台，采用C++实现高性能LLM推理。支持KV缓存优化、RDMA通信和推理服务分解架构，适合部署和优化大型语言模型推理，面向开发者和企业级用户。
Mooncake AI技能包 是一款基于 C++ 开发的开源工具，专注于 LLM推理、高性能服务、分布式推理 等核心功能。作为 GitHub 开源项目，它拥有活跃的社区支持和持续的版本迭代，代码完全透明可审计，支持本地部署以保护数据隐私。无论是个人使用还是集成到企业工作流，都能提供稳定可靠的解决方案。
- 开源免费，支持本地部署，数据完全自主可控
- 活跃的 GitHub 开源社区，持续迭代更新
- 提供详细文档和使用示例，新手友好
- 支持自定义配置，灵活适配不同使用环境
- 可作为基础组件集成进现有技术栈或进行二次开发
- 本地部署运行，保护数据隐私，满足合规要求
- 自定义集成到现有系统，扩展技术栈能力
- 作为开源基础组件进行商业化二次开发
# 克隆仓库 git clone https://github.com/kvcache-ai/Mooncake cd Mooncake # 查看安装说明 cat README.md # 按 README 完成环境依赖安装后即可使用
- 访问 GitHub 仓库页面
- 按照 README 文档完成依赖安装
- 根据系统环境完成初始化配置
- 参考官方示例或文档开始使用
- 遇到问题可在 GitHub Issues 中查找解答
# 查看帮助 mooncake --help # 基本运行 mooncake [options] <input> # 详细使用说明请查阅文档 # https://github.com/kvcache-ai/Mooncake
# mooncake 配置说明 # 查看配置选项 mooncake --config-example > config.yml # 常见配置项 # output_dir: ./output # log_level: info # workers: 4 # 环境变量（覆盖配置文件） export MOONCAKE_CONFIG="/path/to/config.yml"
简介
A KVCache-centric Disaggregated Architecture for LLM Serving
Paper | Slides | Traces | Documentation | Blog | Slack
Mooncake is the serving platform for <a href="https://kimi.ai/"><img src="image/kimi.png" alt="icon" style="height: 16px; vertical-align: middle;"> Kimi</a>, a leading LLM service provided by <a href="https://www.moonshot.cn/"><img src="image/moonshot.jpg" alt="icon" style="height: 16px; vertical-align: middle;"> Moonshot AI</a>. Under real workloads, Mooncake’s innovative architecture enables Kimi to handle 75% more requests while adhering to SLOs.
🔄 Updates
- Aug 20, 2026: Mooncake is integrated into Miles as a rollout data-transfer backend for the fragmented, heterogeneous data moving between rollout and training in disaggregated RL. Blogs: KVCache.AI, lmsys.
- Aug 17, 2026: Mooncake is integrated into Speculators as a distributed backend for multi-node online training, efficiently moves hidden-state between vLLM inference workers and Speculators trainers through RDMA, eliminating the need for massive hidden-state storage in offline training. Benchmark on GB300 NVL72.
- May 7, 2026: 🚀 vLLM officially features Mooncake Store — a deep dive into how Mooncake's distributed KVCache engine supercharges vLLM inference with high-throughput, memory-efficient, cross-instance KV cache sharing!
- Apr 29, 2026: SGLang introduces RDMA-based P2P weight transfer for large-scale distributed RL using Mooncake TransferEngine, achieving 7x faster weight updates for the 1T-parameter Kimi-K2 model (53s → 7.2s) with zero-copy RDMA transfer across thousands of GPUs.
- Mar 19, 2026: TorchSpec: Speculative Decoding Training at Scale is open sourced, using Mooncake to decouple inference and training via efficient hidden states management.
<details> <summary>More</summary>
- Mar 5, 2026: LightX2V now supports disaggregated deployment based on Mooncake, enabling encoder/transformer service decoupling with Mooncake Transfer Engine for high-performance cross-device and cross-machine data transfer. Details in blog.
- Feb 25, 2026: SGLang merged Encoder Global Cache Manager, introducing a Mooncake-powered global multimodal embedding cache that enables cross-instance sharing of ViT embeddings to avoid redundant GPU computation.
- Feb 24, 2026: vLLM-Omni introduces disaggregated inference connectors with support for both
MooncakeStoreConnectorandMooncakeTransferEngineConnectorfor multi-node omni-modality pipelines. - Feb 12, 2026: Mooncake Joins PyTorch Ecosystem We are thrilled to announce that Mooncake has officially joined the PyTorch Ecosystem!
- Jan 28, 2026: FlexKV, a distributed KV store and cache system from Tencent and NVIDIA in collaboration with the community, now supports distributed KVCache reuse with the Mooncake Transfer Engine.
- Dec 27, 2025: Collaboration with ROLL! Check out the paper here.
- Dec 23, 2025: SGLang introduces Encode-Prefill-Decode (EPD) Disaggregation with Mooncake as a transfer backend. This integration allows decoupling compute-intensive multimodal encoders (e.g., Vision Transformers) from language model nodes, utilizing Mooncake's RDMA engine for zero-copy transfer of large multimodal embeddings.
- Dec 19, 2025: Mooncake Transfer Engine has been integrated into TensorRT LLM for KVCache transfer in PD-disaggregated inference.
- Dec 19, 2025: Mooncake Transfer Engine has been directly integrated into vLLM v1 as a KV Connector in PD-disaggregated setups.
- Nov 07, 2025: RBG + SGLang HiCache + Mooncake, a role-based out-of-the-box solution for cloud native deployment, which is elastic, scalable, and high-performance.
- Sept 18, 2025: Mooncake Store empowers vLLM Ascend by serving as the distributed KV cache pool backend.
- Sept 10, 2025: SGLang officially supports Mooncake Store as a hierarchical KV caching storage backend. The integration extends RadixAttention with multi-tier KV cache storage across device, host, and remote storage layers.
- Sept 10, 2025: The official & high-performance version of Mooncake P2P Store is open-sourced as checkpoint-engine. It has been successfully applied in K1.5 and K2 production training, updating Kimi-K2 model (1T parameters) across thousands of GPUs in ~20s.
- Aug 23, 2025: xLLM high-performance inference engine builds hybrid KV cache management based on Mooncake, supporting global KV cache management with intelligent offloading and prefetching.
- Aug 18, 2025: vLLM-Ascend integrates Mooncake Transfer Engine for KV cache register and disaggregate prefill, enabling efficient distributed inference on Ascend NPUs.
- Jul 20, 2025: Mooncake powers the deployment of Kimi K2 on 128 H200 GPUs with PD disaggregation and large-scale expert parallelism, achieving 224k tokens/sec prefill throughput and 288k tokens/sec decode throughput.
- Jun 20, 2025: Mooncake becomes a PD disaggregation backend for LMDeploy.
- May 9, 2025: NIXL officially supports Mooncake Transfer Engine as a backend plugin.
- May 8, 2025: Mooncake x LMCache unite to pioneer KVCache-centric LLM serving system.
- May 5, 2025: Supported by Mooncake Team, SGLang release <a href="https://lmsys.org/blog/2025-05-05-large-scale-ep/" target="_blank">guidance</a> to deploy DeepSeek with PD Disaggregation on 96 H100 GPUs.
- Apr 22, 2025: LMCache officially supports Mooncake Store as a <a href="https://blog.lmcache.ai/2025-04-22-tencent/" target="_blank">remote connector</a>.
- Apr 10, 2025: SGLang officially supports Mooncake Transfer Engine for disaggregated prefilling and KV cache transfer.
- Mar 7, 2025: We open-sourced the Mooncake Store, a distributed KVCache based on Transfer Engine. vLLM's xPyD disaggregated prefilling & decoding based on Mooncake Store will be released soon.
- Feb 25, 2025: Mooncake receives the Best Paper Award at FAST 2025!
- Feb 21, 2025: The updated <a href="FAST25-release/traces" target="_blank">traces</a> used in our FAST'25 paper have been released.
- Dec 16, 2024: vLLM officially supports Mooncake Transfer Engine for disaggregated prefilling and KV cache transfer.
- Nov 28, 2024: We open-sourced the Transfer Engine, the central component of Mooncake. We also provide two demonstrations of Transfer Engine: a P2P Store and vLLM integration.
- July 9, 2024: We open-sourced the trace as a <a href="https://github.com/kvcache-ai/Mooncake/blob/main/FAST25-release/arxiv-trace/mooncake_trace.jsonl" target="_blank">JSONL file</a>.
- June 27, 2024: We present a series of Chinese blogs with more discussions on <a href="https://zhuanlan.zhihu.com/p/705754254">zhihu 1</a>, <a href="https://zhuanlan.zhihu.com/p/705910725">2</a>, <a href="https://zhuanlan.zhihu.com/p/706204757">3</a>, <a href="https://zhuanlan.zhihu.com/p/707997501">4</a>, <a href="https://zhuanlan.zhihu.com/p/9461861451">5</a>, <a href="https://zhuanlan.zhihu.com/p/1939988652114580803">6</a>, <a href="https://zhuanlan.zhihu.com/p/1959366095443064318">7</a>.
- June 26, 2024: Initial technical report release.
</details>
🎉 Overview
Mooncake is an infrastructure project for large-scale LLM inference and training. It features a KV cache-centric disaggregated architecture that separates prefill and decode clusters, while leveraging otherwise underutilized CPU, DRAM, and SSD resources in GPU clusters to build a disaggregated KV cache pool.
Mooncake includes a high-performance Transfer Engine for low-latency data movement across heterogeneous networks and accelerators; Mooncake Store for distributed KV cache and model-weight management; and Mooncake EP & PG for elastic MoE serving. Deeply integrated with ecosystems such as SGLang and vLLM, Mooncake helps LLM systems improve cache reuse, reduce serving latency, and scale efficiently across multi-node clusters.
🔥 Show Cases
SGLang Integration ([Guide](https://kvcache-ai.github.io/Mooncake/deployment/integrations/sglang/index.html))
Mooncake is deeply integrated into SGLang as a high-performance communication and storage backend. These integrations enable efficient KV cache transfer in PD-disaggregated serving, scalable multi-level KV caching through HiCache, fault-tolerant expert-parallel inference, high-performance multimodal pipeline data movement, and fast RDMA-based weight synchronization for large-scale RL training. Together, Mooncake and SGLang provide a production-oriented foundation for building elastic, high-throughput, and resource-efficient LLM and multimodal serving systems.
<details> <summary>Details</summary>
- PD Disaggregated Serving: SGLang officially supports Mooncake Transfer Engine as a backend for disaggregated serving and KV cache transfer, enabling prefill and decode workers to exchange KV cache data efficiently across devices and machines.
- Hierarchical KV Caching: Mooncake Store serves as an external storage backend in SGLang's HiCache system, extending RadixAttention with multi-level KV cache storage across device, host, and remote storage layers.
- Elastic Expert Parallel: Mooncake's collective communication backend and expert parallel kernels are integrated into SGLang to enable fault-tolerant expert parallel inference (Elastic EP).
- Cloud-Native SGLang HiCache Deployment with RBG: The RBG + SGLang HiCache + Mooncake integration provides a role-based, out-of-the-box cloud-native deployment solution that is elastic, scalable, and optimized for high-performance inference workloads.
- Encode-Prefill-Decode Disaggregation for Multimodal Serving: SGLang introduces Encode-Prefill-Decode disaggregation with Mooncake as a transfer backend. This enables compute-intensive multimodal encoders, such as Vision Transformers, to be decoupled from language model workers while transferring large embeddings efficiently through Mooncake’s RDMA-based engine.
- SGLang-Omni Multi-Stage Pipeline Data Transfer: SGLang-Omni integrates Mooncake as a relay backend for efficient cross-stage tensor and blob transfer in multimodal serving pipelines. This enables high-performance data movement between heterogeneous components such as thinker, talker, codec, and vocoder stages.
- RDMA-Based P2P Weight Transfer for Distributed RL: SGLang adopts Mooncake TransferEngine for RDMA-based peer-to-peer weight transfer in large-scale distributed reinforcement learning. This enables zero-copy weight updates across thousands of GPUs and significantly accelerates synchronization for trillion-parameter models.
</details>
vLLM Integration ([Guide](https://kvcache-ai.github.io/Mooncake/deployment/integrations/vllm/index.html))
Mooncake integrates with vLLM to accelerate large language model serving through high-performance KV cache transfer and distributed KV cache storage. The integration supports both disaggregated prefill-decode serving and cross-instance KV cache sharing, helping vLLM deployments reduce TTFT, improve cache reuse, and scale more efficiently across multi-node inference clusters.
<details> <summary>Details</summary>
- Disaggregated prefill-decode serving: Mooncake enables vLLM to split prefill and decode workloads across different nodes. Through MooncakeConnector, vLLM transfers KV cache blocks from prefill workers to decode workers using Mooncake’s high-performance transfer engine, allowing prefill and decode resources to scale independently while keeping cross-node KV transfer overhead low.
- Distributed KV cache pooling and sharing: Mooncake Store extends vLLM from isolated per-instance KV caches to a shared, cluster-level KV cache pool. Through MooncakeStoreConnector, multiple vLLM instances can store, retrieve, and reuse KV cache blocks based on hash-based prefix caching, reducing redundant prefill computation and improving cache efficiency for workloads with repeated prefixes, especially agentic and multi-turn serving scenarios.
- vLLM-Omni stage communication: Mooncake also integrates with vLLM-Omni through
MooncakeTransferEngineConnectorandMooncakeStoreConnector, enabling efficient cross-node data exchange between vLLM-Omni stages.
</details>
🖥️ Supported Hardware
Mooncake supports hardware backends across accelerator vendors, cloud fabrics, and standard datacenter interconnects, as listed below. See the supported protocols and Transfer Engine design docs for details.
| <img src="image/partners/nvidia_logo.png" width="120" alt="NVIDIA"/> | <img src="image/partners/huawei_logo.png" width="120" alt="Huawei"/> | <img src="image/partners/amd_logo.png" width="120" alt="AMD"/> | <img src="image/hardwares/cambricon_logo.png" width="120" alt="Cambricon"/> | <img src="image/partners/moore_thread_logo.jpg" width="120" alt="Moore Threads"/> | <img src="image/partners/aws-logo.png" width="120" alt="AWS"/> |
|---|---|---|---|---|---|
| <img src="image/hardwares/MetaX_logo.png" width="120" alt="MetaX"/> | <img src="image/hardwares/T-Head_logo.png" width="120" alt="T-Head"/> | <img src="image/partners/aliyun_logo.png" width="120" alt="Alibaba Cloud"/> | <img src="image/partners/sunrise_logo.png" width="120" alt="Sunrise"/> | <img src="image/partners/hygon_logo.png" width="120" alt="Hygon"/> | <img src="image/hardwares/biren_logo.png" width="120" alt="Biren Technology"/> |
🚀 Getting Started
Install Mooncake using pip. The mooncake-transfer-engine package includes Mooncake Transfer Engine, Mooncake Store, Mooncake EP and PG:
- CUDA < 13.0
pip install mooncake-transfer-enginepip install mooncake-transfer-engine-cuda13In addition to CUDA, Mooncake also supports other accelerator backends, along with flexible installation and deployment options. See the guides below for details:
高性能LLM推理平台，采用业界先进优化技术。代码质量优秀、更新活跃，是探索推理加速的优选方案。
- 构建企业知识库 / RAG 检索应用的团队
- 生产部署优先使用 Docker Compose 隔离依赖，并挂载 volume 持久化数据
- API key 直接提交到 git 仓库（请用 .env 并加入 .gitignore）
- 容器内无法访问宿主机 localhost — 使用 host.docker.internal
- Docker：Mooncake 提供官方镜像，docker compose up 一键启动
- 云端托管：可放在 Vercel / Railway / Fly.io 等 PaaS 平台
⚡ 核心功能
- 开源免费，支持本地部署，数据完全自主可控
- 活跃的 GitHub 开源社区，持续迭代更新
- 提供详细文档和使用示例，新手友好
- 支持自定义配置，灵活适配不同使用环境
- 可作为基础组件集成进现有技术栈或进行二次开发
- 构建企业知识库 / RAG 检索应用的团队
- 生产部署优先使用 Docker Compose 隔离依赖，并挂载 volume 持久化数据
- API key 直接提交到 git 仓库（请用 .env 并加入 .gitignore）
- 容器内无法访问宿主机 localhost — 使用 host.docker.internal
👥 适合人群
🎯 使用场景
- 本地部署运行，保护数据隐私，满足合规要求
- 自定义集成到现有系统，扩展技术栈能力
- 作为开源基础组件进行商业化二次开发
⚖️ 优点与不足
- +GitHub 5.3k Star，社区高度认可
- +Apache-2.0 协议，可免费商用
- +完全开源免费，无授权费用
- +本地部署，数据完全自主可控
- +开发者社区支持，遇问题可查可问
- −安装和初始配置可能需要一定技术基础
- −功能完整性通常不如成熟商业产品
- −技术支持主要依赖开源社区，响应速度不稳定
AI Skill Hub 为第三方内容聚合平台，本页面信息基于公开数据整理，不对工具功能和质量作任何法律背书。
建议在沙箱或测试环境中充分验证后，再部署至生产环境，并做好必要的安全评估。
✅ Apache 2.0 — 宽松开源协议，可商用，需保留版权声明和 NOTICE 文件，含专利授权条款。
🔗 相关工具推荐
❓ 常见问题 FAQ
AI Skill Hub 点评：Mooncake AI技能包 的核心功能完整，质量优秀。对于AI 技术爱好者来说，这是一个值得纳入个人工具库的选择。建议先在非生产环境试用，再逐步推广。
| 原始名称 | Mooncake |
| 原始描述 | 开源AI工具：Mooncake is the serving platform for Kimi, a leading LLM service provided by Moo。⭐5.3k · C++ |
| Topics | LLM推理高性能服务分布式推理KV缓存优化RDMA网络 |
| GitHub | https://github.com/kvcache-ai/Mooncake |
| License | Apache-2.0 |
| 语言 | C++ |
收录时间：2026-05-16 · 更新时间：2026-05-19 · License：Apache-2.0 · AI Skill Hub 不对第三方内容的准确性作法律背书。