简介

🚀 What's New!

Big updates have landed in LLM Compressor! To get a more in-depth look, check out the LLM Compressor overview.

Some of the exciting new features include:

• DeepSeek-V4-Flash and Kimi-K2.6 Quantized Checkpoints: Quantized checkpoints for DeepSeek-V4-Flash and Kimi-K2.6 have been generated by the RedHat team and posted to the HF hub. Consider using:
• DeepSeek-V4-Flash-NVFP4-FP8 — 163B DeepSeek-V4-Flash quantized to NVFP4 weights with FP8 KV cache
• Kimi-K2.6-NVFP4 — Kimi-K2.6 quantized to NVFP4 (weights and activations), targeting NVIDIA Blackwell GPUs
• Kimi-K2.6-FP8-BLOCK — 1T parameter Kimi-K2.6 quantized to FP8 block format (weights and activations), compatible with DeepGEMM FP8 kernels
• Qwen3.6 NVFP4 Generated Checkpoint: An NVFP4 quantized checkpoint has been generated by the RedHat team and posted to the HF hub. Qwen3.6 follows the same architecture as Qwen3.5, so existing LLM Compressor examples can be used for this model by swapping out the target model string.
• Gemma4 Support: Gemma 4 can now be quantized using LLM Compressor. Support is available through main and will require updating to transformers 5.5 (uv pip install transformers>=5.5). For models quantized and published by the RedHat team, consider using:
• gemma-4-31B-it-NVFP4
• gemma-4-31B-it-FP8-block
• gemma-4-31B-it-FP8-Dynamic
• gemma-4-26B-A4B-it-FP8-Dynamic
• gemma-4-26B-A4B-it-NVFP4
• Qwen3.5 Support: Qwen 3.5 can now be quantized using LLM Compressor. You will need to update your local transformers version using uv pip install --upgrade transformers and install LLM Compressor from source if using <0.11. Once updated, you should be able to run examples for the MoE and non-MoE variants of Qwen 3.5 end-to-end. For models quantized and published by the RedHat team, consider using the NVFP4 and FP8 checkpoints for Qwen3.5-122B and Qwen3.5-397B.
• Updated offloading and model loading support: Loading transformers models that are offloaded to disk and/or offloaded across distributed process ranks is now supported. Disk offloading allows users to load and compress very large models which normally would not fit in CPU memory. Offloading functionality is no longer supported through accelerate but through model loading utilities added to compressed-tensors. For a full summary of updated loading and offloading functionality, for both single-process and distributed flows, see the Big Models and Distributed Support guide.
• Distributed GPTQ Support: GPTQ now supports Distributed Data Parallel (DDP) functionality to significantly improve calibration runtime. An example using DDP with GPTQ can be found here.
• Updated FP4 Microscale Support: GPTQ now supports FP4 quantization schemes, including both MXFP4 and NVFP4. MXFP4 support has also been improved with updated weight scale generation. Models with weight-only quantization in the MXFP4 format can now run in vLLM as of vLLM v0.14.0. MXFP4 models with activation quantization are not yet supported in vLLM for compressed-tensors models
• New Model-Free PTQ Pathway: A new model-free PTQ pathway has been added to LLM Compressor, called model_free_ptq. This pathway allows you to quantize your model without the requirement of Hugging Face model definition and is especially useful in cases where oneshot may fail. This pathway is currently supported for data-free pathways only i.e FP8 quantization and was leveraged to quantize the Mistral Large 3 model. Additional examples have been added illustrating how LLM Compressor can be used for Kimi K2
• MXFP8 Microscale Support: LLM Compressor now supports MXFP8 quantization via PTQ. Both W8A8 (MXFP8) and W8A16 weight-only (MXFP8A16) modes are available.
• Extended KV Cache and Attention Quantization Support: LLM Compressor now supports attention quantization, as well as fine-grained KV Cache quantization. Previously only per-tensor KV cache quantization was supported. Now, you can quantize KV cache with per-head scales and run with vLLM. Examples of more generalized attention and kv cache quantization can be found in the experimental folder.