LlamaFactory Agent工作流

Requirement

Hardware Requirement

\* estimated

Install dependencies

pip install -r requirements-dev.txt

Install the dependencies for the compilation tools. Note that the commands for this step may vary depending on the operating system. The following are provided for reference

apt-get install -y build-essential cmake

Used by [Amazon](https://aws.amazon.com/cn/blogs/machine-learning/how-apoidea-group-enhances-visual-information-extraction-from-banking-documents-with-multimodal-models-using-llama-factory-on-amazon-sagemaker-hyperpod/), [NVIDIA](https://build.nvidia.com/spark/llama-factory), [Aliyun](https://help.aliyun.com/zh/pai/use-cases/fine-tune-a-llama-3-model-with-llama-factory), etc.

----

Check our new open-source project —<br>🐧 [PenguinHarness](https://github.com/Prism-Shadow/penguin-harness): Local-first multi-agent app development platform - create, fine-tune, deploy AI apps automatically!

Follow us and give us a star ⭐: https://github.com/Prism-Shadow/penguin-harness

</div>

https://github.com/user-attachments/assets/9b7033e8-f08a-4c3f-bd33-547896664e6e

----

Getting Started

Installation

[!IMPORTANT] Installation is mandatory.

Install from Source

git clone --depth 1 https://github.com/hiyouga/LlamaFactory.git
cd LlamaFactory
pip install -e .
pip install -r requirements/metrics.txt

Optional dependencies available: metrics, deepspeed. Install with: pip install -e . && pip install -r requirements/metrics.txt -r requirements/deepspeed.txt

Additional dependencies for specific features are available in examples/requirements/.

Install from Docker Image

docker run -it --rm --gpus=all --ipc=host hiyouga/llamafactory:latest

This image is built on Ubuntu 22.04 (x86\_64), CUDA 12.4, Python 3.11, PyTorch 2.6.0, and Flash-attn 2.7.4.

Find the pre-built images: https://hub.docker.com/r/hiyouga/llamafactory/tags

Please refer to build docker to build the image yourself.

<details><summary>Setting up a virtual environment with <b>uv</b></summary>

Create an isolated Python environment with uv:

uv run llamafactory-cli webui

</details>

<details><summary>For Windows users</summary>

Install PyTorch

You need to manually install the GPU version of PyTorch on the Windows platform. Please refer to the official website and the following command to install PyTorch with CUDA support:

pip uninstall torch torchvision torchaudio
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu126
python -c "import torch; print(torch.cuda.is_available())"

If you see True then you have successfully installed PyTorch with CUDA support.

Try dataloader_num_workers: 0 if you encounter Can't pickle local object error.

Install BitsAndBytes

To enable Quantized LoRA (QLoRA) on Windows, you need to install bitsandbytes.

For most users, it is recommended to install the latest official release:

pip install bitsandbytes

If you are using uv to manage your virtual environment, it is recommended to install bitsandbytes after installing the GPU-enabled version of PyTorch:

uv pip install bitsandbytes --no-deps

[!IMPORTANT] Pay attention to the CUDA Toolkit version when installing bitsandbytes. Official bitsandbytes releases are built for specific CUDA Toolkit versions. On Windows x86-64, separate builds are currently provided for CUDA 11.8–12.6 and CUDA 12.8–12.9. Support for NVIDIA RTX 50 Series GPUs (e.g., RTX 5060 Ti, sm_120) requires the CUDA 12.8–12.9 builds.

If your environment uses an older CUDA version, or you need compatibility with older Windows / PyTorch combinations, you can install the third-party precompiled Windows wheel:

pip install https://github.com/jllllll/bitsandbytes-windows-webui/releases/download/wheels/bitsandbytes-0.41.2.post2-py3-none-win_amd64.whl

Install Flash Attention-2

To enable FlashAttention-2 on the Windows platform, please use the script from flash-attention-windows-wheel to compile and install it by yourself.

</details>

<details><summary>For Ascend NPU users</summary>

To install LlamaFactory on Ascend NPU devices, please use Python 3.12 and install the extra dependencies with pip install -r requirements/npu.txt. Additionally, you need to install the Ascend CANN Toolkit and Kernels. Please follow the installation tutorial.

You can also download the pre-built Docker images:

```bash

Docker Hub

docker pull hiyouga/llamafactory:latest-910b-ubuntu docker pull hiyouga/llamafactory:latest-a3-ubuntu docker pull hiyouga/llamafactory:latest-910b-openeuler docker pull hiyouga/llamafactory:latest-a3-openeuler

Install bitsandbytes from source

Compile & install

cmake -DCOMPUTE_BACKEND=npu -S . make pip install .

2. Install transformers from the main branch.

1. Set double_quantization: false in the configuration. You can refer to the example.

</details>

Build Docker

For CUDA users:

cd docker/docker-cuda/
docker compose up -d
docker compose exec llamafactory bash

For Ascend NPU users:

```bash cd docker/docker-npu/

Deploy with OpenAI-style API and vLLM

API_PORT=8000 llamafactory-cli api examples/inference/qwen3.yaml infer_backend=vllm vllm_enforce_eager=true

[!TIP] Visit this page for API document. Examples: Image understanding | Function calling

Easily fine-tune 100+ large language models with zero-code [CLI](#quickstart) and [Web UI](#fine-tuning-with-llama-board-gui-powered-by-gradio)

</div>

👋 Join our WeChat and NPU user groups.

\ English | [中文 \]

Fine-tuning a large language model can be easy as...

https://github.com/user-attachments/assets/3991a3a8-4276-4d30-9cab-4cb0c4b9b99e

Start local training: - Please refer to usage

Start cloud training: - Colab (free): https://colab.research.google.com/drive/1eRTPn37ltBbYsISy9Aw2NuI2Aq5CQrD9?usp=sharing - PAI-DSW (free trial): https://gallery.pai-ml.com/#/preview/deepLearning/nlp/llama_factory - AMD GPU Cloud (free credits): https://github.com/AMD-AIM/AMD_Developers_Notebooks/blob/main/en/AMD_developer_LLaMAFactory_note_en.md

Read technical notes: - Documentation (WIP): https://llamafactory.readthedocs.io/en/latest/ - Documentation (AMD GPU): https://rocm.docs.amd.com/projects/ai-developer-hub/en/latest/notebooks/fine_tune/llama_factory_llama3.html - Documentation (ASCEND NPU): https://llamafactory.readthedocs.io/en/latest/multibackend/npu/index.html - Official Blog: https://blog.llamafactory.net/en/

[!NOTE] Except for the above links, all other websites are unauthorized third-party websites. Please carefully use them.

Quickstart

Use the following 3 commands to run LoRA fine-tuning, inference and merging of the Qwen3-4B-Instruct model, respectively.

llamafactory-cli train examples/train_lora/qwen3_lora_sft.yaml
llamafactory-cli chat examples/inference/qwen3_lora_sft.yaml
llamafactory-cli export examples/merge_lora/qwen3_lora_sft.yaml

See examples/README.md for advanced usage (including distributed training).

[!TIP] Use llamafactory-cli help to show help information. Read FAQs first if you encounter any problems.

Supported Models

[!NOTE] For the "base" models, the template argument can be chosen from default, alpaca, vicuna etc. But make sure to use the corresponding template for the "instruct/chat" models. If the model has both reasoning and non-reasoning versions, please use the _nothink suffix to distinguish between them. For example, qwen3 and qwen3_nothink. Remember to use the SAME template in training and inference. \: You should install the transformers from main branch and use DISABLE_VERSION_CHECK=1 to skip version check. \\*: You need to install a specific version of transformers to use the corresponding model.

Please refer to constants.py for a full list of models we supported.

You also can add a custom chat template to template.py.