---
title: "Large Language Models (LLMs)"
description: ""
position: 3
---
## Introduction
Large Language Models (LLMs) power a variety of tasks such as dialogue systems, text classification, summarization, and more.
NeMo AutoModel provides a simple interface for loading and fine-tuning LLMs hosted on the Hugging Face Hub.

## Run LLMs with NeMo AutoModel
To run LLMs with NeMo AutoModel, make sure you're using NeMo container version [`25.11.00`](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/nemo-automodel?version=25.11.00) or later. If the model you intend to fine-tune requires a newer version of Transformers, you may need to upgrade to the latest version of NeMo AutoModel by using:

```bash
pip3 install --upgrade git+git@github.com:NVIDIA-NeMo/AutoModel.git
```

For other installation options (e.g., uv), see the [NeMo AutoModel Installation Guide](/get-started/installation).

## Supported Models

NeMo AutoModel supports the [AutoModelForCausalLM](https://huggingface.co/transformers/v3.5.1/model_doc/auto.html#automodelforcausallm) in the [Text Generation](https://huggingface.co/models?pipeline_tag=text-generation&sort=trending) category. During preprocessing, it uses `transformers.AutoTokenizer`, which is sufficient for most LLM cases. If your model requires custom text handling, override the tokenizer in your recipe YAML or provide a custom dataset `_target_`. See [LLM datasets](/datasets/text-dataset) and [dataset overview](/datasets/overview).

| Owner | Model Family | Architectures |
|---|---|---|
| Meta | [Llama](/model-coverage/large-language-models/llama) | `LlamaForCausalLM` |
| Google | [Gemma](/model-coverage/large-language-models/gemma) | `GemmaForCausalLM`, `Gemma2ForCausalLM`, `Gemma3ForCausalLM` |
| Qwen / Alibaba Cloud | [Qwen2](/model-coverage/large-language-models/qwen2) | `Qwen2ForCausalLM` |
| Qwen / Alibaba Cloud | [Qwen2 MoE](/model-coverage/large-language-models/qwen2-moe) | `Qwen2MoeForCausalLM` |
| Qwen / Alibaba Cloud | [Qwen3](/model-coverage/large-language-models/qwen3) | `Qwen3ForCausalLM` |
| Qwen / Alibaba Cloud | [Qwen3 MoE](/model-coverage/large-language-models/qwen3-moe) | `Qwen3MoeForCausalLM` |
| Qwen / Alibaba Cloud | [Qwen3-Next](/model-coverage/large-language-models/qwen3-next) | `Qwen3NextForCausalLM` |
| DeepSeek | [DeepSeek](/model-coverage/large-language-models/deepseek) | `DeepseekForCausalLM` |
| DeepSeek | [DeepSeek-V3](/model-coverage/large-language-models/deepseek-v3) | `DeepseekV3ForCausalLM`, `DeepseekV32ForCausalLM` |
| DeepSeek | [DeepSeek V4 Flash](/model-coverage/large-language-models/deepseek-v4-flash) | `DeepseekV4ForCausalLM` |
| Mistral AI | [Mistral](/model-coverage/large-language-models/mistral) | `MistralForCausalLM` |
| Mistral AI | [Mixtral](/model-coverage/large-language-models/mixtral) | `MixtralForCausalLM` |
| Mistral AI | [Ministral3 / Devstral](/model-coverage/large-language-models/ministral3-devstral) | `Mistral3ForConditionalGeneration` |
| Microsoft | [Phi](/model-coverage/large-language-models/phi) | `PhiForCausalLM` |
| Microsoft | [Phi-3 / Phi-4](/model-coverage/large-language-models/phi-3-phi-4) | `Phi3ForCausalLM` |
| Microsoft | [Phi-3-Small](/model-coverage/large-language-models/phi-3-small) | `Phi3SmallForCausalLM` |
| NVIDIA | [Nemotron / Minitron](/model-coverage/large-language-models/nemotron-minitron) | `NemotronForCausalLM` |
| NVIDIA | [Nemotron-H](/model-coverage/large-language-models/nemotron-h) | `NemotronHForCausalLM` |
| NVIDIA | [Nemotron-Flash](/model-coverage/large-language-models/nemotron-flash) | `NemotronFlashForCausalLM` |
| NVIDIA | [Nemotron-Super](/model-coverage/large-language-models/nemotron-super-llama-3-3-nemotron-super-49b) | `DeciLMForCausalLM` |
| THUDM / Zhipu AI | [ChatGLM](/model-coverage/large-language-models/chatglm) | `ChatGLMModel` |
| THUDM / Zhipu AI | [GLM-4](/model-coverage/large-language-models/glm-4) | `GlmForCausalLM`, `Glm4ForCausalLM` |
| THUDM / ZAI | [GLM-4 MoE](/model-coverage/large-language-models/glm-4-moe-glm-4-5-glm-4-7) | `Glm4MoeForCausalLM`, `Glm4MoeLiteForCausalLM` |
| THUDM / ZAI | [GLM-5 / GLM-5.1](/model-coverage/large-language-models/glm-5-moe-dsa) | `GlmMoeDsaForCausalLM` |
| IBM | [Granite](/model-coverage/large-language-models/granite) | `GraniteForCausalLM` |
| IBM | [Granite MoE](/model-coverage/large-language-models/granite-moe) | `GraniteMoeForCausalLM`, `GraniteMoeSharedForCausalLM` |
| IBM | [Bamba](/model-coverage/large-language-models/bamba) | `BambaForCausalLM` |
| Allen AI | [OLMo](/model-coverage/large-language-models/olmo) | `OLMoForCausalLM` |
| Allen AI | [OLMo2](/model-coverage/large-language-models/olmo2) | `OLMo2ForCausalLM` |
| Allen AI | [OLMoE](/model-coverage/large-language-models/olmoe) | `OLMoEForCausalLM` |
| OpenAI | [GPT-OSS](/model-coverage/large-language-models/gpt-oss) | `GptOssForCausalLM` |
| OpenAI | [GPT-2](/model-coverage/large-language-models/gpt-2) | `GPT2LMHeadModel` |
| EleutherAI | [GPT-J](/model-coverage/large-language-models/gpt-j) | `GPTJForCausalLM` |
| EleutherAI | [GPT-NeoX / Pythia](/model-coverage/large-language-models/gpt-neox-pythia) | `GPTNeoXForCausalLM` |
| BigCode | [StarCoder](/model-coverage/large-language-models/starcoder) | `GPTBigCodeForCausalLM` |
| BigCode | [StarCoder2](/model-coverage/large-language-models/starcoder2) | `Starcoder2ForCausalLM` |
| BAAI | [Aquila / Aquila2](/model-coverage/large-language-models/aquila-aquila2) | `AquilaForCausalLM` |
| Baichuan Inc | [Baichuan / Baichuan2](/model-coverage/large-language-models/baichuan-baichuan2) | `BaiChuanForCausalLM` |
| Cohere | [Command-R](/model-coverage/large-language-models/command-r) | `CohereForCausalLM`, `Cohere2ForCausalLM` |
| TII | [Falcon](/model-coverage/large-language-models/falcon) | `FalconForCausalLM` |
| LG AI Research | [EXAONE](/model-coverage/large-language-models/exaone) | `ExaoneForCausalLM` |
| InternLM | [InternLM](/model-coverage/large-language-models/internlm) | `InternLMForCausalLM`, `InternLM2ForCausalLM`, `InternLM3ForCausalLM` |
| Inception AI | [Jais](/model-coverage/large-language-models/jais) | `JAISLMHeadModel` |
| MiniMax | [MiniMax-M2](/model-coverage/large-language-models/minimax-m2) | `MiniMaxM2ForCausalLM` |
| OpenBMB | [MiniCPM](/model-coverage/large-language-models/minicpm) | `MiniCPMForCausalLM`, `MiniCPM3ForCausalLM` |
| Moonshot AI | [Moonlight](/model-coverage/large-language-models/moonlight) | `DeepseekV3ForCausalLM` |
| ByteDance Seed | [Seed (ByteDance)](/model-coverage/large-language-models/seed-bytedance) | `Qwen2ForCausalLM` |
| Upstage | [Solar Pro](/model-coverage/large-language-models/solar-pro) | `SolarForCausalLM` |
| OrionStar | [Orion](/model-coverage/large-language-models/orion) | `OrionForCausalLM` |
| Stability AI | [StableLM](/model-coverage/large-language-models/stablelm) | `StableLmForCausalLM` |
| Stepfun AI | [Step-3.5](/model-coverage/large-language-models/step-3-5) | `Step3p5ForCausalLM` |
| Parasail AI | [GritLM](/model-coverage/large-language-models/gritlm) | `GritLM` |
| Tencent | [Hy3-preview](/model-coverage/large-language-models/hy3-preview) | `HYV3ForCausalLM` |

## Fine-Tuning LLMs with NeMo AutoModel

The models listed above can be fine-tuned using NeMo AutoModel. We support two primary fine-tuning approaches:

1. **Parameter-Efficient Fine-Tuning (PEFT)**: Updates only a small subset of parameters (typically &lt;1%) using techniques like Low-Rank Adaptation (LoRA).
2. **Supervised Fine-Tuning (SFT)**: Updates all or most model parameters for deeper adaptation.

See the [Fine-Tuning Guide](/recipes-e2e-examples/sft-peft) to learn how to apply both methods to your data.

<Tip>
In these guides, we use the `SQuAD v1.1` dataset for demonstration purposes, but you can use your own data. Update the recipe YAML `dataset` / `validation_dataset` sections accordingly. See [LLM datasets](/datasets/text-dataset) and [dataset overview](/datasets/overview).

</Tip>
