H2O LLM Studio

Setup

H2O LLM Studio requires a machine with Ubuntu 16.04+ and at least one recent NVIDIA GPU with NVIDIA drivers version >= 470.57.02. For larger models, we recommend at least 24GB of GPU memory. For more information about installation prerequisites, see the Set up H2O LLM Studio guide in the documentation. For a performance comparison of different GPUs, see the H2O LLM Studio performance guide in the documentation.

Recommended Install

The recommended way to install H2O LLM Studio is using uv with Python 3.10. To install Python 3.10 on Ubuntu 20.04+, execute the following commands: #### Installing NVIDIA Drivers (if required) If deploying on a 'bare metal' machine running Ubuntu, one may need to install the required NVIDIA drivers and CUDA. The following commands show how to retrieve the latest drivers for a machine running Ubuntu 20.04 as an example. One can update the following based on their OS.

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-ubuntu2204.pin
sudo mv cuda-ubuntu2204.pin /etc/apt/preferences.d/cuda-repository-pin-600
wget https://developer.download.nvidia.com/compute/cuda/12.4.0/local_installers/cuda-repo-ubuntu2204-12-4-local_12.4.0-550.54.14-1_amd64.deb
sudo dpkg -i cuda-repo-ubuntu2204-12-4-local_12.4.0-550.54.14-1_amd64.deb
sudo cp /var/cuda-repo-ubuntu2204-12-4-local/cuda-*-keyring.gpg /usr/share/keyrings/
sudo apt-get update
sudo apt-get -y install cuda-toolkit-12-4

Run H2O LLM Studio GUI using Docker

Install Docker first by following instructions from NVIDIA Containers. Make sure to have nvidia-container-toolkit installed on your machine as outlined in the instructions. H2O LLM Studio images are stored in the h2oai Docker Hub container repository. ```bash mkdir -p pwd/llmstudio_mnt chmod 777 pwd/llmstudio_mnt

Quickstart

For questions, discussing, or just hanging out, come and join our Discord! Use cloud-based runpod.io instance to run the latest version of H2O LLM Studio with GUI. Using CLI for fine-tuning LLMs:

Data format and example data

For details on the data format required when importing your data or example data that you can use to try out H2O LLM Studio, see Data format in the H2O LLM Studio documentation.

Example: Run on OASST data via CLI

As an example, you can run an experiment on the OASST data via CLI. For instructions, see Run an experiment on the OASST data guide in the H2O LLM Studio documentation.

Virtual environments

We offer various ways of setting up the necessary python environment. #### UV virtual environment The following command will create a virtual environment using uv and will install the dependencies:

make setup

Run H2O LLM Studio with command line interface (CLI)

You can also use H2O LLM Studio with the command line interface (CLI) and specify the configuration .yaml file that contains all the experiment parameters. To fine-tune using H2O LLM Studio with CLI use the following command:

uv run python llm_studio/train.py -Y {path_to_config_yaml_file}

bash distributed_train.sh {NR_OF_GPUS} -Y {path_to_config_yaml_file}

uv run python llm_studio/prompt.py -e {experiment_name}

uv run python llm_studio/publish_to_hugging_face.py -p {path_to_experiment} -d {device} -a {api_key} -u {user_id} -m {model_name} -s {safe_serialization}

Troubleshooting

If running on cloud-based machines such as runpod, you may need to set the following environment variable to allow the H2O Wave server to accept connections from the proxy:

H2O_WAVE_ALLOWED_ORIGINS="*"

H2O_WAVE_APP_CONNECT_TIMEOUT="15"
H2O_WAVE_APP_WRITE_TIMEOUT="15"
H2O_WAVE_APP_READ_TIMEOUT="15"
H2O_WAVE_APP_POOL_TIMEOUT="15"