### [LocalAI](https://github.com/go-skynet/LocalAI)

> Handle: `localai`<br/>
> URL: [http://localhost:34451](http://localhost:34451)

A drop-in replacement for OpenAI API that runs models locally and supports various model formats and architectures.

#### Starting

```bash
# [Optional] Pre-pull the image
# ⚠️ These images are VERY large (~18Gb for CUDA version)
harbor pull localai

# Start the service
# --open is optional, to open in browser
harbor up localai --open
```

See [troubleshooting guide](./1.-Harbor-User-Guide.md#troubleshooting) if you encounter any issues.

- Unfortunately `localai` doesn't reuse cache with other services providing inference, however you can manually place the models into `$(harbor home)/services/localai/data` folder and follow ["Run Models Manually"](https://localai.io/docs/getting-started/models/#run-models-manually) guide
- If you have sufficient Disk space - consider switching to [AIO](https://localai.io/basics/container/#available-aio-images) image version for all supported kinds of models pre-configured in the service
- Harbor will detect `nvidia` / `rocm` capability on the host and use appropriate image tag (configurable, see below)
- Harbor will share your HuggingFace token with the service if it is set - to download the gated/private models

#### Usage

You'll perform most actions in the LocalAI WebUI, run [`harbor open`](./3.-Harbor-CLI-Reference.md#harbor-open-service) to access it.

- After initial start, go to the Models Gallery and download a model of your choice.
- After downloading, model will appear in LocalAI home page, where you can start a chat with it

#### Configuration

See official [Environment Variables guide](https://localai.io/advanced/#environment-variables) for reference.

Following options can be set via [`harbor config`](./3.-Harbor-CLI-Reference.md#harbor-config):

```bash
# The port on the host where LocalAI endpoint will be available
HARBOR_LOCALAI_HOST_PORT       34451

# Docker image to use for the LocalAI service
HARBOR_LOCALAI_IMAGE           localai/localai

# Docker tag when no Nvidia/ROCm support is detected
HARBOR_LOCALAI_CPU_VERSION     latest-cpu

# Docker tag when Nvidia support is detected
HARBOR_LOCALAI_NVIDIA_VERSION  latest-gpu-nvidia-cuda-12

# Docker tag when ROCm support is detected
HARBOR_LOCALAI_ROCM_VERSION    latest-gpu-hipblas

# Location on the host to store files for the LocalAI service
# Should be either relative to $(harbor home) or absolute
HARBOR_LOCALAI_WORKSPACE       ./services/localai/data
```

See [environment configuration guide](./1.-Harbor-User-Guide.md#environment-variables) to set arbitrary environment variables for the service.
