### [LiteLLM](https://docs.litellm.ai/docs/)

> Handle: `litellm`<br/>
> URL: [http://localhost:33841/](http://localhost:33841/)

LLM API Proxy/Gateway.

LiteLLM is very useful for setups where the target LLM backend is either:
- Not supported by Harbor directly
- Doesn't have an OpenAI-compatible API that can be plugged into Open WebUI directly (for example, `text-generation-inference`)

`litellm` is also a way to use API-based LLM providers with Harbor.

#### Starting

```bash
# [Optional] Pull the litellm images
# ahead of starting the service
harbor pull litellm

# Start the service
harbor up litellm
```

You'll likely want to start it with at least one of the compatible LLM backends, or pointing to an external API ahead of time, see below.

Harbor's LiteLLM service is configured with a DB and an API key, so that you can access LiteLLM UI. The UI is available on the `/ui` endpoint of the service.

```bash
# Open LiteLLM API docs
harbor open litellm
# Open LiteLLM UI directly
harbor litellm ui
```

You can login with the default credentials: `admin` / `admin`. To adjust, use the `harbor` CLI:
```bash
# Set the new credentials
harbor litellm username paole
harbor litellm password $(tr -dc 'A-Za-z0-9!?%=' < /dev/urandom | head -c 10)
```

#### Configuration

Harbor runs LiteLLM in the [proxy mode](https://docs.litellm.ai/docs/proxy/configs). In order to configure it, you'll need to edit [`services/litellm/litellm.config.yaml`](../services/litellm/litellm.config.yaml) according to the documentation.

For example:

```yaml
model_list:
  # What LiteLLM client will see
  - model_name: sllama
    litellm_params:
      # What LiteLLM will send to downstream API
      model: huggingface/repo/model
      # This can be pointed to one of the compatible Harbor
      # backends or to the external API compatible with the LiteLLM
      api_base: http://tgi:80
  - model_name: llamaster
    litellm_params:
      model: bedrock/meta.llama3-1-405b-instruct-v1:0
      aws_region_name: us-west-2
```

- After changing the `litellm.config.yaml` - you will need to restart the service `harbor restart litellm`
- URLs must be internal to Docker network, can be obtained with [`harbor url -i <service>`](./3.-Harbor-CLI-Reference.md#harbor-url-service)
- You can use `harbor env litellm <name> <value>` to set environment variables to reference in the `config.yaml` file

Following options can be set via [`harbor config`](./3.-Harbor-CLI-Reference.md#harbor-config):

```bash
# The port on the host where LiteLLM endpoint will be available
HARBOR_LITELLM_HOST_PORT      33841

# The port on the host where LiteLLM DB is available
HARBOR_LITELLM_DB_HOST_PORT   33842

# Master key for LiteLLM API access
HARBOR_LITELLM_MASTER_KEY     sk-litellm

# Credentials for LiteLLM UI
HARBOR_LITELLM_UI_USERNAME    admin
HARBOR_LITELLM_UI_PASSWORD    admin
```

Please see [official LiteLLM documentation](https://docs.litellm.ai/docs/) for plenty of additional examples.
