### [Ollama](https://ollama.com/)

> Handle: `ollama`<br/>
> URL: [http://localhost:33821](http://localhost:33821)

![Official Ollama logo](https://ollama.com/public/blog/embedding-models.png)

Ergonomic wrapper around llama.cpp with plenty of QoL features.

#### Starting

Ollama is one of the [default services](./3.-Harbor-CLI-Reference.md#harbor-defaults), so you don't need to specify anything special to start it.

```bash
# Will pull images and start the service
harbor up

# [Optional] monitor the logs
harbor logs ollama
```

- Harbor's `ollama` instance is configured to pull default models (see configuration below)
- By default, Ollama cache is stored in `~/.ollama` directory, but you can configure it to match your host's Olama cache location (see `HARBOR_OLLAMA_CACHE` in [configuration](#configuration) below)

#### Integrations

`ollama` (API on port 11434) is automatically wired into dozens of other services via cross-compose files (`services/compose.x.*.ollama.yml`). When you run `harbor up` with `ollama` plus a consumer, Harbor injects the endpoint via `HARBOR_OLLAMA_INTERNAL_URL` (default `http://ollama:11434`) so consumers use it with no manual setup. Many also consume the OpenAI-compatible API at `/v1` (with key `sk-ollama`).

**Frontends and tools that auto-configure for ollama**

- `webui` — mounts `config.ollama.json` and sets `OLLAMA_BASE_URL` env (via `compose.x.webui.ollama.yml` and `compose.x.ollama.webui.yml`).
- `chatui` — mounts `chatui.ollama.yml` defining the model endpoint (OpenAI /v1, driven by `HARBOR_CHATUI_OLLAMA_MODEL`).
- `anythingllm` — sets `LLM_PROVIDER=ollama`, `OLLAMA_BASE_PATH`, and embedding paths.
- `aider` — mounts `aider.ollama.yml` (OpenAI compat /v1 + `HARBOR_AIDER_MODEL`).
- `boost` — injects `HARBOR_BOOST_OPENAI_URL_OLLAMA` (pointing to `/v1`) and key.
- `cmdh`, `hermes`, `opint`, `optillm`, `plandex`, `sillytavern`, `astrbot`, `mindsdb`, `lobechat`, `morphic`, `khoj`, `perplexica`, `fabric`, `gptme`, `karakeep`, `ldr`, `activepieces`, `agentzero`, `browseruse`, `drawio`, `kotaemon`, `npcsh`, `mi`, `openterminal`, `openhands`, `opennotebook`, `parllama`, `photoprism`, `presenton`, `promptfoo`, `raglite`, `sim`, `txtairag` and many others — receive `HARBOR_OLLAMA_INTERNAL_URL` (or derived) via environment injection; several declare `depends_on: ollama: {condition: service_healthy}`.
- `bifrost` — bootstrap sidecar (`compose.x.bifrost.ollama.yml`) registers ollama as a provider using `HARBOR_BIFROST_OLLAMA_MODELS`.
- `hollama` — lightweight dedicated Ollama chat UI (traefik-exposed when enabled).
- `open-design` — uses a dedicated socat proxy sidecar so the service sees ollama on localhost:11434.

**Reverse proxy / external access**

- With `traefik` present: `compose.x.traefik.ollama.yml` (and `compose.x.traefik.hollama.yml`) add labels exposing the service at `https://ollama.${HARBOR_TRAEFIK_DOMAIN}` and `https://hollama.${HARBOR_TRAEFIK_DOMAIN}` (websecure).

**Hardware acceleration (GPU / accelerator)**

- `compose.x.ollama.nvidia.yml` adds full NVIDIA GPU reservations.
- `compose.x.ollama.rocm.yml` adds AMD ROCm devices (`/dev/kfd`, `/dev/dri`) and switches both `ollama` and `ollama-init` containers to the `ollama/ollama:rocm` image.
- `compose.x.ollama.cdi.yml` uses CDI for NVIDIA GPU passthrough.
- Choose the appropriate image tag via `harbor config set ollama.version ...-rocm` (or nvidia variants) as needed.

**Host / volume interactions**

- **Ollama models**: `${HARBOR_OLLAMA_CACHE}` (defaults to `~/.ollama` on host) mounted at `/root/.ollama`. All pulls (`harbor ollama pull`, UI, init) store here.
- **Hugging Face cache**: `${HARBOR_HF_CACHE}` mounted at `/root/.cache/huggingface` — used for `hf.co/...` pulls and GGUF imports documented in the guide; shared across Harbor services.
- **Custom Modelfiles**: `./services/ollama/modelfiles:/modelfiles` — drop your `.Modelfile` sources here for easy `harbor ollama create -f /modelfiles/...` access.
- **ollama-init sidecar** (defined in base `compose.ollama.yml`): depends on `ollama` healthy, runs `ollama pull` for every model in the comma-separated `HARBOR_OLLAMA_DEFAULT_MODELS` (default `nomic-embed-text:latest`), then stays healthy so `harbor up` with `--wait` succeeds even during long first-time pulls. The sidecar re-uses the same cache volume.
- Additional legacy caches (`HARBOR_LLAMACPP_CACHE`, `HARBOR_VLLM_CACHE`) are also mounted for cross-backend convenience.
- The main service has a TCP healthcheck; consumers can safely `depends_on` it with `condition: service_healthy`.

**Special behaviors**

- **OpenAI compatibility layer**: The `/v1` path on the internal URL is the preferred integration point for any tool built against the OpenAI SDK. Harbor wires many consumers this way with the conventional `sk-ollama` key.
- **Pointing everything at external Ollama**: Set `HARBOR_OLLAMA_INTERNAL_URL` (via `harbor config set ollama.internal_url http://host.docker.internal:11434` or your LAN IP) and restart consumers — every auto-wired service will follow.
- Default models are pulled exactly once on first startup via the init sidecar (controlled by `HARBOR_OLLAMA_DEFAULT_MODELS`).

All wiring is declarative in the cross-compose files. After editing any `HARBOR_OLLAMA_*` value with `harbor config set`, run `harbor config update` and `harbor restart` (or `harbor up`) for the relevant services.

#### Models

You can discover new models via Ollama's [model library](https://ollama.com/library).

Management of the models is possible right from the [Open WebUI Admin Settings](http://localhost:33801/admin/settings/). The models are stored in the global ollama cache on your local machine.

Alternatively, you can use `ollama` CLI itself.

```bash
# Show the list of available models
harbor ollama list
harbor ollama ls

# Pull a new model
harbor ollama pull phi4
```

You can also pull models directly from the HuggingFace Hub.

```bash
# [Optional] will open HF model search with "llama gguf" query
harbor hf find llama gguf

# Pull the model
# "hf.co" is a special prefix telling Ollama to pull from HuggingFace
# "unsloth/DeepSeek-R1-Distill-Llama-8B-GGUF" is the HuggingFace repository name
# "Q8_0" points to the "DeepSeek-R1-Distill-Llama-8B-Q8_0.gguf" file in the repository
harbor ollama pull hf.co/unsloth/DeepSeek-R1-Distill-Llama-8B-GGUF:Q8_0
```

#### TroubleShooting

If you encounter any issues with `ollama` service, you can check the logs with `harbor logs ollama`.

```bash
# Check the logs
harbor logs ollama
```

If you need to restart the service, you can do it with `harbor restart ollama`.

```bash
# Restart the service
harbor restart ollama
```

You'll find many more examples and scenarios covered in the [troubleshooting guide](./1.-Harbor-User-Guide.md#troubleshooting).


#### Ollama CLI

You can use a full `ollama` CLI, when `ollama` service is running.

```bash
# Ollama service should be running to access the cli
harbor ollama --help

# See the environment variables
# supported by ollama service
harbor ollama serve --help

# Access Ollama CLI commands
harbor ollama version
```

#### Configuration

See environment variables [configuration guide](./1.-Harbor-User-Guide.md#environment-variables).

```bash
# Env variable reference
harbor ollama serve --help
```

In addition to that, some high-level configurations are available via [`harbor config`](./3.-Harbor-CLI-Reference.md#harbor-config)

```bash
# See all harbor config properties for ollama
harbor config ls | grep OLLAMA

# Configure ollama version, accepts a docker tag
harbor config set ollama.version 0.3.7-rc5-rocm

# Available configuration options
# Location of the ollama own cache, either global
# or relative to $(harbor home)
HARBOR_OLLAMA_CACHE            ~/.ollama

# The port on the host machine where the ollama service
# will be available
HARBOR_OLLAMA_HOST_PORT        33821

# The Docker image tag to use
HARBOR_OLLAMA_VERSION          latest

# This URL will be given to the services connected to Ollama
# you can substitute with your own custom URL to switch
# Harbor to your own Ollama deployment
HARBOR_OLLAMA_INTERNAL_URL     http://ollama:11434

# A comma-separated list of models to pull/update on startup
# The default value is also used in default configs for dependent services
HARBOR_OLLAMA_DEFAULT_MODELS   nomic-embed-text:latest

# Maps to the "OLLAMA_CONTEXT_LENGTH" environment variable
# for the container and sets global default ctx length for all models.
# Don't use "harbor config" to set this as it won't be propagated to the env var.
HARBOR_OLLAMA_CONTEXT_LENGTH   4096
```

#### Environment Variables

Follow Harbor's [environment variables guide](./1.-Harbor-User-Guide.md#environment-variables) to set arbitrary variables for `ollama` service.

#### API

Retrieve the endpoint for `ollama` service with:
```bash
# For the host machine
harbor url ollama

# For the LAN
harbor url -a ollama

# For the Docker network
harbor url -i ollama
```

Additionally, you can find a small [HTTP playbook](../services/http-catalog/ollama.http) in the [`services/http-catalog/`](https://github.com/av/harbor/tree/main/services/http-catalog) folder.

#### Custom Modelfiles

You can create custom modelfiles to run models with specific parameters. Harbor will run Ollama CLI within the same folder you called it from, so all the files will be available.

```bash
# 1. Create a new modelfile at a convenient location
touch mymodel.Modelfile

# 2. Edit the modelfile with your favorite editor
open mymodel.Modelfile

# 3. Import the model into Ollama
harbor ollama create -f mymodel.Modelfile mymodel
```

The most common use-case is to override default Ollama parameters with custom values, for example to increase the default context length.

You can also source your modelfile from the existing one, to use as a template.

```bash
harbor ollama show modelname:latest --modelfile > mymodel.modelfile
```

#### Default context length

Starting from Ollama `v0.5.13` you can set a global default context length for all models. To do so with Harbor, you have two options:

```bash
# Option 1. Use CLI alias
# - Get current config value
harbor ollama ctx
# - Set the new value (4096 in this case)
harbor ollama ctx 4096

# Option 2. Use "harbor env"
# - Get the current value
harbor env ollama OLLAMA_CONTEXT_LENGTH
# - Set the new value (4096 in this case)
harbor env ollama OLLAMA_CONTEXT_LENGTH 4096
```

Note that `harbor ollama ctx` is synchronised to the env, but not vice versa.

#### Switching to external Ollama

It's possible to switch all related service in Harbor to use an external Ollama instance, to do so, you need to change the `HARBOR_OLLAMA_INTERNAL_URL` configuration.

> [!NOTE]
> The URL is relative to the Docker network, use `host.docker.internal` or `172.17.0.1` instead of `localhost`.

```bash
# Set the custom Ollama URL
harbor config set ollama.internal_url http://172.17.0.1:11434
```

#### Changing the cache location

By default, Ollama stores the models in the `~/.ollama` directory. You can change the location by setting the `HARBOR_OLLAMA_CACHE` configuration.

```bash
# Set the custom cache location
harbor config set ollama.cache /path/to/custom/cache
```

The path must be either absolute or relative to the [`harbor home`](./3.-Harbor-CLI-Reference.md#harbor-home) directory.

#### Importing models from HuggingFace

When compatible with Ollama's HF integration:

```bash
harbor ollama pull hf.co/unsloth/DeepSeek-R1-Distill-Llama-8B-GGUF:Q8_0
```

> [!TIP]
> You can then copy the model to have a more convenient ID:<br/>
> `harbor ollama cp hf.co/unsloth/DeepSeek-R1-Distill-Llama-8B-GGUF:Q8_0 r1-8b`

When not compatible, via direct import from file:

```bash
# 1. Download the model
harbor hf download flowaicom/Flow-Judge-v0.1-GGUF

# 2. Locate the gguf file
# The gguf file is located in the model directory
harbor find Flow-Judge-v0.1 | grep .gguf
# /home/user/.cache/huggingface/hub/models--flowaicom--Flow-Judge-v0.1-GGUF/snapshots/3ca...575/flow-judge-v0.1-Q4_K_M.gguf

# 3. Translate the path
# Harbor mounts HF cache to Ollama service
# /home/user/.cache/huggingface -> /root/.cache/huggingface
# The path becomes:
# /root/.cache/huggingface/hub/models--flowaicom--Flow-Judge-v0.1-GGUF/snapshots/3ca...575/flow-judge-v0.1-Q4_K_M.gguf

# 4. Create a modelfile
# You can use any convenient folder to store modelfiles
# By default, Harbor has a directory for modelfiles: ollama/modelfiles
# Below are few _options_ on quickly accessing the directory
harbor vscode       # Open Harbor workspace in VS Code and go from there
open $(harbor home) # Open Harbor workspace in default file manager
open $(harbor home)/services/ollama/modelfiles # This is the directory for modelfiles
code $(harbor home)/services/ollama/modelfiles # Open the directory in VS Code

# 5. Sample modelfile contents
# TIP: Use original base modelfile as a reference:
#      harbor ollama show --modelfile <model name>
# Save as "<your name>.Modelfile" in the modelfiles directory
FROM /root/.cache/huggingface/hub/models--flowaicom--Flow-Judge-v0.1-GGUF/snapshots/3ca...575/flow-judge-v0.1-Q4_K_M.gguf

# 6. Create the model
# 6.1. From Harbor's modelfiles directory
harbor ollama create -f /modelfiles/<your name>.Modelfile <your name>
# 6.2. From current directory
harbor ollama create -f ./<your name>.Modelfile <your name>
# Successful output example
# 13:27:37 [INFO] Service ollama is running. Executing command...
# transferring model data 100%
# using existing layer sha256:939...815
# creating new layer sha256:aaa...169
# writing manifest
# success

# 7. Check the model
harbor ollama run <your name>

# 8. To upload to ollama.com, follow official tutorial
# on sharing the models:
# https://github.com/ollama/ollama/blob/main/docs/import.md#sharing-your-model-on-ollamacom
```
