Known compatibility issues between the services and models.

## Format

The format of the page is as follows:

```markdown
## Service | Model

Short description of nature/cause of the issue

### Affected Service, Model (or combination)

Description of the issue affects the service or combination of services, possible workaround.
```
## Gemma 2 - System Prompt

Gemma 2 models lack a system prompt.

### `vllm` x `searxng` x `webui`

When WebRAG is enabled, Open WebUI will send requests to the respective backend that will use the system role (for RAG). Such requests will fail with Gemma 2 (or other models without system prompt support) when running against VLLM.

### `cmdh`

`cmdh` needs a system prompt to outline the task.

### `chatui`

HuggingFace ChatUI uses system prompt for chat title generation.

#### Workaround

Unfortunately - switch to another model. Alternatively, disable WebRAG.

## LiteLLM - Dynamic Dependencies

When LiteLLM starts, it tries to install Node.js and some other packages right at the runtime. This may cause issues when running in a restricted environment or when related CDNs are unavailable.

Diagnostic:

`harbor logs litellm` is stuck at:
```bash
# First candidate
litellm  | Installing Prisma CLI

# Another candidate
litellm  |  * Install prebuilt node (22.5.1) ..... done.
```
### `litellm` x `webui`

When LiteLLM is stuck at the setup phase, WebUI won't load any of the proxied models.

#### Workaround

Restart the service a few times until it starts successfully.

## WebUI - Missing models in the list

WebUI v0.3.11 fails to load models from OpenAI-compatible endpoints when the API key is specified as an empty string (or missing).

Example configuration that'll not work:

```json
{
  "openai": {
		"api_base_urls": [
			"http://mistralrs:8021/v1"
		],
		"api_keys": [
			""
		],
		"enabled": true
	}
}
```

#### Workaround

Setup the endpoint to use an actual API key or a fake API key if supported by the service.

```json
{
  "openai": {
		"api_base_urls": [
			"http://mistralrs:8021/v1"
		],
		"api_keys": [
			"sk-mistralrs"
		],
		"enabled": true
	}
}
```

## WebUI - Same audio after audio config change

`webui` has a built-in cache for tts. It's sometimes used at a sentence level, so after changing the audio model, generating audio for a previously seen sentence will result in the same audio.

#### Workaround

Use a new sentence for testing audio config changes. For example by re-generating the model response.


## Exllama2 - GPTQ

Exllama2 (and related engines - Aphrodite engine, TabbyAPI, etc) only support GPTQ in 4-bits. You can detect this problem when running, for example, a 2-bit GPTQ model and seeing logs like these:

```bash
RuntimeError: q_weight and gptq_qzeros have incompatible shapes
```

#### Workaround

Switch to a 4-bit GPTQ model if possible. Otherwise, switch to another inference backend.

## vLLM - Out of workspace memory in AlignedAlloactor

This is an error between vLLM and FlashInfer attention backend.

#### Workaround

Ensure you're running the latest vLLM version.

```bash
harbor pull vllm
```

## TTS - xtts-v2

### `webui`

`openedai-speech` only starts downloading the `xtts-v2` model when the first generation request is made, not on startup. There are no logs or indication on download progress.

#### Workaround

Here're sample logs/steps when the `xtts-v2` is not downloaded yet:

```bash
# Initial startup, xtts-v2 isn't downloaded yet
harbor.tts  | First startup may download 2GB of speech models. Please wait.
harbor.tts  | INFO:     Started server process [27]
harbor.tts  | INFO:     Waiting for application startup.
harbor.tts  | INFO:     Application startup complete.
harbor.tts  | INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
harbor.tts  | INFO:     192.168.64.3:40698 - "POST /v1/audio/speech HTTP/1.1" 200 OK

# 1. Configure Open WebUI to use tts-1-hd
# 2. Generate speech from some uncached text
# 3. openedai-speech will log this:
harbor.tts  | 2024-08-26 08:51:44.737 | INFO     | __main__:__init__:59 - Loading model xtts to cuda
# ... Takes some time to download the model

# Check the folder size to see download progress
# voices/tts/tts_models--multilingual--multi-dataset--xtts
du -h $(harbor home)/services/tts

# Sample output when download is complete
user@os:~/code/harbor$ ▼ du -h $(harbor home)/services/tts
12K	/home/user/code/harbor/services/tts/config
1.8G	/home/user/code/harbor/services/tts/voices/tts/tts_models--multilingual--multi-dataset--xtts
1.8G	/home/user/code/harbor/services/tts/voices/tts
1.9G	/home/user/code/harbor/services/tts/voices
1.9G	/home/user/code/harbor/services/tts
```

## Ollama - truncated input

When using OpenAI-compatible endpoints - there's no way to specify `num_ctx` (context size) for the model. This parameter affects how the model is loaded into memory, so must be known/set ahead of inference and Ollama can't change it on a per-request basis or dynamically.

#### Workaround

Create a new Modelfile from the base model specifying desired `num_ctx` parameter by default. Here's an example:

```bash
# 1. Export Modelfile for the LLM:
harbor ollama show --modelfile model > Modelfile

# 2. Edit the Modelfile to include the desired num_ctx:
# FROM model
# PARAMETER num_ctx 128000
code ./Modelfile

# 3. Put modelfile into a folder that is shared with ollama service:
cp Modelfile $(harbor home)/services/ollama/modelfiles/Modelfile

# 4. Import the Modelfile back:
# "/modelfiles" is where the shared folder from above is mounted
harbor ollama create -f /modelfiles/Modelfile model-128k

# 4. Verify the import
harbor ollama show model-128k
```

## ComfyUI - Open WebUI can't connect

Open WebUI might display connection errors when trying to reach default ComfyUI installation in Harbor.

#### Workarond

Unfortunately, Open WebUI doesn't support any kind of auth settings for ComfyUI connections. So Harbor's ComfyUI auth must be disabled for the integration to work.

```bash
harbor comfyui auth false
```

## OptiLLM

### `optillm` x `webui`

- OptiLLM streaming implementation is not compatible with Open WebUI
- Connecting to both OptiLLM and original backend will only allow using original models

#### Workaround

- Turn off streaming in the chat settings when using OptiLLM models (or proxying them via other service).
- Use a proxy that allows overwrite model IDs (for example, `litellm`)