### [vLLM](https://github.com/vllm-project/vllm)

> Handle: `vllm`<br/>
> URL: [http://localhost:33911](http://localhost:33911)

<p align="center">
  <picture>
    <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/vllm-project/vllm/main/docs/source/assets/logos/vllm-logo-text-dark.png">
    <img alt="vLLM" src="https://raw.githubusercontent.com/vllm-project/vllm/main/docs/source/assets/logos/vllm-logo-text-light.png" width=55%>
  </picture>
</p>

A high-throughput and memory-efficient inference and serving engine for LLMs

#### Starting

```bash
# [Optional] pre-build the vLLM image
harbor build vllm

# Start the vLLM service
harbor up vllm
```

- Harbor builds custom `vllm` image with `bitsandbytes`
- `vllm` will be connected to `webui`, `aider`, `boost`, `chatui` and some other services when running together
- Official docker images require specific CUDA versions - beware
- When ROCm is detected, Harbor mounts AMD devices and starts vLLM explicitly via `python3 -m vllm.entrypoints.openai.api_server`, which supports AMD ROCm images that do not provide an OpenAI-server entrypoint

#### Integrations

`vllm` is a high-throughput OpenAI-compatible inference server (listening on port 8000 inside the container) that many Harbor frontends and satellite tools auto-discover and use when started together. Wiring is performed exclusively through the cross-compose overlay files (`compose.x.*.vllm.yml` and a few always-on config mergers).

**Frontends and tools that auto-configure for vllm**

- `webui` — mounts `./services/webui/configs/config.vllm.json` (registers OpenAI provider at `http://vllm:8000/v1` with key `sk-vllm`).
- `chatui` — mounts `./services/chatui/configs/chatui.vllm.yml` (adds a `MODELS` entry for vllm using the same internal URL/key).
- `aider` — mounts `./services/aider/configs/aider.vllm.yml` (OpenAI base + key; model chosen via `HARBOR_AIDER_MODEL`).
- `boost` — injects `HARBOR_BOOST_OPENAI_URL_VLLM` and `HARBOR_BOOST_OPENAI_KEY_VLLM` so boost can target vllm as one of its backends.
- `litellm` — mounts `./services/litellm/litellm.vllm.yaml` (adds a model entry under the `vllm` name; the example model can be changed by the user).
- `opint` (Open Interpreter) — completely overrides the container entrypoint to pass `--api_base http://vllm:8000/v1 --api_key sk-vllm` plus the user command.
- `optillm` — sets `OPTILLM_BASE_URL=http://vllm:8000/v1` and the matching API key.
- `raglite` — sets `VLLM_BASE_URL=http://vllm:8000` (note: no `/v1` suffix for this client).
- `astrbot` — sets `HARBOR_ASTRBOT_VLLM_ENDPOINT` (comment in the overlay also reminds users they can configure it in the AstrBot WebUI).
- `cognee` — declares `depends_on: [vllm]`, forces `LLM_PROVIDER=openai`, and wires `LLM_ENDPOINT`, `LLM_API_KEY`, plus `LLM_MODEL=${HARBOR_VLLM_MODEL}`.
- `mindsdb` — sets `LLM_FUNCTION_BASE_URL=http://vllm:8000/v1` and the OpenAI key so MindsDB functions can call the model.
- `mi` — `depends_on: [vllm]`, sets `OPENAI_BASE_URL=http://vllm:8000` (no `/v1`) + `OPENAI_API_KEY` and `MODEL=${HARBOR_MI_VLLM_MODEL}` (plus a second overlay for the harbor mi variant that supplies a default model).
- `ml-intern` — `depends_on: [vllm]`, supplies both `VLLM_*` and `LOCAL_LLM_*` pairs plus `ML_INTERN_MODEL=vllm/${HARBOR_ML_INTERN_VLLM_MODEL}`.
- `npcsh` — `depends_on: [vllm]`, configures a large set of `*MODEL` / `*PROVIDER` variables (all pointing at the same vllm model via `HARBOR_NPCSH_VLLM_MODEL`) together with multiple OpenAI-compatible base URLs and the key.
- `open-design` — `depends_on: [vllm]`, sets protocol, base URL, key and model defaults using the `HARBOR_OPEN_DESIGN_VLLM_MODEL` and `HARBOR_OPEN_DESIGN_*` family.
- `aichat` — the custom `start_aichat.sh` + yaml merger always concatenates every `aichat.*.yml` file present in the image (including `aichat.vllm.yml`). This gives aichat an openai-compatible client for vllm (using `${HARBOR_AICHAT_MODEL}`) whenever the vllm service is reachable on the network.

**Reverse proxy / external access**

- When `traefik` is also running, `compose.x.traefik.vllm.yml` (auto-generated) attaches the standard labels so vllm becomes available at the public URL `https://vllm.${HARBOR_TRAEFIK_DOMAIN}` on the `websecure` entrypoint. It joins the `traefik-public` network in addition to the internal `harbor-network`.

**Hardware acceleration (GPU / accelerator)**

- NVIDIA: `compose.x.vllm.nvidia.yml` requests `count: all` GPUs with the `nvidia` driver and `gpu` capability.
- ROCm / AMD: `compose.x.vllm.rocm.yml` mounts the two device nodes and forces the explicit `python3 -m vllm.entrypoints.openai.api_server` entrypoint (required for some ROCm images).
- CDI (modern container device interface): `compose.x.vllm.cdi.yml` (generated) uses the `cdi` driver with `nvidia.com/gpu=all`.
- The image itself is always produced by the local `services/vllm/Dockerfile` (which does `pip install bitsandbytes` on top of whatever `HARBOR_VLLM_IMAGE:HARBOR_VLLM_VERSION` you have chosen). GPU overlays are applied after the build step.

**Source builds**

Running `harbor build vllm` (recommended before first GPU use or when you change the version) executes the local Dockerfile. The resulting image carries the bitsandbytes addition and is tagged the same way the official image would have been, so every consumer overlay and GPU compose file continues to work without modification.

**Host / volume interactions**

- **Model weights (Hugging Face cache)**: `${HARBOR_HF_CACHE:-./services/vllm/.hf-cache}` on the host is bind-mounted to `/root/.cache/huggingface` inside the container. All `harbor hf download`, `harbor pull`, and `harbor vllm model` commands populate this cache; vLLM reads its models directly from it.
- **vLLM runtime cache**: `${HARBOR_VLLM_CACHE}` → `/root/.cache/vllm` (stores compiled CUDA kernels, prefix caches, etc.).
- The same vLLM cache volume is also mounted by `ollama`, `localai`, `sglang` and `kobold` (shared cache directory convention).
- `ipc: host` gives the container direct access to host IPC namespace — required for optimal multi-GPU and shared-memory performance with vLLM.
- No persistent `./services/vllm/data` volume (vLLM loads a single model via the specifier or extra args; multi-model serving is usually done via an external router such as litellm or boost).

When you execute a command such as `harbor up vllm webui chatui aider boost cognee` the cross-compose machinery guarantees that every listed frontend immediately sees a working vLLM backend at the internal address with the conventional key, and any GPU/ROCm/CDI requirements are satisfied automatically.

#### Models

- [Official vLLM models](https://docs.vllm.ai/en/latest/models/supported_models.html)
- [Quantization kernels](https://docs.vllm.ai/en/latest/quantization/supported_hardware.html)

Once you've found a model you want to run, you can configure it with Harbor:

```bash
# Quickly lookup some of the compatible quants
harbor hf find awq
harbor hf find gptq

# This propagates the settings
# to the relevant configuration files
harbor vllm model Qwen/Qwen3.5-4B

# To run a gated model, ensure that you've
# also set your Huggingface API Token
harbor hf token <your-token>
```

#### Configuration

- [Engine CLI Arguments](https://docs.vllm.ai/en/stable/serving/engine_args.html)
- [Environment Variables](https://docs.vllm.ai/en/stable/serving/env_vars.html)

You can configure specific portions of vllm via Harbor CLI:

```bash
# See original CLI help
harbor run vllm --help

# Get/Set the extra arguments
harbor vllm args
harbor vllm args '--dtype bfloat16 --code-revision 3.5'

harbor config set vllm.host.port 4090
```

**Version and update**

```bash
# Get/set desired vLLM version
harbor vllm version
# Command accepts a docker tag
harbor vllm version latest
# Customize docker image
harbor config set vllm.image custom/vllm
# Force-pull new version of the base image
# if you have set version to "latest"
docker pull $(harbor config get vllm.image):$(harbor config get vllm.version)
```

**ROCm / AMD**

Use AMD ROCm images instead of CUDA images when running on AMD hardware. Pin to an image that supports your GPU architecture when possible. Harbor includes a ROCm compose override that mounts `/dev/kfd` and `/dev/dri` and starts vLLM explicitly with the OpenAI API server entrypoint; some AMD images do not provide that entrypoint themselves.

```bash
# Example for AMD Strix Halo / gfx1151
harbor config set vllm.image rocm/vllm-dev
harbor vllm version preview_v0.20.0_20260429
harbor build vllm
```

Some newer multimodal Qwen checkpoints need text-only mode for chat inference to avoid multimodal warmup/profile memory spikes:

```bash
harbor vllm model Qwen/Qwen3.6-35B-A3B
harbor vllm args '--max-model-len 4096 --gpu-memory-utilization 0.80 --dtype half --language-model-only --attention-backend TRITON_ATTN'
```

If graph capture fails, add `--enforce-eager` as a fallback. If it succeeds, leaving eager mode off is usually faster.

**AMD Strix Halo / Radeon 8060S / gfx1151 notes**

Strix Halo works with vLLM, but it is not yet a normal upstream ROCm experience. Treat vLLM on this hardware as a throughput-oriented, version-sensitive setup rather than the fastest single-user chat backend.

Key learnings from local testing:

- `rocm/vllm:latest` may be unusable on gfx1151; a tiny PyTorch GPU operation segfaulted in testing.
- `rocm/vllm-dev:preview_v0.20.0_20260429` worked for `Qwen/Qwen3.6-35B-A3B` HF safetensors.
- vLLM does not currently load the Qwen3.6 GGUF path used by llama.cpp: `qwen35moe` GGUF is not supported.
- ROCm AITER MoE did not work on gfx1151 in the tested image; use Triton/auto MoE fallback unless you are using a Strix-specific patched stack.
- Single-request latency is still behind llama.cpp GGUF. vLLM becomes more interesting for concurrent/batched OpenAI-compatible serving.

A known Strix-specific stack is [`kyuz0/vllm-therock-gfx1151`](https://github.com/kyuz0/amd-strix-halo-vllm-toolboxes), built on TheRock ROCm nightlies with gfx1151 patches. It can be run outside Harbor for comparison:

```bash
docker run --rm -it \
  --device /dev/kfd --device /dev/dri \
  --group-add video --group-add render \
  --security-opt seccomp=unconfined \
  --ipc=host \
  -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v ~/.cache/vllm-strix:/root/.cache/vllm \
  kyuz0/vllm-therock-gfx1151:stable \
  vllm serve Qwen/Qwen3.6-35B-A3B \
    --host 0.0.0.0 \
    --port 8000 \
    --max-model-len 4096 \
    --gpu-memory-utilization 0.90 \
    --max-num-seqs 1 \
    --dtype auto \
    --trust-remote-code \
    --language-model-only \
    --attention-backend TRITON_ATTN \
    --mm-encoder-attn-backend TRITON_ATTN
```

For better vLLM single-stream speed on Strix Halo, an AWQ checkpoint was faster than BF16 in local testing:

```bash
docker run --rm -it \
  --device /dev/kfd --device /dev/dri \
  --group-add video --group-add render \
  --security-opt seccomp=unconfined \
  --ipc=host \
  -p 8000:8000 \
  -e VLLM_USE_TRITON_AWQ=1 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v ~/.cache/vllm-strix-awq:/root/.cache/vllm \
  kyuz0/vllm-therock-gfx1151:stable \
  vllm serve cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit \
    --host 0.0.0.0 \
    --port 8000 \
    --max-model-len 4096 \
    --gpu-memory-utilization 0.90 \
    --max-num-seqs 8 \
    --dtype auto \
    --trust-remote-code \
    --language-model-only \
    --attention-backend TRITON_ATTN \
    --mm-encoder-attn-backend TRITON_ATTN
```

Measured locally on a Strix Halo / Radeon 8060S system:

| Setup | Single request | Concurrent aggregate |
| --- | ---: | ---: |
| `rocm/vllm-dev` + Qwen3.6 BF16 | ~15 tok/s | not measured |
| `rocm/vllm-dev` + Qwen3.6 BF16 + MTP | ~21-22 tok/s | not measured |
| `kyuz0/vllm-therock-gfx1151` + Qwen3.6 BF16 | ~14-20 tok/s, depending on MTP | not measured |
| `kyuz0/vllm-therock-gfx1151` + Qwen3.6 AWQ 4-bit | ~25-26 tok/s | ~53 tok/s at concurrency 8 |
| llama.cpp + Qwen3.6 Q8_0 GGUF on Vulkan/RADV | ~48-49 tok/s | not measured |
| llama.cpp + Qwen3.6 Q4_0-class GGUF on Vulkan/RADV | ~75 tok/s | not measured |

For interactive local chat on Strix Halo, prefer [llama.cpp](./2.2.2-Backend&colon-llama.cpp.md). Use vLLM when you need its OpenAI-compatible server, batching, multi-client concurrency, or vLLM-specific model support.

**Speculative decoding / MTP**

Speculative decoding is configured through vLLM's JSON-style `--speculative-config` flags. It is model- and backend-sensitive; benchmark before making it your default.

For checkpoints that include MTP weights, such as `Qwen/Qwen3.6-35B-A3B`, the target model can draft its own next token:

```bash
harbor vllm args '--max-model-len 4096 --gpu-memory-utilization 0.80 --dtype half --language-model-only --attention-backend TRITON_ATTN --max-num-seqs 256 --speculative-config.method mtp --speculative-config.num_speculative_tokens 1'
```

Draft models must use the same tokenizer vocabulary as the target model. For example, `Qwen/Qwen3-0.6B` is not a compatible draft for `Qwen/Qwen3.6-35B-A3B` because the vocabulary sizes differ. `Qwen/Qwen3.5-0.8B` has a matching vocabulary, but vLLM 0.20 currently treats Qwen3.5 draft models as MTP modules on this path and fails to load them as ordinary draft models when their hidden size differs from the target.

DFlash is another option when a matching drafter exists. On ROCm, do not force `TRITON_ATTN` for DFlash, and use `bfloat16` to avoid drafter dtype mismatches:

```bash
harbor vllm args '--max-model-len 4096 --gpu-memory-utilization 0.80 --dtype bfloat16 --language-model-only --max-num-seqs 256 --trust-remote-code --speculative-config.method dflash --speculative-config.model z-lab/Qwen3.6-35B-A3B-DFlash --speculative-config.num_speculative_tokens 8 --max-num-batched-tokens 32768'
```

`ngram` speculative decoding does not require an extra model, but it mainly helps prompt-copying/repetitive workloads and may be slower for general chat.

You can specify more options directly in `services/vllm/override.env`.

Following options can be set via [`harbor config`](./3.-Harbor-CLI-Reference.md#harbor-config):

```bash
# The port on the host machine where the vLLM service
# will be available
HARBOR_VLLM_HOST_PORT          33911

# Docker image to use for vLLM
HARBOR_VLLM_IMAGE              vllm/vllm-openai

# Docker image tag to use for vLLM
HARBOR_VLLM_VERSION            v0.8.5

# Default model to load on startup
HARBOR_VLLM_MODEL              Qwen/Qwen3.5-4B
```

#### VRAM

Below are some steps to take if running out of VRAM (no magic, though).

##### Limit Context Length

You can limit the context length to reduce the memory footprint. This can be done via the `--max-model-len` flag.

```bash
harbor vllm args --max-model-len 2048
```

##### Quantization

vLLM supports many different quantization formats. You would typically configure this via `--load-format` and `--quantization` flags. For example:

```bash
harbor vllm args --load-format bitsandbytes --quantization bitsandbytes
```

##### Offloading

vLLM supports partial offloading to the CPU, similar to [llama.cpp](./2.2.2-Backend&colon-llama.cpp.md) and some other backends. This can be configured via the `--cpu-offload-gb` flag.

```bash
harbor vllm args --cpu-offload-gb 4
```

##### Disable CUDA Graphs

When loading the model, VRAM usage can spike when computing the CUDA graphs. This can be disabled via `--enforce-eager` flag.

```bash
harbor vllm args --enforce-eager
```

##### GPU Memory Utilization

Reduce the amount of VRAM allocated for the model executor. Can be ranged from 0 to 1.0, `0.9` by default.

```bash
harbor vllm args --gpu-memory-utilization 0
```

##### Run on CPU

You can move to CPU by setting the `--device cpu` flag.

```bash
harbor vllm args --device cpu
```
