Worker-VLLM

Prerequisites

• Docker

Option 1: Deploy Any Model Using Pre-Built Docker Image [Recommended]

🚀 Deploy Guide: Follow our step-by-step deployment guide to deploy using the Runpod Console.

📦 Docker Image: runpod/worker-v1-vllm:<version>

• Available Versions: See GitHub Releases
• CUDA Compatibility: Requires CUDA >= 13.0

Option 2: Build Docker Image with Model Inside

To build an image with the model baked in, you must specify the following docker arguments when building the image.

Example: Building an image with OpenChat-3.5

docker build -t username/image:tag --build-arg MODEL_NAME="openchat/openchat_3.5" --build-arg BASE_PATH="/models" .

Example: Building with vLLM Nightly

To use the latest unreleased vLLM build (installs from the nightly wheel index and transformers from source):

docker build -t username/image:tag --build-arg VLLM_NIGHTLY=true .

You can combine it with other arguments:

docker build -t username/image:tag --build-arg VLLM_NIGHTLY=true --build-arg MODEL_NAME="meta-llama/Llama-3.1-8B-Instruct" --build-arg BASE_PATH="/models" .

Modifying your OpenAI Codebase to use your deployed vLLM Worker

Python (similar to Node.js, etc.):

1. When initializing the OpenAI Client in your code, change the api_key to your Runpod API Key and the base_url to your Runpod Serverless Endpoint URL in the following format: https://api.runpod.ai/v2/<YOUR ENDPOINT ID>/openai/v1, filling in your deployed endpoint ID. For example, if your Endpoint ID is abc1234, the URL would be https://api.runpod.ai/v2/abc1234/openai/v1.

• Before:

   from openai import OpenAI

   client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY"))
   

• After:

   from openai import OpenAI

   client = OpenAI(
       api_key=os.environ.get("RUNPOD_API_KEY"),
       base_url="https://api.runpod.ai/v2/<YOUR ENDPOINT ID>/openai/v1",
   )
   

2. Change the model parameter to your deployed model's name whenever using Completions or Chat Completions. - Before:

   response = client.chat.completions.create(
       model="gpt-3.5-turbo",
       messages=[{"role": "user", "content": "Why is Runpod the best platform?"}],
       temperature=0,
       max_tokens=100,
   )
   

   response = client.chat.completions.create(
       model="<YOUR DEPLOYED MODEL REPO/NAME>",
       messages=[{"role": "user", "content": "Why is Runpod the best platform?"}],
       temperature=0,
       max_tokens=100,
   )
   

Using http requests:

1. Change the Authorization header to your Runpod API Key and the url to your Runpod Serverless Endpoint URL in the following format: https://api.runpod.ai/v2/<YOUR ENDPOINT ID>/openai/v1 - Before:

   curl https://api.openai.com/v1/chat/completions \
   -H "Content-Type: application/json" \
   -H "Authorization: Bearer $OPENAI_API_KEY" \
   -d '{
   "model": "gpt-4",
   "messages": [
     {
       "role": "user",
       "content": "Why is Runpod the best platform?"
     }
   ],
   "temperature": 0,
   "max_tokens": 100
   }'
   

   curl https://api.runpod.ai/v2/<YOUR ENDPOINT ID>/openai/v1/chat/completions \
   -H "Content-Type: application/json" \
   -H "Authorization: Bearer <YOUR OPENAI API KEY>" \
   -d '{
   "model": "<YOUR DEPLOYED MODEL REPO/NAME>",
   "messages": [
     {
       "role": "user",
       "content": "Why is Runpod the best platform?"
     }
   ],
   "temperature": 0,
   "max_tokens": 100
   }'
   

Usage: OpenAI Compatibility

The vLLM Worker is fully compatible with OpenAI's API, and you can use it with any OpenAI Codebase by changing only 3 lines in total. The supported routes are <ins>Chat Completions</ins>, <ins>Models</ins>, <ins>Responses</ins>, and <ins>Messages</ins> - with both streaming and non-streaming.

Examples: Using your Runpod endpoint with OpenAI

First, initialize the OpenAI Client with your Runpod API Key and Endpoint URL:

```python from openai import OpenAI import os

Usage: Standard (Non-OpenAI)

Setting up the Serverless Worker

Configuration

Configure worker-vllm using environment variables:

Pass any vLLM engine arg not listed above by setting an environment variable with the UPPERCASED field name (same names vLLM uses). The worker auto-discovers all AsyncEngineArgs fields from env. For example:

Any env var whose name matches a valid AsyncEngineArgs field (uppercased) is applied automatically. Backward-compat aliases: MODEL_NAME, TOKENIZER_NAME, MAX_CONTEXT_LEN_TO_CAPTURE. This lets you configure any vLLM option without waiting for explicit worker support.

For the complete list of all available environment variables, examples, and detailed descriptions: Configuration

(Optional) Including Huggingface Token

If the model you would like to deploy is private or gated, you will need to include it during build time as a Docker secret, which will protect it from being exposed in the image and on DockerHub.

1. Enable Docker BuildKit (required for secrets).

export DOCKER_BUILDKIT=1

1. Export your Hugging Face token as an environment variable

export HF_TOKEN="your_token_here"

1. Add the token as a secret when building

docker build -t username/image:tag --secret id=HF_TOKEN --build-arg MODEL_NAME="openchat/openchat_3.5" .

OpenAI-Compatible vLLM Serverless Endpoint Worker

Deploy OpenAI-Compatible Blazing-Fast LLM Endpoints powered by the vLLM Inference Engine on Runpod Serverless with just a few clicks.

</div>

Current vLLM version: 0.20.2

Check out our Load Balancer implementation here: vLLM Load Balancer

Initialize the OpenAI Client with your Runpod API Key and Endpoint URL

client = OpenAI( api_key=os.environ.get("RUNPOD_API_KEY"), base_url="https://api.runpod.ai/v2/<YOUR ENDPOINT ID>/openai/v1", ) ```

OpenAI Responses API

Path: /openai/v1/responses (full URL: https://api.runpod.ai/v2/<YOUR ENDPOINT ID>/openai/v1/responses)

Supports the OpenAI Responses API request shape. Like other /openai/ routes, this is served directly—use the /openai/ prefix rather than the RunPod native job queue for these calls.

{
  "model": "meta-llama/Llama-3.1-8B-Instruct",
  "input": "Tell me a joke."
}

Using HTTP requests:

curl https://api.runpod.ai/v2/<YOUR ENDPOINT ID>/openai/v1/responses \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <YOUR RUNPOD API KEY>" \
  -d '{
    "model": "<YOUR DEPLOYED MODEL REPO/NAME>",
    "input": "Tell me a joke."
  }'

Anthropic Messages API

Path: /openai/v1/messages (full URL: https://api.runpod.ai/v2/<YOUR ENDPOINT ID>/openai/v1/messages)

Supports the Anthropic Messages API format. Served directly, bypassing the RunPod queue.

{
  "model": "meta-llama/Llama-3.1-8B-Instruct",
  "max_tokens": 256,
  "messages": [
    {"role": "user", "content": "Hello!"}
  ]
}

Using HTTP requests:

curl https://api.runpod.ai/v2/<YOUR ENDPOINT ID>/openai/v1/messages \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <YOUR RUNPOD API KEY>" \
  -d '{
    "model": "<YOUR DEPLOYED MODEL REPO/NAME>",
    "max_tokens": 256,
    "messages": [
      {"role": "user", "content": "Hello!"}
    ]
  }'