# Chat Completions Endpoint Documentation

## Overview

The chat completions endpoints create model responses for chat-based conversations. Given a list of messages comprising a conversation, the model will return a response. These endpoints support streaming, tool calling, and multiple providers.

## Endpoints

```
POST /v1/chat/completions
POST /{provider}/v1/chat/completions
POST /v1/responses
```

### Endpoint Variants

- **`/v1/chat/completions`**: Standard OpenAI-compatible endpoint. Provider is determined from the model name.
- **`/{provider}/v1/chat/completions`**: Provider-specific endpoint (e.g., `/ollama/v1/chat/completions`).
- **`/v1/responses`**: Real-time responses API (similar to OpenAI Responses API), optimized for real-time interaction with streaming enabled.

## Request Body

### Parameters

#### `messages` (array, **Required**)

A list of messages comprising the conversation so far.

**Message Object:**
- `role` (string, **Required**): The role of the message author. One of: `system`, `user`, `assistant`, or `tool`.
- `content` (string or null, **Optional**): The contents of the message. Can be null for assistant messages with tool calls.
- `name` (string, **Optional**): An optional name for the participant. Provides the model information to differentiate between participants of the same role.
- `tool_calls` (array, **Optional**): The tool calls generated by the model (only for assistant messages).
- `tool_call_id` (string, **Optional**): Tool call that this message is responding to (only for tool messages).

**Example:**
```json
"messages": [
  {
    "role": "system",
    "content": "You are a helpful assistant."
  },
  {
    "role": "user",
    "content": "What is the weather like in San Francisco?"
  }
]
```

#### `model` (string, **Required**)

ID of the model to use. Use `provider/model` format for automatic routing.

**Details:**
- Format: `provider/model-name`
- Provider is auto-detected if not specified in the path
- You can use the List models API to see all available models
- Filter generation models: `GET /v1/models?type=text-generation`

**Supported Providers:**
- **OpenAI**: `openai/gpt-4`, `openai/gpt-3.5-turbo`
- **Anthropic**: `anthropic/claude-3-opus-20240229`, `anthropic/claude-3-sonnet-20240229`
- **Ollama**: `ollama/llama3:latest`, `ollama/qwen3-coder:30b`
- **LMStudio**: `lmstudio/local-model`
- **MLX**: `mlx/model-name`
- **HuggingFace**: `huggingface/model-name`

**Examples:**
```json
"model": "openai/gpt-4"
"model": "ollama/llama3:latest"
"model": "anthropic/claude-3-opus-20240229"
```

#### `temperature` (number, **Optional**)

What sampling temperature to use, between 0 and 2.

**Details:**
- Higher values like 0.8 will make the output more random
- Lower values like 0.2 will make it more focused and deterministic
- Generally recommended to alter this or `top_p` but not both

**Default:** `0.7`  
**Range:** `0.0` to `2.0`

**Example:**
```json
"temperature": 0.7
```

#### `max_tokens` (integer, **Optional**)

The maximum number of tokens that can be generated in the chat completion.

**Details:**
- The total length of input tokens and generated tokens is limited by the model's context length
- If not specified, uses the model's default maximum output tokens
- For most models, this is 2048-4096 tokens

**Default:** `null` (model default)

**Example:**
```json
"max_tokens": 2048
```

#### `top_p` (number, **Optional**)

An alternative to sampling with temperature, called nucleus sampling.

**Details:**
- The model considers the results of the tokens with top_p probability mass
- For example, 0.1 means only the tokens comprising the top 10% probability mass are considered
- Generally recommended to alter this or `temperature` but not both

**Default:** `1.0`  
**Range:** `0.0` to `1.0`

**Example:**
```json
"top_p": 1.0
```

#### `stream` (boolean, **Optional**)

If set, partial message deltas will be sent as server-sent events.

**Details:**
- Tokens will be sent as data-only server-sent events as they become available
- The stream will be terminated by a `data: [DONE]` message
- Useful for real-time chat interfaces

**Default:** `false`

**Example:**
```json
"stream": true
```

#### `stop` (string or array, **Optional**)

Up to 4 sequences where the API will stop generating further tokens.

**Details:**
- Can be a string or array of strings
- The returned text will not contain the stop sequence

**Default:** `null`

**Examples:**
```json
"stop": "\n"
"stop": ["\n", "User:", "Assistant:"]
```

#### `tools` (array, **Optional**)

A list of tools the model may call. Currently, only functions are supported as a tool.

**Details:**
- Use this to provide a list of functions the model may generate JSON inputs for
- Maximum of 128 functions supported

**Tool Object:**
- `type` (string): Always `"function"`
- `function` (object):
  - `name` (string): The name of the function
  - `description` (string): A description of what the function does
  - `parameters` (object): JSON Schema object describing the function parameters

**Example:**
```json
"tools": [
  {
    "type": "function",
    "function": {
      "name": "get_weather",
      "description": "Get the current weather in a given location",
      "parameters": {
        "type": "object",
        "properties": {
          "location": {
            "type": "string",
            "description": "The city and state, e.g. San Francisco, CA"
          },
          "unit": {
            "type": "string",
            "enum": ["celsius", "fahrenheit"]
          }
        },
        "required": ["location"]
      }
    }
  }
]
```

#### `tool_choice` (string or object, **Optional**)

Controls which (if any) tool is called by the model.

**Options:**
- `"none"`: The model will not call any tool and will generate a message
- `"auto"`: The model can pick between generating a message or calling one or more tools
- `"required"`: The model must call one or more tools
- Object: Specifies a particular tool via `{"type": "function", "function": {"name": "my_function"}}`

**Default:** `"auto"`

**Examples:**
```json
"tool_choice": "auto"
"tool_choice": "none"
"tool_choice": {"type": "function", "function": {"name": "get_weather"}}
```

#### `frequency_penalty` (number, **Optional**)

Number between -2.0 and 2.0. Positive values penalize new tokens based on their existing frequency in the text so far.

**Details:**
- Decreases the model's likelihood to repeat the same line verbatim
- Higher values (e.g., 1.0) reduce repetition
- Negative values encourage repetition

**Default:** `0.0`  
**Range:** `-2.0` to `2.0`

**Example:**
```json
"frequency_penalty": 0.0
```

#### `presence_penalty` (number, **Optional**)

Number between -2.0 and 2.0. Positive values penalize new tokens based on whether they appear in the text so far.

**Details:**
- Increases the model's likelihood to talk about new topics
- Higher values (e.g., 1.0) encourage new topics
- Negative values encourage staying on topic

**Default:** `0.0`  
**Range:** `-2.0` to `2.0`

**Example:**
```json
"presence_penalty": 0.0
```

#### `seed` (integer, **Optional**)

If specified, the system will make a best effort to sample deterministically.

**Details:**
- Useful for reproducible outputs
- Determinism is not guaranteed but will be best-effort
- Identical requests with the same seed should return similar results

**Default:** `null`

**Example:**
```json
"seed": 12345
```

#### `agent_format` (string, **Optional**)

Target agent format for tool call syntax conversion (AbstractCore-specific feature).

**Options:**
- `"auto"`: Auto-detect format from model and user-agent
- `"openai"`: OpenAI tool call format
- `"codex"`: Codex tool call format
- `"qwen3"`: Qwen3 tool call format
- `"llama3"`: LLaMA3 tool call format
- `"passthrough"`: No conversion

**Default:** `"auto"`

**Example:**
```json
"agent_format": "auto"
```

## Example Requests

### Basic Chat Completion

```bash
curl -X POST https://api.yourdomain.com/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-4",
    "messages": [
      {
        "role": "system",
        "content": "You are a helpful assistant."
      },
      {
        "role": "user",
        "content": "What is the capital of France?"
      }
    ],
    "temperature": 0.7,
    "max_tokens": 150
  }'
```

### Streaming Response

```bash
curl -X POST https://api.yourdomain.com/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ollama/llama3:latest",
    "messages": [
      {"role": "user", "content": "Tell me a story"}
    ],
    "stream": true
  }'
```

### With Tool Calling

```bash
curl -X POST https://api.yourdomain.com/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-4",
    "messages": [
      {"role": "user", "content": "What is the weather in San Francisco?"}
    ],
    "tools": [
      {
        "type": "function",
        "function": {
          "name": "get_weather",
          "description": "Get current weather",
          "parameters": {
            "type": "object",
            "properties": {
              "location": {
                "type": "string",
                "description": "City and state"
              }
            },
            "required": ["location"]
          }
        }
      }
    ],
    "tool_choice": "auto"
  }'
```

### Provider-Specific Endpoint

```bash
curl -X POST https://api.yourdomain.com/ollama/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3:latest",
    "messages": [
      {"role": "user", "content": "Hello!"}
    ]
  }'
```

## Response Format

### Non-Streaming Response

```json
{
  "id": "chatcmpl-abc123",
  "object": "chat.completion",
  "created": 1699896916,
  "model": "openai/gpt-4",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "The capital of France is Paris."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 20,
    "completion_tokens": 10,
    "total_tokens": 30
  }
}
```

### Response Fields

#### `id` (string)
A unique identifier for the chat completion.

#### `object` (string)
The object type, always `"chat.completion"`.

#### `created` (integer)
The Unix timestamp (in seconds) of when the chat completion was created.

#### `model` (string)
The model used for the chat completion (with provider prefix).

#### `choices` (array)
A list of chat completion choices. Can contain more than one if `n` is greater than 1.

**Choice Object:**
- `index` (integer): The index of this choice
- `message` (object): The message generated by the model
  - `role` (string): Always `"assistant"`
  - `content` (string or null): The content of the message
  - `tool_calls` (array, optional): Tool calls generated by the model
- `finish_reason` (string): The reason the model stopped generating
  - `"stop"`: Natural stop point or stop sequence
  - `"length"`: Maximum token limit reached
  - `"tool_calls"`: Model called one or more tools
  - `"content_filter"`: Content was filtered

#### `usage` (object)
Usage statistics for the completion request.

**Fields:**
- `prompt_tokens` (integer): Number of tokens in the prompt
- `completion_tokens` (integer): Number of tokens in the completion
- `total_tokens` (integer): Total tokens used

### Streaming Response

When `stream: true`, responses are sent as server-sent events:

```
data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1699896916,"model":"openai/gpt-4","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]}

data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1699896916,"model":"openai/gpt-4","choices":[{"index":0,"delta":{"content":"The"},"finish_reason":null}]}

data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1699896916,"model":"openai/gpt-4","choices":[{"index":0,"delta":{"content":" capital"},"finish_reason":null}]}

...

data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1699896916,"model":"openai/gpt-4","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: [DONE]
```

**Chunk Object:**
- `object`: Always `"chat.completion.chunk"`
- `choices[].delta`: Contains incremental updates
  - First chunk: Contains `role`
  - Subsequent chunks: Contains `content` pieces
  - Last chunk: Empty `delta` with `finish_reason`

### Tool Calls Response

When the model calls a tool:

```json
{
  "id": "chatcmpl-abc123",
  "object": "chat.completion",
  "created": 1699896916,
  "model": "openai/gpt-4",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": null,
        "tool_calls": [
          {
            "id": "call_abc123",
            "type": "function",
            "function": {
              "name": "get_weather",
              "arguments": "{\"location\": \"San Francisco, CA\"}"
            }
          }
        ]
      },
      "finish_reason": "tool_calls"
    }
  ],
  "usage": {
    "prompt_tokens": 35,
    "completion_tokens": 25,
    "total_tokens": 60
  }
}
```

## Use Cases

### 1. Conversational AI / Chatbots

```python
import requests

def chat(user_message, history=[]):
    history.append({"role": "user", "content": user_message})
    
    response = requests.post("http://localhost:8000/v1/chat/completions", json={
        "model": "openai/gpt-4",
        "messages": [
            {"role": "system", "content": "You are a helpful assistant."}
        ] + history,
        "temperature": 0.7
    })
    
    assistant_message = response.json()["choices"][0]["message"]["content"]
    history.append({"role": "assistant", "content": assistant_message})
    
    return assistant_message, history

# Usage
response, history = chat("Hello!")
print(response)
response, history = chat("What's the weather like?", history)
print(response)
```

### 2. Streaming Chat Interface

```python
import requests

def stream_chat(message):
    response = requests.post(
        "http://localhost:8000/v1/chat/completions",
        json={
            "model": "ollama/llama3:latest",
            "messages": [{"role": "user", "content": message}],
            "stream": True
        },
        stream=True
    )
    
    for line in response.iter_lines():
        if line:
            line = line.decode('utf-8')
            if line.startswith('data: '):
                data = line[6:]
                if data == '[DONE]':
                    break
                import json
                chunk = json.loads(data)
                content = chunk['choices'][0]['delta'].get('content', '')
                if content:
                    print(content, end='', flush=True)

stream_chat("Tell me a story")
```

### 3. Tool/Function Calling

```python
import requests
import json

def chat_with_tools(message):
    response = requests.post(
        "http://localhost:8000/v1/chat/completions",
        json={
            "model": "openai/gpt-4",
            "messages": [{"role": "user", "content": message}],
            "tools": [
                {
                    "type": "function",
                    "function": {
                        "name": "get_weather",
                        "description": "Get weather for a location",
                        "parameters": {
                            "type": "object",
                            "properties": {
                                "location": {"type": "string"}
                            },
                            "required": ["location"]
                        }
                    }
                }
            ]
        }
    )
    
    result = response.json()
    message = result["choices"][0]["message"]
    
    if message.get("tool_calls"):
        # Model wants to call a tool
        for tool_call in message["tool_calls"]:
            function_name = tool_call["function"]["name"]
            arguments = json.loads(tool_call["function"]["arguments"])
            print(f"Model wants to call: {function_name}({arguments})")
    else:
        print(f"Response: {message['content']}")

chat_with_tools("What's the weather in Tokyo?")
```

### 4. Multi-Provider Support

```python
import requests

def chat_multi_provider(message, provider="openai"):
    models = {
        "openai": "openai/gpt-4",
        "anthropic": "anthropic/claude-3-opus-20240229",
        "ollama": "ollama/llama3:latest"
    }
    
    response = requests.post(
        "http://localhost:8000/v1/chat/completions",
        json={
            "model": models[provider],
            "messages": [{"role": "user", "content": message}]
        }
    )
    
    return response.json()["choices"][0]["message"]["content"]

# Use different providers
print(chat_multi_provider("Hello!", "openai"))
print(chat_multi_provider("Hello!", "anthropic"))
print(chat_multi_provider("Hello!", "ollama"))
```

## Discovering Available Models

### List All Text Generation Models

```bash
curl http://localhost:8000/v1/models?type=text-generation
```

### List Provider-Specific Models

```bash
# Ollama models
curl http://localhost:8000/v1/models?provider=ollama&type=text-generation

# OpenAI models
curl http://localhost:8000/v1/models?provider=openai&type=text-generation

# Anthropic models
curl http://localhost:8000/v1/models?provider=anthropic&type=text-generation
```

## Error Responses

### Invalid Model

```json
{
  "error": {
    "message": "Model 'invalid-model' not found",
    "type": "invalid_request_error",
    "code": "model_not_found"
  }
}
```

### Missing Required Parameter

```json
{
  "detail": [
    {
      "loc": ["body", "messages"],
      "msg": "field required",
      "type": "value_error.missing"
    }
  ]
}
```

### Rate Limit Exceeded

```json
{
  "error": {
    "message": "Rate limit exceeded",
    "type": "rate_limit_error"
  }
}
```

## Best Practices

### 1. Use System Messages

Always include a system message to set the behavior:

```json
{
  "messages": [
    {"role": "system", "content": "You are a helpful coding assistant."},
    {"role": "user", "content": "How do I write a Python function?"}
  ]
}
```

### 2. Manage Context Length

Monitor token usage and trim conversation history when needed:

```python
def trim_history(messages, max_tokens=4000):
    # Keep system message and recent messages
    system_messages = [m for m in messages if m["role"] == "system"]
    recent_messages = messages[-10:]  # Keep last 10 messages
    return system_messages + recent_messages
```

### 3. Handle Tool Calls Properly

When the model returns tool calls, execute them and send results back:

```python
# 1. Get tool call from model
response = chat_with_tools(user_message)

# 2. Execute tool
result = execute_tool(tool_call)

# 3. Send result back to model
messages.append({
    "role": "tool",
    "tool_call_id": tool_call["id"],
    "content": json.dumps(result)
})

# 4. Get final response
final_response = chat(messages)
```

### 4. Use Streaming for Better UX

For user-facing applications, use streaming to show progress:

```python
response = requests.post(
    "/v1/chat/completions",
    json={"model": "...", "messages": [...], "stream": True},
    stream=True
)
```

### 5. Set Appropriate Temperature

- **Low temperature (0.1-0.3)**: Factual answers, code generation
- **Medium temperature (0.5-0.8)**: Balanced creativity
- **High temperature (0.9-1.5)**: Creative writing, brainstorming

### 6. Provider-Specific Optimizations

Choose the right provider for your use case:
- **OpenAI**: Best quality, highest cost
- **Anthropic**: Long context, strong reasoning
- **Ollama**: Local/private, no API costs
- **LMStudio**: Local development, model testing

## Endpoint Differences

### `/v1/chat/completions`
- Standard OpenAI-compatible endpoint
- Provider determined from model name
- Most flexible

### `/{provider}/v1/chat/completions`
- Provider-specific routing
- Model name doesn't need provider prefix
- Useful for provider-specific features

### `/v1/responses`
- Real-time responses API
- Always streams responses (`stream: true` forced)
- Optimized for real-time interaction
- Similar to OpenAI's Realtime API

## Related Endpoints

- **List Models**: `GET /v1/models?type=text-generation`
- **Embeddings**: `POST /v1/embeddings`
- **Health Check**: `GET /health`
- **Providers**: `GET /providers`

---

**OpenAI Compatibility**: These endpoints follow the OpenAI Chat Completions API format, making them compatible with existing tools and libraries that support OpenAI's API, including LangChain, LlamaIndex, and more.

