---
sidebar_position: 4
---

import Terminal from '@site/src/components/Terminal';

# Expose your LLM Provider

Now that your provider is fully set up, you can expose it to your organization. The idea is to do it through an [Otoroshi route](https://maif.github.io/otoroshi/manual/entities/routes.html) with a plugin of type `backend` that will handle passing incoming requests to the actual LLM provider.

## OpenAI compatible plugins

We provide a set of plugins capable of exposing any supported LLM provider through a [compatible OpenAI API](https://platform.openai.com/docs/api-reference/chat). This is the standard way of doing LLM exposition.

### Chat completions proxy

Plugin: **Cloud APIM - LLM OpenAI chat/completions Proxy**

```
cp:otoroshi_plugins.com.cloud.apim.otoroshi.extensions.aigateway.plugins.OpenAiCompatProxy
```

This plugin is compatible with the [OpenAI chat completion API](https://platform.openai.com/docs/api-reference/chat/create), including [streaming](https://platform.openai.com/docs/api-reference/chat-streaming).

#### Configuration

```js
{
  "refs": ["provider_entity_id_1", "provider_entity_id_2"]
}
```

| Parameter | Type | Description |
|-----------|------|-------------|
| `refs` | array | List of LLM Provider entity IDs |

![](/img/openai-chat-completion.png)

#### Usage

```sh
curl https://my-own-llm-endpoint.on.otoroshi/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OTOROSHI_API_KEY" \
  -d '{
    "model": "gpt-4o",
    "messages": [
      {
        "role": "user",
        "content": "Hello how are you!"
      }
    ]
  }'
```

Response:

```js
{
  "id": "chatcmpl-B9MBs8CjcvOU2jLn4n570S5qMJKcT",
  "object": "chat.completion",
  "created": 1741569952,
  "model": "gpt-4o",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Hello! How can I assist you today?"
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 19,
    "completion_tokens": 10,
    "total_tokens": 29
  }
}
```

#### Model routing

When multiple providers are configured in `refs`, you can target a specific provider using the `model` field:

- **Slash syntax**: `providerName/modelName` (e.g., `my_openai_provider/gpt-4o`)
- **Hash syntax**: `providerId###modelName` (e.g., `provider_xxx###gpt-4o`)

If no provider prefix is specified, the first configured ref is used.

### Text completions proxy

Plugin: **Cloud APIM - LLM OpenAI completions Proxy**

```
cp:otoroshi_plugins.com.cloud.apim.otoroshi.extensions.aigateway.plugins.OpenAiCompletionProxy
```

This plugin is compatible with the [OpenAI completions API](https://platform.openai.com/docs/api-reference/completions), including streaming. It converts the legacy completions format (`prompt` field) to the internal chat format.

#### Configuration

```js
{
  "refs": ["provider_entity_id"]
}
```

| Parameter | Type | Description |
|-----------|------|-------------|
| `refs` | array | List of LLM Provider entity IDs |

#### Usage

```sh
curl https://my-own-llm-endpoint.on.otoroshi/v1/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OTOROSHI_API_KEY" \
  -d '{
    "model": "gpt-4o",
    "prompt": "Once upon a time",
    "max_tokens": 100
  }'
```

Supports `echo` (include the prompt in the response) and `suffix` parameters.

### Models list

Plugin: **Cloud APIM - LLM OpenAI Compat. Models list**

```
cp:otoroshi_plugins.com.cloud.apim.otoroshi.extensions.aigateway.plugins.OpenAiCompatModels
```

Exposes the provider models list compatible with the [OpenAI models API](https://platform.openai.com/docs/api-reference/models).

#### Configuration

```js
{
  "refs": ["provider_entity_id"]
}
```

| Parameter | Type | Description |
|-----------|------|-------------|
| `refs` | array | List of LLM Provider entity IDs |

![](/img/openai-models.png)

#### Usage

```sh
curl https://my-own-llm-endpoint.on.otoroshi/v1/models \
  -H "Authorization: Bearer $OTOROSHI_API_KEY"
```

Response:

```js
{
  "object": "list",
  "data": [
    {
      "id": "o1-mini",
      "object": "model",
      "created": 1686935002,
      "owned_by": "openai"
    },
    {
      "id": "gpt-4",
      "object": "model",
      "created": 1686935002,
      "owned_by": "openai"
    }
  ]
}
```

When multiple providers are configured, model IDs are prefixed with the provider slug name (e.g., `my_openai/gpt-4o`). Use `?raw=true` to get the raw model identifiers without prefixes.

### Models list with provider info

Plugin: **Cloud APIM - LLM OpenAI Compat. Provider with Models list**

```
cp:otoroshi_plugins.com.cloud.apim.otoroshi.extensions.aigateway.plugins.OpenAiCompatProvidersWithModels
```

Similar to the models list, but enriches each model entry with provider metadata:

```js
{
  "object": "list",
  "data": [
    {
      "id": "my_openai/gpt-4o",
      "combined_id": "my_openai/gpt-4o",
      "simple_id": "gpt-4o",
      "provider_id": "my_openai",
      "owned_by": "OpenAI",
      "owned_by_with_model": "OpenAI / gpt-4o",
      "object": "model",
      "created": 1686935002
    }
  ]
}
```

### LLM response endpoint

Plugin: **Cloud APIM - LLM Response endpoint**

```
cp:otoroshi_plugins.com.cloud.apim.otoroshi.extensions.aigateway.plugins.LlmResponseEndpoint
```

Provides an endpoint that returns raw LLM responses from a predefined prompt. The prompt supports [Expression Language](https://maif.github.io/otoroshi/manual/topics/expression-language.html) for dynamic values based on the request context.

#### Configuration

```js
{
  "ref": "provider_entity_id",
  "prompt": "Summarize the following text: ${req.body.text}",
  "prompt_ref": null,
  "context_ref": null
}
```

| Parameter | Type | Description |
|-----------|------|-------------|
| `ref` | string | LLM Provider entity ID |
| `prompt` | string | Prompt text with optional Expression Language variables |
| `prompt_ref` | string | Optional reference to a stored prompt configuration |
| `context_ref` | string | Optional reference to a stored context configuration |

#### Response

```js
{
  "generations": [
    {
      "message": {
        "role": "assistant",
        "content": "..."
      }
    }
  ]
}
```

## Anthropic compatible plugin

We also provide an [Anthropic Messages API compatible proxy](/docs/llm-gateway/anthropic-proxy) that allows any Anthropic API client (including **Claude Code**) to use any LLM provider managed by Otoroshi. See the [dedicated documentation page](/docs/llm-gateway/anthropic-proxy) for details.

## Open Responses compatible plugin

We provide an [Open Responses API compatible proxy](/docs/llm-gateway/open-response-proxy) that allows any client speaking the [Open Responses](https://www.openresponses.org/) specification to use any LLM provider managed by Otoroshi. See the [dedicated documentation page](/docs/llm-gateway/open-response-proxy) for details.

## Route example

A complete route configuration exposing a chat completions endpoint with a models list:

```js
{
  "frontend": {
    "domains": ["llm.my-domain.com"]
  },
  "backend": {
    "targets": [
      {
        "hostname": "request.otoroshi.io",
        "port": 443,
        "tls": true
      }
    ]
  },
  "plugins": [
    {
      "enabled": true,
      "plugin": "cp:otoroshi.next.plugins.OverrideHost",
      "config": {}
    },
    {
      "enabled": true,
      "plugin": "cp:otoroshi_plugins.com.cloud.apim.otoroshi.extensions.aigateway.plugins.OpenAiCompatProxy",
      "includes": [
        "/v1/chat/completions"
      ],
      "config": {
        "refs": ["provider_openai_1"]
      }
    },
    {
      "enabled": true,
      "plugin": "cp:otoroshi_plugins.com.cloud.apim.otoroshi.extensions.aigateway.plugins.OpenAiCompatModels",
      "includes": [
        "/v1/models"
      ],
      "config": {
        "refs": ["provider_openai_1"]
      }
    }
  ]
}
```
