Skip to main content

# Expose your LLM Provider

Now that your provider is fully set up, you can expose it to your organization. The idea is to do it through an Otoroshi route with a plugin of type `backend` that will handle passing incoming requests to the actual LLM provider.

## OpenAI compatible plugins​

We provide a set of plugins capable of exposing any supported LLM provider through a compatible OpenAI API. This is the standard way of doing LLM exposition.

### Chat completions proxy​

Plugin: Cloud APIM - LLM OpenAI chat/completions Proxy

    cp:otoroshi_plugins.com.cloud.apim.otoroshi.extensions.aigateway.plugins.OpenAiCompatProxy

This plugin is compatible with the OpenAI chat completion API, including streaming.

#### Configuration​

    {
      "refs": ["provider_entity_id_1", "provider_entity_id_2"]
    }

Parameter | Type | Description
--- | --- | ---
`refs` | array | List of LLM Provider entity IDs

#### Usage​

    curl https://my-own-llm-endpoint.on.otoroshi/v1/chat/completions \
      -H "Content-Type: application/json" \
      -H "Authorization: Bearer $OTOROSHI_API_KEY" \
      -d '{
        "model": "gpt-4o",
        "messages": [
          {
            "role": "user",
            "content": "Hello how are you!"
          }
        ]
      }'

Response:

    {
      "id": "chatcmpl-B9MBs8CjcvOU2jLn4n570S5qMJKcT",
      "object": "chat.completion",
      "created": 1741569952,
      "model": "gpt-4o",
      "choices": [
        {
          "index": 0,
          "message": {
            "role": "assistant",
            "content": "Hello! How can I assist you today?"
          },
          "finish_reason": "stop"
        }
      ],
      "usage": {
        "prompt_tokens": 19,
        "completion_tokens": 10,
        "total_tokens": 29
      }
    }

#### Model routing​

When multiple providers are configured in `refs`, you can target a specific provider using the `model` field:

- Slash syntax: `providerName/modelName` (e.g., `my_openai_provider/gpt-4o`)
- Hash syntax: `providerId###modelName` (e.g., `provider_xxx###gpt-4o`)

If no provider prefix is specified, the first configured ref is used.

### Text completions proxy​

Plugin: Cloud APIM - LLM OpenAI completions Proxy

    cp:otoroshi_plugins.com.cloud.apim.otoroshi.extensions.aigateway.plugins.OpenAiCompletionProxy

This plugin is compatible with the OpenAI completions API, including streaming. It converts the legacy completions format (`prompt` field) to the internal chat format.

#### Configuration​

    {
      "refs": ["provider_entity_id"]
    }

Parameter | Type | Description
--- | --- | ---
`refs` | array | List of LLM Provider entity IDs

#### Usage​

    curl https://my-own-llm-endpoint.on.otoroshi/v1/completions \
      -H "Content-Type: application/json" \
      -H "Authorization: Bearer $OTOROSHI_API_KEY" \
      -d '{
        "model": "gpt-4o",
        "prompt": "Once upon a time",
        "max_tokens": 100
      }'

Supports `echo` (include the prompt in the response) and `suffix` parameters.

### Models list​

Plugin: Cloud APIM - LLM OpenAI Compat. Models list

    cp:otoroshi_plugins.com.cloud.apim.otoroshi.extensions.aigateway.plugins.OpenAiCompatModels

Exposes the provider models list compatible with the OpenAI models API.

#### Configuration​

    {
      "refs": ["provider_entity_id"]
    }

Parameter | Type | Description
--- | --- | ---
`refs` | array | List of LLM Provider entity IDs

#### Usage​

    curl https://my-own-llm-endpoint.on.otoroshi/v1/models \
      -H "Authorization: Bearer $OTOROSHI_API_KEY"

Response:

    {
      "object": "list",
      "data": [
        {
          "id": "o1-mini",
          "object": "model",
          "created": 1686935002,
          "owned_by": "openai"
        },
        {
          "id": "gpt-4",
          "object": "model",
          "created": 1686935002,
          "owned_by": "openai"
        }
      ]
    }

When multiple providers are configured, model IDs are prefixed with the provider slug name (e.g., `my_openai/gpt-4o`). Use `?raw=true` to get the raw model identifiers without prefixes.

### Models list with provider info​

Plugin: Cloud APIM - LLM OpenAI Compat. Provider with Models list

    cp:otoroshi_plugins.com.cloud.apim.otoroshi.extensions.aigateway.plugins.OpenAiCompatProvidersWithModels

Similar to the models list, but enriches each model entry with provider metadata:

    {
      "object": "list",
      "data": [
        {
          "id": "my_openai/gpt-4o",
          "combined_id": "my_openai/gpt-4o",
          "simple_id": "gpt-4o",
          "provider_id": "my_openai",
          "owned_by": "OpenAI",
          "owned_by_with_model": "OpenAI / gpt-4o",
          "object": "model",
          "created": 1686935002
        }
      ]
    }

### LLM response endpoint​

Plugin: Cloud APIM - LLM Response endpoint

    cp:otoroshi_plugins.com.cloud.apim.otoroshi.extensions.aigateway.plugins.LlmResponseEndpoint

Provides an endpoint that returns raw LLM responses from a predefined prompt. The prompt supports Expression Language for dynamic values based on the request context.

#### Configuration​

    {
      "ref": "provider_entity_id",
      "prompt": "Summarize the following text: ${req.body.text}",
      "prompt_ref": null,
      "context_ref": null
    }

Parameter | Type | Description
--- | --- | ---
`ref` | string | LLM Provider entity ID
`prompt` | string | Prompt text with optional Expression Language variables
`prompt_ref` | string | Optional reference to a stored prompt configuration
`context_ref` | string | Optional reference to a stored context configuration

#### Response​

    {
      "generations": [
        {
          "message": {
            "role": "assistant",
            "content": "..."
          }
        }
      ]
    }

## Anthropic compatible plugin​

We also provide an Anthropic Messages API compatible proxy that allows any Anthropic API client (including Claude Code) to use any LLM provider managed by Otoroshi. See the dedicated documentation page for details.

## Open Responses compatible plugin​

We provide an Open Responses API compatible proxy that allows any client speaking the Open Responses specification to use any LLM provider managed by Otoroshi. See the dedicated documentation page for details.

## Route example​

A complete route configuration exposing a chat completions endpoint with a models list:

    {
      "frontend": {
        "domains": ["llm.my-domain.com"]
      },
      "backend": {
        "targets": [
          {
            "hostname": "request.otoroshi.io",
            "port": 443,
            "tls": true
          }
        ]
      },
      "plugins": [
        {
          "enabled": true,
          "plugin": "cp:otoroshi.next.plugins.OverrideHost",
          "config": {}
        },
        {
          "enabled": true,
          "plugin": "cp:otoroshi_plugins.com.cloud.apim.otoroshi.extensions.aigateway.plugins.OpenAiCompatProxy",
          "includes": [
            "/v1/chat/completions"
          ],
          "config": {
            "refs": ["provider_openai_1"]
          }
        },
        {
          "enabled": true,
          "plugin": "cp:otoroshi_plugins.com.cloud.apim.otoroshi.extensions.aigateway.plugins.OpenAiCompatModels",
          "includes": [
            "/v1/models"
          ],
          "config": {
            "refs": ["provider_openai_1"]
          }
        }
      ]
    }