Skip to main content

# LLM Gateway

All documentation to use our LLM Providers entities

## 📄️ Philosophy

Today, generative AI is simply unavoidable. It’s transforming industries, unlocking new possibilities, and accelerating innovation

## 📄️ Supported LLM Providers

Here is a list of all LLM chat providers available in the Otoroshi LLM Extension :

## 📄️ Setup a new LLM Provider

Quick start

## 📄️ Expose your LLM Provider

Now that your provider is fully set up, you can expose it to your organization. The idea is to do it through an Otoroshi route with a plugin of type backend that will handle passing incoming requests to the actual LLM provider.

## 📄️ 🔐 Secure your LLM provider endpoint

Video Turorial

## 📄️ Anthropic Messages API Proxy

The LLM Anthropic Messages Proxy plugin exposes any LLM provider managed by Otoroshi through an Anthropic Messages API-compatible endpoint. This means any client that speaks the Anthropic API format (including Claude Code) can seamlessly use any LLM provider proxied by Otoroshi, whether it's OpenAI, Mistral, Ollama, Azure OpenAI, Cohere, or any other supported provider.

## 📄️ Open Responses API Proxy

The LLM OpenResponse Proxy plugin exposes any LLM provider managed by Otoroshi through an Open Responses-compatible API endpoint. Open Responses is an open-source specification for multi-provider, interoperable LLM interfaces, backed by NVIDIA, Vercel, OpenRouter, Hugging Face, Databricks, Red Hat, Ollama, OpenAI, and others.

## 📄️ OpenAI Compatible API (unified)

The LLM OpenAI Compatible API plugin is a unified backend plugin that exposes a single Otoroshi route as a full-featured, multi-endpoint AI API. Instead of configuring separate plugins for chat completions, audio, images, embeddings, moderation, and responses, this single plugin handles all of them with a consistent configuration.

## 📄️ OpenAI Responses API Proxy

The LLM OpenAI Responses Proxy plugin exposes any LLM provider managed by Otoroshi through an OpenAI Responses API-compatible endpoint. This is a lightweight proxy that converts the Responses API format to the standard chat completions format internally, making it compatible with any provider supported by Otoroshi.

## 📄️ Managing secrets

You can use a secret vault to save your LLM provider tokens. The main advantage here is to avoid spreading those token and giving otoroshi apikeys to your own people instead.

## 📄️ Load Balancing

Load balancing distributes LLM requests across multiple providers, optimizing performance, ensuring availability, and preventing any single provider from being overloaded.

## 📄️ Resilience

Resilience ensures that LLM interactions remain highly available and fault-tolerant, even when providers experience outages or failures.

## 📄️ Fallback

The fallback mechanism ensures continuity of service by automatically switching to an alternative LLM provider when the primary provider fails.

## 📄️ Observability

Every interaction with a Large Language Model (LLM) generates crucial data that can be monitored, analyzed, and optimized.