---
sidebar_position: 1
---

import Terminal from '@site/src/components/Terminal';

# Overview

The Otoroshi LLM extension provides a unified, OpenAI-compatible API for computing text embeddings across multiple providers. Embeddings are vector representations of text that capture semantic meaning, enabling similarity search, clustering, and RAG (Retrieval-Augmented Generation) pipelines.

## Features

- **16+ embedding providers** including cloud APIs and a local ONNX model
- **OpenAI-compatible API** — standard `/v1/embeddings` endpoint
- **Batch embedding** — embed multiple texts in a single request
- **Encoding formats** — `float` (JSON array) or `base64` (compact binary)
- **Model routing** — route to different providers using `provider/model` syntax
- **Model constraints** — restrict which models consumers can use via include/exclude regex patterns
- **Budget enforcement** — embedding costs are tracked and budgets are enforced
- **Cost tracking** — per-request cost tracking integrated with [cost tracking](/docs/cost-optimizations/cost-tracking)
- **Embedding stores** — local in-memory vector stores for similarity search
- **Token round-robin** — distribute load across multiple API tokens

## API endpoint

| Endpoint | Method | Description |
|----------|--------|-------------|
| `/v1/embeddings` | POST | Compute embeddings for one or more text inputs |

## Request

```sh
curl --request POST \
  --url http://myroute.oto.tools:8080/v1/embeddings \
  --header 'content-type: application/json' \
  --data '{
  "input": ["Hello world", "How are you?"],
  "model": "text-embedding-3-small",
  "encoding_format": "float"
}'
```

### Request parameters

| Parameter | Type | Description |
|-----------|------|-------------|
| `input` | string or array | The text(s) to embed. Can be a single string or an array of strings for batch embedding |
| `model` | string | Model name. Can include a provider prefix for [model routing](/docs/embeddings/plugins#model-routing) |
| `dimensions` | integer | Requested embedding dimensions (supported by some models like `text-embedding-3-small`) |
| `encoding_format` | string | Output format: `"float"` (default, JSON array of numbers) or `"base64"` (compact binary encoding) |
| `user` | string | End-user identifier for tracking |

## Response

```js
{
  "object": "list",
  "data": [
    {
      "object": "embedding",
      "index": 0,
      "embedding": [0.0023064255, -0.009327292, ...]
    },
    {
      "object": "embedding",
      "index": 1,
      "embedding": [-0.0015486241, 0.0073928963, ...]
    }
  ],
  "model": "text-embedding-3-small",
  "usage": {
    "prompt_tokens": 8,
    "total_tokens": 8
  }
}
```

With `"encoding_format": "base64"`, each embedding is returned as a base64-encoded string of little-endian float bytes instead of a JSON array.

## Embedding stores

The extension provides **embedding stores** for storing and searching embedding vectors. Multiple backends are supported, from a local in-memory store to production-grade vector databases.

Embedding stores are used internally by the [semantic cache](/docs/cost-optimizations/semantic-cache) and can be used in [workflows](/docs/workflows/overview) via dedicated functions:
- `vector_store_add` — add a document with its embedding to a store
- `vector_store_remove` — remove a document by ID
- `vector_store_search` — search by embedding vector similarity

### Supported providers

| Provider | Value | Description |
|----------|-------|-------------|
| **Local (in-memory)** | `local` | LangChain4j `InMemoryEmbeddingStore`, no external dependency |
| **ChromaDB** | `chromadb` | Open-source vector database via HTTP API |
| **Elasticsearch** | `elasticsearch` | kNN vector search on dense_vector fields |
| **OpenSearch** | `opensearch` | kNN vector search with HNSW |
| **Qdrant** | `qdrant` | Purpose-built vector database via REST API |
| **Weaviate** | `weaviate` | Vector database with GraphQL search |
| **Pinecone** | `pinecone` | Cloud-native vector database |
| **Redis Stack** | `redis` | RediSearch module with vector similarity (requires Redis Stack) |
| **PostgreSQL** | `postgresql` | pgvector extension for cosine similarity search |

### Local store configuration

```js
{
  "provider": "local",
  "config": {
    "connection": {
      "name": "my-store",
      "session_id": "optional-session-id",
      "init_content": "https://example.com/initial-data.json"
    },
    "options": {
      "max_results": 3,
      "min_score": 0.7
    }
  }
}
```

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `connection.name` | string | — | Store name |
| `connection.session_id` | string | — | Optional session ID for per-session isolation |
| `connection.init_content` | string | — | URL to initial content (HTTP/HTTPS, `file://`, or `s3://`) |
| `options.max_results` | integer | `3` | Maximum number of results returned by search |
| `options.min_score` | number | `0.7` | Minimum cosine similarity score for search matches |

### ChromaDB configuration

```js
{
  "provider": "chromadb",
  "config": {
    "connection": {
      "url": "http://localhost:8000",
      "collection": "my-collection",
      "api_key": "optional-api-key",
      "tenant": "default_tenant",
      "database": "default_database"
    }
  }
}
```

### Elasticsearch / OpenSearch configuration

```js
{
  "provider": "elasticsearch",
  "config": {
    "connection": {
      "url": "http://localhost:9200",
      "index": "embeddings",
      "username": "elastic",
      "password": "changeme",
      "dims": 384,
      "similarity": "cosine"
    }
  }
}
```

Authentication supports `username`/`password` (Basic), `api_key` (ApiKey header), or no auth. The index and mapping are created automatically on first use.

For OpenSearch, use `"provider": "opensearch"` with additional options `engine` (default `lucene`) and `space_type` (default `cosinesimil`).

### Qdrant configuration

```js
{
  "provider": "qdrant",
  "config": {
    "connection": {
      "url": "http://localhost:6333",
      "collection": "my-collection",
      "api_key": "optional-api-key",
      "dims": 384,
      "distance": "Cosine"
    }
  }
}
```

### Weaviate configuration

```js
{
  "provider": "weaviate",
  "config": {
    "connection": {
      "url": "http://localhost:8080",
      "class_name": "Embedding",
      "api_key": "optional-api-key"
    }
  }
}
```

### Pinecone configuration

```js
{
  "provider": "pinecone",
  "config": {
    "connection": {
      "url": "https://index-xxxxx.svc.environment.pinecone.io",
      "api_key": "your-api-key",
      "namespace": ""
    }
  }
}
```

### Redis Stack configuration

Requires Redis with the **Search** module (redis-stack). Embeddings are stored as binary vectors in HASH keys and indexed with RediSearch for KNN similarity search.

```js
{
  "provider": "redis",
  "config": {
    "connection": {
      "url": "redis://localhost:6379",
      "prefix": "otoroshi:ai:emb",
      "dims": 384,
      "distance_metric": "COSINE"
    }
  }
}
```

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `url` | string | `redis://localhost:6379` | Redis connection URI |
| `prefix` | string | `otoroshi:ai:emb` | Key prefix for hash keys and index |
| `dims` | integer | `384` | Vector dimensions |
| `distance_metric` | string | `COSINE` | Distance metric (`COSINE`, `L2`, `IP`) |

### PostgreSQL (pgvector) configuration

Requires PostgreSQL with the **pgvector** extension. The extension and table are created automatically on first use.

```js
{
  "provider": "postgresql",
  "config": {
    "connection": {
      "uri": "postgresql://user:password@localhost:5432/mydb",
      "table": "otoroshi_ai_embeddings",
      "dims": 384
    }
  }
}
```

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `uri` | string | `postgresql://otoroshi:otoroshi@localhost:5432/otoroshi` | PostgreSQL connection URI |
| `table` | string | `otoroshi_ai_embeddings` | Table name |
| `dims` | integer | `384` | Vector dimensions |
