Skip to main content

# Overview

The Otoroshi LLM extension provides a unified, OpenAI-compatible API for computing text embeddings across multiple providers. Embeddings are vector representations of text that capture semantic meaning, enabling similarity search, clustering, and RAG (Retrieval-Augmented Generation) pipelines.

## Features​

- 16+ embedding providers including cloud APIs and a local ONNX model
- OpenAI-compatible API — standard `/v1/embeddings` endpoint
- Batch embedding — embed multiple texts in a single request
- Encoding formats — `float` (JSON array) or `base64` (compact binary)
- Model routing — route to different providers using `provider/model` syntax
- Model constraints — restrict which models consumers can use via include/exclude regex patterns
- Budget enforcement — embedding costs are tracked and budgets are enforced
- Cost tracking — per-request cost tracking integrated with cost tracking
- Embedding stores — local in-memory vector stores for similarity search
- Token round-robin — distribute load across multiple API tokens

## API endpoint​

Endpoint | Method | Description
--- | --- | ---
`/v1/embeddings` | POST | Compute embeddings for one or more text inputs

## Request​

    curl --request POST \
      --url http://myroute.oto.tools:8080/v1/embeddings \
      --header 'content-type: application/json' \
      --data '{
      "input": ["Hello world", "How are you?"],
      "model": "text-embedding-3-small",
      "encoding_format": "float"
    }'

### Request parameters​

Parameter | Type | Description
--- | --- | ---
`input` | string or array | The text(s) to embed. Can be a single string or an array of strings for batch embedding
`model` | string | Model name. Can include a provider prefix for model routing
`dimensions` | integer | Requested embedding dimensions (supported by some models like `text-embedding-3-small`)
`encoding_format` | string | Output format: `"float"` (default, JSON array of numbers) or `"base64"` (compact binary encoding)
`user` | string | End-user identifier for tracking

## Response​

    {
      "object": "list",
      "data": [
        {
          "object": "embedding",
          "index": 0,
          "embedding": [0.0023064255, -0.009327292, ...]
        },
        {
          "object": "embedding",
          "index": 1,
          "embedding": [-0.0015486241, 0.0073928963, ...]
        }
      ],
      "model": "text-embedding-3-small",
      "usage": {
        "prompt_tokens": 8,
        "total_tokens": 8
      }
    }

With `"encoding_format": "base64"`, each embedding is returned as a base64-encoded string of little-endian float bytes instead of a JSON array.

## Embedding stores​

The extension provides embedding stores for storing and searching embedding vectors. Multiple backends are supported, from a local in-memory store to production-grade vector databases.

Embedding stores are used internally by the semantic cache and can be used in workflows via dedicated functions:

- `vector_store_add` — add a document with its embedding to a store
- `vector_store_remove` — remove a document by ID
- `vector_store_search` — search by embedding vector similarity

### Supported providers​

Provider | Value | Description
--- | --- | ---
Local (in-memory) | `local` | LangChain4j `InMemoryEmbeddingStore`, no external dependency
ChromaDB | `chromadb` | Open-source vector database via HTTP API
Elasticsearch | `elasticsearch` | kNN vector search on dense_vector fields
OpenSearch | `opensearch` | kNN vector search with HNSW
Qdrant | `qdrant` | Purpose-built vector database via REST API
Weaviate | `weaviate` | Vector database with GraphQL search
Pinecone | `pinecone` | Cloud-native vector database
Redis Stack | `redis` | RediSearch module with vector similarity (requires Redis Stack)
PostgreSQL | `postgresql` | pgvector extension for cosine similarity search

### Local store configuration​

    {
      "provider": "local",
      "config": {
        "connection": {
          "name": "my-store",
          "session_id": "optional-session-id",
          "init_content": "https://example.com/initial-data.json"
        },
        "options": {
          "max_results": 3,
          "min_score": 0.7
        }
      }
    }

Parameter | Type | Default | Description
--- | --- | --- | ---
`connection.name` | string | — | Store name
`connection.session_id` | string | — | Optional session ID for per-session isolation
`connection.init_content` | string | — | URL to initial content (HTTP/HTTPS, `file://`, or `s3://`)
`options.max_results` | integer | `3` | Maximum number of results returned by search
`options.min_score` | number | `0.7` | Minimum cosine similarity score for search matches

### ChromaDB configuration​

    {
      "provider": "chromadb",
      "config": {
        "connection": {
          "url": "http://localhost:8000",
          "collection": "my-collection",
          "api_key": "optional-api-key",
          "tenant": "default_tenant",
          "database": "default_database"
        }
      }
    }

### Elasticsearch / OpenSearch configuration​

    {
      "provider": "elasticsearch",
      "config": {
        "connection": {
          "url": "http://localhost:9200",
          "index": "embeddings",
          "username": "elastic",
          "password": "changeme",
          "dims": 384,
          "similarity": "cosine"
        }
      }
    }

Authentication supports `username`/`password` (Basic), `api_key` (ApiKey header), or no auth. The index and mapping are created automatically on first use.

For OpenSearch, use `"provider": "opensearch"` with additional options `engine` (default `lucene`) and `space_type` (default `cosinesimil`).

### Qdrant configuration​

    {
      "provider": "qdrant",
      "config": {
        "connection": {
          "url": "http://localhost:6333",
          "collection": "my-collection",
          "api_key": "optional-api-key",
          "dims": 384,
          "distance": "Cosine"
        }
      }
    }

### Weaviate configuration​

    {
      "provider": "weaviate",
      "config": {
        "connection": {
          "url": "http://localhost:8080",
          "class_name": "Embedding",
          "api_key": "optional-api-key"
        }
      }
    }

### Pinecone configuration​

    {
      "provider": "pinecone",
      "config": {
        "connection": {
          "url": "https://index-xxxxx.svc.environment.pinecone.io",
          "api_key": "your-api-key",
          "namespace": ""
        }
      }
    }

### Redis Stack configuration​

Requires Redis with the Search module (redis-stack). Embeddings are stored as binary vectors in HASH keys and indexed with RediSearch for KNN similarity search.

    {
      "provider": "redis",
      "config": {
        "connection": {
          "url": "redis://localhost:6379",
          "prefix": "otoroshi:ai:emb",
          "dims": 384,
          "distance_metric": "COSINE"
        }
      }
    }

Parameter | Type | Default | Description
--- | --- | --- | ---
`url` | string | `redis://localhost:6379` | Redis connection URI
`prefix` | string | `otoroshi:ai:emb` | Key prefix for hash keys and index
`dims` | integer | `384` | Vector dimensions
`distance_metric` | string | `COSINE` | Distance metric (`COSINE`, `L2`, `IP`)

### PostgreSQL (pgvector) configuration​

Requires PostgreSQL with the pgvector extension. The extension and table are created automatically on first use.

    {
      "provider": "postgresql",
      "config": {
        "connection": {
          "uri": "postgresql://user:password@localhost:5432/mydb",
          "table": "otoroshi_ai_embeddings",
          "dims": 384
        }
      }
    }

Parameter | Type | Default | Description
--- | --- | --- | ---
`uri` | string | `postgresql://otoroshi:otoroshi@localhost:5432/otoroshi` | PostgreSQL connection URI
`table` | string | `otoroshi_ai_embeddings` | Table name
`dims` | integer | `384` | Vector dimensions