Skip to main content

# Observability

Every interaction with a Large Language Model (LLM) generates crucial data that can be monitored, analyzed, and optimized.

Our LLM Gateway provides real-time tracking, security, and performance insights, acting as a centralized observability layer to streamline LLM interactions.

By routing all requests through our LLM gateway, you gain:

- Full-stack observability by capturing every LLM request before it reaches the provider
- Granular control over who can access what, when, and how
- Cost efficiency through detailed usage tracking and budgeting
- Real-time insights into token consumption, latency, and error rates

## Key metrics tracked​

Every LLM request logs critical telemetry data, including:

- LLM Provider - Identify the AI service in use (e.g., OpenAI, Anthropic, Mistral)
- Model Version - Track which model is processing requests
- Prompt & Response Data - Log inputs and outputs for debugging & quality control
- Token Usage Metrics - Measure input/output/reasoning token consumption
- User Identity - Associate API usage with specific users for accountability
- API Key Tracking - Monitor & secure API access to prevent unauthorized use
- Cost Tracking - Compute costs per request, per model, per user
- Ecological Impact - Estimate energy consumption and CO2 emissions
- Budget Consumption - Track spending against defined budget limits

## Using data exporters​

You can use Otoroshi data exporters to extract LLM usage information and send it to any external system (Elasticsearch, Kafka, webhooks, etc.).

Make sure to filter events on `LLMUsageAudit`:

    {
      "include": [{
        "audit": "LLMUsageAudit"
      }],
      "exclude": []
    }

For detailed information about `LLMUsageAudit` events, event fields, operation types, and dashboard examples, see the Reporting documentation.

## Dashboarding​

You can use `LLMUsageAudit` events to build dashboards with tools like Kibana, Grafana, or any visualization tool.

See the Reporting section for Elasticsearch data exporter configuration examples and event structure details.