高性能AI网关

Installation

Since 0.8.0, default features select the reusable library with metrics and tracing. Gateway and database dependencies are opt-in. The previous 0.7.0 release used gateway defaults; gateway consumers must explicitly select a storage feature such as sqlite or postgres.

```toml

Build/test uses too much CPU or memory

• Use API-only defaults first: cargo test --lib --tests --no-default-features --features "lite"
• Limit local parallelism when needed: CARGO_BUILD_JOBS=4 cargo test --lib --tests --no-default-features --features "lite" -- --test-threads=4
• Avoid --all-features unless you are doing release/nightly validation

Quick Start: Self-Hosted Gateway

Run the primary supported product from source:

git clone https://github.com/majiayu000/litellm-rs.git
cd litellm-rs
cp config/gateway.dev.yaml.example config/gateway.yaml
cargo run --bin gateway --features sqlite

Or install the gateway binary:

cargo install litellm-rs --version 0.8.2 --locked --bin gateway --no-default-features \
  --features postgres,sqlite,redis,s3,metrics,tracing,websockets,providers-extra,providers-extended,mcp-validation,a2a
mkdir -p config
curl -L https://raw.githubusercontent.com/majiayu000/litellm-rs/v0.8.2/config/gateway.dev.yaml.example -o config/gateway.yaml
gateway

The development config starts without provider credentials or auth secrets and uses the local vllm catalog provider. Use config/gateway.yaml.example for production-style deployments with real provider keys and auth enabled. The explicit sqlite feature enables the storage needed by the gateway binary. The registry install selects the same feature profile as release archives and containers, including MCP, A2A and Realtime. Binary and example config versions must match. See the 0.8 artifact verification for source and published-artifact verification.

The gateway serves its stable inference contract at GET /openapi.json; the versioned source is docs/openapi/inference.json. The admin control-plane contract is served at GET /admin/openapi.json (admin-authenticated) from docs/openapi/admin.json.

API-key budgets currently use process-local definitions and counters. Persisting a key's budget_id does not restore its budget after restart or share its balance with another gateway; an unavailable binding fails closed. The key management API rejects max_budget. Redis-backed provider/model budgets are a separate scope; see the budget capability boundary.

For a first HTTP request or setup failure, use the gateway setup and compatibility questions.

Library Example

Add Tokio with macros and rt-multi-thread features. The application must install its configured RuntimeBinding with install_default_runtime before calling completion; the verified consumer example shows runtime setup and successful calls against a local mock.

use litellm_rs::{completion, user_message, system_message};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let response = completion(
        "gpt-4",
        vec![
            system_message("You are a helpful assistant."),
            user_message("Hello!"),
        ],
        None,
    ).await?;

    println!("{}", response.choices[0].message.content.as_ref().unwrap());
    Ok(())
}

Examples

Gateway Configuration

The gateway router config maps these fields into the runtime router:

• router.strategy selects the deployment routing strategy.
• router.circuit_breaker.failure_threshold controls consecutive failures before cooldown.
• router.circuit_breaker.recovery_timeout controls cooldown duration in seconds.
• router.circuit_breaker.min_requests sets the sample size required before cooldown.
• router.circuit_breaker.success_threshold sets the successes required to recover from cooldown.
• router.load_balancer.health_check_enabled enables pre-call deployment health checks.

Active provider probes are opt-in: a provider's health_check must differ from the defaults. Native probes are limited to provider types whose configured deployment is unambiguously a chat model (currently Anthropic and GitHub Copilot); they send a one-token request and may incur provider charges. OpenAI, Bedrock, OpenAI-compatible, Vertex AI, FalAI, and other multi-capability providers require a custom health_check.endpoint. Without an active probe policy, readiness remains fail-closed (Unknown).

router.load_balancer.sticky_sessions and router.load_balancer.session_timeout are reserved for future session affinity. Non-default values fail config validation until runtime affinity is implemented.

Gateway YAML can publish stable model names and deterministic primary/fallback tiers:

providers:
  - name: openai-primary
    provider_type: openai
    api_key: "${OPENAI_API_KEY}"
    models: [gpt-4o]
    priority: 0
  - name: openai-fallback
    provider_type: openai
    api_key: "${OPENAI_API_KEY}"
    models: [gpt-4o]
    priority: 10

model_aliases:
  production-chat: gpt-4o
  stable-chat: production-chat

router:
  strategy: priority_based

Alias chains are validated and flattened at startup; empty values, cycles, canonical-name collisions, and targets without an enabled deployment fail startup. Alias names appear in /v1/models alongside canonical models. Lower numeric priority wins under priority_based; omitted provider priorities default to 0. When rolling back to a binary that predates these fields, remove model_aliases and priority from YAML before rolling back the binary, because unknown fields are rejected.

Core Subsystem Runtime Status

Runtime wiring decisions are tracked in src/core/subsystem_registry.rs, and tests assert that every module exported from src/core/mod.rs is either referenced by the gateway runtime or explicitly classified. The current issue-838 subsystem decisions are:

Library without optional observability

[dependencies] litellm-rs = { version = "0.8", default-features = false }

Environment Variables

```bash

Optional

LITELLM_VERBOSE=true # Enable verbose logging ```

Provider API Keys

OPENAI_API_KEY=sk-... ANTHROPIC_API_KEY=sk-ant-... GOOGLE_API_KEY=... AZURE_OPENAI_API_KEY=... AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=... AWS_REGION=us-east-1 GROQ_API_KEY=... AI21_API_KEY=... HF_TOKEN=... BASETEN_API_KEY=... DEEPSEEK_API_KEY=... MOONSHOT_API_KEY=... ZHIPU_API_KEY=... MINIMAX_API_KEY=...

I only need provider API aggregation, not gateway

• Prefer default-features = false with features = ["lite"]
• Use gateway runtime commands only when you need HTTP server/auth/storage middleware

Troubleshooting