高性能AI网关

Installation

```toml

Build/test uses too much CPU or memory

• Use API-only defaults first: cargo test --lib --tests --no-default-features --features "lite"
• Limit local parallelism when needed: CARGO_BUILD_JOBS=4 cargo test --lib --tests --no-default-features --features "lite" -- --test-threads=4
• Avoid --all-features unless you are doing release/nightly validation

Quick Start: Self-Hosted Gateway

Run the primary supported product from source:

git clone https://github.com/majiayu000/litellm-rs.git
cd litellm-rs
cp config/gateway.dev.yaml.example config/gateway.yaml
cargo run --bin gateway

Or install the gateway binary:

cargo install litellm-rs --bin gateway
mkdir -p config
curl -L https://raw.githubusercontent.com/majiayu000/litellm-rs/main/config/gateway.dev.yaml.example -o config/gateway.yaml
gateway

The development config starts without provider credentials or auth secrets and uses the local vllm catalog provider. Use config/gateway.yaml.example for production-style deployments with real provider keys and auth enabled. Default features include SQLite storage, which satisfies the gateway binary's storage requirement.

The gateway serves its stable inference contract at GET /openapi.json; the versioned source is docs/openapi/inference.json. The admin control-plane contract is served at GET /admin/openapi.json (admin-authenticated) from docs/openapi/admin.json.

Library Example

use litellm_rs::{completion, user_message, system_message};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let response = completion(
        "gpt-4",
        vec![
            system_message("You are a helpful assistant."),
            user_message("Hello!"),
        ],
        None,
    ).await?;

    println!("{}", response.choices[0].message.content.as_ref().unwrap());
    Ok(())
}

Examples

Gateway Configuration

The gateway router config maps these fields into the runtime router:

• router.strategy selects the deployment routing strategy.
• router.circuit_breaker.failure_threshold controls consecutive failures before cooldown.
• router.circuit_breaker.recovery_timeout controls cooldown duration in seconds.
• router.circuit_breaker.min_requests sets the sample size required before cooldown.
• router.circuit_breaker.success_threshold sets the successes required to recover from cooldown.
• router.load_balancer.health_check_enabled enables pre-call deployment health checks.

Active provider probes are opt-in: a provider's health_check must differ from the defaults. Native probes are limited to provider types whose configured deployment is unambiguously a chat model (currently Anthropic and GitHub Copilot); they send a one-token request and may incur provider charges. OpenAI, Bedrock, OpenAI-compatible, Vertex AI, FalAI, and other multi-capability providers require a custom health_check.endpoint. Without an active probe policy, readiness remains fail-closed (Unknown).

router.load_balancer.sticky_sessions and router.load_balancer.session_timeout are reserved for future session affinity. Non-default values fail config validation until runtime affinity is implemented.

Gateway YAML can publish stable model names and deterministic primary/fallback tiers:

providers:
  - name: openai-primary
    provider_type: openai
    api_key: "${OPENAI_API_KEY}"
    models: [gpt-4o]
    priority: 0
  - name: openai-fallback
    provider_type: openai
    api_key: "${OPENAI_API_KEY}"
    models: [gpt-4o]
    priority: 10

model_aliases:
  production-chat: gpt-4o
  stable-chat: production-chat

router:
  strategy: priority_based

Alias chains are validated and flattened at startup; empty values, cycles, canonical-name collisions, and targets without an enabled deployment fail startup. Alias names appear in /v1/models alongside canonical models. Lower numeric priority wins under priority_based; omitted provider priorities default to 0. When rolling back to a binary that predates these fields, remove model_aliases and priority from YAML before rolling back the binary, because unknown fields are rejected.

Core Subsystem Runtime Status

Runtime wiring decisions are tracked in src/core/subsystem_registry.rs, and tests assert that every module exported from src/core/mod.rs is either referenced by the gateway runtime or explicitly classified. The current issue-838 subsystem decisions are:

Environment Variables

```bash

Optional

LITELLM_VERBOSE=true # Enable verbose logging ```

API-only - lightweight, no actix-web/argon2/aes-gcm/clap

[dependencies] litellm-rs = { version = "0.6", default-features = false }

API-only with metrics

[dependencies] litellm-rs = { version = "0.6", default-features = false, features = ["lite"] }

Provider API Keys

OPENAI_API_KEY=sk-... ANTHROPIC_API_KEY=sk-ant-... GOOGLE_API_KEY=... AZURE_OPENAI_API_KEY=... AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=... AWS_REGION=us-east-1 GROQ_API_KEY=... AI21_API_KEY=... HF_TOKEN=... BASETEN_API_KEY=... DEEPSEEK_API_KEY=... MOONSHOT_API_KEY=... ZHIPU_API_KEY=... MINIMAX_API_KEY=...

I only need provider API aggregation, not gateway

• Prefer default-features = false with features = ["lite"]
• Use gateway runtime commands only when you need HTTP server/auth/storage middleware

Gateway modules in library context (not standalone gateway binary runtime)

[dependencies] litellm-rs = { version = "0.6", default-features = false, features = ["gateway"] } ```

Experimental / module-only

The following modules exist under src/core/providers/ (gated on providers-extra or providers-extended) but are not wired into the unified Provider enum or the factory today. They compile but cannot be selected through create_provider/from_config_async. Treat them as experimental scaffolding subject to change:

custom_api

For self-hosted or unlisted OpenAI-compatible endpoints, prefer the generic openai_compatible provider type instead.

Troubleshooting