本地优先混合LLM路由器

Prerequisites

• Python 3.11+
• conda recommended for the smoothest setup on macOS

Install

Recommended user install

pip install routelabs-router
router start

Contributor install

Clone the repo and install from source:

git clone https://github.com/routelabsai/router.git
cd router
conda create -n routelabs-router python=3.11 -y
conda activate routelabs-router
python -m pip install --upgrade pip setuptools wheel
pip install -e '.[dev]'
router start --reload

Setup And Usage

Install from PyPI

conda create -n routelabs-router python=3.11 -y
conda activate routelabs-router
python -m pip install --upgrade pip
pip install routelabs-router

Install from source

Use this path if you want to contribute or modify the router itself.

git clone https://github.com/routelabsai/router.git
cd router
conda create -n routelabs-router python=3.11 -y
conda activate routelabs-router
python -m pip install --upgrade pip setuptools wheel
pip install -e '.[dev]'

Show the fastest next setup path

router quickstart

This shows:

• virtual models like route-auto
• configured local and cloud models
• installed Ollama models discovered live
• whether each model is installed, configured, or not_configured

60-Second Quickstart

Install from PyPI, start the runtime, and send one request. Cloud keys are optional.

pip install routelabs-router
export OPENAI_API_KEY=your_api_key_here  # optional, enables cloud execution
export ANTHROPIC_API_KEY=your_api_key_here  # optional, enables Anthropic cloud execution
router start --reload

Then in another terminal:

curl -X POST http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "messages":[{"role":"user","content":"Summarize this in one sentence: RouteLabs Router chooses between local and cloud models based on privacy, cost, latency, and task complexity."}],
    "private":false
  }'

If you prefer Anthropic-style clients:

curl -X POST http://127.0.0.1:8000/v1/messages \
  -H "Content-Type: application/json" \
  -d '{
    "model":"claude-sonnet-4-20250514",
    "max_tokens":256,
    "messages":[{"role":"user","content":"Summarize RouteLabs Router in one sentence."}],
    "private":false
  }'

Use Cases

• Local-first copilots that should only escalate when a task gets difficult
• Privacy-sensitive workflows where private data should never leave the device
• Browser or desktop assistants that need one middleware layer above multiple runtimes
• Agent systems that want future step-level routing instead of a single fixed model

OpenAI-compatible drop-in example

If you already use the OpenAI Python SDK, you can point it at RouteLabs:

from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:8000/v1",
    api_key="not-needed-for-local-dev",
)

response = client.chat.completions.create(
    model="route-auto",
    messages=[
        {
            "role": "user",
            "content": "Summarize this in one sentence: RouteLabs Router chooses between local and cloud models based on privacy, cost, latency, and task complexity.",
        }
    ],
)

See examples/openai-compatible-client.py. You may need to install the OpenAI SDK separately:

pip install openai

Anthropic SDK drop-in example

If you already use the Anthropic Python SDK, you can point it at RouteLabs:

from anthropic import Anthropic

client = Anthropic(
    base_url="http://127.0.0.1:8000",
    api_key="not-needed-for-local-dev",
)

response = client.messages.create(
    model="claude-sonnet-4-20250514",
    max_tokens=256,
    messages=[
        {
            "role": "user",
            "content": "Summarize RouteLabs Router in one sentence.",
        }
    ],
)

See examples/anthropic-sdk-client.py. You may need to install the Anthropic SDK separately:

pip install anthropic

LangChain drop-in example

If you already use LangChain, you can point ChatOpenAI at RouteLabs:

from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    model="route-auto",
    base_url="http://127.0.0.1:8000/v1",
    api_key="not-needed-for-local-dev",
)

response = llm.invoke("Summarize RouteLabs Router in one sentence.")
print(response.content)

See examples/langchain-openai-compatible.py. You may need to install LangChain packages separately:

pip install langchain-openai

For a multi-step tool-calling example, see:

• examples/agent-loop.py
• examples/agent-loop.md

The stats response includes simple estimated fields such as:

• estimated_total_cost_usd
• estimated_baseline_cloud_cost_usd
• estimated_cost_saved_usd
• estimated_cloud_requests_avoided

OpenAI-style chat completion:

curl -X POST http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "messages":[{"role":"user","content":"Summarize this in one sentence: RouteLabs Router chooses between local and cloud models based on privacy, cost, latency, and task complexity."}],
    "private":false
  }'

If Ollama is running locally, the chat endpoint will execute against your configured local model. If OPENAI_API_KEY is set, high-complexity tasks can execute through the configured OpenAI-compatible cloud provider. If it is not set, cloud-routed chat requests return a clear configuration error. If the local provider is unavailable and the request is not forced to stay private, RouteLabs can now fall back to the cloud automatically and record that decision in the trace. The stats endpoint gives a simple first pass at the eventual cost/latency visibility story by showing how many requests stayed local, how many escalated, and how often verification failed. It also includes a lightweight savings estimate based on configurable per-request local and cloud cost assumptions. The logs endpoint exposes recent request-level decisions so users can inspect privacy detection, verification, escalation, final route choice, and estimated per-request cost directly.

Hybrid mode example

With both Ollama and OPENAI_API_KEY configured:

• simple tasks usually run locally
• private tasks prefer local execution
• high-complexity tasks can route to the cloud

Example cloud-leaning route check:

curl -X POST http://127.0.0.1:8000/v1/route \
  -H "Content-Type: application/json" \
  -d '{"task":"design architecture for a multi-step agent","private":false}'

More examples

• curl walkthrough: examples/curl-quickstart.md
• product framing and common scenarios: examples/use-cases.md
• agent loop walkthrough: examples/agent-loop.md

Example Routing Philosophy

• send simple, low-risk tasks to local models first
• prefer local execution when privacy rules require it
• escalate to stronger models when verification or confidence checks fail
• keep the decision trace visible so routing can be audited and improved

Quick Demo

Once the server is running, you can inspect decisions directly:

curl -X POST http://127.0.0.1:8000/v1/route \
  -H "Content-Type: application/json" \
  -d '{"task":"summarize a short product description","private":false}'

Expected shape:

{
  "target": "local",
  "provider": "ollama",
  "model": "qwen3:4b",
  "reason": "task is suitable for local-first execution",
  "complexity": "medium",
  "verify": true,
  "provider_available": true,
  "provider_status": "ready",
  "fallback_available": false,
  "fallback_status": "not_configured"
}

What this tells you:

• the router chose local
• it selected ollama
• it picked a model
• it marked the request as worth verification
• it reports whether the planned provider is actually reachable right now
• it reports whether cloud fallback is available if the local route fails

/v1/route is a planning endpoint, not an execution endpoint. It tells you what RouteLabs would try first and whether that path currently looks available.

And you can send an OpenAI-style chat request:

curl -X POST http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "messages":[{"role":"user","content":"Summarize this in one sentence: RouteLabs Router chooses between local and cloud models based on privacy, cost, latency, and task complexity."}],
    "private":false
  }'

If Ollama is running locally, that request executes against your configured local model. If OPENAI_API_KEY is set, high-complexity requests can route through the configured OpenAI-compatible cloud provider. The response includes a trace showing the initial route, verification result, and any escalation.

Newer OpenAI-style agent clients can also use /v1/responses:

curl -X POST http://127.0.0.1:8000/v1/responses \
  -H "Content-Type: application/json" \
  -d '{
    "model":"route-auto",
    "input":"Summarize RouteLabs Router in one sentence.",
    "private":false
  }'

Today, RouteLabs accepts these practical /v1/responses input shapes:

• a plain string in input
• a list of message-like items with role and content
• a list of type: "message" items with nested input_text content
• a top-level list of type: "input_text" items for simple text prompts

When stream=true, RouteLabs emits semantic Responses-style SSE events including:

• response.created
• response.output_text.delta
• response.output_text.done
• response.function_call_arguments.done when relevant
• response.completed

Anthropic-style clients can use /v1/messages:

curl -X POST http://127.0.0.1:8000/v1/messages \
  -H "Content-Type: application/json" \
  -d '{
    "model":"claude-sonnet-4-20250514",
    "max_tokens":256,
    "messages":[{"role":"user","content":"Summarize RouteLabs Router in one sentence."}],
    "private":false
  }'

When stream=true, RouteLabs emits Anthropic-style message events including:

• message_start
• content_block_start
• content_block_delta
• content_block_stop
• message_delta
• message_stop

Configure cloud execution

If you want cloud-routed requests to execute instead of returning a configuration error, set:

export OPENAI_API_KEY=your_api_key_here

The default cloud adapter uses the OpenAI-compatible endpoint configured in config/router.yaml.

Optional profile configs

The repo includes starter profiles in config/profiles/:

• balanced.yaml
• local-first.yaml
• openclaw.yaml
• privacy-first.yaml
• unsloth-local.yaml

The fastest way to scaffold a config now is:

router init --profile balanced

Or generate a config that defaults to Anthropic cloud fallback:

router init --profile balanced --cloud anthropic

By default this writes to config/router.yaml. Use --output to write somewhere else or --force to overwrite an existing file.

Run environment checks

router doctor

This shows:

• local and cloud provider readiness
• configured chat and embedding models
• installed Ollama models when RouteLabs can detect them
• missing configured local models
• the next setup action if something is unavailable

3. As an OpenAI-compatible or Anthropic-compatible endpoint

If you already have code using an OpenAI-style client, point it at RouteLabs via base_url. Use model="route-auto" when you want RouteLabs to choose the concrete backend model for each request.

That is one of the easiest ways to adopt it without rewriting your app.

If you already have Anthropic Messages API clients, point them at RouteLabs via base_url and call /v1/messages.

Test the API

Health check:

curl http://127.0.0.1:8000/healthz

Expected shape:

{
  "status": "ok",
  "providers": {
    "ollama": {
      "available": true,
      "status": "ready"
    },
    "openai-compatible": {
      "available": false,
      "status": "not_configured"
    }
  }
}

Health status semantics:

• ok: the local-first path is available
• degraded: local is unavailable, but cloud execution is still possible
• error: neither local nor cloud execution is currently usable

Route inspection:

curl -X POST http://127.0.0.1:8000/v1/route \
  -H "Content-Type: application/json" \
  -d '{"task":"summarize a short product description","private":false}'

Stats endpoint:

curl http://127.0.0.1:8000/v1/stats

It includes:

• chat vs embeddings request counts
• average total latency
• average chat latency
• average embeddings latency
• average local vs cloud latency
• average completion token speed for chat requests

Recent route logs:

curl http://127.0.0.1:8000/v1/logs

Each log entry includes:

• request kind
• total request latency
• completion tokens per second when available
• per-attempt timing in the trace

Model discovery:

curl http://127.0.0.1:8000/v1/models

Ecosystem workflows:

• OpenClaw: examples/openclaw.md
• Unsloth: examples/unsloth.md

Embeddings:

curl -X POST http://127.0.0.1:8000/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{
    "input":"RouteLabs Router chooses between local and cloud models based on privacy and task complexity.",
    "private":false
  }'

If local embeddings fail and cloud embeddings are not configured, RouteLabs now returns a clearer configuration error instead of a misleading “provider does not support embeddings” message.