自托管AI栈

PostgreSQL with pgvector (required by LiteLLM; pgvector enables vector storage for RAG)

docker run -d --name litellm-db --restart always \ --network ai-stack \ -e POSTGRES_USER=litellm \ -e POSTGRES_PASSWORD=litellm \ -e POSTGRES_DB=litellm \ -v litellm-db:/var/lib/postgresql \ pgvector/pgvector:pg18-trixie

Stop and remove all containers (data is preserved in Docker volumes)

docker compose down ```

Running without Docker Compose

If you prefer using docker run commands directly, first create a shared network so services can communicate:

docker network create ai-stack

Then start each service on the shared network:

Note: With manual docker run, wait for each dependency to become ready before starting services that use it (for example, wait for PostgreSQL and any other dependencies, such as Ollama or MCP, before LiteLLM; if using AnythingLLM, wait for LiteLLM before starting it). For production or shared Docker networks, change the default PostgreSQL password before first start and update every matching connection string.

```bash

Internet-facing deployments

By default, all services listen over plain HTTP. For internet-facing deployments, use the included Caddy overlay to add automatic HTTPS. In proxy mode, Caddy is the only public listener on ports 80 and 443; the direct AnythingLLM and LiteLLM ports are rebound to 127.0.0.1.

Prerequisites:

• Docker Compose 2.24.4+ (required for the proxy overlay's port override)
• A DNS A/AAAA record for your domain pointing to this server
• Inbound 80/tcp, 443/tcp, and ideally 443/udp open in your firewall/security group
• No other service already using ports 80 or 443 on the host

CPU stack:

DOMAIN=chat.example.com ACME_EMAIL=you@example.com \
  docker compose -f docker-compose.yml -f docker-compose.proxy.yml up -d

CUDA stack:

DOMAIN=chat.example.com ACME_EMAIL=you@example.com \
  docker compose -f docker-compose.cuda.yml -f docker-compose.proxy.yml up -d

Open https://chat.example.com (replace with your DOMAIN) to access AnythingLLM. In proxy mode, http://127.0.0.1:3001 and http://127.0.0.1:4000/ui remain available on the host, but the direct 3001 and 4000 ports are not reachable from outside the server.

The standard compose files publish LiteLLM on port 4000. The proxy overlay changes that direct port to localhost-only, and the included Caddyfile routes only AnythingLLM by default. Uncommenting the optional LiteLLM hostname block exposes LiteLLM through Caddy, so keep the LiteLLM master key secret.

Troubleshooting:

```bash docker logs ai-stack-caddy

Stop and remove all containers (data is preserved in Docker volumes)

docker compose down mkdir -p backups for vol in ollama-data litellm-data litellm-db embeddings-data whisper-data whisper-live-data kokoro-data mcp-data docling-data anythingllm-data caddy-data caddy-config; do docker volume inspect "$vol" >/dev/null 2>&1 && \ docker run --rm -v "${vol}:/source:ro" -v "$(pwd)/backups:/backup" \ alpine tar czf "/backup/${vol}.tar.gz" -C /source . done ```

Note: The ollama-shared, mcp-shared, and litellm-shared volumes are ephemeral key-sharing volumes and do not need to be backed up.

For restore instructions, server migration, and the full pre-upgrade checklist, see the Backup and Restore guide.

Quick start

Requirements:

• A Linux server (local or cloud) with Docker installed
• At least 8 GB of RAM (with small models). For larger LLM models (8B+), 16 GB or more is recommended.
• You can comment out services you don't need to reduce memory usage.

Start the full stack:

```bash

Voice pipeline example

Transcribe a spoken question, get a local LLM response via Ollama, and convert it to speech:

Note: Kokoro (TTS) is disabled by default. To use this example, first uncomment the kokoro service in docker-compose.yml, then run docker compose up -d.

Tip: Need a sample audio file? Download this English speech sample (WAV, MIT License) from the Azure Samples repository:

curl -L -o sample_speech.wav \
    "https://github.com/Azure-Samples/cognitive-services-speech-sdk/raw/master/sampledata/audiofiles/katiesteve.wav"

```bash LITELLM_KEY=$(docker exec litellm litellm_manage --getkey)

RAG pipeline example

Embed documents for semantic search, retrieve context, then answer questions with a local Ollama model:

```bash LITELLM_KEY=$(docker exec litellm litellm_manage --getkey)

MCP tools example

Use MCP Gateway to give your AI assistant access to files, web, and GitHub:

```bash MCP_KEY=$(docker exec mcp mcp_manage --showkey | grep '^mcp-' | head -1)

Set Authorization header: Bearer <api_key>

Or test the MCP endpoint directly with an initialize request

curl -s http://localhost:3000/mcp \ -X POST \ -H "Authorization: Bearer $MCP_KEY" \ -H "Content-Type: application/json" \ -H "Accept: application/json, text/event-stream" \ -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-03-26","capabilities":{},"clientInfo":{"name":"test","version":"1.0"}}}' ```

Export API keys (while containers are running)

docker exec ollama ollama_manage --showkey docker exec litellm litellm_manage --showkey docker exec mcp mcp_manage --showkey

Use MCP endpoint with an AI client (e.g., Cline in VS Code)

Step 2: At query time, embed the question, retrieve the top matching chunks from

the vector DB, then send the question and retrieved context to Ollama via LiteLLM.

curl -s http://localhost:4000/v1/chat/completions \ -H "Authorization: Bearer $LITELLM_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "ollama/llama3.2:3b", "messages": [ {"role": "system", "content": "Answer using only the provided context."}, {"role": "user", "content": "What does Docker do?\n\nContext: Docker simplifies deployment by packaging apps in containers."} ] }' \ | jq -r '.choices[0].message.content' ```