LLMesh

Prerequisites

• Docker + Docker Compose v2 (Docker Desktop on macOS/Windows, docker-compose-plugin on Linux)
• At least one machine running Ollama (or vLLM / MLX) — this is the agent side; not Docker
• .env and server_config.json files (templates below)

Prerequisites

• Python 3.10+
• Ollama running locally (if running an agent node)

Quick Start (Docker — recommended)

Hub-side install via Docker Compose. Brings up the FastAPI hub + a Postgres-backed session store. Agents still run on each compute host bare-metal (they need GPU/Ollama/MLX access on the host, so they don't containerise sensibly).

Installation

1. Clone the repository and navigate to the project root:

   cd llmesh
   

   python -m venv .venv
   source .venv/bin/activate
   pip install --upgrade pip
   

Hub or headless agent (server / Linux deployment):

   pip install .
   

Desktop tray client (macOS / Windows developer machine):

   pip install '.[desktop]'
   

Building the desktop binary (PyInstaller):

   pip install '.[desktop,build]'
   

Running the test suite:

   pip install '.[dev]'
   

4. Install git hooks — Ledger Law check + gitleaks secrets scan:

   bash scripts/install_git_hooks.sh
   

   brew install gitleaks
   

3. Optional: Session Memory & Configuration

The Hub maintains per-session conversation history so clients do not need to resend the full message history on every turn. History is stored in a local sessions.db SQLite file (created automatically — no setup needed).

How it works: Include an X-Session-ID header in your requests. The Hub returns the assigned session ID in the same response header:

```bash

4. Optional: Ollama context window (`OLLAMA_NUM_CTX`)

When Ollama runs a model, it allocates a KV cache sized to the context window at load time. The default context window is 4096 tokens, regardless of what the model was trained on. Models trained on larger contexts (e.g. 32768 tokens for Llama 3) produce a log warning:

llama_context: n_ctx_per_seq (4096) < n_ctx_train (32768)
-- the full capacity of the model will not be utilized

This is not a crash — inference still works — but it means the model cannot attend to more than 4096 tokens of input at once. Long prompts, multi-turn conversations, or document-grounded queries will silently truncate if they exceed this limit.

The agent sets num_ctx on every Ollama request via the OLLAMA_NUM_CTX environment variable:

Why 8192 and not the full training context?

num_ctx directly controls VRAM allocation — Ollama pre-allocates the full KV cache at load time. Setting it too high can prevent the model from loading at all on machines with limited VRAM:

8192 covers the vast majority of chat and code tasks. The per-machine override lets high-VRAM nodes serve larger contexts without forcing that requirement on every node in the mesh.

Setting per machine (in the agent's .env):

```bash

6. Optional: Metrics retention

Inference events and node snapshots accumulate indefinitely by default. Pruning runs automatically in the hub's cleanup loop. Configure via server_config.json or env vars:

{
    "metrics": {
        "retention_days_events": 30,
        "retention_days_snapshots": 7
    }
}

`0.21.1` — Unauthenticated `/version` endpoint

• GET /version (unauthenticated, D097). Returns {"version": "<APP_VERSION>"} for fast post-deploy verification (curl -s https://mesh.qcoda.com/version) without an API key. /health stays version-less per its CVE-targeting comment; version-string enumeration adds no surface beyond what pip index versions or GitHub Releases already expose.

`0.21.3` — Multi-backend + Anthropic-endpoint tool coverage

• /v1/messages (Anthropic endpoint) now forwards tools + tool_choice (D099). Previously the Anthropic path accepted the schema fields but never forwarded them to the agent (text-only response, stop_reason:end_turn). Now forwards both with the same hub-enforced filter/post-validate as the OpenAI path and emits native Anthropic tool_use content blocks (non-streaming + SSE), flipping stop_reason to "tool_use".
• vLLM + MLX backends now forward tools (D099). The agent's vLLM/MLX non-streaming path now sends tools and parses tool_calls/reasoning_content from the upstream response — closing the last silent-drop on non-Ollama nodes. tool_choice stays hub-enforced (not forwarded), matching the Ollama contract.
• Still not covered. qwen3-thinking <think>...</think> blocks (MLX) are a separate gap from gpt-oss harmony. Streaming tool_call.arguments arrives as a single synthesized delta (Ollama emits the whole call in one frame), not true per-token incremental deltas. Full breakdown in .qcoda/api.md.

API Endpoints

Once the Hub is running, it exposes four standard LLM API endpoints (chat, embeddings, Anthropic messages, image generation). Point any compatible SDK at http://localhost:8000 using your API key from server_config.json.

Core Project Components

Based on our recent implementation phases, the system is designed around two primary components:

1. Hub (lib/hub/): A FastAPI-based central orchestrator. It manages node registrations, tracks active hardware resources, queues inference requests, and dynamically routes those tasks to the optimal available node. It also provides a web-based dashboard for real-time monitoring of tasks and connected nodes.
2. Agent (lib/agent/): A lightweight Python client running on contributor or execution machines. The agent detects locally running inference backends (Ollama, and optionally vLLM/MLX), registers hardware capabilities and available models with the Hub, maintains a heartbeat, and continuously polls for tasks. Upon receiving a task it dispatches to the appropriate local backend and transmits the result back to the Hub.

Our project history reflects ongoing architectural evolution, particularly focusing on migrating from a single-owner MVP model to a scalable, multi-tenant SaaS architecture.

Bare-metal install (alternative)

Use this when you want SQLite (default), are developing on the hub itself, or are installing the agent on a compute host.