MiniClosedAI

Requirements

Five Python dependencies — fastapi, uvicorn, httpx, pypdf, python-multipart. That's it. The same five whether you run heavy or lite.

Lite mode (no local Ollama) drops every requirement except Python + the five pip packages. See Lite install below.

---

Install

Quickest — one-line install (macOS, Linux, WSL)

curl -fsSL https://raw.githubusercontent.com/edantonio505/miniclosedai/main/install.sh | bash

What it does: clones to ~/miniclosedai, creates a Python venv, installs the three dependencies, and starts the server detached on port 8095. When it returns, open <http://localhost:8095>. Re-run the same command later to update — it git pulls the latest and reinstalls deps in place.

No curl? Same thing with wget:

wget -qO- https://raw.githubusercontent.com/edantonio505/miniclosedai/main/install.sh | bash

Tunables (export before piping, or prepend on the same line):

Example — install to a custom path, skip auto-start:

curl -fsSL https://raw.githubusercontent.com/edantonio505/miniclosedai/main/install.sh \
  | MINICLOSEDAI_DIR=/opt/miniclosedai MINICLOSEDAI_START=0 bash

Requirements the script checks before doing anything: git, python3 ≥ 3.10. On a fresh macOS, both ship with the developer tools (xcode-select --install once if you've never used git). On Debian/Ubuntu, sudo apt install git python3 python3-venv covers it. Docker is not required — this is the bare-metal lite install.

Windows: use WSL (the command above works inside an Ubuntu/Debian WSL shell). Native Windows install is supported via the manual path below — there's no PowerShell installer yet.

All installation paths

Two methods (Docker or bare-metal), each with two modes — heavy (Ollama + baked models, ~10 GB image) or lite (no local Ollama; you point at any external Ollama / OpenAI-compatible endpoint via the Settings page, ~160 MB image, runs on any laptop). Pick whichever combination fits your hardware and use case:

Lite mode is the right pick when you have an inference server somewhere else — another machine on your LAN, a cloud Ollama relay, an LM Studio, vLLM, or any OpenAI-compatible URL — and you just want the playground UI on this machine.

Docker quick start (with baked models)

One-command setup that boots MiniClosedAI and Ollama with three small-but-capable models (llama3.2:3b, qwen2.5:3b, gemma2:2b — about 5.5 GB of weights) already on disk inside the image. No ollama pull step. No host Ollama install. Works on any Linux with Docker + NVIDIA GPU; CPU-only fallback below.

Requirements

Bring it up

git clone https://github.com/edantonio505/miniclosedai.git && cd miniclosedai
docker compose up -d --build

First build takes ~8–15 min (downloads three models from registry.ollama.ai). Subsequent up -d calls boot in ~30 s. When both services report healthy in docker compose ps, open <http://127.0.0.1:8095> — the model dropdown lists the three baked models under Ollama (built-in).

CPU-only hosts

If you don't have (or don't want to use) an NVIDIA GPU:

docker compose -f docker-compose.yml -f docker-compose.cpu.yml up -d --build

The override strips the GPU reservation. Inference on the 3B models will be noticeably slower; gemma2:2b stays usable in real time.

Adding or removing models after build

Models pulled at runtime persist in the ollama_models named volume across restarts and image rebuilds:

```bash

Docker — lite (no built-in Ollama)

Single-service compose file. Brings up only the MiniClosedAI web app (~160 MB image) — no Ollama container, no GPU passthrough, no nvidia-container-toolkit requirement, no model layers. The dashboard starts empty; you register your external endpoint(s) through the Settings page and the dropdown lights up.

```bash git clone https://github.com/edantonio505/miniclosedai.git && cd miniclosedai docker compose -f docker-compose.lite.yml up -d --build

Systemd installs automatically. To verify:

systemctl status ollama curl http://localhost:11434/api/tags # → {"models":[]}

**Windows**

Download and run `OllamaSetup.exe` from <https://ollama.com/download>. It installs as a background service. Verify with PowerShell:

Lite install (no local Ollama)

If you don't want to install Ollama on this machine — for example, you'll point at a remote Ollama, an LM Studio on your LAN, or any OpenAI-compatible endpoint — skip steps 1 and 2 above and run just step 3, then start the app with the MINICLOSEDAI_NO_OLLAMA=1 env var:

git clone https://github.com/edantonio505/miniclosedai.git && cd miniclosedai
python -m venv .venv
source .venv/bin/activate             # Windows: .venv\Scripts\activate
pip install -r requirements.txt       # 5 deps total, no system packages

MINICLOSEDAI_NO_OLLAMA=1 \
  python -m uvicorn app:app --host 0.0.0.0 --port 8095

(On Windows PowerShell: $env:MINICLOSEDAI_NO_OLLAMA=1; python -m uvicorn app:app --host 0.0.0.0 --port 8095)

Open <http://localhost:8095>. The dashboard's empty state shows "Welcome — let's add your first endpoint" with a button that takes you straight to Settings → Add endpoint. Fill in:

• Name — anything you like (e.g. Home server Ollama).
• Kind — ollama for native Ollama servers (recommended; supports think: false properly), openai for LM Studio / vLLM / llama.cpp / Bonsai / any OpenAI-compatible URL.
• Base URL — the full URL of your server. For native-Ollama use the host root (e.g. https://my-server.example.com — no /v1 suffix). For OpenAI-compat include /v1 (e.g. http://192.168.1.50:1234/v1).
• API key (optional) — sent as Authorization: Bearer <key> on every request. Use this if your endpoint is gated by a Bearer token.
• Custom headers (optional) — any extra headers your endpoint requires.

Save. The endpoint's models appear in the dashboard's model dropdown immediately. From there everything else (chat, attachments, API Code modal, fine-tuning data export) works exactly the same as the heavy install — only the inference happens elsewhere.

What MINICLOSEDAI_NO_OLLAMA=1 does:

• On a fresh database, the built-in Ollama backend isn't seeded. The dashboard starts with zero endpoints registered.
• On an existing database that previously ran in heavy mode, the built-in row is auto-disabled at startup (its enabled flag is set to 0) so it doesn't show as a permanently-broken endpoint in the dropdown.
• Without the env var, behavior is identical to the heavy default — no breaking changes for existing users.

Other env vars you may want for the lite setup:

---

One-shot setup — builds llama.cpp and downloads the default GGUF model (Bonsai-8B)

./setup.sh

UI guide

Clone PrismML's demo repo

git clone https://github.com/PrismML-Eng/Bonsai-demo.git cd Bonsai-demo

→ click ⚙️ Settings → Add endpoint

Build is ~30 seconds (one Python deps layer, no model pulls). The compose file sets `MINICLOSEDAI_NO_OLLAMA=1` in the container env so the built-in Ollama backend isn't seeded — the dashboard's empty state shows a "Welcome — let's add your first endpoint" CTA that flips to the Settings tab in one click.

| When to pick this | When to pick the heavy default |
|---|---|
| You have an Ollama (or OpenAI-compat server) on another machine, in a VPS, or behind a relay. | You want everything on one host. |
| Laptop with no GPU, or with limited disk. | Workstation or server with NVIDIA GPU + ≥20 GB free disk. |
| You don't want a 10 GB image to build/store. | You want zero external dependencies. |
| Your hardware can't run a 7–9B model locally. | Your hardware can. |

To switch back to the heavy default later, just bring the lite stack down and start the standard one — the SQLite DB lives in the same `miniclosedai_db` volume, so your conversations persist:

The auto-disable on the built-in row is reversible — running the standard compose without MINICLOSEDAI_NO_OLLAMA won't re-flip the row's enabled flag automatically. If you've previously been in lite mode and the built-in row is disabled, re-enable it once via the Settings page (or set its enabled = 1 in the DB).

To run without Docker — keep reading.

API Code modal

Three independent toggles produce 12 snippet variants:

• Language: cURL · Python · JavaScript
• Mode: Streaming · Non-streaming
• Style: Native · OpenAI-compat

A clickable "Bot #N" pill sits in the modal header next to the title. One click copies just the raw conversation id (e.g. 42) to the clipboard — useful when your microservice config only needs the id, not the full URL or code. The pill briefly flashes "Copied!" in accent color, then fades back. Disabled (greyed) when no conversation has been created yet.

The modal can be opened from two places: the chat topbar's </> icon (scopes to the currently-open bot), OR a bot card's row </> action on the Bots list (scopes to that specific bot without changing your current open chat). The pill always reflects whichever conversation the modal is currently scoped to.

Copy button works on both HTTPS/localhost (via navigator.clipboard) and plain-HTTP LAN (falls back to document.execCommand("copy")).

<p align="center"> <img src="docs/images/miniclosedai3.png" alt="API Integration Code modal with three toggle groups — Language, Mode, Style — showing the JavaScript / Non-streaming / OpenAI-compat variant pointed at this instance" width="820"> <br><em>Every saved chat is a microservice. Copy the snippet as cURL, Python, or JavaScript — native or OpenAI-SDK-compatible.</em> </p>

Connecting LM Studio and other OpenAI-compatible endpoints

MiniClosedAI ships with Ollama as a built-in endpoint and lets you register any number of additional servers alongside it: OpenAI-compatible ones like LM Studio, vLLM, llama.cpp's server binary, Text Generation WebUI's OpenAI extension, or the real OpenAI API itself, plus Ollama-compatible cloud services such as Interdata Lab that expose a remote Ollama at a public URL.

Each saved conversation picks one endpoint + one model; the Dashboard's model dropdown groups everything into a single OpenWebUI-style <optgroup> picker so you can chat with a Qwen3.6 on LM Studio and a Llama3.2 on Ollama in separate tabs without swapping anything.

<p align="center"> <img src="docs/images/miniclosedai6.png" alt="MiniClosedAI Settings page with three registered LLM endpoints: Ollama (built-in) at localhost:11434 showing 23 reachable models, LM Studio at 192.168.0.110:1234/v1 with 7 models and an API key set, and Bonsai at localhost:8080/v1 with 1 model — each card has Edit and (for non-built-in) Delete buttons" width="820"> <br><em>Settings → LLM Endpoints. The built-in Ollama, an LM Studio instance on your LAN, and PrismML's 1-bit Bonsai server all coexist. Each card shows its kind, base URL, API-key status, and a live reachability count.</em> </p>

Serves at http://localhost:8080 — health check at /health, API at /v1/chat/completions

Pick a different size by setting `BONSAI_MODEL` before the script: `8B` (default), `4B`, or `1.7B`.

aarch64 / ARM: the shipped binaries are x86_64. Build from source with ./scripts/build_cuda_linux.sh (Linux+NVIDIA) or ./scripts/build_mac.sh (macOS+Metal) before the start_llama_server step.

2. Register Bonsai as an endpoint in MiniClosedAI

Settings (gear icon, bottom of activity bar) → + Add endpoint:

Click Test connection — should say "✓ Reachable · 1 model(s)" and list Bonsai-8B.gguf (or whichever size you loaded). Save.

3. Chat with Bonsai

Back on the Dashboard, open the model dropdown → there's a Bonsai optgroup with Bonsai-8B.gguf inside it. Create a new chat, pick that model, write a prompt. Every subsequent turn routes automatically to http://localhost:8080/v1/chat/completions because the conversation is pinned to (backend_id=<Bonsai>, model="Bonsai-8B.gguf").

The per-conv microservice pattern applies unchanged: POST /api/conversations/{id}/chat is your Bonsai bot's stable callable URL, and the API Code modal emits cURL / Python / JavaScript snippets that point at it.

Ready-made Bonsai microservice recipe: see RAG Query Router.md for a complete system prompt, recommended settings, and Python integration code for a latency-critical query-classification bot that's purpose-built for Bonsai's speed profile. Covered in the Recipes section below as #8 RAG query router.

Notes and gotchas specific to Bonsai

• Thinking: Off (or Default). start_llama_server.sh already boots with --reasoning-budget 0 --reasoning-format none --chat-template-kwargs '{"enable_thinking": false}' — thinking is disabled upstream. Leave MiniClosedAI's Thinking control on Off or Default to match. Flipping it to On just wastes tokens; the server still won't emit reasoning.
• Context window. Bonsai-8B is trained at 65,536 tokens; the start script uses llama.cpp's -c 0 (auto-fit). For very long contexts your GPU VRAM will be the binding constraint, not MiniClosedAI.
• Server-side sampling defaults (set in start_llama_server.sh): temp=0.5, top-p=0.85, top-k=20, min-p=0. MiniClosedAI's per-conversation sliders override these on every request, so tune per-chat as usual.
• Pitfall — wrong port = feedback loop. If you accidentally set Bonsai's Base URL to http://localhost:8095/v1 (MiniClosedAI's own port), the endpoint's /v1/models call loops back and returns MiniClosedAI's saved conversations as "models". The model dropdown will show conversation IDs (e.g. "30") under the Bonsai optgroup; picking one sends the chat through that conversation's bot instead of Bonsai, producing nonsensically on-topic responses (the Lead Qualifier's JSON, etc.). Fix: edit the endpoint, change the port to 8080.
• Stop the server when you're done: kill $(lsof -ti TCP:8080) (or Ctrl+C in the terminal you started it in).

Managing endpoints

From the Settings page:

• Test connection (on each card or in the Add/Edit modal) probes the endpoint through the MiniClosedAI server — avoids browser CORS blocks on cross-origin calls to LM Studio.
• Edit lets you change the name, URL, API key, or custom headers. kind is immutable once saved.
• Delete removes a non-built-in endpoint. Two paths depending on whether bots are pinned to it:
• No bots pinned → single confirm, gone.
• N bots pinned → first dialog lists the bot titles and recommends rebinding instead. If you OK, a second dialog gates the cascade (Really delete N bot(s)? This is permanent.); confirming both deletes the backend AND every bot pinned to it in one transaction. Use this for stale endpoints whose bots aren't worth migrating.
• The built-in Ollama endpoint can be renamed or have its URL changed (useful if Ollama runs on a different port or host), but can't be deleted (the 403 fires regardless of force=true so you can't accidentally wipe the only Ollama row in lite mode).

API reference — native endpoints

Base URL: http://<host>:8095. All endpoints return JSON unless noted. Interactive OpenAPI docs at /docs.

Backends (endpoint lifecycle)

GET    /api/backends              → list all (api_key scrubbed to api_key_set bool)
POST   /api/backends              → create. Strip trailing /, normalize URL.
PATCH  /api/backends/{id}         → update (kind is immutable)
DELETE /api/backends/{id}         → 403 on is_builtin, 409 if bound (force=true cascades)
GET    /api/backends/{id}/models  → list that backend's models only
GET    /api/backends/{id}/status  → is it reachable?
POST   /api/backends/test         → probe a draft config without saving

Create body:

curl -X POST http://localhost:8095/api/backends \
  -H "Content-Type: application/json" \
  -d '{
    "name": "LM Studio",
    "kind": "openai",
    "base_url": "http://localhost:1234/v1",
    "api_key": "optional-bearer-token",
    "headers": {"X-Custom": "optional"}
  }'

Valid kind values: "ollama" · "openai". The OpenAI kind speaks the /v1/chat/completions wire format and works with any compliant server.

Delete guardrails:

• DELETE /api/backends/1 (or any row with is_builtin=1) → 403 Forbidden, regardless of force=true. The built-in is undeletable so a lite-mode user can't accidentally end up with zero Ollama rows.
• DELETE /api/backends/<id> when one or more conversations still point at it → 409 Conflict with a bound_conversations list. Rebind those conversations first (change their saved model to one on a different backend), then retry. Or cascade-delete with ?force=true to remove the backend AND every bot pinned to it in one transaction. Response body includes deleted_conversations: <count> so callers can confirm what was wiped.

```bash