简介

<p align="center"> <img src="https://raw.githubusercontent.com/mohitsoni48/TurboLLM/main/turbollm/web/public/brand/turbollm-icon-512.jpeg?v=2" width="92" height="92" alt="TurboLLM" /> </p>

TurboLLM

<p align="center"> <strong>Run <em>any</em> local LLM engine, auto-tuned to your GPU — with a polished web UI and an OpenAI/Anthropic-compatible API.</strong><br/> Bring your own llama.cpp fork. No compiling. No Electron. No Python. Point Claude Code at your own machine in one command — fully offline. </p>

<p align="center"> <a href="https://turbollm.dev"><img src="https://img.shields.io/badge/turbollm.dev-docs%20·%20models%20·%20guides-e2552e" alt="turbollm.dev" /></a> <a href="https://www.npmjs.com/package/turbollm"><img src="https://img.shields.io/npm/v/turbollm.svg?color=e2552e" alt="npm version" /></a> <a href="https://www.npmjs.com/package/turbollm"><img src="https://img.shields.io/npm/dm/turbollm.svg?color=e2552e" alt="npm downloads" /></a> <img src="https://img.shields.io/badge/node-%E2%89%A522-3c873a.svg" alt="node >= 22" /> <img src="https://img.shields.io/badge/license-FSL--1.1--ALv2-blue.svg" alt="license" /> <img src="https://img.shields.io/badge/platform-Windows%20%C2%B7%20macOS%20%C2%B7%20Linux-555.svg" alt="platforms" /> <a href="https://ko-fi.com/mohitsoni"><img src="https://img.shields.io/badge/Ko--fi-support%20us-FF5E5B?logo=kofi&logoColor=white" alt="Ko-fi" /></a> <a href="https://github.com/sponsors/mohitsoni48"><img src="https://img.shields.io/badge/GitHub%20Sponsors-support%20us-EA4AAA?logo=githubsponsors&logoColor=white" alt="GitHub Sponsors" /></a> <a href="https://discord.gg/v6kRbV7nC"><img src="https://img.shields.io/badge/Discord-join%20chat-5865F2?logo=discord&logoColor=white" alt="Discord" /></a> </p>

<p align="center"> _{<strong>Source-available</strong> under <a href="#license">FSL-1.1</a> — free for personal and internal business use; every release converts to <strong>Apache-2.0</strong> two years after it ships.} </p>

npx turbollm

That one command starts a local daemon, opens a browser UI, and serves your models over an API any tool can talk to. TurboLLM is the performance & bleeding-edge layer for local LLMs — built for people who today hand-compile forks and hunt forums for the right flags. The speed is measured, not promised: **74.7 vs 61 t/s on the same official llama.cpp as LM Studio — and 2.2× on forks it can't load** (the numbers).

<p align="center"> <strong>📖 Full docs · what runs on your GPU · model picks → <a href="https://turbollm.dev">turbollm.dev</a></strong> </p>

<p align="center"> <img src="https://raw.githubusercontent.com/mohitsoni48/TurboLLM/main/assets/how-it-works.svg?v=2" width="860" alt="How TurboLLM works: clients -> one lightweight daemon -> any engine on your GPU" /> </p>

---

Features

The headline — running any engine, including community forks — has its own section below. Everything else is grouped here; each summary is the gist, expand for the detail:

<details> <summary><strong>📦 Models — bring your own, or browse Hugging Face</strong></summary>

<br/>

- Use the folders you already have. Point TurboLLM at any directory of GGUFs — your existing LM Studio folders or manual downloads — no re-downloading. It parses GGUF metadata (arch, params, quant, context, vision) for every file. (Models pulled by ollama pull live in Ollama's own blob store, which TurboLLM doesn't read.) - Browse & download from Hugging Face, in-app: a live, sortable list (trending / downloads / likes / recently updated / newest) alongside a permanent detail pane — pick a quant, read the rendered model card, and download with resume + SHA-256 verification. Each model lands in its own folder (mirroring Hugging Face's own layout) with its vision projector and every shard of a split/multipart quant fetched alongside it automatically. Gated models (Llama, Gemma) work via your own HF token, which never leaves your machine. - Import from any URL — not just Hugging Face. Paste a direct .gguf link (model-author sites, mirrors, private servers) and it disk-space-checks and downloads through the same manager; paste a Hugging Face model page link instead and it opens that repo's quant picker so you can choose which one to download. - Quant recommendation per GPU and a VRAM-fit verdict so you pick a quant that actually fits before you commit. - Primary download folder, real-time measured t/s per model, delete-from-disk, and pin your favourites to the top of the list.

</details>

<details> <summary><strong>⚡ Auto-tuning &amp; performance</strong></summary>

<br/>

- Auto-benchmark on load derives fast defaults for your exact GPU. - Recommended sampling from Hugging Face — auto-tune checks a repo's structured params / generation_config.json sidecar first when the quantizer publishes one (exact values, no guessing), then falls back to reading the model's card (and the original model behind a requant) and prefills the author's recommended temperature / top_k / top_p / min_p. No recommendation → your sampling is left untouched. - Real measured tokens/sec in the model list — live while generating, last-session when idle (never a synthetic estimate). - Full load-parameter UI, a superset of what other tools expose: context length, GPU offload (-ngl), MoE CPU-offload (--n-cpu-moe), parallel slots, independent K and V cache-quant type (incl. low-bit on supporting forks), CPU threads, flash attention, speculative decoding (NextN / MTP / draft / DFlash, with a configurable draft min/max window), a pinned engine port per model, and a custom/raw flags field for anything not exposed as its own control. - Copy the exact launch command for a loaded model — runs the same config standalone with llama.cpp/the fork directly, no TurboLLM required. - Auto-fit GPU layers / MoE offload — an optional toggle (off by default) that hands the GPU/CPU split decision to llama.cpp's own memory-fitting logic at load time instead of a fixed number, honored by Auto-tune too. Useful when a large context or model doesn't fit your usual fixed setting. - Fast by default: flash attention on, NextN self-speculative decoding on for models that carry a draft head, threads auto — safely gated to what your engine actually accepts. - Multi-GPU, per model — split a model across cards (layer/row split + main-GPU pick on llama.cpp, tensor-parallel on vLLM). Defaults are no-ops, so single-GPU rigs are untouched. - Saved per-model profiles, per engine — tune once per (model, engine) pair, so switching engines (or between two installs of the same engine, e.g. a fork) never overwrites another engine's tuning for the same model. - Named load presets, many per model — keep "max speed 32k" and "200k long context" side by side instead of the second tune overwriting the first. Pick one from the Model Detail panel to fill the load parameters (it's remembered and auto-applied on the next load), Save as… captures the current settings, and rename/delete live in the ⋯ menu. Auto-tune mints a preset every time you save one — named Auto-tune <date> and badged with its measured tok/s — so every tune becomes browsable, restorable history rather than a one-slot overwrite. The ten newest auto-tune presets are kept; your own saves are never pruned. Existing profiles carry over as presets automatically, and a model with no presets behaves exactly as before. - Configurable VRAM headroom (Settings → Models & loading → Advanced, 300 MB–2 GB, default 1 GB) — tell auto-tune how much VRAM to keep free for other GPU workloads instead of a fixed margin. Drag it to 0 to opt into an experimental MoE "VRAM-spill" search — auto-tune keeps pushing more experts onto the GPU past the safe margin as long as both generation and prompt-processing speed keep improving.

</details>

<details> <summary><strong>💬 Chat &amp; agentic tools — a genuinely good UI, not an afterthought</strong></summary>

<br/>

- Streaming with a stop button, live tokens/sec, prompt-processing % and prefill t/s, time-to-first-token, total time, exact token counts, and a context-usage meter (filled / max) on every reply. - Thinking control — toggle reasoning off for a direct answer, or leave it on with collapsible, timed "thought for N s" blocks. - Markdown + syntax-highlighted code with one-click copy — plus inline Unicode charts the model draws when a comparison, trend, or hierarchy is genuinely worth a visual. - Live artifacts — html, svg, and mermaid replies render as sandboxed, offline previews shown as an image, with one-click export to PNG / JPEG / SVG / animated GIF / HTML. - Agents — pick a style (Default · Designer · Concise · Detailed · Blunt · Formal · Tutor · Creative · Research · Lite · Code) per conversation, no prompt-wrangling required. The Designer persona produces polished, self-contained, previewable designs by default; Lite strips the hidden prompt to the bare minimum for the fastest responses. See below for editing built-ins or creating your own. - Edit or regenerate any message without losing history — both branch instead of overwriting, with a ‹ 1/2 › switcher to flip between versions (including nested branch points); delete and copy still work as before. Persistent, searchable conversations with rename, delete, and auto-generated titles, organized into drag-resizable, collapsible folders you create/rename/move conversations into. - Switching chats never cancels a reply — an in-flight generation keeps running in the background and saves normally; the sidebar shows a live indicator on any chat still generating, and a dot on one that finished while you were elsewhere. - Preserve thinking across turns (on by default) — the model's past reasoning is resent on later turns, not just its final answers, so follow-ups have real context to work with. - Per-chat system prompt and per-chat sampling overrides — temperature, top-p/k, min-p, repeat/presence/frequency penalties, and stop strings — prefilled from the loaded model's own recommended values, available even before you send the first message. - Image input for vision models, PDF and code/text attachments (real extracted text, not raw bytes), and TurboLLM Expert — a built-in assistant that knows the app and your hardware for onboarding and troubleshooting without leaving the UI. - Agentic tools — built-in web_search (Tavily), fetch_url, and sandboxed run_code, plus an MCP marketplace in Customize: one-click connect for hosted MCPs (GitHub, Linear, Stripe, Atlassian, Neon, Supabase, Cloudflare, Zapier, Apify, Mixpanel) and open-source local MCPs (filesystem, git, postgres, playwright, …), plus your own custom servers. Connected tools appear in every chat with no restart. A Research persona forces multi-step web search and cites sources inline. - Tool-call approval gate — every tool call asks for your approval by default before it runs, with Deny, Allow, Allow for this chat, or Always Allow on an inline bar above the composer. Set per-tool defaults globally from Settings → Tools & safety, or flip Auto-allow all to skip prompts entirely — a tool set to Deny still stays blocked either way. - Usage dashboard — a GitHub-style activity heatmap of your local generation history (adaptive 1h/12h/24h boxes for the 7-day/30-day/all-time views), streaks, peak hour, a per-model breakdown, a lifetime token-milestone tracker, and a separate API tab tracking tokens hitting the gateway from external tools like Claude Code — not just in-app chat. - Auto-memory (experimental, off by default) — silently extracts durable facts you mention in chat (name, preferences, hardware) using your own loaded model, and carries them into future new conversations. Nothing leaves your device; the full fact list is reviewable and deletable from Settings → Memory, and turning the toggle off stops new chats from seeing it immediately. - Turbo Link (experimental, off by default) — link a second machine running TurboLLM with one pasted link string and use its models as if they were local: they appear in the chat model picker (grouped by machine), in /v1/models and in launch --model, and Models, Downloads and Engines merge both machines' rows into one list. You grant an explicit capability set per link — use models only, run a server, or full control — optionally pinned to specific models, and revocable at any time from the host. Not yet verified against a real two-machine setup, so it stays behind Settings → Experimental. - Thinking-budget control — a graduated slider, not just on/off: cap reasoning to a specific token count, disable it entirely, or leave it unlimited. Works in Chat and Code alike. - Reasoning-effort control for models whose own chat template supports it (e.g. Qwen3.8) — an Off/Low/Medium/xhigh slider swaps in automatically in place of the thinking-budget one, detected per model.

</details>

<details> <summary><strong>🧑‍💻 Code — a local coding agent, in a real project directory</strong></summary>

<br/>

Workspace → Code hands a task to an agent running on the same model you already have loaded — point it at a repo folder, describe what you want, and it reads, edits, and runs commands to get there. Entirely local; nothing leaves your machine.

- Real repo access — plans and edits end-to-end (or asks before mutating, or plan-only, depending on mode), with an optional isolated git worktree so your actual checkout stays untouched, and a real diff summary of what changed. - Persistent sessions — archive/filter past runs, revert to any earlier message (with optional real file-edit reversal), attach files as context, and a "Coding activity" dashboard (sessions, tasks shipped, files touched, diff shipped, streaks) built from real history, not mock data. - Real LSP integration for TypeScript/JavaScript and Python — detects the language, installs the language server if needed, and uses it for edits. - The same tools Chat gets from Customize — any connected MCP server, plus the sandboxed run_code tool — are available to Code too, alongside honest skill invocation and AGENTS.md/agents.md support. - Independent access control — gated behind its own API key on non-host devices, separate from Chat's gate. - Expose it to other tools over MCP — turbollm mcp-server runs a stdio MCP server so any MCP-compatible tool (Claude Desktop, Cursor, Windsurf, Cline, Claude Code, and more) can delegate a real coding task to Code, not just chat completions. Setup snippets for any host are in the Developer tab. - Or run Claude Code, pi or opencode itself, inside Code — set Settings → Code agent and a session opens the real CLI in a full-screen terminal, on your local model, in your project folder. The model picker, context ring, thinking-budget slider and stats row stay right where they are: the CLI replaces the transcript, not the controls. Any of the three not installed yet is shown with an Install button rather than failing when you open the session, and a CLI launched this way sees only your local models — never your cloud API keys. It starts in the mode you picked (auto / plan / ask), the thinking budget applies live with no restart, switching models keeps your scrollback, and the conversation survives a restart of TurboLLM. The terminal backend is an optional native component — if it can't be built for your platform it's skipped (TurboLLM installs and runs normally), and this agent simply isn't offered on that machine.

</details>

<details> <summary><strong>🤖 Customize → Agents — edit any built-in, or build your own</strong></summary>

<br/>

- Edit any built-in agent in place — system prompt, which shared skills it uses, and which tools it may call — with a one-click Reset back to the original. - Create your own agent from scratch: a name, description, and system prompt, plus a checklist of which shared skills and which tools it's allowed to use (everything on by default). Pick it from the same in-chat agent picker as the built-ins. - MCP tools grouped by server in the tool checklist — one toggle selects or deselects an entire server's tools at once, or expand it to pick individually.

</details>

<details> <summary><strong>🔌 APIs &amp; integrations — OpenAI + Anthropic, a model-loading gateway, and a stateful chat API</strong></summary>

<br/>

With a model loaded, TurboLLM serves two compatible APIs on the same port:

```bash

⭐ Bring any engine — the headline feature

No other local-LLM app lets you run whatever inference engine you want. TurboLLM treats the engine as a swappable component.

Add a custom engine (Engines screen → Add your own engine):

1. Compile or download any llama-server-compatible binary — stock llama.cpp, a community fork, or your own build. 2. Point TurboLLM at the folder — it scans for the llama-server binary, runs a capability probe, and learns exactly which flags and features that build supports. (Optional: paste the source repo URL so TurboLLM flags when a newer build ships.) 3. Activate it. The load-parameter UI adapts to that engine — features the build doesn't support are hidden; ones it adds (e.g. low-bit KV cache, NextN) light up.

No prebuilt for your OS? The build-from-source guide checks your toolchain (git / CMake / CUDA / a compiler — MSVC on Windows, gcc/clang on Linux), hands you the exact build commands (or a 1-click "Build it for me" on Windows and Linux), then drops you into the folder scan above.

Or skip the manual clone entirely — Engines screen → Add via git repo: paste any llama.cpp-compatible fork's git URL (+ optional branch, defaults to the repo's own default) and build it in-app with the same 1-click flow, no separate "point at a folder" step needed.

Branch pickers and engine update checks both read GitHub's API, which allows 60 anonymous requests an hour. If you run into that ceiling, add a GitHub token in Settings → Models & loading to raise it to 5,000/hour — it is write-only and never leaves your machine.

Auto-provisioned default. Don't want to fetch anything? On first run TurboLLM downloads the right upstream prebuilt for your GPU automatically — and a backend picker lets you switch between CUDA / ROCm / Metal / SYCL / Vulkan / CPU at any time (it downloads the variant you choose, LM Studio-style).

Engine types. llama.cpp / GGUF, KoboldCpp and llamafile (GGUF, every OS), MLX and MLX-VLM (macOS), and vLLM (Linux + NVIDIA) are all first-class engine kinds — install from the curated catalog, pick the right one per model, and switch from a single dropdown.

Fully supervised. Every engine runs under a real state machine: health-gated readiness, graceful stop, an idle auto-stop watchdog, and live logs + clear error surfacing in the UI when something fails to load.

Why it matters: fork-exclusive features — speculative decoding (NextN / MTP / draft), low-bit KV cache, new quant formats — are usable on day 0, with zero compiler knowledge on your part beyond producing the binary (and often not even that).

---