# Sinain <img src="https://raw.githubusercontent.com/anthillnet/sinain-hud/main/media/screen-recording-2026-03-26.gif" alt="Sinain HUD" width="120" align="right">

[![MIT License](https://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/anthillnet/sinain-hud/blob/main/LICENSE)
[![CI](https://github.com/anthillnet/sinain-hud/actions/workflows/ci.yml/badge.svg)](https://github.com/anthillnet/sinain-hud/actions/workflows/ci.yml)
[![npm](https://img.shields.io/npm/v/@geravant/sinain)](https://www.npmjs.com/package/@geravant/sinain)
[![macOS 12.3+](https://img.shields.io/badge/macOS-12.3%2B-black?logo=apple)](https://support.apple.com/macos)
[![LongMemEval IPR 82.8%](https://img.shields.io/badge/LongMemEval%20IPR-82.8%25-success)](https://github.com/anthillnet/sinain-hud/blob/main/docs/LONGMEMEVAL-AUDIT.md)

**Context OS** — ambient intelligence for builders. Captures what you see and hear, distills it into a private knowledge graph, accessible from MCP, a web UI, and a screen-recording-invisible HUD overlay.

<p align="center">
  <img src="https://raw.githubusercontent.com/anthillnet/sinain-hud/main/media/sinain-demo-full.gif" alt="Sinain demo" width="720">
</p>

**[Quick Start](#quick-start)** · **[Docs](https://github.com/anthillnet/sinain-hud/tree/main/docs)** · **[Privacy](https://github.com/anthillnet/sinain-hud/blob/main/docs/privacy-protection-design.md)** · **[Configuration](https://github.com/anthillnet/sinain-hud/blob/main/docs/CONFIGURATION.md)** · **[Contributing](https://github.com/anthillnet/sinain-hud/blob/main/CONTRIBUTING.md)**

---

### You, Augmented

Sinain captures your screen and audio continuously, distills the stream into a **structured live knowledge graph** of facts, entities, and decisions, and exposes that graph through every interface where you might need it.

- **Capture** — screen frames + system audio, with `<private>` tag stripping and auto-redaction of credentials and tokens *before* anything leaves your machine.
- **Distill** — facts (atomic claims), entities (people / projects / topics), decisions (what was chosen and why) — extracted by an LLM, integrated by deterministic graph operations (no LLM in the integration step → **[82.8% Information Preservation Rate](https://github.com/anthillnet/sinain-hud/blob/main/docs/LONGMEMEVAL-AUDIT.md)** on the [LongMemEval benchmark](https://github.com/xiaowu0162/LongMemEval) (ICLR 2025, 500 questions), measured against full-context oracle).
- **Access from anywhere** — **MCP server** for agents (Claude Code, Codex, Goose, OpenClaude, Junie), **web UI** at `localhost:9500/knowledge/ui/` for browsing, and a **HUD overlay** for in-the-moment recall.

The HUD overlay is invisible to screen capture — never appears in screenshots, recordings, or screen shares.

### Agent-Agnostic

Sinain feeds the same screen and audio context to any MCP-compatible agent. Switch agents on the fly — no restart, no context loss.

- Tested with Claude Code, OpenClaude, Codex, Goose, and Junie. Any MCP-compatible agent works.
- Pick agents in the overlay's flash-icon selector — spawn tasks can route to any profile in your roster.
- Add custom profiles (personal Claude config, alternate models) by editing [`agents.json`](https://github.com/anthillnet/sinain-hud/blob/main/docs/AGENT-ROSTER.md). The roster is the source of truth.
- Knowledge modules travel with you — export from one machine, import on another.

### Privacy Controls

By default, sinain uses cloud APIs (OpenRouter) for transcription and analysis. When you need tighter control, switch privacy modes — no code changes, one env var.

- `off` → `standard` → `strict` → `paranoid` — four modes in `~/.sinain/.env`.
- `paranoid` mode: Ollama + whisper.cpp. **Zero network calls at runtime** — the embedding model is pre-cached during the setup wizard (`sinain setup-embedding`), so subsequent runs are fully offline.
- HUD overlay is invisible to screen capture (`NSWindow.sharingType = .none` — see [`overlay/macos/Runner/AppDelegate.swift:70`](https://github.com/anthillnet/sinain-hud/blob/main/overlay/macos/Runner/AppDelegate.swift#L70)). Verifiable via split-screen recording with QuickTime + OBS + Loom — the HUD is absent from all three.

## Quick Start

```bash
npx @geravant/sinain@latest start
```

That's it. On first run, sinain will:
1. Run an **interactive setup wizard** — transcription backend, API key, agent, privacy mode
2. **Auto-download** the overlay app (~17MB), sck-capture binary (~5MB), embedding model (~90MB), and Python dependencies
3. **Start all services** — sinain-core, sense_client, overlay, and agent

All assets are cached locally after the first install. In `paranoid` mode, subsequent runs are fully offline — no network calls at runtime.

> **Pin `@latest`** on every invocation. `npx @geravant/sinain` (without `@latest`) caches *forever* against the unversioned spec — you'd silently keep running an old version for months. Sinain self-updates automatically when stale, but pinning `@latest` makes it explicit and saves a redundant relaunch.

> **Re-run the wizard** anytime: `npx @geravant/sinain@latest start --setup`

### Prerequisites

- **macOS 12.3+** — Sinain uses ScreenCaptureKit (introduced in 12.3). Earlier versions are not supported in this release. Apple Silicon and Intel both work.
- **Node.js 18+** — [nodejs.org](https://nodejs.org/) (LTS recommended)
- **Python 3.10+** — `brew install python3` (macOS) or [python.org](https://www.python.org/downloads/)
- **OpenRouter API key** (optional for local-only mode) — [openrouter.ai](https://openrouter.ai)
- **Network access during first install** — the wizard downloads ~112MB total (overlay app, sck-capture binary, sentence-transformer embedding model). All cached locally; subsequent runs need network only for cloud LLM API calls (or zero network in `paranoid` mode).

> **Fully local?** No API key needed. Ollama + whisper-cli = zero cloud at runtime. See [Running Fully Local](#running-fully-local).

> **First install reproducibility?** See [docs/cold-install-verification.md](https://github.com/anthillnet/sinain-hud/blob/main/docs/cold-install-verification.md) for a step-by-step verified-on-fresh-user-account guide, including the timing measurement and the failure modes the audit caught + fixed.

### macOS Permissions

1. **System Settings → Privacy & Security → Screen Recording** — add your Terminal, then **quit and reopen Terminal** (macOS TCC entitlements only apply to processes started after the grant)
2. **System Settings → Privacy & Security → Microphone** — same: add Terminal, then restart Terminal

> Sinain detects when these permissions are missing and surfaces a clear restart-instruction banner; you'll never get a silent degraded mode.

### Managing sinain

```bash
npx @geravant/sinain@latest stop             # stop all services
npx @geravant/sinain@latest status           # check what's running
npx @geravant/sinain@latest start --setup    # re-run setup wizard
npx @geravant/sinain@latest start --no-sense # skip screen capture
npx @geravant/sinain@latest start --no-overlay  # headless mode
```

> Always pin `@latest` — see the note in [Quick Start](#quick-start) above.

## Architecture

```
┌─── Your Device ─────────────────────────────────────────────────────┐
│                                                                     │
│  sck-capture (Swift)                                                │
│    ├─ system audio (PCM) ──► sinain-core :9500                      │
│    └─ screen frames (JPEG) ──► sense_client ─── POST /sense ──►    │
│                                                      │              │
│                              ┌────────────────────────┘              │
│                              │                                      │
│                         sinain-core                                 │
│                           ├─ audio pipeline → transcription         │
│                           ├─ agent loop → digest + HUD text         │
│                           ├─ knowledge graph (private, on-device)   │
│                           └─ WebSocket feed                         │
│                                  │                                  │
│                                  ▼                                  │
│                           overlay (Flutter)                         │
│                           private, invisible to screen capture      │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘
```

## Components

| Component | Language | What it does | Docs |
|---|---|---|---|
| **sinain-core** | TypeScript | Central hub: audio pipeline, agent loop, knowledge graph, WS feed | [README](https://github.com/anthillnet/sinain-hud/blob/main/sinain-core/README.md) |
| **overlay** | Dart / Swift | Private HUD (macOS), display modes, hotkeys | [Hotkeys](https://github.com/anthillnet/sinain-hud/blob/main/docs/HOTKEYS.md) |
| **sense_client** | Python | Screen capture, SSIM diff, OCR, privacy filter | [sense_client/](https://github.com/anthillnet/sinain-hud/tree/main/sense_client) |
| **sck-capture** | Swift | ScreenCaptureKit: system audio + screen frames | [tools/sck-capture/](https://github.com/anthillnet/sinain-hud/tree/main/tools/sck-capture) |
| **sinain-agent** | Bash | Shell harness that connects any agent to sinain-core | [sinain-agent/](https://github.com/anthillnet/sinain-hud/tree/main/sinain-agent) |
| **sinain-knowledge** | TypeScript | Curation, playbook, eval, portable knowledge modules | [Knowledge System](https://github.com/anthillnet/sinain-hud/blob/main/docs/knowledge-system.md) |
| **sinain-mcp-server** | TypeScript | MCP server exposing sinain tools to agents | [sinain-mcp-server/](https://github.com/anthillnet/sinain-hud/tree/main/sinain-mcp-server) |

## Configuration

Sinain splits config across two files in `~/.sinain/`:

- **`.env`** — secrets (API keys) and infrastructure (ports, audio device, privacy mode, analyzer LLM).
- **`agents.json`** — agent roster (default agent, allowed-tools whitelists, analyzer pacing).

Both are created by the setup wizard. To re-run: `npx @geravant/sinain start --setup`.

### Agents & profiles → `agents.json`

The agent roster lives in `~/.sinain/agents.json`. Each entry is a profile mapping a name to a binary + behavior type + optional env, settings, and model overrides. The overlay's flash-icon selector lets you pick which profile handles spawn tasks at runtime. Custom profiles like `pclaude` (personal claude with its own config dir) are first-class — the dispatch decision keys off `profile.type`, not the profile name.

See **[Agent Roster & Profiles](https://github.com/anthillnet/sinain-hud/blob/main/docs/AGENT-ROSTER.md)** for the complete schema, recipes, and routing model.

### Context Analysis (HUD summarizer) → `.env`

The context analysis loop runs every 3–30 seconds, sending recent audio/screen context to an LLM. It produces the short HUD text shown on the overlay plus a richer digest stored in the feed buffer for the knowledge graph.

| Variable | Default | Description |
|---|---|---|
| `ANALYSIS_PROVIDER` | `openrouter` | `openrouter` (cloud) or `ollama` (local, free) |
| `ANALYSIS_MODEL` | `google/gemini-2.5-flash-lite` | Primary model for text analysis |
| `ANALYSIS_VISION_MODEL` | `google/gemini-2.5-flash` | Auto-selected when screen images are present |
| `ANALYSIS_ENDPOINT` | *(auto per provider)* | Override for custom OpenAI-compatible endpoints |
| `ANALYSIS_API_KEY` | *(from OPENROUTER_API_KEY)* | API key; not needed for ollama |
| `ANALYSIS_FALLBACK_MODELS` | `gemini-2.5-flash,...` | Comma-separated fallback chain |

### Other Key Settings → `.env`

| Variable | Default | Description |
|---|---|---|
| `OPENROUTER_API_KEY` | — | Required (unless `ANALYSIS_PROVIDER=ollama` + local transcription) |
| `PRIVACY_MODE` | `off` | `off` / `standard` / `strict` / `paranoid` |

See [docs/CONFIGURATION.md](https://github.com/anthillnet/sinain-hud/blob/main/docs/CONFIGURATION.md) for the full reference.

## Privacy Modes

| Mode | What it does |
|---|---|
| `off` | All data flows freely — maximum insight quality |
| `standard` | Auto-redacts credentials before cloud APIs (wizard default) |
| `strict` | Only summaries leave your machine — no raw text sent to cloud |
| `paranoid` | Fully local: Ollama + whisper.cpp. Zero network calls at runtime (embedding model pre-cached at install). |

See [Privacy Threat Model](https://github.com/anthillnet/sinain-hud/blob/main/docs/privacy-protection-design.md) for the full design.

## Hotkeys

Global hotkeys use **Cmd+Shift**:

| Shortcut | Action |
|---|---|
| `Cmd+Shift+Space` | Toggle overlay visibility |
| `Cmd+Shift+M` | Cycle display mode |
| `Cmd+Shift+/` | Open command input |
| `Cmd+Shift+H` | Quit overlay |

See [docs/HOTKEYS.md](https://github.com/anthillnet/sinain-hud/blob/main/docs/HOTKEYS.md) for all 15 shortcuts.

## Running Fully Local

No cloud APIs needed. Local models handle everything:

```bash
# 1. Install local transcription
./setup-local-stt.sh

# 2. Install Ollama + vision model
brew install ollama && ollama pull llava

# 3. Start in local mode
./start-local.sh
```

| Model | Size | Speed | Best for |
|---|---|---|---|
| `llava` | 4.7 GB | ~2s/frame | General use (recommended) |
| `llama3.2-vision` | 7.9 GB | ~4s/frame | Best accuracy |
| `moondream` | 1.7 GB | ~1s/frame | Fastest, lower quality |

## Setup Guides

| Setup | Guide |
|---|---|
| **Agent Roster & Profiles** | [docs/AGENT-ROSTER.md](https://github.com/anthillnet/sinain-hud/blob/main/docs/AGENT-ROSTER.md) — pick agents, add custom profiles |
| **Bare Agent** | [docs/INSTALL-BARE-AGENT.md](https://github.com/anthillnet/sinain-hud/blob/main/docs/INSTALL-BARE-AGENT.md) — the default install path |
| From Source | `git clone`, `cp .env.example ~/.sinain/.env`, `./start.sh` |

## Knowledge System

Sinain builds a persistent knowledge graph from everything it captures — audio transcriptions, screen OCR, and agent interactions. Facts are distilled incrementally (on buffer full and session end), stored in an EAV triplestore with graph relationships, and retrieved via hybrid search (FTS5 + tag-based + entity graph backrefs with RRF fusion).

The integration step is fully deterministic — no LLM decides what to store. Every extracted fact is preserved.

```bash
npx @geravant/sinain export-knowledge   # export playbook, modules, graph
npx @geravant/sinain import-knowledge ~/sinain-knowledge-export.tar.gz
```

See [Knowledge System docs](https://github.com/anthillnet/sinain-hud/blob/main/docs/knowledge-system.md) for architecture details.

### Querying knowledge from any MCP agent

Sinain's knowledge graph is exposed to any MCP-aware agent via the bundled MCP server. See **[Connect Your Coding Agent (MCP)](#connect-your-coding-agent-mcp)** below for setup.

## Connect Your Coding Agent (MCP)

Sinain ships an MCP server that exposes 15 `sinain_*` tools — including `sinain_knowledge_query`, `sinain_get_knowledge`, `sinain_distill_session`, `sinain_get_context`, and `sinain_respond` — to any MCP-aware agent. Register it once and the agent can read your knowledge graph and surface text on the HUD from any project.

```bash
npx @geravant/sinain@latest mcp install
```

The wizard detects which MCP agents you have installed and registers sinain for the ones you select. Re-runnable any time; idempotent.

| Agent | Setup | Config it touches |
|---|---|---|
| **Claude Code** | `mcp install` (auto via wizard) | `~/.claude.json` (`claude mcp add`) |
| **Claude Desktop** | `mcp install` (auto via wizard) | `~/Library/Application Support/Claude/claude_desktop_config.json` (mac) |
| **Cursor** | `mcp install` (auto via wizard) | `~/.cursor/mcp.json` |
| **Codex** | `mcp install` (auto via wizard) | `~/.codex/config.toml` (`codex mcp add`) |
| **Goose** | `mcp install` (auto via wizard) | `~/.config/goose/config.yaml` |
| **Junie** | `mcp install` (auto via wizard) | `~/.junie/mcp/mcp.json` |

> **Already in `sinain onboard`** — step 6 of the advanced flow runs the same registration. Quickstart asks once if any MCP agent is detected.

- See [docs/MCP-INTEGRATION.md](https://github.com/anthillnet/sinain-hud/blob/main/docs/MCP-INTEGRATION.md) for setup details, troubleshooting, and the manual `pclaude` / alternate `CLAUDE_CONFIG_DIR` recipe.
- See [docs/MCP-CAPABILITIES.md](https://github.com/anthillnet/sinain-hud/blob/main/docs/MCP-CAPABILITIES.md) for what each tool enables, with example prompts and end-to-end recipes.

## Deep Dives

| Topic | Doc |
|---|---|
| Knowledge System | [docs/knowledge-system.md](https://github.com/anthillnet/sinain-hud/blob/main/docs/knowledge-system.md) |
| Knowledge API (HTTP) | [docs/KNOWLEDGE-API.md](https://github.com/anthillnet/sinain-hud/blob/main/docs/KNOWLEDGE-API.md) |
| MCP Integration (setup) | [docs/MCP-INTEGRATION.md](https://github.com/anthillnet/sinain-hud/blob/main/docs/MCP-INTEGRATION.md) |
| MCP Capabilities (tools + recipes) | [docs/MCP-CAPABILITIES.md](https://github.com/anthillnet/sinain-hud/blob/main/docs/MCP-CAPABILITIES.md) |
| LongMemEval Audit | [docs/LONGMEMEVAL-AUDIT.md](https://github.com/anthillnet/sinain-hud/blob/main/docs/LONGMEMEVAL-AUDIT.md) |
| Privacy Threat Model | [docs/privacy-protection-design.md](https://github.com/anthillnet/sinain-hud/blob/main/docs/privacy-protection-design.md) |
| Full Configuration | [docs/CONFIGURATION.md](https://github.com/anthillnet/sinain-hud/blob/main/docs/CONFIGURATION.md) |
| All Hotkeys | [docs/HOTKEYS.md](https://github.com/anthillnet/sinain-hud/blob/main/docs/HOTKEYS.md) |

## Source

Full source, issues, and contributing guide: **[github.com/anthillnet/sinain-hud](https://github.com/anthillnet/sinain-hud)**

## Contributing

See [CONTRIBUTING.md](https://github.com/anthillnet/sinain-hud/blob/main/CONTRIBUTING.md).

## License

MIT
