# Tokensave User Guide

Thanks for downloading Tokensave!

Tokensave is a code intelligence tool that builds a semantic knowledge graph of your codebase. It gives AI coding agents (like Claude Code) instant, structured access to your code's symbols, relationships, and dependencies — so they spend fewer tokens scanning files and more time writing code.

Everything runs locally. Your code never leaves your machine.

---

## Table of Contents

1. [Installing Tokensave](#installing-tokensave)
2. [Your First Index](#your-first-index)
3. [Connecting to Your Agent](#connecting-to-your-agent)
4. [Exploring Your Codebase from the CLI](#exploring-your-codebase-from-the-cli)
5. [Keeping the Index Fresh](#keeping-the-index-fresh)
6. [The Background Daemon](#the-background-daemon)
7. [Checking Your Setup with Doctor](#checking-your-setup-with-doctor)
8. [Finding Affected Tests](#finding-affected-tests)
9. [MCP Tools for AI Agents](#mcp-tools-for-ai-agents)
10. [Supported Languages](#supported-languages)
11. [Privacy and Network](#privacy-and-network)
12. [Updating Tokensave](#updating-tokensave)
13. [Configuration Files](#configuration-files)
14. [Troubleshooting](#troubleshooting)

---

## Installing Tokensave

Pick whichever method suits your platform.

**Homebrew (macOS):**

```bash
brew install aovestdipaperino/tap/tokensave
```

**Scoop (Windows):**

```powershell
scoop bucket add tokensave https://github.com/aovestdipaperino/scoop-bucket
scoop install tokensave
```

**Cargo (any platform):**

```bash
cargo install tokensave
```

If you only work with a subset of languages, you can install a smaller binary:

```bash
cargo install tokensave --features medium        # 20 languages
cargo install tokensave --no-default-features    # 11 languages (lite)
```

**Prebuilt binaries:**

Download from the [latest release](https://github.com/aovestdipaperino/tokensave/releases/latest) and place the binary somewhere on your `PATH`. Archives are available for macOS (Apple Silicon), Linux (x86_64 and ARM64), and Windows (x86_64).

---

## Your First Index

Navigate to any project directory and run:

```bash
cd /path/to/your/project
tokensave init
```

Tokensave will scan every supported source file, extract symbols (functions, classes, methods, imports, type relationships, complexity metrics), and store everything in a local database at `.tokensave/tokensave.db`. You'll see a spinner with file-by-file progress and an ETA.

Once it finishes, run `tokensave status` to see what was indexed:

```bash
tokensave status
```

This prints an overview of your project: the number of files, symbols, edges (relationships between symbols), language distribution, and how many tokens the index has saved you so far. If you just want the summary line without the ASCII art, pass `--short`:

```bash
tokensave status --short
```

For machine-readable output, use `--json`.

### Why `init` and `sync` are separate

Initialization (`tokensave init`) and incremental updates (`tokensave sync`) are deliberately separate commands.

Tokensave installs a global git post-commit hook that runs `tokensave sync` after every commit to keep the index fresh. If `sync` were allowed to create a new database when none existed, it would silently bootstrap a `.tokensave/` directory in every git repository on your machine -- even ones you never intended to index. By requiring an explicit `init`, only projects you opt into get a database. The hook runs harmlessly (exits with a non-zero status, output suppressed) in all other repos.

In short:
- **`tokensave init`** -- one-time setup. Creates the database and performs a full index. Errors if already initialized.
- **`tokensave sync`** -- ongoing updates. Requires an existing database. Errors if the project was never initialized.

### Incremental syncs

After the initial full index, every subsequent `tokensave sync` is incremental. It detects which files changed since the last sync (via content hashing) and only re-indexes those files. On a typical commit-sized change, this takes under a second.

### Force re-index

If you ever need to rebuild the entire index from scratch (for example, after a major Tokensave upgrade), pass `--force`:

```bash
tokensave sync --force
```

### Skipping folders

If there are directories you never want indexed (vendored code, generated output, etc.), pass `--skip-folder`:

```bash
tokensave sync --skip-folder vendor --skip-folder generated
```

### Seeing what changed

The `--doctor` flag lists every file that was added, modified, or removed during the sync, so you can verify exactly what the index updated:

```bash
tokensave sync --doctor
```

### Diagnosing slow syncs

If a sync appears stuck or is taking longer than expected, add `--verbose` (`-v`) to see per-phase diagnostics with file counts and timings:

```bash
tokensave sync --verbose
```

Example output:

```
  [verbose] scanned 10432 files in 2.3s
  [verbose] stat-checked 10432 files in 0.1s
  [verbose] changes: 3 new, 847 stat-changed, 0 removed, 9582 unchanged
  [verbose] hashed 850 files in 1.2s (0 read errors)
  [verbose] content check: 12 modified, 838 mtime-only
  [verbose] indexed 15 files (204 nodes, 189 edges) in 0.3s
  [verbose] resolved 39841 references in 0.5s
✔ sync done — 3 added, 12 modified, 0 removed in 4412ms
```

This also works with `--force` for full re-index diagnostics.

### Respecting .gitignore

By default, tokensave respects your `.gitignore` rules and skips ignored files during indexing. You can check the current setting or toggle it:

```bash
tokensave gitignore              # show current setting
tokensave gitignore on           # enable (default)
tokensave gitignore off          # disable — index everything
```

Don't forget to add `.tokensave` to your `.gitignore` so the database doesn't get committed:

```bash
echo .tokensave >> .gitignore
```

---

## Connecting to Your Agent

Tokensave works as an MCP (Model Context Protocol) server. AI coding agents connect to it to query your codebase instead of scanning files directly. The `install` command sets everything up automatically.

### Claude Code

```bash
tokensave install
```

This is the default. It registers the MCP server in `~/.claude/settings.json`, grants tool permissions so Claude doesn't have to ask you every time, installs a `PreToolUse` hook that redirects Claude away from spawning expensive Explore agents, and adds prompt rules to `~/.claude/CLAUDE.md` that tell Claude to prefer tokensave tools.

### Other agents

Tokensave supports fourteen agents. Pass `--agent` to install for a specific one:

```bash
tokensave install --agent claude      # Claude Code (default)
tokensave install --agent opencode    # OpenCode
tokensave install --agent codex       # OpenAI Codex CLI
tokensave install --agent gemini      # Gemini CLI
tokensave install --agent copilot     # GitHub Copilot CLI
tokensave install --agent cursor      # Cursor
tokensave install --agent zed         # Zed
tokensave install --agent cline       # Cline
tokensave install --agent roo-code    # Roo Code
tokensave install --agent antigravity # Antigravity (Windsurf)
tokensave install --agent kilo        # Kilo CLI
tokensave install --agent kiro        # AWS Kiro
tokensave install --agent kimi        # Moonshot Kimi CLI
tokensave install --agent vibe        # Mistral Vibe
```

Each agent gets an appropriate configuration: MCP server registration, tool permissions (where the agent supports them), and prompt rules in the agent's instruction file.

Kiro setup registers tokensave in `~/.kiro/settings/mcp.json`, adds steering in
`~/.kiro/steering/AGENTS.md`, and writes a tokensave-managed agent with hooks
that block research delegation until tokensave MCP tools have been tried and
sync the index after Kiro writes files. If you already have a different custom
default agent or a user-managed `tokensave` agent, tokensave leaves it alone and
prints a warning.

The install is idempotent — safe to run again after upgrading tokensave. You'll also be offered the option to set up a global git post-commit hook and the background daemon (more on those below).

#### Config backups

Whenever tokensave rewrites an agent config file — on `install`, on `uninstall`, or when the `doctor` auto-repairs hooks — it first copies the original to a sibling `.bak` file in the same directory. For example:

- `~/.codex/config.toml` → `~/.codex/config.toml.bak`
- `~/.cursor/mcp.json` → `~/.cursor/mcp.json.bak`
- `~/.claude.json` → `~/.claude.json.bak`

If anything goes wrong (a typo, an unexpected rewrite, an unknown bug), restore with `cp <path>.bak <path>`. The `.bak` is always the **exact bytes** of whatever was on disk just before the write; tokensave never deletes or rotates it, so the most recent backup is the file you want.

### Removing an integration

```bash
tokensave uninstall                   # remove Claude Code integration
tokensave uninstall --agent codex     # remove Codex integration
```

---

## Exploring Your Codebase from the CLI

You don't need an AI agent to use tokensave. The CLI has several commands for direct exploration.

### Searching for symbols

```bash
tokensave query "authenticate"
```

This searches the full-text index for symbols matching your query. It returns function names, class names, method names, and their file locations and signatures. Limit results with `-l`:

```bash
tokensave query "authenticate" -l 5
```

### Building task context

```bash
tokensave context "implement user authentication"
```

This is the same context builder that AI agents use. Given a natural language task description, it finds the most relevant entry points, related symbols, and code structure. Output defaults to Markdown; use `--format json` for structured output.

```bash
tokensave context "implement user authentication" --format json -n 30
```

The `-n` flag controls how many symbols are included (default: 20).

### Listing indexed files

```bash
tokensave files                           # all files
tokensave files --filter src/mcp          # only files under src/mcp/
tokensave files --pattern "**/*.rs"       # only Rust files
tokensave files --json                    # machine-readable output
```

### Running the MCP server directly

```bash
tokensave serve
```

This starts the MCP server over stdio. You normally don't need to run this yourself — the agent integration handles it. But it's useful for debugging or connecting custom tools.

### Working from a subdirectory

You can open your AI agent from any subdirectory of an indexed project. Tokensave will walk up the directory tree to find the nearest `.tokensave/` database — similar to how git finds `.git/`.

When the MCP server starts from a subdirectory, listing tools like `tokensave_files`, `tokensave_search`, and `tokensave_context` automatically scope their results to that subdirectory. This is useful in monorepos or large projects where you want to focus on one area.

Graph traversal tools (`tokensave_callers`, `tokensave_callees`, `tokensave_impact`, etc.) remain unscoped so you can still follow connections across directory boundaries.

You can always override the automatic scope by passing an explicit `path` parameter to any tool. `tokensave_status` shows the active scope prefix when one is in effect.

---

## Keeping the Index Fresh

Tokensave gives you three ways to keep the index up to date.

### Manual sync

Run `tokensave sync` whenever you want. It's incremental and fast.

### Post-commit hook

During `tokensave install`, you'll be offered a global git `post-commit` hook. If you accept, tokensave will automatically sync in the background after every git commit across all your repos. The hook is a no-op in repos that don't have a `.tokensave/` directory.

You can also set it up manually:

**Global (all repos):**

```bash
git config --global core.hooksPath ~/.git-hooks
mkdir -p ~/.git-hooks
cp scripts/post-commit ~/.git-hooks/post-commit
chmod +x ~/.git-hooks/post-commit
```

**Per-repo:**

```bash
cp scripts/post-commit .git/hooks/post-commit
chmod +x .git/hooks/post-commit
```

### Background daemon

The daemon watches all your tracked projects for file changes and syncs automatically. See the next section.

---

## The Background Daemon

The background daemon monitors all projects you've indexed and runs incremental syncs whenever files change on disk. This means your index is always up to date, even between commits.

### Starting the daemon

```bash
tokensave daemon
```

This forks into the background. To run it in the foreground (useful for debugging), pass `--foreground`:

```bash
tokensave daemon --foreground
```

### Checking daemon status

```bash
tokensave daemon --status
```

### Stopping the daemon

```bash
tokensave daemon --stop
```

### Autostart on boot

You can register the daemon as a system service so it starts automatically when you log in:

```bash
tokensave daemon --enable-autostart    # install launchd (macOS) / systemd (Linux) / Windows Service
tokensave daemon --disable-autostart   # remove the autostart service
```

### Upgrade-aware restarts

When you upgrade tokensave (via `brew upgrade`, `cargo install`, `scoop update`, or any other method), the running daemon detects that its binary has been replaced and automatically restarts with the new version. You don't need to manually stop and start it.

---

## Checking Your Setup with Doctor

The `doctor` command runs a comprehensive health check:

```bash
tokensave doctor
```

It verifies:

- **Binary** — location and version
- **Current project** — whether a `.tokensave/` index exists and the database is healthy
- **Global database** — the cross-project database at `~/.tokensave/global.db`
- **User config** — `~/.tokensave/config.toml` and upload settings
- **Agent integrations** — MCP server registration, hook installation, tool permissions, prompt rules
- **Network** — connectivity to the worldwide counter and GitHub releases API

If any tool permissions are missing after an upgrade, doctor will tell you to run `tokensave install` again.

To check only a specific agent:

```bash
tokensave doctor --agent claude
tokensave doctor --agent codex
tokensave doctor --agent kiro
```

The accepted agent values are the same values supported by `tokensave install --agent`.

---

## Finding Affected Tests

When you change source files, you often want to know which tests might be affected. The `affected` command traces through the file dependency graph to find them.

```bash
tokensave affected src/main.rs src/db/connection.rs
```

This performs a breadth-first search from the changed files through import/dependency edges to find test files that directly or transitively depend on those files.

### Piping from git

This is especially useful in CI pipelines:

```bash
git diff --name-only HEAD~1 | tokensave affected --stdin
```

### Options

```bash
tokensave affected src/lib.rs --depth 3         # limit traversal depth (default: 5)
tokensave affected src/lib.rs --filter "*_test.rs"  # custom test file pattern
tokensave affected src/lib.rs --json             # JSON output
tokensave affected src/lib.rs --quiet            # just file paths, no decoration
```

---

## MCP Tools for AI Agents

When running as an MCP server, tokensave exposes 41 tools that AI agents can call. Here's what they do, grouped by purpose.

### Core exploration

| Tool | What it does |
|------|-------------|
| `tokensave_context` | Given a task description, returns relevant symbols, relationships, and code snippets. This is the go-to starting point for any coding task. |
| `tokensave_search` | Find symbols by name. Supports filtering by kind (function, class, method, etc.). |
| `tokensave_node` | Get full details for a specific symbol: source code, location, complexity metrics, and relationships. |
| `tokensave_files` | List indexed files, optionally filtered by directory or glob pattern. |
| `tokensave_status` | Index statistics: file counts, symbol counts, language distribution, and tokens saved. |

### Navigating relationships

| Tool | What it does |
|------|-------------|
| `tokensave_callers` | Find what calls a given function or method. Configurable traversal depth. |
| `tokensave_callees` | Find what a function or method calls. |
| `tokensave_impact` | Trace the full blast radius of changing a symbol — everything that could be affected. |
| `tokensave_affected` | Find test files affected by source file changes. |
| `tokensave_similar` | Find symbols with similar names (useful for naming patterns or related code). |
| `tokensave_rename_preview` | Preview all references to a symbol before renaming it. |

### Code quality analysis

| Tool | What it does |
|------|-------------|
| `tokensave_dead_code` | Find unreachable symbols — functions with no callers. |
| `tokensave_unused_imports` | Find import statements that are never referenced. |
| `tokensave_circular` | Detect circular file dependencies. |
| `tokensave_recursion` | Detect recursive and mutually-recursive call cycles. |
| `tokensave_complexity` | Rank functions by composite complexity score, including cyclomatic complexity from the AST. |
| `tokensave_god_class` | Find classes with the most members — candidates for decomposition. |
| `tokensave_hotspots` | Find the most connected symbols (highest call count). These are high-risk areas. |
| `tokensave_doc_coverage` | Find public symbols missing documentation. |
| `tokensave_simplify_scan` | Quality analysis of changed files: duplications, dead code, complexity, coupling. |

### Health & quality signals

| Tool | What it does |
|------|-------------|
| `tokensave_health` | Composite quality signal (0–10000) from five structural dimensions (acyclicity, depth, equality, redundancy, modularity) with a low-weight penalty for `/// skip-test-coverage` overuse. The single number to track over time. |
| `tokensave_gini` | Gini inequality coefficient for any metric (complexity, lines, fan-in, fan-out, members). Finds god files and uneven distributions. |
| `tokensave_dependency_depth` | Longest file-level dependency chains — the critical paths where upstream changes ripple through the most layers. |
| `tokensave_dsm` | Design Structure Matrix showing file dependencies as clusters, density stats, or an NxN grid. Reveals hidden coupling patterns. |
| `tokensave_test_risk` | Risk-weighted test gaps combining complexity, coupling, git churn, and test coverage. Answers "where should the next test go?" |

### Test Coverage Conventions

#### `/// skip-test-coverage`

Mark functions that are genuinely untestable in unit tests (e.g. infrastructure-dependent, framework-invoked, or private helpers tested only transitively):

```rust
/// skip-test-coverage
pub async fn produce(&mut self, topic: &str, batch: Bytes) -> io::Result<i64> { ... }
```

Marked functions are excluded from `tokensave_test_risk` coverage calculations, giving you an accurate picture of testable-code coverage. The `skipped` count appears in the summary so you can track how many functions use the annotation.

**Health penalty:** The `coverage_discipline` dimension (visible in `tokensave_health` and `tokensave_session_start`/`session_end`) penalises overuse. Each skipped function lowers the score proportionally — a few genuine exclusions have negligible impact, but marking 50%+ of your codebase as untestable will visibly reduce your quality signal. This encourages using the annotation for its intended purpose rather than as a way to game coverage numbers.

### Structural analysis

| Tool | What it does |
|------|-------------|
| `tokensave_module_api` | Public API surface of a file or directory. |
| `tokensave_coupling` | Rank files by coupling (fan-in or fan-out). |
| `tokensave_inheritance_depth` | Find the deepest class inheritance hierarchies. |
| `tokensave_type_hierarchy` | Recursive type hierarchy tree for traits, interfaces, and classes. |
| `tokensave_distribution` | Node kind breakdown (classes, methods, fields) per file or directory. |
| `tokensave_rank` | Rank nodes by relationship count (most-implemented interface, most-extended class, etc.). |
| `tokensave_largest` | Rank nodes by size — largest classes, longest methods. |

### Git-aware tools

| Tool | What it does |
|------|-------------|
| `tokensave_diff_context` | Semantic context for changed files: modified symbols, dependencies, and affected tests. |
| `tokensave_changelog` | Semantic diff between two git refs — which symbols were added, removed, or modified. |
| `tokensave_commit_context` | Semantic summary of uncommitted changes, useful for drafting commit messages. |
| `tokensave_pr_context` | Semantic diff between git refs for pull request descriptions. |
| `tokensave_test_map` | Source-to-test mapping at the symbol level, with uncovered symbol detection. |

### Porting tools

| Tool | What it does |
|------|-------------|
| `tokensave_port_status` | Compare symbols between source/target directories to track cross-language porting progress. |
| `tokensave_port_order` | Topological sort of symbols for porting — tells you what to port first based on dependencies. |

### Session management

| Tool | What it does |
|------|-------------|
| `tokensave_session_start` | Save current health metrics as a baseline before starting work. |
| `tokensave_session_end` | Compare current health against the baseline to detect structural degradation during the session. |

Discovery and analysis tools are read-only and safe to call in parallel. Session baseline tools write/remove `.tokensave/session_baseline.json`, memory-recording tools update the project database, and edit tools modify source files.

---

## Supported Languages

Tokensave supports 31 languages, organized into three tiers. Each tier includes all the languages from the tier below it.

### Lite (11 languages)

Always compiled. The smallest binary for the most popular languages.

Rust, Go, Java, Scala, TypeScript, JavaScript, Python, C, C++, Kotlin, C#, Swift

### Medium (Lite + 9 = 20 languages)

Adds scripting, config, and additional systems languages.

Dart, Pascal, PHP, Ruby, Bash, Protobuf, PowerShell, Nix, VB.NET

### Full (Medium + 11 = 31 languages)

Everything, including legacy and niche languages.

Lua, Zig, Objective-C, Perl, Batch/CMD, Fortran, COBOL, MS BASIC 2.0, GW-BASIC, QBasic, QuickBASIC 4.5

### Mixing individual languages

You can also cherry-pick individual languages without taking a full tier:

```bash
cargo install tokensave --no-default-features --features lang-nix,lang-bash
```

### What gets extracted

For each supported language, tokensave extracts:

- Function and method definitions (with signatures)
- Class, struct, trait, interface, and enum definitions
- Fields and properties
- Import and export statements
- Call relationships and type references
- Docstrings and annotations
- Complexity metrics (branches, loops, returns, max nesting, cyclomatic complexity)
- Cross-file dependency edges

---

## Privacy and Network

Tokensave's core functionality is 100% local. Indexing, search, graph queries, and the MCP server all run on your machine against a local database. No API keys are needed.

There are two optional network calls.

### Worldwide token counter

Tokensave tracks how many tokens it has saved you. During `sync` and `status`, it uploads that count (a single number like `4823`) to an anonymous worldwide counter. No code, file names, project names, or identifying information is sent. The Cloudflare Worker also logs the country derived from your IP for aggregate geographic statistics — your actual IP is not stored.

This powers the "Worldwide" counter shown in `tokensave status`.

**To opt out:**

```bash
tokensave disable-upload-counter
```

When disabled, tokensave never uploads your count but still fetches and displays the worldwide total. Re-enable at any time:

```bash
tokensave enable-upload-counter
```

### Version check

Tokensave checks GitHub for new releases so it can show you an upgrade notice. This is a single GET request to the GitHub API with no identifying information. It has a 1-second timeout and failures are silently ignored. This check cannot be disabled, but it never blocks your workflow.

---

## Updating Tokensave

When a new version is available, tokensave tells you during `sync` and `status`:

```
Update available: v3.3.3 -> v3.4.0
  Run: tokensave upgrade
```

The `upgrade` command downloads the latest release from GitHub and replaces the binary in place:

```bash
tokensave upgrade
```

It automatically stops the background daemon before replacing the binary and restarts it afterwards if it was running. Beta and stable are separate update channels — a beta build only sees beta releases and vice versa.

You can also update through your package manager:

```bash
brew upgrade tokensave          # Homebrew
scoop update tokensave          # Scoop
cargo install tokensave         # Cargo
```

After upgrading, it's good practice to re-run install (to pick up any new tool permissions or prompt rules) and force a re-index:

```bash
tokensave install
tokensave sync --force
```

---

## Configuration Files

Tokensave stores data in two places.

### Per-project: `.tokensave/`

Created inside each project you index. Contains:

- `tokensave.db` — the libSQL database with all symbols, edges, files, and vector embeddings

Add `.tokensave` to your `.gitignore`.

### Per-user: `~/.tokensave/`

Created in your home directory. Contains:

- `config.toml` — user preferences (upload opt-in/out, cached version info, pending upload count)
- `global.db` — cross-project database that tracks tokens saved across all your projects

The `config.toml` is plain TOML and fully transparent:

```toml
upload_enabled = true       # set to false to stop uploading
pending_upload = 4823       # tokens waiting to be uploaded
last_upload_at = 1711375200 # last successful upload timestamp
last_worldwide_total = 1000000
last_worldwide_fetch_at = 1711375200
```

---

## Troubleshooting

### "tokensave not initialized"

The `.tokensave/` directory doesn't exist in your current project. Run:

```bash
tokensave init
```

### MCP server not connecting

Your AI agent doesn't see tokensave tools.

1. Run `tokensave doctor` to check the integration
2. Verify `tokensave` is on your PATH: `which tokensave`
3. Re-run `tokensave install` and restart your agent completely

### Missing symbols in search

Some symbols aren't showing up.

- Run `tokensave sync` to update the index
- Check that the language is supported (see the tiers above)
- Verify the file isn't being skipped by `.gitignore` (`tokensave gitignore` to check)

### Indexing is slow on first run

The initial full index of a large project can take a few seconds. This is normal. Use `tokensave sync --verbose` to see which phase is taking the longest.

- Subsequent syncs are incremental and much faster
- Use `tokensave sync` (not `--force`) for day-to-day updates
- The post-commit hook and daemon run in the background so they never block you

### Stale install warning

If you see a warning about your install being stale after an upgrade, run:

```bash
tokensave install
```

This updates tool permissions, hooks, and prompt rules to match the new version.

### Getting help

If you run into something not covered here, check the [GitHub repository](https://github.com/aovestdipaperino/tokensave) or open an issue.
