# cartog-indexer

Code indexing and change detection for cartog.

## Overview

Walks a directory tree, detects which files changed, extracts symbols and edges via `cartog-languages`, and writes the results to `cartog-db`. Indexes both code files and Markdown documents (`.md`). Uses a multi-tier change detection strategy and Merkle tree hashing for surgical symbol-level updates.

> Architecture overview: [docs/architecture/incremental-indexing.md](../../docs/architecture/incremental-indexing.md). This README focuses on the crate's public API and module layout.

## How it works

### Change detection (3 tiers)

1. **Git-based** — `git diff last_commit..HEAD` identifies changed files; unchanged files are skipped without reading them from disk
2. **SHA-256 fallback** — for non-git repos or when git detection fails, each file is hashed and compared to the stored hash
3. **Force mode** — `force=true` bypasses all checks and re-indexes everything

### Merkle tree hashing

Each symbol gets two hashes for fine-grained diff detection:

```text
content_hash = SHA256(kind + ":" + name + ":" + signature + ":" + body_source)
subtree_hash = SHA256(content_hash + sorted(children_subtree_hashes))
```

Computed bottom-up (post-order traversal). When re-indexing a file, symbols are classified as:

- **added** — new symbol ID not in previous index
- **modified** — `content_hash` changed (body or signature edited)
- **children_changed** — `content_hash` same but `subtree_hash` differs (child added/removed/modified)
- **unchanged** — both hashes match, skip entirely
- **removed** — symbol ID in old index but not in new extraction

Edges are always fully re-inserted for dirty files (no edge-level diff).

### LSP resolution (optional)

When the `lsp` feature is enabled, a second pass resolves edges that the heuristic resolver in `cartog-db` left unresolved, using real language servers via `cartog-lsp`. Skipped on no-op reindexes (no file added, modified, or removed) — the unresolved set is identical to the previous run. Use `--force` to retry resolution after toggling `--no-lsp` off.

Edges that LSP classifies are persisted to `resolution_state` so future runs skip them: `2` (unresolvable — typo, dyn dispatch, macro expansion) and `3` (external — stdlib, deps, node_modules). A no-lsp run (`--no-lsp`, `watch`) has no LSP pass to retry the heuristic's leftovers, so the indexer seals them at `4` (heuristic-exhausted) — without this every incremental re-index re-walks the whole permanent-failure backlog. When a new symbol is added whose name matches a sealed edge (e.g. the user vendors a dep in-tree), the indexer auto-resets the marker so the next pass retries — see `Database::reset_unresolvable_for_names`. A later LSP-enabled reindex reopens state `4` first (`Database::reopen_heuristic_exhausted`) so the language server gets a shot. `--force` resets all sticky markers (see `Database::reset_all_unresolvable`).

## Public API

| Export | Description |
|--------|-------------|
| `index_directory()` | Main entry point — index a directory into the database. Takes `db`, `root`, `force`, `lsp`, `progress: Option<ProgressCallback>` (fires at coarse phase boundaries: `Walking`, `Parsing`, `Storing`, `ResolvingLsp`), `cancel: Option<CancelProbe>` (cooperative abort), `redact: RedactionConfig` (secret scrubbing), `lsp_overrides: &HashMap<String, Vec<String>>` (per-language `[lsp.<lang>]` command overrides; empty = PATH-resolved servers), and `filter: &WalkFilter` (`.gitignore`/`.cartogignore` + `[index] exclude` globs; `unrestricted()` = gitignore honored, no extra excludes). Pass `None` for the no-op progress/cancel defaults. |
| `IndexResult` | Summary: files indexed/skipped/removed, symbols added/modified/unchanged/removed, edges resolved (heuristic + `edges_lsp_resolved`, `edges_marked_unresolvable`, `edges_marked_external`), `dirty_files`, `files_unsupported` + `unsupported_by_ext` (files whose language isn't supported; cartog's own `.cartog.db*` / `db.sqlite*` sidecars are excluded), plus `files_redacted_skipped` (sensitive files never read/parsed) and `redaction_backfilled` (set when a first full re-index scrubs pre-redaction content) |
| `render_index_summary()` | Render a human-readable one-block summary of an `IndexResult`; shared by `cartog index` CLI output and the `cartog_index` MCP tool |
| `RedactionConfig` | Secret-redaction policy passed to `index_directory()` (default-on; `disabled()` for a verbatim no-op) |
| `WalkFilter` | Walk policy passed to `index_directory()`: `{ exclude: ExcludeGlobs, respect_gitignore: bool }`. `unrestricted()`/`default()` = no excludes with `.gitignore` honored. The walker honors `.gitignore`/`.cartogignore` (via the `ignore` crate) unless `respect_gitignore` is false; the hardcoded floor + `exclude` prune on top. |
| `ExcludeGlobs` | Compiled `[index] exclude` path globs inside a `WalkFilter` (`empty()` for none; `from_globs()` errors on a malformed/empty glob). `literal_separator` (`*` is segment-local); matches files and prunes matched directories. |
| `ProgressUpdate` | `pub enum` of in-flight phases (`Walking`, `Parsing { total }`, `Storing { total }`, `ResolvingLsp`) |
| `ProgressCallback` / `CancelProbe` | Type aliases (`Fn` trait-object references) for phase reporting and cooperative cancellation (transport-agnostic) |
| `is_ignored_dirname()` | Check if a directory name should be skipped (`.git`, `node_modules`, `target`, etc.) |
| `git_recently_changed_files()` | List files changed in the last N git commits |

## Crate dependencies

`cartog-core`, `cartog-db`, `cartog-languages`, optionally `cartog-lsp`
