快捷混合代码搜索

Embedded-corpus build pattern (v0.6.0)

The library form of ken-mcp lets SDK authors ship docs as a single static MCP server binary. Write ~20 lines of main.go, //go:embed your docs/ and the Model2Vec model, go build — push a binary to a GitHub release. Users brew install, add one line to their agent config, and their coding agent has high-quality local retrieval over your SDK's docs. No backend, no vector DB to operate, no network egress per query, no "is the cache stale" question — the binary IS the corpus, version-pinned by build artifact.

package main

import (
    "context"
    "embed"
    "io/fs"
    "log"
    "os"

    "github.com/townsendmerino/ken/mcp"

    _ "github.com/townsendmerino/ken/internal/chunk/markdown"
)

//go:embed docs/*.md
var docsFS embed.FS

//go:embed model/tokenizer.json model/config.json model/model.safetensors
var modelFS embed.FS

func main() {
    docsSub, _  := fs.Sub(docsFS, "docs")
    modelSub, _ := fs.Sub(modelFS, "model")
    if err := mcp.Run(context.Background(), docsSub, mcp.Options{
        Mode:        "hybrid",
        ChunkerName: "markdown",
        ModelFS:     modelSub,
        LogWriter:   os.Stderr,
    }); err != nil {
        log.Fatal(err)
    }
}

cmd/ken-mcp-docs/ is the canonical worked example — it bakes ken's own docs/*.md and the Model2Vec model into a 74 MB static binary built via scripts/build-docs-mcp.sh. Design and rationale: ADR-016.

Install both binaries (Go 1.26+).

go install github.com/townsendmerino/ken/cmd/ken@latest go install github.com/townsendmerino/ken/cmd/ken-mcp@latest

Install in your agent

```bash

Quickstart

```bash

Environment

Embedded DB support for SDK authors (v0.8.0 Part 3, opt-in)

SDK authors using mcp.Run (the v0.6.0 embedded-corpus entrypoint) can wire Tier 2 DB support — schema introspection, optional LISTEN/NOTIFY, optional interval reindex, and the reindex_db MCP tool — via the new opt-in mcp/db package:

package main

import (
    "context"
    "log"
    "os"
    "time"

    "github.com/townsendmerino/ken/mcp"
    mcpdb "github.com/townsendmerino/ken/mcp/db"
)

func main() {
    ctx := context.Background()

    // Opt-in: only SDK authors who want DB support import mcp/db.
    refresher, err := mcpdb.Setup(ctx, mcpdb.Config{
        DSN:             os.Getenv("MY_DB_DSN"),
        SampleRows:      0,
        ReindexInterval: 5 * time.Minute,
        EnableListen:    true, // requires one-time `mcpdb.ListenNotifyScript | psql $DSN` setup
    })
    if err != nil {
        log.Fatal(err)
    }

    // refresher is nil when MY_DB_DSN is unset → opts.DB stays nil →
    // reindex_db tool NOT registered (the v0.6.0 docs-only behavior).
    // When non-nil, mcp.Run calls refresher.Start internally and
    // defers the returned cleanup.
    if err := mcp.Run(ctx, myEmbeddedDocsCorpus, mcp.Options{
        Mode:        "hybrid",
        ChunkerName: "markdown",
        DB:          refresher, // *mcpdb.Refresher satisfies mcp.DBIntegration
    }); err != nil {
        log.Fatal(err)
    }
}

v0.6.0 binary-size contract preserved. SDK authors who DON'T import mcp/db get a binary identical in dep-tree shape to v0.7.2's mcp.Run use case — no pgx, no SQLite, no MySQL driver, no internal/db in the link graph. The opt-in package boundary is enforced at CI time by TestBinary_MCPPackageStaysDBFree, which shells out to go list -deps github.com/townsendmerino/ken/mcp and fails if any DB driver path appears.

SDK authors who want print-listen-script in their own CLI can grab the embedded SQL script from mcpdb.ListenNotifyScript (a re-export of internal/db.ListenNotifyScript) without depending on the internal/ package:

if len(os.Args) > 1 && os.Args[1] == "print-listen-script" {
    _, _ = io.WriteString(os.Stdout, mcpdb.ListenNotifyScript)
    return
}

Chunk integration is end-to-end. Calling reindex_db from an agent against an mcp.Run + mcp/db.Setup binary runs the introspection AND makes the new DB chunks searchable in the agent's next search / find_related call. The pipeline: mcp.Run wraps the embedded *search.Index in atomic.Pointer[search.Index]; mcp/db.Refresher.Start (called by mcp.Run on startup) wires the swap callback to *search.Index.WithExtraChunks + atomic-pointer store; each refresh rebuilds against the original corpus + the latest DB chunks. cmd/ken-mcp continues to use *WatchedIndex.SetExtraChunks for its fsnotify-rooted path; the SDK-author + CLI surfaces converge on the same Refresher + reindex_db semantics. See ADR-020 Part 3 for the full design + the rejected alternatives.

Benchmarks — external reference (CoIR-CSN-Python)

A single externally-reproducible NDCG@10 number on CoIR's CodeSearchNet-python task, independent of semble's own benchmark — gives readers a comparable anchor against published code-IR baselines.

Result (v0.2.0, 1000-query subsample, regex chunker):

Reproduce:

python scripts/bench_coir.py                                # ~45 s download + 280k corpus files
KEN_COIR_QUERY_LIMIT=1000 go test -tags=bench ./bench/ndcg/ -run TestCoIR -v   # ~13 min

A nuance worth surfacing up front: on CSN-Python, BM25 beats hybrid by 0.09 — opposite of what semble's bench shows. CSN-Python's queries (as CoIR re-hosts the dataset) are full Python function sources, and the relevant document for each query is the docstring extracted from that same function. Because the docstring lives inside the function source as a literal substring (the function's own """...""" block), any lexical retriever with identifier-aware tokenization wins — BM25 has the answer string as input. ken's α=0.5 RRF fusion then drags the hybrid number down by averaging in the weaker semantic ranking. Not a ken bug; it's a structural artifact of how CoIR reframed CodeSearchNet for retrieval, and doesn't generalize to natural NL-to-code distributions. Detailed empirical findings and the comparison to potion-code-16M's published aggregate are in docs/BENCH.md.

Comparison to semble

† Measured at v0.1.0 / v0.2.0 against semble's published benchmark (63 repos, 1251 queries, semble's own benchmarks.metrics.ndcg_at_k + target_rank). Reproduce: see docs/BENCH.md. Ablation breakdown vs semble's published raw retrieval numbers: > > | Mode | semble (raw) | ken regex (default) | ken treesitter (opt-in) | > |---|---:|---:|---:| > | Semantic only (potion-code-16M) | 0.650 | 0.647 | — | > | BM25 only | 0.675 | 0.624 | 0.621 | > | Hybrid (full ranker) | 0.854 | 0.842 | 0.838 | > > The semantic-raw match within 0.003 isolates and validates the embedding + tokenizer + ANN port. The BM25 tokenizer was also re-aligned to a verbatim port of semble's tokens.py (snake-case compound preservation, ASCII-only identifier extraction, compound-first emission order). The v0.2.0 tree-sitter chunker (--chunker=treesitter via gotreesitter) trades NDCG per-language without net movement — clear wins on Kotlin / Zig / TypeScript / Java / PHP, losses on Python / Rust / C / Lua / Scala — so the default chunker stays regex and treesitter is opt-in. See "Choosing a chunker" for the per-language recommendation and docs/DECISIONS.md ADR-011 for the full rationale.

†† CoIR-CSN-Python numbers reported separately because they tell a different story than semble's bench: on CSN, BM25 beats hybrid by ~0.09 due to a substring-leak artifact in how CoIR reframes the CodeSearchNet dataset (queries are Python function sources; documents are docstrings extracted from those same functions, so the answer is a literal substring of the query). See the "Benchmarks — external reference" section and docs/BENCH.md for the corrected explanation. semble's bench is the verbatim-port confirmation; CoIR-CSN is the externally-reproducible anchor against published code-IR baselines but is read as a dataset-construction case study, not as evidence about ken's hybrid retrieval on natural NL-to-code queries.

††† Measured at v0.3.0 against semble's 63-repo benchmark (914 NL queries from semble's 1,251-query corpus, ranked by ken's regex chunker, K=10). The honest framing: ken trades ~17 percentage points of recall for ~44× fewer agent-input tokens. Exhaustive enumeration (refactors, pre-rename audits) still belongs to grep — ken is for "find the chunk that answers this." Full per-query-class table (symbol + NL) and the methodology + caveats are in docs/BENCH.md.

semble timings cited above are from semble's own README "Benchmarks" section; ken's are measured on the included testdata/repo polyglot fixture and on a sibling shallow clone of /tmp/semble. Cold-start was timed by /usr/bin/time -p ken search testdata/repo "validate" -k 1 --mode bm25 over three trials (M2 MacBook Air, Go 1.26.3, darwin/amd64 build under Rosetta).