# Octocode benchmark suite

Three feature areas, each a self-contained folder with its own real-file
samples and a check that exercises the feature through the shipped
`octocode-engine` napi binary. This suite lives in the private
`@octocodeai/octocode-benchmark` workspace package.

> **Full format support:** [`SUPPORT.md`](https://github.com/bgauryy/octocode/blob/main/packages/octocode-benchmark/benchmark/SUPPORT.md) is the canonical,
> machine-verified matrix of every supported extension (AST / signature / LSP /
> minify). It is generated by `yarn support:gen` (`check-matrix.mjs --write`) and
> is what the repo README points to.

| Folder | Feature | Samples | Check |
| --- | --- | --- | --- |
| [`ast/`](https://github.com/bgauryy/octocode/blob/main/packages/octocode-benchmark/benchmark/ast/) | Structural (AST) search + signature extraction | `ast/samples/` — one real file per grammar (provenance in `ast/manifest.json`) | `ast/check-ast.mjs` — parse + match + signature for all 19 grammars |
| [`ast-grep/`](https://github.com/bgauryy/octocode/blob/main/packages/octocode-benchmark/benchmark/ast-grep/) | Optional external ast-grep CLI comparison | ast-grep upstream outline benchmark scenario repos | `ast-grep/compare-upstream-scenarios.mjs` — same-corpus ast-grep CLI vs Octocode CLI benchmark |
| [`lsp/`](https://github.com/bgauryy/octocode/blob/main/packages/octocode-benchmark/benchmark/lsp/) | LSP language-id, server resolution, native semantics | `lsp/samples/` — one real file per LSP language | `lsp/check-lsp.mjs` — language-id + server config + native semantics for 18 languages |
| [`minify/`](https://github.com/bgauryy/octocode/blob/main/packages/octocode-benchmark/benchmark/minify/) | Minifier (the original size/strategy report) | `minify/<lang>/raw|minified|symbol/` — one excerpt per discovered extension | `minify/check-minify.mjs` — minify runs for all 46 formats; `generate-real-code-report.mjs` produces the size report |

## Run

```bash
yarn workspace @octocodeai/octocode-engine build:dev
node benchmark/run-all.mjs        # all three checks
# or individually:
node benchmark/ast/check-ast.mjs
node benchmark/ast-grep/compare-upstream-scenarios.mjs --sync-repos --scenario gin-middleware-routing
node benchmark/lsp/check-lsp.mjs
node benchmark/minify/check-minify.mjs
```

Equivalent yarn scripts: `yarn ast:check`, `yarn lsp:check`, `yarn minify:check`,
`yarn ast:compare:upstream`, `yarn benchmark` (run-all). Engine package
compatibility scripts delegate here. The ast-grep comparison is optional and is
not part of `yarn benchmark` because it needs external CLIs and cloned upstream
scenario repositories.

## Real samples, not invented

The `ast/` and `lsp/` samples are real third-party source files fetched from
public repositories (TypeScript compiler, Rust core, Spring, LLVM, nvm,
Bootstrap, Home Assistant, …). Each `manifest.json` records the exact repo,
branch, path, byte size, and sha256 so a sample can be re-fetched or audited.
Files retain their upstream licenses and are used only to exercise the engine.

## Notes

- Tree-sitter is the **only** signature path; there is no regex/heuristic
  fallback (the former `src/signatures/heuristic.rs` was orphaned and deleted).
  Structural-only grammars (HTML/CSS/SCSS/Less/Scala/JSON/YAML/TOML) produce no
  signature outline by design.
- Scala has a grammar (structural search works) but **no configured language
  server**, so it appears in `ast/` but not `lsp/`.
- The LSP check verifies wiring + native (in-process) semantics only. Real
  server-backed features need the external servers installed; the check reports
  their availability but never fails on a missing server.
