# Benchmarking

This repository uses two separate tools for two separate jobs:

- Vitest is for unit and correctness tests.
- ESBench is for repeatable local benchmarks, HTML reports, and baseline comparisons.

The initial ESBench suite lives in `bench/publish-async-get.bench.ts`. It uses
the same publish/get runner contract as the CLI benchmark and a deterministic
in-memory DKG client from `bench/support/layered-dkg-client.ts`. That client
models the important layers explicitly:

- working memory, where benchmark triples are first written
- shared working memory, where payloads are staged for publishing
- verifiable memory, where sync or async publish finalizes readable content
- publisher jobs, where async enqueue is separated from runtime finalization

The ESBench suite measures focused memory-layer flows:

- `get/read retrieval`
- `synchronous publish with finalization`
- `asynchronous publish enqueue and finalization`
- `upload payload to local working memory`
- `lift local working memory to shared working memory`

Each flow runs against generated payload sizes of `10kb`, `100kb`, `2mb`, and
`200mb`. The labels use binary units: `1kb = 1024` bytes and `1mb = 1024 * 1024`
bytes.

Add new suites under `bench/**/*.bench.ts` as performance-sensitive paths become
obvious.

## Local Usage

Run correctness tests:

```bash
pnpm test
```

Run benchmarks and write the raw result to `bench/results/latest.json`:

```bash
pnpm bench
```

Run a quick subset while iterating on benchmark wiring:

```bash
DKG_ESBENCH_PAYLOAD_SIZES=10kb pnpm bench
```

The root ESBench config is `esbench.config.mjs`. It was verified against ESBench `0.8.1`, whose CLI runs suites with `esbench --config <file>` and generates reports from saved result files with `esbench report <patterns...> --config <file>`.

## HTML Reports

Generate a benchmark run plus an interactive HTML report:

```bash
pnpm bench:html
```

The default run includes the `200mb` generated payload scene and is intentionally
heavy. Use `DKG_ESBENCH_PAYLOAD_SIZES=10kb,100kb` for a faster local smoke run,
or pass one of `10kb`, `100kb`, `2mb`, `200mb` to isolate a single size.

The `bench:html` script sets both `ESBENCH_HTML=1` and
`ESBENCH_PUBLISH_ASYNC_GET_HTML=1`, so the combined report and the focused
publish/async/get pages are generated by the same ESBench run.

This writes:

- `bench/results/latest.json`
- `bench/results/latest.html`
- `bench/results/publish-async-get/get-read-retrieval.html`
- `bench/results/publish-async-get/sync-publish-finalization.html`
- `bench/results/publish-async-get/async-publish-finalization.html`
- `bench/results/publish-async-get/working-memory-upload.html`
- `bench/results/publish-async-get/working-to-shared-memory.html`

Open the combined HTML file for the full table, or one of the per-flow pages
when you only need a single DKG memory/publish/read path. The per-flow pages are
generated from the same result object through
`ESBENCH_PUBLISH_ASYNC_GET_HTML=1`; they contain benchmark results only and do
not embed local auth tokens or daemon paths. The combined report and per-flow
pages include a fixed report navigation bar so the generated HTML files can be
opened directly from disk without losing the link between the pages.

## CPU Profiles And Flame Graphs

Generate the ESBench reports plus a V8 CPU profile and generated flame graph:

```bash
pnpm bench:profile
```

Generate the per-flow method trace without CPU profiling:

```bash
pnpm bench:analysis
```

Profile a single large generated payload when investigating the heavy path:

```bash
DKG_ESBENCH_PAYLOAD_SIZES=200mb pnpm bench:profile
```

This writes:

- `bench/results/profiles/publish-async-get-<timestamp>.esbench.json`
- `bench/results/profiles/publish-async-get-<timestamp>.esbench.html`
- `bench/results/profiles/publish-async-get-<timestamp>.cpuprofile`
- `bench/results/profiles/publish-async-get-<timestamp>.flamegraph.html`
- `bench/results/profiles/method-analysis.latest.html`
- `bench/results/profiles/method-analysis.latest.json`
- `bench/results/profiles/index.html`

The generated flame graph is a local HTML view built from V8 sampled CPU stacks.
Width represents aggregated sampled CPU time. Use it to find where CPU time is
going inside publish, async publish, get, and memory-layer benchmark runs.
When `bench/results/latest.html` and the focused per-flow pages already exist,
`pnpm bench:profile` updates their navigation bars with `CPU profiles` and
`Method analysis` links.

The method analysis report is the place to inspect which benchmark-layer
methods were invoked for each flow. It separates setup, measured, validation,
and cleanup phases and reports per-method wall-clock timing with context such as
payload size, root entity, marker, and quad count. Use it alongside the flame
graph: the method analysis explains the awaited DKG-layer sequence, while the
flame graph explains sampled CPU stacks inside the process.

The raw `.cpuprofile` can also be opened in Chrome or Edge DevTools Performance,
or uploaded locally to Speedscope. CPU profiling adds overhead, so use normal
`pnpm bench` or `pnpm bench:html` output as the timing baseline and use
`pnpm bench:profile` for deeper attribution.

## Baseline Workflow

Record a local baseline before changing performance-sensitive code:

```bash
pnpm bench:baseline
```

This writes:

- `bench/results/baseline.json`
- `bench/results/baseline.html`

Keep baseline files local unless a PR explicitly needs to share benchmark artifacts. They are generated outputs and are ignored by git.

## Branch Comparison Workflow

Use the same machine, Node version, and terminal session shape for both runs.

```bash
git checkout main
pnpm install --frozen-lockfile
pnpm bench:baseline

git checkout <feature-branch>
pnpm install --frozen-lockfile
pnpm bench:compare
```

`pnpm bench:compare` reads `bench/results/baseline.json`, writes the current run to `bench/results/compare.json`, and writes an HTML report to `bench/results/compare.html`. ESBench includes the previous-run diff in the text and HTML reports.

## Node Version And CI Noise

Pin Node before comparing numbers. This repo has `.nvmrc` set to Node `22`; use that version for local comparisons:

```bash
nvm use
node --version
```

Microbenchmarks are sensitive to CPU scaling, thermal throttling, background processes, dependency versions, and VM/container scheduling. Treat CI benchmark numbers as smoke signals, not hard gates, unless the runner is pinned and quiet. For regression review, prefer repeated local runs on the same machine with the same Node and pnpm versions.
