# Benchmarks

This page defines practical benchmark patterns for BioMCP.

The objective is reproducible command behavior, not synthetic leaderboard numbers.

For BioASQ's public-corpus ingestion and official Task B lane documentation, see
[BioASQ Benchmark](bioasq-benchmark.md).

## Benchmark goals

- Compare latency across entities.
- Track output-size drift.
- Validate cache behavior.
- Catch regressions in command startup overhead.

## Baseline command set

Use a fixed baseline so runs are comparable over time.

```bash
biomcp get gene BRAF
biomcp get variant "BRAF V600E"
biomcp get trial NCT02576665
biomcp search article -g BRAF --limit 5
```

## Latency measurement

Use repeated runs and report median + spread.

Example with `hyperfine`:

```bash
hyperfine -m 10 'biomcp get gene BRAF' 'biomcp search trial -c melanoma --limit 5'
```

Recommendations:

- run on stable network,
- avoid mixed-background workloads,
- capture raw command outputs for auditing.

## Output-size tracking

Track markdown and JSON sizes independently.

```bash
biomcp get gene BRAF | wc -c
biomcp --json get gene BRAF | wc -c
```

Why this matters:

- markdown size impacts prompt context cost,
- JSON size affects API and queue throughput,
- sudden growth can signal response-shape regressions.

## Cache-behavior checks

Compare first call vs second call for cache-eligible endpoints.

```bash
biomcp get article 22663011
biomcp get article 22663011
```

Capture timing for both runs to verify expected improvement.

## Date-validation contract checks

Invalid dates should fail before network calls.

Examples:

```bash
biomcp search article -g BRAF --since 2024-13-01 --limit 1
biomcp search article -g BRAF --since 2024-02-30 --limit 1
```

Expected behavior:

- immediate validation error,
- no long wait for upstream API timeouts.

## Cache-directory contract checks

Downloaded artifacts should use platform cache paths.

Use command help and health output to validate paths in your environment.

## Reporting template

When sharing benchmark results, include:

- command list,
- machine details,
- network assumptions,
- cache warm/cold status,
- median and p95 latency,
- output byte counts.

## Caveats

- Public APIs can throttle unpredictably.
- Upstream schema changes can affect command duration.
- Geographic distance to API hosts influences baseline latency.

## Recommended cadence

- Run quick checks on every release candidate.
- Run fuller benchmark suites before major parser or rendering changes.
