# Benchmarking

This directory contains the benchmark suite for the Bengal static site generator. The benchmarks are designed to be run using `pytest` and the `pytest-benchmark` plugin.

## Setup

### 1. Install benchmark dependencies

```bash
pip install -r requirements.txt
```

### 2. Install Bengal and its dependencies

**Multi-repo workspace** (bengal, patitas, kida, etc. as siblings in `b-stack/`):

```bash
cd /path/to/b-stack   # workspace root with pyproject.toml
uv sync
```

This installs bengal and all workspace packages (kida, patitas, etc.) from local paths.

**Standalone** (bengal only):

```bash
cd /path/to/bengal
pip install -e .
```

### 3. Verify

```bash
bengal --version
```

If this fails with `ModuleNotFoundError` (e.g. `No module named 'bengal'` or `No module named 'kida'`), the install is broken. Reinstall: `pip uninstall bengal` then run `uv sync` from the workspace root (or `pip install -e .` from bengal).

## Canonical Benchmark Matrix

`benchmark_matrix.toml` is the source of truth for which benchmark rows count as
canonical evidence. It divides runs into smoke, release, and investigation tiers,
and each row names the command, source paths, metrics, evidence artifacts, and
change triggers it protects.

Use the matrix before making or reviewing performance claims:

```bash
uv run pytest tests/unit/benchmarks/test_benchmark_matrix.py -q
```

Run or list matrix rows by tier:

```bash
uv run python benchmarks/run_matrix.py --tier smoke --dry-run
uv run python benchmarks/run_matrix.py --tier smoke
uv run python benchmarks/run_matrix.py --list
```

Treat the overview below as explanatory. When adding, removing, or renaming a
benchmark that should guide decisions, update `benchmark_matrix.toml` in the same
change.

## Running the Benchmarks

To run the benchmarks, simply run `pytest` from the `benchmarks` directory:

```bash
pytest
```

This will run the benchmarks for all the scenarios defined in the `scenarios` directory. The results will be displayed in the console.

## Profiling Rendering

To profile Bengal's rendering pipeline and find bottlenecks:

```bash
# CPU profile (cProfile) - saves to benchmarks/.benchmarks/cpu_profile.prof
uv run python benchmarks/profile_render.py --profile

# With template profiling (shows slow templates/functions)
uv run python benchmarks/profile_render.py --profile --profile-templates

# Profile a specific site
uv run python benchmarks/profile_render.py --profile --site /path/to/site

# View saved profile
python -m pstats benchmarks/.benchmarks/cpu_profile.prof
# Then: sort cumtime, stats 30
```

## Benchmark Suite Overview

| Suite | Purpose |
|-------|---------|
| `test_build.py` | Core build, incremental, fast mode, memory |
| `test_cold_build_permutations.py` | Build modes × site sizes (100–1000 pages) |
| `test_10k_site.py` | Large site (10k pages), discovery, memory |
| `test_github_pages_optimization.py` | GitHub Pages constraints (2-core, 7GB) |
| `test_directive_heavy_build.py` | Real-world docs (admonitions, code blocks) |
| `test_notebook_build.py` | Native .ipynb cold + incremental builds |
| `test_patitas_performance.py` | Patitas vs Mistune parse comparison |
| `test_directive_performance.py` | Directive parsing performance |
| `test_highlighting_performance.py` | Rosettes vs Pygments |
| `test_kida_vs_jinja.py` | Template engine concurrent performance |
| `test_import_overhead.py` | CLI startup, import pollution |
| `tests/performance/benchmark_realistic_scale.py` | Realistic content at scale |
| `tests/performance/benchmark_template_complexity.py` | Template complexity impact |
| `tests/performance/test_autodoc_render_regression.py` | Autodoc output shape/size regression gate |
| `tests/performance/test_asset_fallback_cost.py` | Fallback asset parsing cost regression gate |
| `benchmark_hmr_latency.py` | End-to-end dev-loop save->reload latency (record-only) |

## Benchmark Categories

### Full Build Performance
- **Baseline builds**: Standard builds for small/large sites
- **Fast mode**: Quiet output + guaranteed parallel (`--fast` flag)

### Incremental Builds
- **Single page change**: Most common developer workflow (rebuilds a fraction of the site; warm-build speedup is workload-dependent and not yet committed as a baseline)
- **Multi-page change**: Batch edits (5 pages)
- **Config change**: Should trigger full rebuild
- **No changes**: Cache validation (<1s expected)

### Parallel Processing
- **Parallel vs Sequential**: Validates 2-4x speedup on multi-core systems
- **Fast mode impact**: Measures logging overhead reduction

### Memory Optimization
- **Memory-optimized builds**: Streaming builds for large sites (`--memory-optimized`)
- **Memory profiling**: Tracks peak memory usage across scenarios

## Expected Performance

The numbers below are the **committed baselines** in `benchmarks/baselines/`
(free-threaded CPython 3.14t, `blog` archetype with taxonomy, cold build, median
of 3). Regenerate with `python benchmarks/benchmark_gil_speedup.py --publish`;
see `baselines/SPEEDUP.md`.

| Archetype | Pages | GIL=0 (free-threading) | GIL=1 (GIL re-enabled) | Free-threading speedup |
|-----------|------:|-----------------------:|-----------------------:|-----------------------:|
| blog | 100 | 4.86 s (20.6 pages/s) | 6.05 s (16.5 pages/s) | 1.24x |
| blog | 1,000 | 56.3 s (17.8 pages/s) | 109.2 s (9.2 pages/s) | **1.94x** |

**The headline is the free-threading speedup, not absolute pages/sec.** On the
same 3.14t interpreter, disabling the GIL (`PYTHON_GIL=0`) is **1.94x faster at
1,000 pages** because rendering — the dominant (~39%) phase — scales across
threads. Absolute throughput is workload-dependent: a render-light "minimal
content" fixture (no taxonomy, no directives, no highlighting) builds far faster
per page than the taxonomy-heavy `blog` archetype above, so the two are not
comparable.

> **Accuracy note.** Earlier revisions of this file advertised "~256 pages/sec"
> (3.14) and "~373 pages/sec" (3.14t). Those render-light best-case figures had
> no committed benchmark behind them and contradicted the baselines above by
> ~13x. Use the committed numbers. Warm/incremental speedup is likewise not yet
> captured as a committed baseline — do not quote a fixed multiplier until it is.

For absolute single-thread throughput, compiled SSGs (e.g. Hugo, written in Go)
remain ~20x faster per page; Bengal's bet is free-threaded parallel **scaling**,
not winning the per-page constant factor.

## HMR Save-to-Reload Latency (dev loop)

`benchmark_hmr_latency.py` measures the FULL dev-loop save->reload cycle, not
just `site.build()`. The committed dev-server warm path times only the build +
buffer swap, hiding the fixed pre-build latency. This harness drives the real
`WatcherRunner -> BuildTrigger -> BufferManager -> ReloadNotifier` chain (wired
as `DevServer._create_watcher`), edits a real content file on disk, and times
from just before the write to the instant the SSE reload signal fires.

The total is decomposed so the number is informative rather than dominated by a
configurable constant:

- `timer_floor_ms` — the deterministic floor a save waits through before any
  build can start: `debounce_ms` (300) + one debounce-loop tick (50ms) + the
  expected half of one watchfiles poll interval (`poll_delay_ms`/2 = 150ms) =
  **500ms**. This is config, not Bengal's work.
- `variable_work_ms` = `total_ms - timer_floor_ms` — build + swap + signal. This
  is the number v0.6.0's proportional-rebuild work (#332/#333) should drive down.

The committed baseline lives at `baselines/hmr_latency.json`. Regenerate with:

```bash
python benchmarks/benchmark_hmr_latency.py --scales 50,200 --samples 7 --publish
```

**Record-only.** Driving the async watcher deterministically is timing-flaky, so
this is a RECORD, not a gate — it never fails CI on a regression.
`compare_hmr_latency.py` re-measures and reports drift vs the committed baseline
without changing its exit code on a measured regression:

```bash
python benchmarks/compare_hmr_latency.py            # measure + report drift (no gate)
```

It is macOS wall-clock only (the dev server forces polling there) and was
captured on the dev machine; re-capture on a stable host if it is ever promoted
to a gate.

## Recent Optimizations (2025-10-20)

- **Page list caching**: 75% reduction in equality checks
- **Parallel related posts**: 7.5x faster (10K pages: 120s → 16s)
- **Parallel taxonomy generation**: Automatic for 100+ page sites
- **Fast mode**: Quiet output + guaranteed parallel processing

## Cold Build Permutation Benchmarks

The `test_cold_build_permutations.py` suite provides comprehensive comparison of all build modes across multiple site sizes.

### Running Cold Build Permutations

```bash
# Run all permutations
pytest test_cold_build_permutations.py -v

# Run specific site size
pytest test_cold_build_permutations.py -k "100_pages" -v

# Run specific build mode across all sizes
pytest test_cold_build_permutations.py -k "fast_mode" -v
```

### Test Matrix

**Site Sizes**: 100, 500, 1000 pages

**Build Modes**:
- Standard (default parallel)
- Fast mode (`--fast`)
- Memory-optimized (`--memory-optimized`)
- Sequential (`--no-parallel`)
- Fast + Memory-optimized (`--fast --memory-optimized`) - 500+ pages only

**Total Tests**: 14 permutations (5 modes × 3 sizes, minus 1 mode that only runs on 500+ pages)

### Expected Insights

- **Fast mode impact**: Measures logging overhead reduction
- **Memory-optimized tradeoffs**: Speed vs memory usage for large sites
- **Parallel speedup**: Validates 2-4x improvement on multi-core systems
- **Scaling characteristics**: How each mode performs as site size increases

### Interpreting Results

After running benchmarks, use `pytest-benchmark` comparison features:

```bash
# Compare results
pytest test_cold_build_permutations.py --benchmark-compare

# Save results for comparison
pytest test_cold_build_permutations.py --benchmark-save=baseline
pytest test_cold_build_permutations.py --benchmark-compare=baseline
```

Look for:
- **Fast mode**: Should be similar or slightly faster than standard (reduced logging)
- **Memory-optimized**: May be slightly slower but uses constant memory
- **Sequential**: Should be 2-4x slower than parallel on multi-core systems

### Running Specific Benchmarks

You can run specific benchmarks using the `-k` flag or by specifying the test node ID:

```bash
# Run incremental build tests
pytest benchmarks/test_build.py -k "incremental" -v

# Run fast mode benchmarks
pytest benchmarks/test_build.py -k "fast_mode" -v
```

### Benchmark Types

- **Full Build**: Standard build with parallel processing enabled (default).
- **Incremental**: Tests single-page edits, batch edits, and config changes.

## 10k Page Site Benchmarks

The `test_10k_site.py` suite tests performance at scale with 10,000+ page sites.

### Running 10k Benchmarks

```bash
# Run all 10k benchmarks (slow, use for nightly CI)
pytest benchmarks/test_10k_site.py -v --benchmark

# Run without benchmark plugin (simpler output)
pytest benchmarks/test_10k_site.py -v

# Run the faster 1k variant
pytest benchmarks/test_10k_site.py -k "1k_site" -v
```

### Performance Gates

| Metric | 10k Pages | 1k Pages |
|--------|-----------|----------|
| Discovery Time | <30s | <5s |
| Peak Memory | <2GB | <500MB |
| Discovery Rate | >300 pages/sec | >200 pages/sec |

### Expected Baseline Performance

**M1 Pro (8-core)**:
- 10k page discovery: ~10-15s
- Peak memory: ~600MB-1GB
- Per-page overhead: ~60-100KB

**Intel i7 (8-core)**:
- 10k page discovery: ~12-20s
- Peak memory: ~700MB-1.2GB
- Per-page overhead: ~70-120KB

### Tests

1. **`test_10k_site_discovery_performance`**: Measures discovery time for 10k pages
2. **`test_10k_site_memory_usage`**: Tracks peak memory usage
3. **`test_1k_site_discovery_performance`**: Faster variant for regular CI
4. **`test_discovery_scaling`**: Verifies linear scaling behavior

### CI Integration

These tests are marked `@pytest.mark.slow` and should run in nightly CI:

```yaml
# .github/workflows/nightly.yml
- name: Run performance benchmarks
  run: pytest benchmarks/test_10k_site.py -v --benchmark
```

For regular CI, use the 1k variant:

```yaml
# .github/workflows/ci.yml
- name: Run quick performance check
  run: pytest benchmarks/test_10k_site.py -k "1k_site" -v
```

## GitHub Pages Optimization Benchmarks

The `test_github_pages_optimization.py` suite comprehensively tests ALL Bengal build option combinations under GitHub Pages worker constraints (2-core CPU, 7GB RAM) using Python 3.14t free-threading.

### Purpose

This suite systematically explores the build option space to find optimal configurations for GitHub Pages deployments, which run on constrained resources.

### Resource Constraints

- **CPU**: 2 cores (emulated via CPU affinity)
- **RAM**: 7GB limit (emulated via resource limits)
- **Python**: 3.14t (free-threading, no GIL)

### Build Options Tested

- `--parallel` / `--no-parallel`
- `--incremental` / `--no-incremental` / auto
- `--memory-optimized`
- `--fast`
- `--profile` (writer/theme-dev/dev)
- `--quiet`
- `--strict`
- `--clean-output`

### Site Sizes

- 50 pages: Small site (typical blog)
- 200 pages: Medium site (documentation)
- 500 pages: Large site (comprehensive docs)
- 1000 pages: Very large site (stress test)

### Running GitHub Pages Benchmarks

```bash
# Run all configurations (comprehensive, takes time)
pytest benchmarks/test_github_pages_optimization.py -v --benchmark

# Run specific site size
pytest benchmarks/test_github_pages_optimization.py -k "50_pages" -v

# Run optimal CI build test
pytest benchmarks/test_github_pages_optimization.py -k "optimal_ci_build" -v

# Save results for analysis
pytest benchmarks/test_github_pages_optimization.py --benchmark-json=gh_pages_results.json

# Analyze results
python benchmarks/analyze_github_pages_results.py gh_pages_results.json
```

### Expected Insights

- **Fast mode impact**: Logging overhead reduction on constrained systems
- **Memory-optimized tradeoffs**: Speed vs memory usage for large sites
- **Parallel speedup**: Expected ~1.5-2x on 2-core systems (Python 3.14t)
- **Optimal CI configuration**: Best flags for GitHub Actions workflows
- **Scaling characteristics**: How each mode performs as site size increases

### Prerequisites

```bash
# Install dependencies (includes psutil for resource constraints)
pip install -r benchmarks/requirements.txt

# Ensure Python 3.14t is available
python --version  # Should show 3.14.x
```

### Analysis Script

The `analyze_github_pages_results.py` script processes benchmark results and generates:
- Best configuration per site size
- Performance comparison tables
- Recommendations for GitHub Pages builds
- Resource usage analysis

```bash
# Generate and analyze results
pytest benchmarks/test_github_pages_optimization.py --benchmark-json=results.json
python benchmarks/analyze_github_pages_results.py results.json
```

---

## Import Overhead Benchmarks

The `test_import_overhead.py` suite measures module startup latency and "import pollution" - when lightweight modules accidentally pull in heavy dependencies.

### Why Import Time Matters

- **CLI responsiveness**: Users notice if `bengal --help` takes 500ms+ to start
- **Dev server startup**: Fast restarts improve DX
- **Incremental builds**: Re-importing unchanged modules wastes time
- **Rosettes as external package**: Users may `import rosettes` directly for standalone highlighting

### Running Import Benchmarks

```bash
# Quick diagnostic report (no pytest)
python benchmarks/test_import_overhead.py

# Full test suite
pytest benchmarks/test_import_overhead.py -v

# Test specific modules
pytest benchmarks/test_import_overhead.py -k "rosettes" -v
pytest benchmarks/test_import_overhead.py -k "lightweight" -v
```

### What It Tests

| Category | Tests | Threshold |
|----------|-------|-----------|
| Lightweight modules | rosettes (external), kida, highlighting | <50ms, no heavy deps |
| Optional dependencies | uvloop, psutil, aiohttp | Only loaded when used |
| Error imports | bengal.errors | Only on error path |
| Package inits | `__init__.py` lazy exports | No eager heavy loading |
| Regression detection | Baselines from known good state | <1.5x baseline |

### Key Concepts

- **Heavy modules**: `bengal.errors`, `bengal.core.page`, `bengal.rendering.pipeline`
- **Optional deps**: `uvloop`, `psutil`, `aiohttp`, `zstd`
- **Import pollution**: When importing module A accidentally loads unrelated module B

### Example Output

```
======================================================================
IMPORT OVERHEAD REPORT
======================================================================

✅ rosettes (external package)
   Time: 11.3ms

✅ bengal.rendering.kida
   Time: 7.3ms

❌ bengal.utils
   Time: 143.6ms
   Heavy deps: bengal.core.site, bengal.errors
   Optional deps: uvloop
```

### Design Patterns (from investigation)

1. **Lazy exports in `__init__.py`**: Use `__getattr__` for heavy re-exports
2. **Deferred error imports**: Import exceptions only in error-raising functions
3. **Optional deps on-demand**: Import `uvloop`, `psutil` etc. only when feature is used
4. **TYPE_CHECKING guards**: Keep type hints without runtime import cost

### CI Integration

```yaml
# .github/workflows/ci.yml
- name: Check import overhead
  run: pytest benchmarks/test_import_overhead.py -v
```

Failing tests indicate a performance regression that should block the PR.

---

## Directive-Heavy Build Benchmarks

The `test_directive_heavy_build.py` suite tracks build performance for real-world
documentation sites that use directives (admonitions, tabs, cards), syntax highlighting,
and TOC. Docs claim "directive-heavy sites typically see 40–60% of benchmark speeds."

### Running Directive-Heavy Benchmarks

```bash
pytest test_directive_heavy_build.py -v --benchmark-only

# Compare directive-heavy vs minimal ratio
pytest test_directive_heavy_build.py -k "vs_minimal" -v -s
```

### Tests

- **`test_directive_heavy_cold_build_100`**: Cold build of 100 pages with directives
- **`test_directive_heavy_incremental_single_page`**: Incremental rebuild after edit
- **`test_directive_heavy_vs_minimal_ratio`**: Reports directive-heavy / minimal ratio

---

## Notebook Build Benchmarks

The `test_notebook_build.py` suite benchmarks native `.ipynb` support. Bengal parses
notebooks via Patitas (stdlib JSON, no nbformat).

### Running Notebook Benchmarks

```bash
pytest test_notebook_build.py -v --benchmark-only
```

### Tests

- **`test_notebook_cold_build_50`**: Cold build of 50 notebooks
- **`test_notebook_incremental_single_change`**: Incremental rebuild after edit
- **`test_notebook_mixed_site`**: Mixed content (25 markdown + 25 notebooks)
