---
title: Large Site Optimization
nav_title: Large Sites
description: Build and render 5K-100K+ pages efficiently with streaming, parallel processing, and query indexes
weight: 10
icon: server
tags:
- performance
- large-sites
- streaming
- query-indexes
keywords:
- large site
- streaming build
- memory optimization
- query indexes
- parallel builds
- performance
- scale
category: how-to
---
# Large Site Optimization

Bengal is designed for sites with thousands of pages. This guide covers strategies for sites beyond 5,000 pages.

## Quick Start

For sites with 5K+ pages:

```bash
# Memory-optimized build
bengal build --memory-optimized --fast

# Full incremental + parallel + fast
bengal build --incremental --fast
```

---

## Strategy Overview

| Site Size | Recommended Strategy | Build Time |
|-----------|---------------------|------------|
| <500 pages | Default (no changes needed) | 1-3s |
| 500-5K pages | Default (parallel + incremental enabled) | 3-15s |
| 5K-20K pages | `--memory-optimized` | 15-60s |
| 20K+ pages | Full optimization stack | 1-5min |

---

## 1. Memory-Optimized Builds (Streaming Mode)

:::{warning}
**Experimental.** `--memory-optimized` is an experimental streaming mode. Its memory benefit is **not yet verified** by Bengal's benchmarks, and on the workloads measured so far it has not reduced peak memory below a standard build. Treat it as a tuning knob to try and measure on your own site, not a guaranteed win. Profile your build (see [Build Profiling](#6-build-profiling)) before and after to confirm any improvement.
:::

For sites with 5K+ pages, you can try streaming mode:

```bash
bengal build --memory-optimized
```

### How It Works

1. **Builds knowledge graph** to understand page connectivity
2. **Renders hubs first** (highly connected pages) and keeps them in memory
3. **Streams leaves** in batches and releases memory immediately

The intent is to keep memory closer to constant as page count grows by releasing leaf pages after they render. Whether this lowers peak memory on a given site depends on its content and link structure — measure to confirm.

### When to Try It

- Very large sites (5K+ pages) where a standard build runs out of memory
- CI runners or containers with strict memory limits

In each case, compare peak memory with and without the flag before relying on it.

:::{warning}
`--memory-optimized` and `--perf-profile` cannot be used together (profiler doesn't work with batched rendering).
:::

---

## 2. Query Indexes (O(1) Lookups)

Replace O(n) page filtering with O(1) index lookups in templates.

### The Problem

```kida
{# O(n) - scans ALL pages on every request #}
{% let blog_posts = site.pages | where('section', 'blog') %}
```

On a 10K page site, this filter runs 10,000 comparisons.

### The Solution

```kida
{# O(1) - instant hash lookup #}
{% let blog_posts = site.indexes.section.get('blog') | resolve_pages %}
```

### Built-in Indexes

| Index | Key Type | Example |
|-------|----------|---------|
| `section` | Section name | `site.indexes.section.get('blog')` |
| `author` | Author name | `site.indexes.author.get('Jane')` |
| `category` | Category | `site.indexes.category.get('tutorial')` |
| `date_range` | Year or Year-Month | `site.indexes.date_range.get('2024')` |

### Usage Examples

**Section-based listing:**

```kida
{% let blog_posts = site.indexes.section.get('blog') | resolve_pages %}
{% for post in blog_posts | sort_by('date', reverse=true) %}
  <h2>{{ post.title }}</h2>
{% end %}
```

**Author archive:**

```kida
{% let author_posts = site.indexes.author.get('Jane Smith') | resolve_pages %}
<p>{{ author_posts | length }} posts by Jane</p>
```

**Monthly archives:**

```kida
{% let jan_posts = site.indexes.date_range.get('2024-01') | resolve_pages %}
{% for post in jan_posts %}
  {{ post.title }}
{% end %}
```

### Performance Impact

| Pages | O(n) Filter | Query Index |
|-------|-------------|-------------|
| 1K | 2ms | <0.1ms |
| 10K | 20ms | <0.1ms |
| 100K | 200ms | <0.1ms |

---

## 3. Parallel Processing

Parallel processing is **auto-detected** based on page count and workload. Adjust worker count if needed:

```toml
# bengal.toml
[build]
max_workers = 8           # Optional: adjust based on CPU cores (auto-detected if omitted)
```

To force sequential processing (useful for debugging):

```bash
bengal build --no-parallel
```

### Free-Threaded Python

Bengal automatically detects Python 3.14t+ (free-threaded):

```bash
# 1.5-2x faster rendering
# Install free-threaded Python:
pyenv install 3.14t
python3.14t -m pip install bengal
```

See [[docs/about/free-threading|Free-Threading]] for architecture details and code references.

When running on free-threaded Python:
- ThreadPoolExecutor gets true parallelism (no GIL contention)
- ~1.78x faster rendering on multi-core machines
- No code changes needed

---

## 4. Incremental Builds

Incremental builds are **automatic** — no configuration needed. First build is full, subsequent builds only rebuild changed content. Force a full rebuild if needed:

```bash
# Force full rebuild (skip cache)
bengal build --no-incremental
```

### What Gets Cached

- **Content parsing** — Markdown AST cached per file
- **Template rendering** — Output cached by content hash
- **Asset hashing** — Fingerprints cached
- **Query indexes** — Updated incrementally
- **Autodoc AST parsing** — Python modules cached to skip AST parsing (30-40% speedup for autodoc-heavy sites)
- **Asset dependencies** — Tracked during render-time (no HTML parsing needed)

### Cache Location

```tree
.bengal/
├── cache.json.zst          # Main build cache (compressed)
├── page_metadata.json.zst  # Page discovery cache
├── taxonomy_index.json.zst # Taxonomy index
├── indexes/                # Query indexes (section, author, etc.)
├── templates/              # Template bytecode cache
└── logs/                   # Build logs
```

### Clear Cache

```bash
# Clear all caches (forces cold rebuild)
bengal clean --cache

# Clear output and cache
bengal clean --all
```

---

## 5. Fast Mode

Combine all optimizations for maximum speed:

```bash
bengal build --fast
```

`--fast` enables:
- Quiet output (minimal console I/O)
- Suppresses verbose logging
- Parallelism auto-detected as normal
- **Skips HTML formatting** (raw HTML output, ~10-15% faster)

:::{note}
Fast mode skips HTML pretty-printing and minification. Output is still valid HTML but not formatted. Use for development and CI builds where formatting doesn't matter.
:::

---

## 6. Build Profiling

Identify bottlenecks:

```bash
# Generate performance profile
bengal build --perf-profile

# View results
python -m pstats .bengal/profiles/profile.stats
```

### Template Profiling

Find slow templates:

```bash
bengal build --profile-templates
```

**Output:**

```text
Template Rendering Times:
  layouts/blog.html: 1.2s (340 pages, 3.5ms avg)
  layouts/docs.html: 0.8s (890 pages, 0.9ms avg)
  partials/nav.html: 0.3s (included 1230 times)
```

---

## 7. Content Organization

### Split Large Sections

If one section has 5K+ pages, consider splitting:

```tree
content/
├── blog/
│   ├── 2024/     # 500 pages
│   ├── 2023/     # 800 pages
│   └── archive/  # 3000+ pages (separate pagination)
```

### Use Pagination

Don't render 1000 items on one page:

```yaml
# Paginate blog listing
pagination:
  per_page: 20
```

### Lazy-Load Heavy Content

Move rarely-accessed content to separate pages:

```kida
{# Don't: render full changelog inline #}
{{ include('changelog.html') }}

{# Do: link to separate page #}
<a href="/changelog/">View full changelog</a>
```

---

## 8. CI/CD Optimization

### GitHub Actions Example

```yaml
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Cache Bengal
        uses: actions/cache@v4
        with:
          path: .bengal
          key: bengal-${{ hashFiles('content/**/*.md') }}

      - name: Build
        run: bengal build --fast --environment production
```

### Docker Memory Limits

```dockerfile
# Use memory-optimized for container builds
CMD ["bengal", "build", "--memory-optimized", "--fast"]
```

---

## 9. Monitoring Build Health

Track build performance over time:

```bash
# Detailed build stats
bengal build --verbose
```

**Output:**

```text
Build Summary:
  Total Pages: 15,432
  Rendered: 342 (incremental)
  Skipped: 15,090 (cached)
  Duration: 12.3s
  Memory Peak: 245MB
  Pages/sec: 1,254
```

---

## Quick Reference

```bash
# Memory-efficient large site build
bengal build --memory-optimized --fast

# Profile to find bottlenecks
bengal build --perf-profile --profile-templates

# Force full rebuild
bengal build --no-incremental

# Clear all caches
bengal clean --cache

# Clear output and cache
bengal clean --all
```

---

## Troubleshooting

### Build runs out of memory

1. Increase available memory or swap space
2. Use `bengal build --dev --verbose` to watch memory usage
3. Try the experimental streaming mode (`--memory-optimized`) and measure peak memory with and without it — it is not guaranteed to help on every site

### Build is slow despite caching

1. Check what's invalidating cache: `bengal build --verbose`
2. Profile templates: `--profile-templates`
3. Check for O(n) filters in templates (use query indexes)

### Incremental not working

1. Ensure `.bengal/` is not gitignored for local dev
2. Run `bengal clean --cache` to reset
3. Check for template changes that invalidate all pages

---

:::{seealso}
- [[docs/building/performance|Performance Overview]]
- [[docs/reference/cheatsheet|CLI Cheatsheet]]
:::
