# Documentation Site Architecture

This file documents non-obvious architectural decisions for the docs site (`docs.pipelex.com`). Read this before modifying anything under `docs/`.

## Site Architecture Overview

- MkDocs Material with `mike` versioning, published at `docs.pipelex.com`
- `site_url` is `https://docs.pipelex.com/` (bare domain, NOT `/latest/`) — mike rewrites sitemap URLs by inserting the version prefix during deployment; if `site_url` already contains `/latest/`, mike creates double paths like `latest/0.20.9/page/`. Canonical URLs are forced to `/latest/` via a template override in `docs/overrides/main.html`.
- Custom domain via `docs/CNAME` (value: `docs.pipelex.com`)
- Theme overrides live in `docs/overrides/`

## URL Structure & Redirects

Legacy URL paths (e.g. `home/1-features/`) are redirected to current clean paths (e.g. `features/`) via two mechanisms:

- The `mkdocs-redirects` plugin maps old→new paths (see `redirect_maps` in `mkdocs.yml`)
- `docs/404.html` contains client-side JS rewrite rules for the same prefixes, covering URLs that bypass MkDocs routing

Both must stay in sync. `mkdocs-redirects` must be installed in CI (`doc-check.yml`).

## The Two 404 Pages (critical — do not conflate)

There are two distinct 404 pages serving different purposes:

1. **`docs/404.html`** — Standalone root-level fallback for GitHub Pages. Copied to `/404.html` on `gh-pages` by `docs-deploy-root`. Contains JS redirect logic for legacy URLs. **NOT** part of the MkDocs build.

2. **`docs/overrides/404.html`** — MkDocs Material template override. Controls the versioned `site/404.html` rendered by MkDocs. Extends `main.html`. Must include both SEO tags (`noindex`, canonical) AND visible user-facing content (heading + link back to `/latest/`).

There must be **NO** `docs/404.md` — a Markdown 404 would be treated as content, appear in the sitemap, and create an indexable `/latest/404/` URL.

## SEO Architecture

- **`docs/overrides/main.html`** — Owns OG tags, Twitter cards, JSON-LD for all pages. Uses null-safe `page.meta` access (`page.meta and page.meta.description`). Does NOT special-case 404 behavior.
- **`docs/overrides/404.html`** — Extends `main.html`, adds `noindex/nofollow` + canonical back to `/latest/`. Must also render visible content.
- **`docs/.meta.yml`** — Default frontmatter for all pages via `meta-manager` plugin (image, description, keywords).
- Per-page `description` frontmatter in individual `.md` files overrides the default.

## robots.txt: Two Files, Different Purposes

- **`docs/robots.txt`** — Lands at `/latest/robots.txt` inside the versioned site. Crawlers ignore it (RFC 9309: only domain-root robots.txt is authoritative). Exists as a courtesy/fallback.
- **`Makefile` `ROOT_ROBOTS_TXT`** — The authoritative robots.txt deployed to domain root by `docs-deploy-root`. Allows only `/latest/` and `/sitemap.xml`, disallows everything else, points sitemap to root `/sitemap.xml` (which contains `/latest/` URLs, generated by `docs-deploy-root` from the `latest/` alias sitemap via sed version-to-`latest` rewrite). **Critical:** `Disallow: /` blocks ALL root paths not explicitly `Allow`ed. The `Sitemap:` directive tells crawlers where the sitemap is but does NOT override `Disallow` — crawlers still need an `Allow:` line to fetch it.

## Deployment (`docs-deploy-root`)

The `docs-deploy-root` Makefile target deploys root assets (`404.html`, `robots.txt`, `index.html`, `sitemap.xml`) directly to the `gh-pages` branch via a temporary git worktree. The root `sitemap.xml` is generated from `latest/sitemap.xml` on gh-pages (the alias directory) by sed-rewriting version numbers to `/latest/`. This ensures the root sitemap always reflects what `/latest/` actually serves, even after pre-release deploys — because mike only updates the `latest/` alias directory during stable deploys, not pre-release ones.

- Called automatically after `docs-deploy-stable` and `docs-deploy-specific-version-pre-release`
- The root `index.html` is a meta-refresh redirect to `/latest/`

## MkDocs Plugins

- `search` — site search with custom separator
- `redirects` — legacy→current URL path mapping
- `meta-manager` — applies `docs/.meta.yml` defaults to all pages
- `glightbox` — image lightbox

## Sidebar Width

Material for MkDocs hard-codes `.md-sidebar { width: 12.1rem }`. The root font-size is **20px** (not 16), so `rem` values are larger than you'd expect (12.1rem = 242px). Custom overrides live in `docs/stylesheets/general.css`.

To widen the sidebar you must override **two** things:
1. `.md-sidebar { width: ... }` — the container itself
2. `.md-sidebar .md-sidebar__inner { padding-right: 4px }` — the theme dynamically inflates `padding-right` on `.md-sidebar__inner` to absorb extra container width, keeping the nav at its original size. Pin this padding to prevent that.

## Mike Versioning Behavior (critical — non-obvious)

Mike (`mike deploy X.Y.Z latest`) creates two directories on gh-pages: `X.Y.Z/` and `latest/`. Understanding what mike does and does NOT rewrite is essential:

**What mike rewrites:** HTML files — mike inserts the version prefix into relative URLs, asset paths, and navigation links so pages work from both `/X.Y.Z/` and `/latest/`.

**What mike does NOT rewrite:** Sitemap URLs. `latest/sitemap.xml` is a copy of `X.Y.Z/sitemap.xml` with the same versioned URLs (e.g., `/X.Y.Z/page/`). Mike does NOT replace the version with the alias name. This means you cannot simply copy `latest/sitemap.xml` and expect `/latest/` URLs — you must sed-rewrite them.

**What mike does NOT rewrite:** Canonical tags, OG URLs, JSON-LD — these come from `site_url` + page path at build time. Mike does not touch them during deploy. That's why we override them in `docs/overrides/main.html`.

**The double-path trap:** If `site_url` is set to `https://docs.pipelex.com/latest/`, MkDocs builds URLs like `/latest/page/`. Then mike inserts the version prefix, producing `/latest/X.Y.Z/page/` — a broken double path. This is why `site_url` must be the bare domain.

## Common Mistakes to Avoid

- Do NOT create `docs/404.md` — it poisons the sitemap with an indexable `/latest/404/` URL
- Do NOT delete `docs/overrides/404.html` — it controls the versioned 404 page
- Do NOT put 404-specific SEO tags in `main.html` — that's `overrides/404.html`'s job
- Do NOT treat `docs/robots.txt` as the production robots policy — the authoritative one is `ROOT_ROBOTS_TXT` in the Makefile
- When changing URL paths, update BOTH `mkdocs.yml` `redirect_maps` AND `docs/404.html` JS rewrite rules
- Always add `mkdocs-redirects` to CI pip install when `redirect_maps` exist
- Do NOT set `site_url` to include `/latest/` — mike will create double paths like `/latest/X.Y.Z/page/`
- Do NOT assume `latest/sitemap.xml` has `/latest/` URLs — mike copies sitemaps verbatim from the version directory without rewriting
- Do NOT source the root sitemap from `$(DOCS_VERSION)/` — use `latest/` so pre-release deploys don't overwrite it with pages not served at `/latest/`
- Do NOT add root-level files to `docs-deploy-root` without also adding a corresponding `Allow:` line in `ROOT_ROBOTS_TXT` — `Disallow: /` blocks them from crawlers even if a `Sitemap:` directive references them

## Spec vs Blueprint in Documentation

The `docs/building-methods/` section documents the MTHDS language — what users write in `.mthds` files. This corresponds to the **blueprint** layer, NOT the spec layer.

- **Blueprints** define the MTHDS language — what `.mthds` files parse into via `PipelexInterpreter`.
- **Specs** (in `pipelex/builder/pipe/`) are a simplified authoring format for AI agents with convenience fields that get transformed into blueprint fields via `to_blueprint()`. Specs are NOT the language definition.

Documentation under `docs/building-methods/` must describe blueprint-level fields, not spec-level convenience fields. For example, `PipeComposeSpec.target_format` is a spec convenience — the MTHDS language uses `category` + `templating_style` on the `TemplateBlueprint` instead.
