# System Architecture

This document explains the runtime structure of `nginx-markdown-for-agents`, the responsibilities of each component, and the reasoning behind the main technology choices.

## Design Goals

The system is designed to do four things well:

- add a Markdown representation to existing HTML responses without changing the application
- keep request handling inside standard NGINX deployment and operations patterns
- keep HTML parsing and conversion safe, deterministic, and testable
- keep failure behavior explicit and observable

## High-Level Structure

At runtime, the system has two main components:

- the NGINX module in C, which participates in the NGINX request and filter pipeline
- the conversion engine in Rust, which parses HTML and produces Markdown through a small FFI boundary

```mermaid
flowchart TD
    Client["Client"] --> Nginx["NGINX Request Processing"]
    Nginx --> Filter["Markdown Filter Module (C)"]
    Filter --> Eligibility{"Eligibility Checks"}
    Eligibility -->|eligible| Buffer["Buffering / Decompression / Header Policy"]
    Eligibility -->|not eligible| PassThrough["Pass Through (original response)"]
    Buffer --> FFI["Rust FFI Call"]
    FFI --> Rust["Rust Converter"]
    Rust --> Parse["HTML Parsing"]
    Parse --> Sanitize["Sanitization"]
    Sanitize --> Generate["Markdown Generation"]
    Generate --> Meta["Metadata / ETag Support"]
    Meta --> Response["NGINX Response Delivery"]
    PassThrough --> Response

    style Filter fill:#009639,color:#fff
    style Rust fill:#ce422b,color:#fff
    style Eligibility fill:#f90,color:#000
```

## Why C + Rust

The split is deliberate.

### Why the NGINX-facing layer is C

NGINX modules are built around NGINX's C APIs, request phases, memory pools, buffers, and filter-chain model. Putting the request-path integration in C means the module can:

- fit naturally into normal NGINX module loading and configuration
- manage NGINX request and response objects directly
- participate in header and body filtering without an additional translation layer
- preserve familiar deployment and debugging workflows for operators

### Why the conversion engine is Rust

HTML parsing and Markdown generation are the parts most exposed to malformed or hostile input and the parts most likely to grow in complexity over time. Rust is a better fit there because it provides:

- stronger memory-safety guarantees for parsing untrusted input
- a strong ecosystem for HTML parsing and supporting utilities
- better ergonomics for testing, property testing, and output normalization
- clearer ownership and error-handling rules than a pure C implementation

### Why not pure C

A pure C implementation would reduce build complexity, but it would move the riskiest parsing and transformation code into the least forgiving part of the stack. For this project, maintainability and input safety matter more than keeping the entire system in one language.

### Why not an external service

An external converter service would simplify NGINX module logic, but it would add network hops, new failure modes, deployment overhead, and an extra operational surface. This project is explicitly designed to keep conversion inline with the existing reverse-proxy path.

## Responsibility Split

### NGINX module responsibilities

The C module is responsible for:

- deciding whether a request/response pair is eligible for conversion
- handling configuration, request policy, and negotiation rules
- buffering response bodies and coordinating decompression when needed
- applying response-header changes for Markdown variants
- selecting fail-open or fail-closed behavior on conversion failure
- exposing runtime metrics

Internally, the module now keeps these concerns in separate implementation units:

- config wiring plus a dedicated directive registry table
- config object lifecycle, markdown_filter resolution, merged-config logging, and shared-metrics bootstrap
- custom directive parsing and validation
- request-path state transitions in the header/body filter entrypoints
- request-body buffering, decompression coordination, and fail-open replay
- conversion input preparation, FFI execution, and Markdown-output shaping
- worker lifecycle setup/teardown
- dedicated metrics-handler formatting and access control

### Rust converter responsibilities

The Rust converter is responsible for:

- parsing incoming HTML
- removing or neutralizing unsafe or non-content elements
- generating deterministic Markdown output
- producing optional metadata such as token estimates and front matter
- returning structured results through a stable C-compatible interface

Inside the converter crate, the Markdown renderer is split into traversal, block handling, inline handling, table rendering, front-matter emission, and normalization helpers. The FFI boundary is also split into ABI, option-decoding, conversion, memory-management, and export units, while metadata extraction and URL resolution now live in separate helper modules. That reduces the amount of logic concentrated in a single source file without changing the public API or the C ABI.

## Why the FFI Boundary Is Small

The C/Rust boundary is intentionally narrow. The NGINX module does not ask Rust to understand NGINX internals, and the Rust converter does not try to manage the HTTP lifecycle.

That keeps the contract easier to reason about:

- C owns request-path orchestration
- Rust owns conversion logic
- the boundary passes bytes, options, and results

This reduces coupling and makes it easier to test each side independently.

## Request Flow

For an eligible Markdown request, the runtime flow is:

```mermaid
sequenceDiagram
    participant C as Client
    participant N as NGINX
    participant M as Markdown Module
    participant R as Rust Converter

    C->>N: HTTP Request
    N->>M: Evaluate markdown_filter
    M->>M: Check eligibility (method, status, content-type, policy)
    alt Eligible
        M->>M: Buffer body + decompress if needed
        M->>R: FFI call (HTML bytes + options)
        R->>R: Parse HTML → Sanitize → Generate Markdown
        R-->>M: Markdown output + metadata
        M->>M: Update headers (Content-Type, Vary, ETag)
        M->>M: Record metrics
        M-->>C: Markdown response
    else Not Eligible
        M-->>C: Original response (unchanged)
    end
```

Step-by-step:

1. NGINX receives the request and evaluates the configured `markdown_filter` behavior.
2. The module checks request and response eligibility such as method, status, content type, and policy exclusions.
3. If needed, the module buffers the upstream body and decompresses supported encodings.
4. The buffered payload is passed to the Rust converter through FFI.
5. The converter returns Markdown output and optional metadata.
6. The module updates headers such as `Content-Type`, `Vary`, and variant `ETag`, records shared metrics, then sends the Markdown response.

For non-eligible requests, the module stays out of the way and the original response continues through NGINX unchanged.

## Key Architectural Tradeoffs

### Full buffering in v1

The current architecture buffers the full eligible response before conversion. That makes correctness, deterministic output, and header handling much simpler, but it also means:

- larger responses consume more memory
- conversion cannot start streaming output immediately
- very large or streaming-style content should usually be bypassed

This tradeoff is documented in [ADR-0002](ADR/0002-full-buffering-approach.md).

### Shared observability state

Runtime metrics are aggregated in shared memory so the metrics endpoint reports cross-worker totals instead of whichever worker handled the metrics request. This keeps alerting and capacity signals aligned with the whole NGINX instance rather than a single worker view.

### Inline conversion instead of offline publishing

The project chooses inline conversion because it keeps representation negotiation close to the request and avoids duplicating content pipelines. The tradeoff is that conversion now sits inside the request path, so limits, failure policy, and observability become first-class concerns.

### Origin-near positioning

Conversion runs at the reverse-proxy layer closest to the application, not at the CDN edge. This means the module converts the direct output of the application or CMS, before any downstream infrastructure has modified the page. It also means the operator controls the module configuration, failure policy, and rollout scope within their own infrastructure.

This positioning aligns with the HTTP content negotiation model: the origin (or its reverse proxy) selects the best representation of a resource based on the client's Accept header. Cache semantics are simpler because the CDN layer caches the already-converted variant like any other response.

The tradeoff is that conversion cost falls on the origin or reverse-proxy server rather than being distributed across edge nodes, and the operator must be able to install and configure the module. Edge-layer conversion (as Cloudflare's Markdown for Agents demonstrates) serves a different operational model — zero-touch enablement without origin changes. The two approaches can coexist.

This decision is documented in [ADR-0003](ADR/0003-inline-origin-near-conversion.md).

## Where to Go Next

- Runtime decision rationale: [ADR/README.md](ADR/README.md)
- Rust-selection decision: [ADR/0001-use-rust-for-conversion.md](ADR/0001-use-rust-for-conversion.md)
- Buffering decision: [ADR/0002-full-buffering-approach.md](ADR/0002-full-buffering-approach.md)
- Origin-near positioning decision: [ADR/0003-inline-origin-near-conversion.md](ADR/0003-inline-origin-near-conversion.md)
- Repository layout: [REPOSITORY_STRUCTURE.md](REPOSITORY_STRUCTURE.md)
- Operator-facing behavior: [../guides/CONFIGURATION.md](../guides/CONFIGURATION.md)


## v0.7.0 Subsystems

The following Rust-first subsystems were introduced in v0.7.0 to move
pure-logic decisions from C into Rust, improving testability and safety:

### Accept Negotiator (`negotiator.rs`)

Parses `Accept` headers per RFC 7231 §5.3.2, performs q-value comparison
between `text/markdown` and `text/html`, and determines whether conversion
should proceed. Exposed via `FFIAcceptResult` and `markdown_negotiate_accept`
FFI.

### Conditional Request Handler (`conditional.rs`)

Implements `If-None-Match` (ETag strong/weak comparison) and
`If-Modified-Since` (HTTP-date parsing and time comparison) for 304 Not
Modified responses. Used internally by the C conditional-request path.

### Decision Engine (`decision/mod.rs` + `decision/reason_code.rs`)

Pure function `make_decision(DecisionContext) -> Decision` (where `Decision`
is `Convert` or `Skip(SkipReason)`) that centralizes the conversion/skip
decision logic. Each decision path maps to a canonical `ReasonCode`
discriminant (the single source of truth in `decision/reason_code.rs`) for
logging and metrics.

### Header Plan (`header_plan.rs`)

Builder for response header mutations (Content-Type, Content-Length, ETag,
Vary). Generates an operation list that can be applied atomically by the
C module.

### Security Extensions (`security.rs` additions)

URL control-character rejection, X-Forwarded-Host/Proto parsing with host
validation, and Markdown link label/destination escaping for injection
prevention.

### Bounded Decompression

`markdown_decompress_max_size` directive limits decompressed output
independently from `markdown_max_size`, preventing zip-bomb attacks.
`DecompressionBudgetExceeded` (FFI code 9) is classified as
`RESOURCE_LIMIT` in C.

### Parser Timeout and Budget

`markdown_parse_timeout` (default 30s) and `markdown_parser_budget`
(default 64m) directives limit parsing time and memory. New error codes
`ParseTimeout` (10) and `ParseBudgetExceeded` (11) map to
`RESOURCE_LIMIT`.

### Diagnostics Endpoint (`ngx_http_markdown_diagnostics.c`)

A dedicated HTTP handler exposes runtime state at
`/nginx-markdown/diagnostics` when `markdown_diagnostics on` is
configured. Returns JSON containing the current config snapshot, recent
decision summaries (reason codes, durations), and a metrics snapshot.
Access is restricted by `allow` CIDR configuration; external access is
denied by default.

### Dynconf Dry-run and Last-Known-Good (`ngx_http_markdown_dynconf.c`)

`markdown_dynconf_dry_run on` validates a new configuration file on HUP
without replacing the active snapshot. Validation results include line
numbers, field names, and error reasons. On successful reload, the
previous active snapshot is preserved as last-known-good (LKG). Manual
rollback restores `active_snapshot` from LKG. `applied_mtime` updates
only after successful application (Rule 35).

### Reason Code FFI Accessor (`reason_code.rs` + FFI)

Reason codes are defined as a Rust enum (single source of truth). C
accesses reason code values and display strings through the
`markdown_reason_code_str()` / `markdown_reason_code_metric_key()` FFI
exports (wrapped C-side by `ngx_http_markdown_get_reason_code_str()` and
`ngx_http_markdown_get_reason_code_metric_key()`). C-side independent
`#define` constants are removed; all consumers go through the FFI accessor.
This ensures Rust enum, C usage, docs, and metrics labels stay aligned.

### Header Plan Atomic Application (`ngx_http_markdown_ffi_helpers.c`)

The C module applies `FFIHeaderPlan` operations (set, delete, modify)
atomically: all header mutations succeed or all are rolled back. This
prevents partial header state when an allocation failure occurs mid-plan.
The plan is built by Rust and returned as a single FFI struct; C iterates
the operation list and applies changes to `r->headers_out`.

## Document Updates

| Version | Date | Author | Changes |
|---------|------|--------|---------|
| 0.5.0 | 2026-04-21 | docs-standardization | Standardized formatting, added mermaid diagrams where applicable, verified directive accuracy against code, added update tracking section |
| 0.6.2 | 2026-05-08 | Kang | Unified version narrative to 0.6.2 current release line |
| 0.7.0 | 2026-05-17 | Kang | Added v0.7.0 subsystems section (negotiator, conditional, decision, header_plan, security extensions, bounded decompression, parser timeout/budget, diagnostics endpoint, dynconf dry-run/LKG, reason code FFI accessor, header plan atomic application) |
