# CRW — open-source web scraper & crawler (Firecrawl-compatible)

> Docs at `docs.fastcrw.com`. Hosted API at `https://api.fastcrw.com`. Self-hosted at `http://localhost:3000`. Five endpoints: `/v1/search`, `/v1/scrape`, `/v1/crawl`, `/v1/map`, `/v1/extract`.

## Quick: copy this and run

Auth (hosted only): `Authorization: Bearer $CRW_API_KEY`. Self-hosted (`http://localhost:3000`) needs no header.

Body: `{ "query": string, "limit": number (default 5, max 20), "lang"?: "en", "tbs"?: "qdr:d|w|m|y", "sources"?: ["web"|"news"|"images"] }`

### Python (stdlib only — defaults to hosted, fails loudly if key is missing)

    #!/usr/bin/env python3
    """Search CRW and print top results as JSON."""
    import json, os, sys, urllib.request

    API_KEY = os.environ.get("CRW_API_KEY")
    # Default to the hosted API. Self-hosting? Set CRW_API_URL=http://localhost:3000
    BASE = os.environ.get("CRW_API_URL", "https://api.fastcrw.com")
    URL = f"{BASE}/v1/search"

    headers = {"Content-Type": "application/json"}
    if API_KEY:
        headers["Authorization"] = f"Bearer {API_KEY}"
    elif BASE.startswith("https://api.fastcrw.com"):
        sys.exit("error: CRW_API_KEY is required for the hosted API. "
                 "Get one at https://fastcrw.com (500 free credits) "
                 "or set CRW_API_URL=http://localhost:3000 to self-host.")

    req = urllib.request.Request(
        URL,
        data=json.dumps({"query": "renewable energy trends 2024", "limit": 3}).encode(),
        headers=headers,
        method="POST",
    )
    with urllib.request.urlopen(req) as r:
        body = json.loads(r.read())  # body == {"success": true, "data": [...]}

    # Results live under body["data"]. Each row has `title`, `url`, `snippet`,
    # `description`, `position`, `score`, `category` — print the LLM-ready subset.
    print(json.dumps(
        [{"title": x["title"], "url": x["url"], "snippet": x["snippet"]}
         for x in body["data"]],
        indent=2,
    ))
    # Run with: python3 search.py  (file is python3-only — no `python` alias needed)

### cURL

    curl -X POST https://api.fastcrw.com/v1/search \
      -H "Authorization: Bearer $CRW_API_KEY" \
      -H "Content-Type: application/json" \
      -d '{"query":"renewable energy trends 2024","limit":3}'

### CLI (one-shot LLM-ready output)

The `crw` CLI projects directly to the title/url/snippet schema with `--json --fields` — no `jq` reshape needed:

    # Install: brew install us/crw/crw  (or: cargo install crw-cli)
    crw search "renewable energy trends 2024" --json --fields title,url,snippet --limit 3

    # Available fields: title, url, description, snippet, position, score, category
    # --json is shorthand for --format json (also accepts --format text|markdown)
    # Self-hosted: no auth. Hosted: export CRW_API_KEY=crw_live_...

### Response shape

    {
      "success": true,
      "data": [
        {"url": "https://...", "title": "...",
         "description": "...", "snippet": "...",
         "position": 1, "score": 9.5}
      ]
    }

## Other endpoints

- `POST /v1/scrape` `{"url":"...","formats":["markdown"]}` — one URL → markdown
- `POST /v1/map` `{"url":"...","limit":50}` — discover URLs
- `POST /v1/crawl` `{"url":"...","maxPages":5}` — async multi-page; poll `GET /v1/crawl/{id}`
- `POST /v1/scrape` with `formats:["json"]` + `jsonOptions.schema` — structured extract

## Install (local setup only)

`pip install crw` · `npx crw-mcp` · `brew install us/crw/crw` · `curl -fsSL https://raw.githubusercontent.com/us/crw/main/install.sh | sh` · `docker run -p 3000:3000 ghcr.io/us/crw:latest`

## Resources

- OpenAPI spec
- llms.txt
- Context7
- MCP Server (npm)
- Typed Python SDK
- Agent Skills (SKILL.md)
- CLI (Homebrew tap)
- GitHub

# Architecture

## Crate Structure

crw is a Rust workspace with 6 crates, each with a focused responsibility:

    crw-server (Axum HTTP API — main binary)
    ├── crw-crawl (BFS crawler, single-page scraper)
    │   ├── crw-extract (HTML cleaning, format conversion)
    │   │   └── crw-core (types, config, errors)
    │   └── crw-renderer (HTTP + CDP fetcher)
    │       └── crw-core
    └── crw-core

    crw-mcp (Stdio MCP proxy — standalone binary)

## Request Flow

### Scrape request

    Client → POST /v1/scrape
      → Auth middleware (Bearer token check)
      → scrape handler
        → FallbackRenderer.fetch(url)
          → HTTP request (reqwest)
          → SPA detection heuristics
          → CDP rendering if needed (LightPanda/Playwright/Chrome)
        → HTML cleaning (lol_html)
          → Remove script/style/iframe/svg
          → Remove nav/footer/header if onlyMainContent
          → Apply includeTags/excludeTags CSS selectors
        → Readability extraction
          → Try: article → main → [role=main] → .post-content → body
        → Format conversion
          → Markdown (fast_html2md with fallback chain)
          → HTML / RawHTML / PlainText / Links
          → JSON (LLM extraction via Anthropic/OpenAI)
        → Metadata extraction
          → title, description, og:*, canonical, lang
      → JSON response

### Crawl request

    Client → POST /v1/crawl
      → Create crawl job (UUID)
      → Spawn async BFS task
        → Fetch robots.txt
        → BFS loop with VecDeque:
          → Dequeue URL
          → Check robots.txt allowance
          → Acquire semaphore (max_concurrency)
          → Rate limit (requests_per_second)
          → Scrape page (same pipeline as single scrape)
          → Extract links from page
          → Filter: same origin, not visited, under max_depth
          → Enqueue new URLs
          → Update crawl state via watch::Sender
        → Until: maxPages reached, maxDepth exhausted, or queue empty
      → Return job ID immediately

    Client → GET /v1/crawl/{id}
      → Read crawl state via watch::Receiver
      → Return progress + results

## Middleware Stack

The Axum server applies middleware in this order:

1. CORS — `CorsLayer::permissive()`
2. Trace — HTTP request logging via `tracing`
3. Body limit — Max 1 MB request body
4. Timeout — Configurable request timeout
5. Auth — Bearer token validation (if `auth.api_keys` is set)

## Feature Flags

Flag | Crate | Effect
--- | --- | ---
`cdp` | `crw-renderer` | Enables CDP rendering via `tokio-tungstenite`
`cdp` | `crw-server` | Passes through to `crw-renderer/cdp`
`test-utils` | `crw-server` | Exposes internal functions for testing

## Key Dependencies

Dependency | Purpose
--- | ---
`axum` 0.8 | HTTP API framework
`tokio` | Async runtime
`reqwest` | HTTP client (rustls)
`tokio-tungstenite` | CDP WebSocket (with `cdp` feature)
`lol_html` | Streaming HTML rewriting
`scraper` | CSS selector-based HTML parsing
`fast_html2md` | HTML → Markdown conversion
`jsonschema` | LLM output validation
`config` | Layered TOML configuration
`tracing` | Structured logging

`esc`