# CRW — open-source web scraper & crawler (Firecrawl-compatible)

> Docs at `docs.fastcrw.com`. Hosted API at `https://api.fastcrw.com`. Self-hosted at `http://localhost:3000`. Five endpoints: `/v1/search`, `/v1/scrape`, `/v1/crawl`, `/v1/map`, `/v1/extract`.

## Quick: copy this and run

Auth (hosted only): `Authorization: Bearer $CRW_API_KEY`. Self-hosted (`http://localhost:3000`) needs no header.

Body: `{ "query": string, "limit": number (default 5, max 20), "lang"?: "en", "tbs"?: "qdr:d|w|m|y", "sources"?: ["web"|"news"|"images"] }`

### Python (stdlib only — defaults to hosted, fails loudly if key is missing)

    #!/usr/bin/env python3
    """Search CRW and print top results as JSON."""
    import json, os, sys, urllib.request

    API_KEY = os.environ.get("CRW_API_KEY")
    # Default to the hosted API. Self-hosting? Set CRW_API_URL=http://localhost:3000
    BASE = os.environ.get("CRW_API_URL", "https://api.fastcrw.com")
    URL = f"{BASE}/v1/search"

    headers = {"Content-Type": "application/json"}
    if API_KEY:
        headers["Authorization"] = f"Bearer {API_KEY}"
    elif BASE.startswith("https://api.fastcrw.com"):
        sys.exit("error: CRW_API_KEY is required for the hosted API. "
                 "Get one at https://fastcrw.com (500 free credits) "
                 "or set CRW_API_URL=http://localhost:3000 to self-host.")

    req = urllib.request.Request(
        URL,
        data=json.dumps({"query": "renewable energy trends 2024", "limit": 3}).encode(),
        headers=headers,
        method="POST",
    )
    with urllib.request.urlopen(req) as r:
        body = json.loads(r.read())  # body == {"success": true, "data": [...]}

    # Results live under body["data"]. Each row has `title`, `url`, `snippet`,
    # `description`, `position`, `score`, `category` — print the LLM-ready subset.
    print(json.dumps(
        [{"title": x["title"], "url": x["url"], "snippet": x["snippet"]}
         for x in body["data"]],
        indent=2,
    ))
    # Run with: python3 search.py  (file is python3-only — no `python` alias needed)

### cURL

    curl -X POST https://api.fastcrw.com/v1/search \
      -H "Authorization: Bearer $CRW_API_KEY" \
      -H "Content-Type: application/json" \
      -d '{"query":"renewable energy trends 2024","limit":3}'

### CLI (one-shot LLM-ready output)

The `crw` CLI projects directly to the title/url/snippet schema with `--json --fields` — no `jq` reshape needed:

    # Install: brew install us/crw/crw  (or: cargo install crw-cli)
    crw search "renewable energy trends 2024" --json --fields title,url,snippet --limit 3

    # Available fields: title, url, description, snippet, position, score, category
    # --json is shorthand for --format json (also accepts --format text|markdown)
    # Self-hosted: no auth. Hosted: export CRW_API_KEY=crw_live_...

### Response shape

    {
      "success": true,
      "data": [
        {"url": "https://...", "title": "...",
         "description": "...", "snippet": "...",
         "position": 1, "score": 9.5}
      ]
    }

## Other endpoints

- `POST /v1/scrape` `{"url":"...","formats":["markdown"]}` — one URL → markdown
- `POST /v1/map` `{"url":"...","limit":50}` — discover URLs
- `POST /v1/crawl` `{"url":"...","maxPages":5}` — async multi-page; poll `GET /v1/crawl/{id}`
- `POST /v1/scrape` with `formats:["json"]` + `jsonOptions.schema` — structured extract

## Install (local setup only)

`pip install crw` · `npx crw-mcp` · `brew install us/crw/crw` · `curl -fsSL https://raw.githubusercontent.com/us/crw/main/install.sh | sh` · `docker run -p 3000:3000 ghcr.io/us/crw:latest`

## Resources

- OpenAPI spec
- llms.txt
- Context7
- MCP Server (npm)
- Typed Python SDK
- Agent Skills (SKILL.md)
- CLI (Homebrew tap)
- GitHub

Get Started

# CRW Docs

Turn websites into usable data with one API. Start with a single `scrape` request, then move into `search`, `map`, `crawl`, `extract`, `browse` (interactive browser automation), or MCP only when your workflow actually needs them.

Fastest first win: one URL, one markdown response
Works for: agents, ETL, RAG, structured extraction
Deploy: cloud first, self-host when ready
Make your first request Self-host CRW
30-second example
The shortest path to a successful response
If this request works, you already understand the core CRW model: known URL in, clean content out. Everything else in the docs builds on that.

    curl -X POST https://fastcrw.com/api/v1/scrape \
      -H "Authorization: Bearer YOUR_API_KEY" \
      -H "Content-Type: application/json" \
      -d '{
        "url": "https://example.com",
        "formats": ["markdown"]
      }'

    {
      "success": true,
      "data": {
        "markdown": "# Example Domain\n\nThis domain is for use in illustrative examples...",
        "metadata": {
          "title": "Example Domain",
          "sourceURL": "https://example.com",
          "statusCode": 200,
          "elapsedMs": 32
        }
      }
    }

## Start here

:::cards ::card{icon="code" title="Scrape a page" href="#scraping" description="Use one URL and get markdown, HTML, links, or JSON back."} ::card{icon="search" title="Search the web" href="#search" description="Find URLs first, then scrape only the results you care about."} ::card{icon="cursor" title="Browse interactively" href="#mcp" description="Drive a real browser from your agent — multi-step flows, clicks, stateful sessions (v0.4.0)."} ::card{icon="plug" title="Add MCP tools" href="#mcp" description="Give Claude, Cursor, Codex, and other hosts live web access."} :::

## Choose your path

:::cards ::card{icon="rocket" title="Cloud API" href="#quick-start" description="The fastest first run: get a key, copy one request, and move."} ::card{icon="plug" title="MCP" href="#mcp" description="Best when your agent runtime already expects MCP tools."} ::card{icon="box" title="Self-host" href="#self-hosting" description="Best when you want your own infrastructure, auth, and deployment controls."} :::

## Why teams switch to CRW

CRW is meant to feel easy on day one without closing off the more serious use cases:

- one API surface for single-page scrape, discovery, bounded crawl, and extraction,
- Firecrawl-compatible request shapes where they matter,
- low-ops self-hosting when you need infra control,
- and a built-in MCP server for agent workflows.

## Benchmarks

Public 3-way run on Firecrawl scrape-content-dataset-v1, full 1000 URL, canonical `diagnose_3way.py` harness (concurrency 5 / timeout 120s):

Metric | CRW | crawl4ai | Firecrawl
--- | --- | --- | ---
Truth-recall (522/819 labeled URLs) | 63.74% | 59.95% | 56.04%
Scrape-success (of 1000) | 877 (87.7%) | 835 (83.5%) | 897 (89.7%)
Thrown errors (3000 requests) | 0 | 0 | 0
p50 latency | 1914ms | 1916ms | 2305ms
p90 latency | 14157ms | 4754ms | 6937ms
Dependencies | single binary | Python + Playwright | Node + Redis + PG + RabbitMQ

The 63.74% denominator is 819 labeled/matchable URLs — not 3,000 requests, not 1,000. The 87.7% scrape-success is stated next to "0 errors" deliberately. crw's p50 beats Firecrawl; its p90 is the disclosed worst-of-three (the recovery fallback that lifts recall is also why the tail is worst). Full result: `bench/server-runs/RESULT_3WAY_1000_FULL.md`.

## What to read next

- Quick Start for the fastest first request
- API Overview for the endpoint map
- Scrape for the canonical single-page flow
- Authentication for key handling and self-host auth

`esc`