# CRW — open-source web scraper & crawler (Firecrawl-compatible)

> Docs at `docs.fastcrw.com`. Hosted API at `https://api.fastcrw.com`. Self-hosted at `http://localhost:3000`. Five endpoints: `/v1/search`, `/v1/scrape`, `/v1/crawl`, `/v1/map`, `/v1/extract`.

## Quick: copy this and run

Auth (hosted only): `Authorization: Bearer $CRW_API_KEY`. Self-hosted (`http://localhost:3000`) needs no header.

Body: `{ "query": string, "limit": number (default 5, max 20), "lang"?: "en", "tbs"?: "qdr:d|w|m|y", "sources"?: ["web"|"news"|"images"] }`

### Python (stdlib only — defaults to hosted, fails loudly if key is missing)

    #!/usr/bin/env python3
    """Search CRW and print top results as JSON."""
    import json, os, sys, urllib.request

    API_KEY = os.environ.get("CRW_API_KEY")
    # Default to the hosted API. Self-hosting? Set CRW_API_URL=http://localhost:3000
    BASE = os.environ.get("CRW_API_URL", "https://api.fastcrw.com")
    URL = f"{BASE}/v1/search"

    headers = {"Content-Type": "application/json"}
    if API_KEY:
        headers["Authorization"] = f"Bearer {API_KEY}"
    elif BASE.startswith("https://api.fastcrw.com"):
        sys.exit("error: CRW_API_KEY is required for the hosted API. "
                 "Get one at https://fastcrw.com (500 free credits) "
                 "or set CRW_API_URL=http://localhost:3000 to self-host.")

    req = urllib.request.Request(
        URL,
        data=json.dumps({"query": "renewable energy trends 2024", "limit": 3}).encode(),
        headers=headers,
        method="POST",
    )
    with urllib.request.urlopen(req) as r:
        body = json.loads(r.read())  # body == {"success": true, "data": [...]}

    # Results live under body["data"]. Each row has `title`, `url`, `snippet`,
    # `description`, `position`, `score`, `category` — print the LLM-ready subset.
    print(json.dumps(
        [{"title": x["title"], "url": x["url"], "snippet": x["snippet"]}
         for x in body["data"]],
        indent=2,
    ))
    # Run with: python3 search.py  (file is python3-only — no `python` alias needed)

### cURL

    curl -X POST https://api.fastcrw.com/v1/search \
      -H "Authorization: Bearer $CRW_API_KEY" \
      -H "Content-Type: application/json" \
      -d '{"query":"renewable energy trends 2024","limit":3}'

### CLI (one-shot LLM-ready output)

The `crw` CLI projects directly to the title/url/snippet schema with `--json --fields` — no `jq` reshape needed:

    # Install: brew install us/crw/crw  (or: cargo install crw-cli)
    crw search "renewable energy trends 2024" --json --fields title,url,snippet --limit 3

    # Available fields: title, url, description, snippet, position, score, category
    # --json is shorthand for --format json (also accepts --format text|markdown)
    # Self-hosted: no auth. Hosted: export CRW_API_KEY=crw_live_...

### Response shape

    {
      "success": true,
      "data": [
        {"url": "https://...", "title": "...",
         "description": "...", "snippet": "...",
         "position": 1, "score": 9.5}
      ]
    }

## Other endpoints

- `POST /v1/scrape` `{"url":"...","formats":["markdown"]}` — one URL → markdown
- `POST /v1/map` `{"url":"...","limit":50}` — discover URLs
- `POST /v1/crawl` `{"url":"...","maxPages":5}` — async multi-page; poll `GET /v1/crawl/{id}`
- `POST /v1/scrape` with `formats:["json"]` + `jsonOptions.schema` — structured extract

## Install (local setup only)

`pip install crw` · `npx crw-mcp` · `brew install us/crw/crw` · `curl -fsSL https://raw.githubusercontent.com/us/crw/main/install.sh | sh` · `docker run -p 3000:3000 ghcr.io/us/crw:latest`

## Resources

- OpenAPI spec
- llms.txt
- Context7
- MCP Server (npm)
- Typed Python SDK
- Agent Skills (SKILL.md)
- CLI (Homebrew tap)
- GitHub

# Compatibility Matrix

## What "Compatible" Means Here

CRW is built around Firecrawl-compatible workflows, not a blanket "drop-in replacement" claim. The goal is to preserve the core request mental model so migrations are manageable, while still documenting the places where behavior differs.

## Supported alignment

Area | Status
--- | ---
`/v1/scrape`, `/v1/crawl`, `/v1/map` route shape | Supported
`limit`, `maxPages`, `max_pages` for crawl caps | Supported
Numeric `waitFor` for JS rendering | Supported
`cssSelector`, `xpath`, `chunkStrategy`, `filterMode` | Supported

## Known differences

Area | Current behavior
--- | ---
Screenshot output | Not supported in this release
`success` semantics | `success: false` when target returns HTTP 4xx/5xx with minimal content; `success: true` with `warning` when target returns error status but has real content
JS waiting | Numeric delay only; no selector-based wait primitive
`extract` format | Accepted as alias for `json`. Use `formats: ["json"]` with `jsonSchema` for structured extraction
SDKs | Raw HTTP examples only, no official language SDK package

Treat this page as the source of truth during migrations.

## Migration Checklist

If you are moving an existing Firecrawl-style integration:

1. verify `scrape`, `crawl`, and `map` request bodies against real targets,
2. confirm how your code interprets `success`, `warning`, and target-side HTTP statuses,
3. remove dependencies on unsupported features such as screenshots,
4. and compare output quality, not just endpoint shape.

Compatibility at the request level is useful, but output semantics and operational behavior matter just as much.

## Example Evaluation Workflow

A practical migration test usually looks like this:

1. take one production URL that already works in the old integration,
2. run it through scrape with the same high-level options,
3. compare markdown quality, warnings, and target status behavior,
4. repeat the test on one JS-heavy page and one failure-prone page,
5. only then update the calling code.

That sequence catches the gap between request-shape compatibility and output-quality compatibility.

## When Compatibility Is "Good Enough"

Compatibility is good enough when your migration goal is:

- preserving the existing mental model,
- keeping endpoint names and common options familiar,
- and minimizing changes in the application layer.

It is not good enough if your current system depends on features this page already marks as unsupported or behaviorally different.

## Common Mistakes

- Assuming route-name compatibility means output semantics are identical.
- Migrating a whole workload before testing `warning` handling and failure cases.
- Ignoring unsupported capabilities such as screenshots and then discovering the gap in production.

Use this page with output formats and error codes so your migration covers both payload shape and operational behavior.

`esc`