# CRW — open-source web scraper & crawler (Firecrawl-compatible)

> Docs at `docs.fastcrw.com`. Hosted API at `https://api.fastcrw.com`. Self-hosted at `http://localhost:3000`. Five endpoints: `/v1/search`, `/v1/scrape`, `/v1/crawl`, `/v1/map`, `/v1/extract`.

## Quick: copy this and run

Auth (hosted only): `Authorization: Bearer $CRW_API_KEY`. Self-hosted (`http://localhost:3000`) needs no header.

Body: `{ "query": string, "limit": number (default 5, max 20), "lang"?: "en", "tbs"?: "qdr:d|w|m|y", "sources"?: ["web"|"news"|"images"] }`

### Python (stdlib only — defaults to hosted, fails loudly if key is missing)

    #!/usr/bin/env python3
    """Search CRW and print top results as JSON."""
    import json, os, sys, urllib.request

    API_KEY = os.environ.get("CRW_API_KEY")
    # Default to the hosted API. Self-hosting? Set CRW_API_URL=http://localhost:3000
    BASE = os.environ.get("CRW_API_URL", "https://api.fastcrw.com")
    URL = f"{BASE}/v1/search"

    headers = {"Content-Type": "application/json"}
    if API_KEY:
        headers["Authorization"] = f"Bearer {API_KEY}"
    elif BASE.startswith("https://api.fastcrw.com"):
        sys.exit("error: CRW_API_KEY is required for the hosted API. "
                 "Get one at https://fastcrw.com (500 free credits) "
                 "or set CRW_API_URL=http://localhost:3000 to self-host.")

    req = urllib.request.Request(
        URL,
        data=json.dumps({"query": "renewable energy trends 2024", "limit": 3}).encode(),
        headers=headers,
        method="POST",
    )
    with urllib.request.urlopen(req) as r:
        body = json.loads(r.read())  # body == {"success": true, "data": [...]}

    # Results live under body["data"]. Each row has `title`, `url`, `snippet`,
    # `description`, `position`, `score`, `category` — print the LLM-ready subset.
    print(json.dumps(
        [{"title": x["title"], "url": x["url"], "snippet": x["snippet"]}
         for x in body["data"]],
        indent=2,
    ))
    # Run with: python3 search.py  (file is python3-only — no `python` alias needed)

### cURL

    curl -X POST https://api.fastcrw.com/v1/search \
      -H "Authorization: Bearer $CRW_API_KEY" \
      -H "Content-Type: application/json" \
      -d '{"query":"renewable energy trends 2024","limit":3}'

### CLI (one-shot LLM-ready output)

The `crw` CLI projects directly to the title/url/snippet schema with `--json --fields` — no `jq` reshape needed:

    # Install: brew install us/crw/crw  (or: cargo install crw-cli)
    crw search "renewable energy trends 2024" --json --fields title,url,snippet --limit 3

    # Available fields: title, url, description, snippet, position, score, category
    # --json is shorthand for --format json (also accepts --format text|markdown)
    # Self-hosted: no auth. Hosted: export CRW_API_KEY=crw_live_...

### Response shape

    {
      "success": true,
      "data": [
        {"url": "https://...", "title": "...",
         "description": "...", "snippet": "...",
         "position": 1, "score": 9.5}
      ]
    }

## Other endpoints

- `POST /v1/scrape` `{"url":"...","formats":["markdown"]}` — one URL → markdown
- `POST /v1/map` `{"url":"...","limit":50}` — discover URLs
- `POST /v1/crawl` `{"url":"...","maxPages":5}` — async multi-page; poll `GET /v1/crawl/{id}`
- `POST /v1/scrape` with `formats:["json"]` + `jsonOptions.schema` — structured extract

## Install (local setup only)

`pip install crw` · `npx crw-mcp` · `brew install us/crw/crw` · `curl -fsSL https://raw.githubusercontent.com/us/crw/main/install.sh | sh` · `docker run -p 3000:3000 ghcr.io/us/crw:latest`

## Resources

- OpenAPI spec
- llms.txt
- Context7
- MCP Server (npm)
- Typed Python SDK
- Agent Skills (SKILL.md)
- CLI (Homebrew tap)
- GitHub

# Errors and Warnings

## Response semantics

- `success: true` means the pipeline produced usable page content.
- `success: false` means the request failed -- either an engine error or the target returned an error status (4xx/5xx) with minimal content.
- `error` is present when `success: false`. It describes what went wrong.
- `warning` flags degraded target outcomes (anti-bot pages, problematic status codes) when `success: true` -- meaning content was produced but may be incomplete.
- `metadata.statusCode` is the target site's HTTP status.
- `data` may still be present when `success: false` if partial content was extracted.

The important distinction is that there are two layers of status:

- the HTTP status returned by CRW,
- and the target site's own status exposed through `metadata.statusCode`.

You need both to debug real scraping failures.

## HTTP status codes returned by the API

Status | Meaning
--- | ---
200 | Success
400 | Invalid request parameters (bad URL, invalid JSON body, invalid selector)
401 | Invalid or missing API key
404 | Endpoint not found
422 | Validation failed (unknown format, invalid schema, extraction error)
429 | Rate limit or credit quota exceeded
502 | Engine internal error
503 | Server at capacity
504 | Request timed out

## How To Read Common Cases

Situation | What it usually means | What to do next
--- | --- | ---
HTTP `200` with `warning` | The request succeeded, but the target result is degraded | Inspect `warning` and `metadata.statusCode`
HTTP `400` | Your request body is invalid | Fix fields, selectors, or schema
HTTP `422` | The request shape is valid JSON but semantically invalid | Check format names, schema, or extraction config
HTTP `429` | Rate limit or credit ceiling hit | Back off and honor `Retry-After`
HTTP `502` / `504` | Upstream or timeout issue | Retry with backoff

## Common engine errors

Error | When it happens
--- | ---
Invalid URL | URL is malformed or targets a blocked address
Invalid selector | CSS selector or XPath expression cannot be parsed
Renderer timeout | JS rendering exceeded the page timeout
Navigation failed | CDP browser could not load the page
Response too large | Page exceeded the maximum allowed size
Invalid JSON schema | Schema provided for extraction is malformed
Extraction failure | LLM extraction failed (no LLM configured, or LLM returned an error)
`summary` format requires LLM config | `formats: ["summary"]` sent without a server-side `[extraction.llm]` block and without per-request `llmApiKey`
BYOK header required | Server is in BYOK-only mode (`require_byok_header`) and the request lacks both the header and `llmApiKey`
No JS renderer available | `renderJs: true` but no CDP browser is configured

## Common warnings

Warning | When it appears
--- | ---
`Target returned 403 Forbidden` | Target site blocked the request
`Target returned 429 Too Many Requests` | Target site rate limited the request
`Blocked by anti-bot protection` | Page contains Cloudflare/captcha markers
`JS rendering was requested but no renderer is available` | Fallback to HTTP-only fetch
`content truncated to N bytes before summarization` | Scraped content exceeded `maxContentChars` (or `[extraction.llm].max_html_bytes`); the head of the page was summarized
`summary failed: <reason>` | A per-result summary in `/v1/search` failed (e.g. provider 429). Other results are still returned
`answer synthesis failed: <reason>` | The answer call in `/v1/search` failed. `answer` is `null` but `summaries` and `results` are still returned
`N of M per-result summaries failed` | Some but not all per-result summaries failed in `/v1/search`

## Retry Guidance

Retrying helps only for some classes of failure:

- retry `429`, `502`, and `504` with backoff,
- do not blindly retry `400` or `422`,
- and treat repeated warnings from the same domain as a target compatibility problem, not a random transient issue.

If a page is repeatedly blocked by anti-bot protection, longer retry loops usually make the situation worse rather than better.

`esc`