# CRW — open-source web scraper & crawler (Firecrawl-compatible)

> Docs at `docs.fastcrw.com`. Hosted API at `https://api.fastcrw.com`. Self-hosted at `http://localhost:3000`. Five endpoints: `/v1/search`, `/v1/scrape`, `/v1/crawl`, `/v1/map`, `/v1/extract`.

## Quick: copy this and run

Auth (hosted only): `Authorization: Bearer $CRW_API_KEY`. Self-hosted (`http://localhost:3000`) needs no header.

Body: `{ "query": string, "limit": number (default 5, max 20), "lang"?: "en", "tbs"?: "qdr:d|w|m|y", "sources"?: ["web"|"news"|"images"] }`

### Python (stdlib only — defaults to hosted, fails loudly if key is missing)

    #!/usr/bin/env python3
    """Search CRW and print top results as JSON."""
    import json, os, sys, urllib.request

    API_KEY = os.environ.get("CRW_API_KEY")
    # Default to the hosted API. Self-hosting? Set CRW_API_URL=http://localhost:3000
    BASE = os.environ.get("CRW_API_URL", "https://api.fastcrw.com")
    URL = f"{BASE}/v1/search"

    headers = {"Content-Type": "application/json"}
    if API_KEY:
        headers["Authorization"] = f"Bearer {API_KEY}"
    elif BASE.startswith("https://api.fastcrw.com"):
        sys.exit("error: CRW_API_KEY is required for the hosted API. "
                 "Get one at https://fastcrw.com (500 free credits) "
                 "or set CRW_API_URL=http://localhost:3000 to self-host.")

    req = urllib.request.Request(
        URL,
        data=json.dumps({"query": "renewable energy trends 2024", "limit": 3}).encode(),
        headers=headers,
        method="POST",
    )
    with urllib.request.urlopen(req) as r:
        body = json.loads(r.read())  # body == {"success": true, "data": [...]}

    # Results live under body["data"]. Each row has `title`, `url`, `snippet`,
    # `description`, `position`, `score`, `category` — print the LLM-ready subset.
    print(json.dumps(
        [{"title": x["title"], "url": x["url"], "snippet": x["snippet"]}
         for x in body["data"]],
        indent=2,
    ))
    # Run with: python3 search.py  (file is python3-only — no `python` alias needed)

### cURL

    curl -X POST https://api.fastcrw.com/v1/search \
      -H "Authorization: Bearer $CRW_API_KEY" \
      -H "Content-Type: application/json" \
      -d '{"query":"renewable energy trends 2024","limit":3}'

### CLI (one-shot LLM-ready output)

The `crw` CLI projects directly to the title/url/snippet schema with `--json --fields` — no `jq` reshape needed:

    # Install: brew install us/crw/crw  (or: cargo install crw-cli)
    crw search "renewable energy trends 2024" --json --fields title,url,snippet --limit 3

    # Available fields: title, url, description, snippet, position, score, category
    # --json is shorthand for --format json (also accepts --format text|markdown)
    # Self-hosted: no auth. Hosted: export CRW_API_KEY=crw_live_...

### Response shape

    {
      "success": true,
      "data": [
        {"url": "https://...", "title": "...",
         "description": "...", "snippet": "...",
         "position": 1, "score": 9.5}
      ]
    }

## Other endpoints

- `POST /v1/scrape` `{"url":"...","formats":["markdown"]}` — one URL → markdown
- `POST /v1/map` `{"url":"...","limit":50}` — discover URLs
- `POST /v1/crawl` `{"url":"...","maxPages":5}` — async multi-page; poll `GET /v1/crawl/{id}`
- `POST /v1/scrape` with `formats:["json"]` + `jsonOptions.schema` — structured extract

## Install (local setup only)

`pip install crw` · `npx crw-mcp` · `brew install us/crw/crw` · `curl -fsSL https://raw.githubusercontent.com/us/crw/main/install.sh | sh` · `docker run -p 3000:3000 ghcr.io/us/crw:latest`

## Resources

- OpenAPI spec
- llms.txt
- Context7
- MCP Server (npm)
- Typed Python SDK
- Agent Skills (SKILL.md)
- CLI (Homebrew tap)
- GitHub

# JS Rendering

crw supports JavaScript rendering for single-page applications (SPAs) and JS-heavy sites via the Chrome DevTools Protocol (CDP).

## Rendering Modes

Set the rendering mode in your config:

    [renderer]
    mode = "auto"          # auto | lightpanda | playwright | chrome | none
    page_timeout_ms = 30000
    pool_size = 4

Mode | Behavior
--- | ---
`auto` | Use HTTP first, detect SPAs, fall back to CDP if needed
`lightpanda` | Always use LightPanda for JS rendering
`playwright` | Always use Playwright for JS rendering
`chrome` | Always use Chrome for JS rendering
`none` | HTTP only, never render JS

## Per-request control

Override the rendering mode per request using `renderJs`:

    {
      "url": "https://spa-app.com",
      "renderJs": true,
      "waitFor": 3000
    }

Value | Behavior
--- | ---
`null` (default) | Auto-detect based on heuristics, or fall back to the global `render_js_default` if set
`true` | Force CDP rendering
`false` | HTTP only

## Global default

To force JS rendering for every request that doesn't specify `renderJs` explicitly, set `render_js_default` in your config:

    [renderer]
    mode = "chrome"
    render_js_default = true   # alias: force_js = true

    # Or via environment variables
    CRW_RENDERER__MODE=chrome
    CRW_RENDERER__RENDER_JS_DEFAULT=true
    # Backward-compat alias:
    CRW_RENDERER__FORCE_JS=true

Precedence: a per-request `renderJs` always wins over the global default. Unset (`null`) on the request falls back to the default; if the default is also unset, the auto-detection heuristics below apply.

## Per-request renderer override

When `mode = "auto"` and you have multiple renderers configured (e.g., LightPanda + Chrome), the auto-detect chain decides which one to use. Sometimes you already know that a specific site needs Chrome (Cloudflare-protected SPAs) or LightPanda (fast static-JS sites). Pin the renderer per request with the `renderer` field:

    {
      "url": "https://x.com/elonmusk",
      "renderer": "chrome"
    }

Value | Behavior
--- | ---
omitted / `auto` | Use the configured fallback chain (existing behavior)
`lightpanda` | Hard-pin to LightPanda — no fallback
`chrome` | Hard-pin to Chrome — no fallback
`playwright` | Hard-pin to Playwright — no fallback

### Pinned implies JS

A non-`auto` `renderer` value implies `renderJs:true`. If you set `renderJs:false` explicitly, the request stays HTTP-only and the pin is silently ignored — `renderJs:false` always wins. This means the availability check is also skipped when `renderJs:false` is set, so combinations like `{"mode":"none","renderJs":false,"renderer":"chrome"}` are accepted.

### Errors and validation

If the named renderer isn't available in the server's pool, the request returns HTTP 400 immediately with `errorCode: "invalid_request"` and a message listing the configured renderers:

    {
      "success": false,
      "error": "renderer 'chrome' not available; configured renderers: [lightpanda]. Update server config or omit the 'renderer' field.",
      "errorCode": "invalid_request"
    }

For `/v1/crawl`, this validation runs once at job acceptance — bad combinations return 400 before the job is queued.

### Pinning reduces resilience

Hard-pinning a renderer means transient failures of that renderer surface as errors instead of silently falling back to HTTP. If you need maximum resilience, omit `renderer` (or set it to `auto`) so the auto-detect chain can fall back. If you need determinism — "I know this site needs Chrome and a LightPanda success would be a wrong answer" — pin it.

For crawls, per-page failures of a pinned renderer are still logged and skipped; the rest of the crawl continues. Pages that fail will not appear in the results.

## Per-request country (residential proxy tier)

When the operator configures a `chrome_proxy` tier backed by a residential proxy provider (e.g. DataImpulse), each request can route through a specific country by setting `country` in the request body:

    {
      "url": "https://geo-restricted.example.com",
      "country": "us"
    }

Value | Behavior
--- | ---
omitted | Uses `CRW_RENDERER__PROXY_DEFAULT_COUNTRY` if configured, otherwise the provider's global pool
2-letter code (`us`, `gb`, `de`, `fr`, `jp`, …) | Lowercased and appended to the base proxy username as a country selector
Invalid (non-alpha, wrong length) | Silently ignored — falls back to the default country, then the global pool

The country selector is composed at request time and supplied to Chrome via the CDP `Fetch.authRequired` event — no per-country container restart needed, and concurrent requests can target different countries without cross-contamination.

This field has no effect when the `chrome_proxy` tier is not configured. It is a renderer-tier routing hint, not a content filter — the engine does not translate or localize the response.

See Configuration — Residential proxy tier for the operator-side env vars (`CRW_RENDERER__PROXY_BASE_USER`, `CRW_RENDERER__PROXY_BASE_PASS`, `CRW_RENDERER__PROXY_DEFAULT_COUNTRY`).

## When To Turn It On

Enable JS rendering when the page content is not present in the initial HTML response. Typical examples:

- single-page applications,
- pages that fetch content after hydration,
- and sites where the meaningful body is assembled client-side.

Do not enable it blindly for every request. HTTP-only fetches are faster and cheaper.

## Choosing a `waitFor` Value

Start with the smallest value that works:

- `500` to `1000` for lightly hydrated pages,
- `2000` for typical JS-heavy pages,
- `3000` to `5000` only when you have confirmed the target hydrates slowly.

:::warning Long waits are not automatically safer. They increase latency and can hide the fact that the page is blocked rather than merely slow. :::

## Auto-detection

When `renderJs` is `null`, crw fetches the page via HTTP first, then checks for SPA signals:

Triggered when body text < 200 chars and:

- Contains `id="root"`, `id="app"`, `id="__next"`, `id="__nuxt"`, `id="__gatsby"`, `id="svelte"`
- Contains `ng-app`, `data-reactroot`
- Contains `window.__initial_state__`, `__next_data__`, `window.__remixcontext`, `window.__astro`

Triggered when:

- `<noscript>` contains "enable javascript"
- Body text < 500 chars and URL matches Framer, Webflow, Wix, or Squarespace domains

## CDP Backends

### LightPanda (Recommended)

Fastest option. Lightweight browser engine purpose-built for scraping.

    # Auto-install
    crw-server setup

    # Manual start
    lightpanda serve --host 127.0.0.1 --port 9222 &

    [renderer.lightpanda]
    ws_url = "ws://127.0.0.1:9222/"

### Playwright

    [renderer.playwright]
    ws_url = "ws://playwright:9222"

### Chrome / Chromium

    [renderer.chrome]
    ws_url = "ws://chrome:9222"

## How CDP rendering works

1. Open a new browser tab via `Target.createTarget`
2. Navigate to the URL
3. Attach to the target via `Target.attachToTarget`
4. Wait for the configured time (`waitFor` or default 2000ms)
5. Execute `Runtime.evaluate("document.documentElement.outerHTML")` to get rendered HTML
6. Close the tab via `Target.closeTarget`

## Cloud vs Self-hosted

- Cloud: JS rendering is always available. The managed infrastructure runs a LightPanda sidecar alongside the engine. Available on fastcrw.com (cloud).
- Self-hosted: You must run `crw-server setup` or configure a CDP browser (LightPanda, Chrome, or Playwright) in your `config.toml` under `[renderer]`. If no JS renderer is configured, requests with `renderJs: true` will fall back to HTTP-only fetching and include a warning.

## What To Inspect in the Response

When rendered output looks wrong, check:

- `metadata.renderedWith` to verify a browser was actually used,
- `metadata.elapsedMs` to understand the cost of the request,
- and `warning` to catch anti-bot or fallback situations.

## Troubleshooting

- Empty content from JS-heavy sites: Increase `waitFor` (e.g., `3000`-`5000`). Some SPAs need extra time to hydrate.
- `renderedWith: "http_only_fallback"` in metadata: JS rendering was requested but no renderer is available. Check your deployment configuration.
- Internal error on `renderJs: true`: Verify the LightPanda sidecar is running and reachable. Check `/health` for renderer status.
- Still poor output after increasing `waitFor`: The issue may be anti-bot protection or authentication flow, not rendering delay.

## Docker Compose

The included `docker-compose.yml` runs crw with a LightPanda sidecar:

    services:
      crw:
        image: ghcr.io/us/crw:latest
        ports:
          - "3000:3000"
        environment:
          - CRW_RENDERER__LIGHTPANDA__WS_URL=ws://lightpanda:9222
      lightpanda:
        image: lightpanda/lightpanda:latest
        command: ["serve", "--host", "0.0.0.0", "--port", "9222"]

`esc`