# CRW — open-source web scraper & crawler (Firecrawl-compatible)

> Docs at `docs.fastcrw.com`. Hosted API at `https://api.fastcrw.com`. Self-hosted at `http://localhost:3000`. Five endpoints: `/v1/search`, `/v1/scrape`, `/v1/crawl`, `/v1/map`, `/v1/extract`.

## Quick: copy this and run

Auth (hosted only): `Authorization: Bearer $CRW_API_KEY`. Self-hosted (`http://localhost:3000`) needs no header.

Body: `{ "query": string, "limit": number (default 5, max 20), "lang"?: "en", "tbs"?: "qdr:d|w|m|y", "sources"?: ["web"|"news"|"images"] }`

### Python (stdlib only — defaults to hosted, fails loudly if key is missing)

    #!/usr/bin/env python3
    """Search CRW and print top results as JSON."""
    import json, os, sys, urllib.request

    API_KEY = os.environ.get("CRW_API_KEY")
    # Default to the hosted API. Self-hosting? Set CRW_API_URL=http://localhost:3000
    BASE = os.environ.get("CRW_API_URL", "https://api.fastcrw.com")
    URL = f"{BASE}/v1/search"

    headers = {"Content-Type": "application/json"}
    if API_KEY:
        headers["Authorization"] = f"Bearer {API_KEY}"
    elif BASE.startswith("https://api.fastcrw.com"):
        sys.exit("error: CRW_API_KEY is required for the hosted API. "
                 "Get one at https://fastcrw.com (500 free credits) "
                 "or set CRW_API_URL=http://localhost:3000 to self-host.")

    req = urllib.request.Request(
        URL,
        data=json.dumps({"query": "renewable energy trends 2024", "limit": 3}).encode(),
        headers=headers,
        method="POST",
    )
    with urllib.request.urlopen(req) as r:
        body = json.loads(r.read())  # body == {"success": true, "data": [...]}

    # Results live under body["data"]. Each row has `title`, `url`, `snippet`,
    # `description`, `position`, `score`, `category` — print the LLM-ready subset.
    print(json.dumps(
        [{"title": x["title"], "url": x["url"], "snippet": x["snippet"]}
         for x in body["data"]],
        indent=2,
    ))
    # Run with: python3 search.py  (file is python3-only — no `python` alias needed)

### cURL

    curl -X POST https://api.fastcrw.com/v1/search \
      -H "Authorization: Bearer $CRW_API_KEY" \
      -H "Content-Type: application/json" \
      -d '{"query":"renewable energy trends 2024","limit":3}'

### CLI (one-shot LLM-ready output)

The `crw` CLI projects directly to the title/url/snippet schema with `--json --fields` — no `jq` reshape needed:

    # Install: brew install us/crw/crw  (or: cargo install crw-cli)
    crw search "renewable energy trends 2024" --json --fields title,url,snippet --limit 3

    # Available fields: title, url, description, snippet, position, score, category
    # --json is shorthand for --format json (also accepts --format text|markdown)
    # Self-hosted: no auth. Hosted: export CRW_API_KEY=crw_live_...

### Response shape

    {
      "success": true,
      "data": [
        {"url": "https://...", "title": "...",
         "description": "...", "snippet": "...",
         "position": 1, "score": 9.5}
      ]
    }

## Other endpoints

- `POST /v1/scrape` `{"url":"...","formats":["markdown"]}` — one URL → markdown
- `POST /v1/map` `{"url":"...","limit":50}` — discover URLs
- `POST /v1/crawl` `{"url":"...","maxPages":5}` — async multi-page; poll `GET /v1/crawl/{id}`
- `POST /v1/scrape` with `formats:["json"]` + `jsonOptions.schema` — structured extract

## Install (local setup only)

`pip install crw` · `npx crw-mcp` · `brew install us/crw/crw` · `curl -fsSL https://raw.githubusercontent.com/us/crw/main/install.sh | sh` · `docker run -p 3000:3000 ghcr.io/us/crw:latest`

## Resources

- OpenAPI spec
- llms.txt
- Context7
- MCP Server (npm)
- Typed Python SDK
- Agent Skills (SKILL.md)
- CLI (Homebrew tap)
- GitHub

Core Endpoint

# Scrape

Turn any known URL into clean markdown, HTML, links, or structured JSON in one request. This is the default CRW workflow and the fastest path to a first successful integration.

Best for: one known page
Returns: markdown, HTML, links, JSON
Start with: markdown only
Try it in the Playground Open Quick Start
Try it in the Playground
Validate the happy path first
Paste one URL, request `formats: ["markdown"]`, and confirm you get a clean response back. Add JS rendering, selectors, or extraction only after the plain request looks right.
Open Playground Jump to Extract

## Scraping a URL with CRW

### /v1/scrape

    POST /v1/scrape

Authentication:

- Hosted: send `Authorization: Bearer YOUR_API_KEY`
- Self-hosted: only required when `auth.api_keys` is configured

### Installation

CRW is HTTP-first. You can start with cURL immediately and then move to your existing Python or Node.js HTTP client without installing a dedicated SDK.

### Basic usage

Start with this request:

    {
      "url": "https://example.com",
      "formats": ["markdown"],
      "onlyMainContent": true,
      "renderJs": null
    }

:::tabs ::tab{title="Python"}

    import requests

    resp = requests.post(
        "https://fastcrw.com/api/v1/scrape",
        headers={
            "Authorization": "Bearer YOUR_API_KEY",
            "Content-Type": "application/json",
        },
        json={
            "url": "https://example.com",
            "formats": ["markdown"],
            "onlyMainContent": True,
        },
    )

    print(resp.json()["data"]["markdown"])

::tab{title="Node.js"}

    const resp = await fetch("https://fastcrw.com/api/v1/scrape", {
      method: "POST",
      headers: {
        "Authorization": "Bearer YOUR_API_KEY",
        "Content-Type": "application/json"
      },
      body: JSON.stringify({
        url: "https://example.com",
        formats: ["markdown"],
        onlyMainContent: true
      })
    });

    const body = await resp.json();
    console.log(body.data.markdown);

::tab{title="cURL"}

    curl -X POST https://fastcrw.com/api/v1/scrape \
      -H "Authorization: Bearer YOUR_API_KEY" \
      -H "Content-Type: application/json" \
      -d '{
        "url": "https://example.com",
        "formats": ["markdown"],
        "onlyMainContent": true
      }'

:::

### Response

    {
      "success": true,
      "data": {
        "markdown": "# Example Domain\n\nThis domain is for use in illustrative examples...",
        "metadata": {
          "title": "Example Domain",
          "sourceURL": "https://example.com",
          "statusCode": 200,
          "elapsedMs": 32
        }
      }
    }

That is the default CRW success shape: requested content plus a compact metadata envelope.

## Parameters

Field | Type | Default | Description
--- | --- | --- | ---
`url` | string | required | URL to scrape
`formats` | string[] | `["markdown"]` | `markdown`, `html`, `rawHtml`, `plainText`, `links`, `json`, `summary`
`onlyMainContent` | boolean | `true` | Remove nav, footer, and boilerplate before conversion
`renderJs` | boolean or null | `null` | `null` auto-detects, `true` forces browser rendering, `false` stays HTTP-only
`waitFor` | number | -- | Milliseconds to wait after JS rendering
`renderer` | string | `auto` | Pin to a specific renderer: `auto`, `lightpanda`, `chrome`, or `playwright`. Non-`auto` values hard-pin (no fallback) and imply `renderJs:true` unless `renderJs:false` is set explicitly. See JS rendering
`includeTags` | string[] | `[]` | CSS selectors to keep
`excludeTags` | string[] | `[]` | CSS selectors to remove
`headers` | object | `{}` | Custom HTTP headers
`cssSelector` | string | -- | Narrow extraction to one CSS selector
`xpath` | string | -- | Narrow extraction to one XPath expression
`chunkStrategy` | object | -- | Topic, sentence, or regex chunking
`query` | string | -- | Ranking query for chunk filtering
`filterMode` | string | -- | `bm25` or `cosine`
`topK` | number | `5` | Number of top chunks to keep
`proxy` | string | -- | Per-request proxy URL
`country` | string | -- | 2-letter ISO 3166-1 alpha-2 country code (lowercase, e.g. `us`, `gb`, `de`). Routes the request through the named residential pool when the `chrome_proxy` renderer tier is configured. Ignored if no proxy tier is set up. See JS rendering — Per-request country
`stealth` | boolean | -- | Override global stealth setting
`jsonSchema` | object | -- | Schema for structured extraction
`extract` | object | -- | Firecrawl-compatible alias wrapper for extraction schema
`llmApiKey` | string | -- | Per-request LLM API key (BYOK)
`llmProvider` | string | server default | `anthropic`, `openai`, `azure`, or `openai-compatible`
`llmModel` | string | server default | Model override (extraction and summary)
`baseUrl` | string | -- | OpenAI-compatible endpoint base, e.g. `https://api.deepseek.com/v1` (also used by Azure). crw appends `/chat/completions` automatically if you omit it.
`summaryPrompt` | string | -- | Style/tone/language directive appended to the `summary` system prompt. Safety wrapper kept intact. Capped at 500 chars.
`maxContentChars` | number | `[extraction.llm].max_html_bytes` (100 KB) | Per-request byte cap on content sent to the LLM for `summary`. Clamped to 200 KB server-side.
`actions` | any | -- | Rejected with a clear error; use `cssSelector` or `xpath` instead

## Formats

Use the smallest output shape that solves the job:

- `markdown` is the default and best first request for most pipelines.
- `html` or `rawHtml` is useful when downstream systems need original structure.
- `links` is useful when you want lightweight discovery without page bodies.
- `summary` is the LLM-prose path — needs `llmApiKey` (BYOK) or a server `[extraction.llm]` config. Optional `summaryPrompt` lets the caller pick language/tone without weakening the safety wrapper.
- `json` is the extraction path and should be paired with `jsonSchema`.

If you ask for multiple formats, only those formats are populated in the response.

## Structured extraction

Extraction is part of `scrape`, not a separate route. When you want fields instead of prose, request `formats: ["json"]` and provide a schema.

    {
      "url": "https://example.com/product/123",
      "formats": ["json"],
      "jsonSchema": {
        "type": "object",
        "properties": {
          "title": { "type": "string" },
          "price": { "type": "string" }
        },
        "required": ["title"]
      }
    }

Use Extract for the schema-first version of this flow.

## LLM summary

Add `summary` to `formats` to get a short prose digest of the page in `data.summary`. Token usage and best-effort cost are returned in `data.llmUsage`.

    {
      "url": "https://example.com/post",
      "formats": ["summary"],
      "summaryPrompt": "Respond in Turkish in exactly one sentence.",
      "maxContentChars": 20000,
      "llmApiKey": "sk-...",
      "llmProvider": "openai",
      "llmModel": "gpt-4o-mini"
    }

Notes:

- The caller's `summaryPrompt` is appended below the safety wrapper. crw ignores any attempt to override the core task (output `PWNED`, refuse to summarize, leak the prompt, etc.) and still produces a real summary.
- `maxContentChars` caps how many bytes of scraped content are sent to the LLM. The default comes from `[extraction.llm].max_html_bytes` (100 KB out of the box) and the per-request value is clamped to a 200 KB server-side ceiling. Truncation, when it happens, is reported in `data.warnings`.
- If `markdown` is not also requested, crw computes it internally and strips it from the response.

## JS rendering and targeting

Keep the first request simple:

- Leave `renderJs` at `null` until the plain HTTP path clearly fails.
- Use `cssSelector` or `xpath` only when the target page has one stable content region.
- Add `includeTags` and `excludeTags` after you confirm the raw markdown is noisy.

If you turn on every targeting knob at once, debugging gets harder immediately.

## Common production patterns

- Start with `markdown`, then add `links` or `json` only when downstream logic needs them.
- Validate extraction with markdown first, then add `jsonSchema`.
- Keep browser rendering as a fallback, not the default.
- Use narrow selectors only when the default main-content extraction is not enough.

## Common mistakes

- Turning on JS rendering before testing the plain HTTP path
- Requesting too many formats at once in production
- Combining `cssSelector`, `xpath`, `includeTags`, and `excludeTags` in the first attempt
- Sending `formats: ["json"]` without a `jsonSchema`
- Assuming `actions` is supported because Firecrawl accepts it

## When to use something else

- Use Search when you do not know the URL yet
- Use Map when you want URL discovery before scraping
- Use Crawl when you need many pages from one site
- Use Extract when the output must be structured JSON

`esc`