# CRW — open-source web scraper & crawler (Firecrawl-compatible)

> Docs at `docs.fastcrw.com`. Hosted API at `https://api.fastcrw.com`. Self-hosted at `http://localhost:3000`. Five endpoints: `/v1/search`, `/v1/scrape`, `/v1/crawl`, `/v1/map`, `/v1/extract`.

## Quick: copy this and run

Auth (hosted only): `Authorization: Bearer $CRW_API_KEY`. Self-hosted (`http://localhost:3000`) needs no header.

Body: `{ "query": string, "limit": number (default 5, max 20), "lang"?: "en", "tbs"?: "qdr:d|w|m|y", "sources"?: ["web"|"news"|"images"] }`

### Python (stdlib only — defaults to hosted, fails loudly if key is missing)

    #!/usr/bin/env python3
    """Search CRW and print top results as JSON."""
    import json, os, sys, urllib.request

    API_KEY = os.environ.get("CRW_API_KEY")
    # Default to the hosted API. Self-hosting? Set CRW_API_URL=http://localhost:3000
    BASE = os.environ.get("CRW_API_URL", "https://api.fastcrw.com")
    URL = f"{BASE}/v1/search"

    headers = {"Content-Type": "application/json"}
    if API_KEY:
        headers["Authorization"] = f"Bearer {API_KEY}"
    elif BASE.startswith("https://api.fastcrw.com"):
        sys.exit("error: CRW_API_KEY is required for the hosted API. "
                 "Get one at https://fastcrw.com (500 free credits) "
                 "or set CRW_API_URL=http://localhost:3000 to self-host.")

    req = urllib.request.Request(
        URL,
        data=json.dumps({"query": "renewable energy trends 2024", "limit": 3}).encode(),
        headers=headers,
        method="POST",
    )
    with urllib.request.urlopen(req) as r:
        body = json.loads(r.read())  # body == {"success": true, "data": [...]}

    # Results live under body["data"]. Each row has `title`, `url`, `snippet`,
    # `description`, `position`, `score`, `category` — print the LLM-ready subset.
    print(json.dumps(
        [{"title": x["title"], "url": x["url"], "snippet": x["snippet"]}
         for x in body["data"]],
        indent=2,
    ))
    # Run with: python3 search.py  (file is python3-only — no `python` alias needed)

### cURL

    curl -X POST https://api.fastcrw.com/v1/search \
      -H "Authorization: Bearer $CRW_API_KEY" \
      -H "Content-Type: application/json" \
      -d '{"query":"renewable energy trends 2024","limit":3}'

### CLI (one-shot LLM-ready output)

The `crw` CLI projects directly to the title/url/snippet schema with `--json --fields` — no `jq` reshape needed:

    # Install: brew install us/crw/crw  (or: cargo install crw-cli)
    crw search "renewable energy trends 2024" --json --fields title,url,snippet --limit 3

    # Available fields: title, url, description, snippet, position, score, category
    # --json is shorthand for --format json (also accepts --format text|markdown)
    # Self-hosted: no auth. Hosted: export CRW_API_KEY=crw_live_...

### Response shape

    {
      "success": true,
      "data": [
        {"url": "https://...", "title": "...",
         "description": "...", "snippet": "...",
         "position": 1, "score": 9.5}
      ]
    }

## Other endpoints

- `POST /v1/scrape` `{"url":"...","formats":["markdown"]}` — one URL → markdown
- `POST /v1/map` `{"url":"...","limit":50}` — discover URLs
- `POST /v1/crawl` `{"url":"...","maxPages":5}` — async multi-page; poll `GET /v1/crawl/{id}`
- `POST /v1/scrape` with `formats:["json"]` + `jsonOptions.schema` — structured extract

## Install (local setup only)

`pip install crw` · `npx crw-mcp` · `brew install us/crw/crw` · `curl -fsSL https://raw.githubusercontent.com/us/crw/main/install.sh | sh` · `docker run -p 3000:3000 ghcr.io/us/crw:latest`

## Resources

- OpenAPI spec
- llms.txt
- Context7
- MCP Server (npm)
- Typed Python SDK
- Agent Skills (SKILL.md)
- CLI (Homebrew tap)
- GitHub

# Credit Costs

:::note Cloud only (fastcrw.com) -- self-hosted instances do not have credit-based billing. :::

## Current billing rules

Operation | Credit cost
--- | ---
`scrape` | 1 credit
`map` | 1 credit
`crawl` start | 1 credit
`crawl` polling | New pages discovered since the previous poll
`search` | 1 credit
`search` + scrape | 1 credit + 1 per scraped result
`browse` session | 1 credit (planned cloud rate; the self-hosted `crw-browse` binary is free)

## Why crawl billing looks different

The crawl start reserves the job. Subsequent polls charge only for newly materialized pages, not for the total accumulated page count each time.

That prevents the same already-seen pages from being charged again and again just because you are checking progress.

## Simple Examples

Scenario | Credit effect
--- | ---
One `scrape` request | 1 credit
One `map` request | 1 credit
Start one crawl job | 1 credit
Poll a crawl and receive 7 new pages | 7 additional credits
Poll again with no new pages | No new page credits
Search for "AI tools" with 5 results | 1 credit
Search + scrape 3 results | 1 + 3 = 4 credits
Search + scrape, 1 scrape fails | 1 + 2 = 3 credits (failed scrape refunded)

## What Usually Does Not Consume Permanent Credits

The billing logic is designed to avoid charging you for requests that never become real usable work. Validation failures and certain upstream failures are refunded rather than treated like successful paid execution.

The safest way to confirm actual consumption is still the balance endpoint before and after a test.

## Balance check

Use `GET /api/v1/account/balance` (cloud only — fastcrw.com) with your API key to inspect included credits, purchased balance, and total available credits.

## Example Monitoring Pattern

A simple integration-safe pattern is:

1. read balance before a new workflow rollout,
2. run a bounded test batch,
3. read balance again,
4. compare expected consumption with actual consumption.

That is especially useful for crawl jobs because the start request and the page-materialization charges happen at different times.

## When To Watch Credits Closely

Watch credits closely when:

- you are polling crawl jobs at high frequency,
- many workers share one account balance,
- or you are benchmarking output quality across multiple target sites.

If request rate is the only thing you monitor, you can still be surprised by crawl-heavy usage. Pair billing checks with rate limits so throughput and credit consumption are interpreted together.

## Operational Advice

- Use small test batches before large crawls.
- Check balance before and after integration changes.
- Separate "request volume" monitoring from "credit consumption" monitoring; they are related but not identical.

## Common Mistakes

- Assuming every crawl poll re-bills the full job instead of only newly materialized pages.
- Launching large crawls before validating cost on a much smaller limit.
- Treating validation failures and refunded work as if they were successful billable jobs.

`esc`