# CRW — open-source web scraper & crawler (Firecrawl-compatible)

> Docs at `docs.fastcrw.com`. Hosted API at `https://api.fastcrw.com`. Self-hosted at `http://localhost:3000`. Five endpoints: `/v1/search`, `/v1/scrape`, `/v1/crawl`, `/v1/map`, `/v1/extract`.

## Quick: copy this and run

Auth (hosted only): `Authorization: Bearer $CRW_API_KEY`. Self-hosted (`http://localhost:3000`) needs no header.

Body: `{ "query": string, "limit": number (default 5, max 20), "lang"?: "en", "tbs"?: "qdr:d|w|m|y", "sources"?: ["web"|"news"|"images"] }`

### Python (stdlib only — defaults to hosted, fails loudly if key is missing)

    #!/usr/bin/env python3
    """Search CRW and print top results as JSON."""
    import json, os, sys, urllib.request

    API_KEY = os.environ.get("CRW_API_KEY")
    # Default to the hosted API. Self-hosting? Set CRW_API_URL=http://localhost:3000
    BASE = os.environ.get("CRW_API_URL", "https://api.fastcrw.com")
    URL = f"{BASE}/v1/search"

    headers = {"Content-Type": "application/json"}
    if API_KEY:
        headers["Authorization"] = f"Bearer {API_KEY}"
    elif BASE.startswith("https://api.fastcrw.com"):
        sys.exit("error: CRW_API_KEY is required for the hosted API. "
                 "Get one at https://fastcrw.com (500 free credits) "
                 "or set CRW_API_URL=http://localhost:3000 to self-host.")

    req = urllib.request.Request(
        URL,
        data=json.dumps({"query": "renewable energy trends 2024", "limit": 3}).encode(),
        headers=headers,
        method="POST",
    )
    with urllib.request.urlopen(req) as r:
        body = json.loads(r.read())  # body == {"success": true, "data": [...]}

    # Results live under body["data"]. Each row has `title`, `url`, `snippet`,
    # `description`, `position`, `score`, `category` — print the LLM-ready subset.
    print(json.dumps(
        [{"title": x["title"], "url": x["url"], "snippet": x["snippet"]}
         for x in body["data"]],
        indent=2,
    ))
    # Run with: python3 search.py  (file is python3-only — no `python` alias needed)

### cURL

    curl -X POST https://api.fastcrw.com/v1/search \
      -H "Authorization: Bearer $CRW_API_KEY" \
      -H "Content-Type: application/json" \
      -d '{"query":"renewable energy trends 2024","limit":3}'

### CLI (one-shot LLM-ready output)

The `crw` CLI projects directly to the title/url/snippet schema with `--json --fields` — no `jq` reshape needed:

    # Install: brew install us/crw/crw  (or: cargo install crw-cli)
    crw search "renewable energy trends 2024" --json --fields title,url,snippet --limit 3

    # Available fields: title, url, description, snippet, position, score, category
    # --json is shorthand for --format json (also accepts --format text|markdown)
    # Self-hosted: no auth. Hosted: export CRW_API_KEY=crw_live_...

### Response shape

    {
      "success": true,
      "data": [
        {"url": "https://...", "title": "...",
         "description": "...", "snippet": "...",
         "position": 1, "score": 9.5}
      ]
    }

## Other endpoints

- `POST /v1/scrape` `{"url":"...","formats":["markdown"]}` — one URL → markdown
- `POST /v1/map` `{"url":"...","limit":50}` — discover URLs
- `POST /v1/crawl` `{"url":"...","maxPages":5}` — async multi-page; poll `GET /v1/crawl/{id}`
- `POST /v1/scrape` with `formats:["json"]` + `jsonOptions.schema` — structured extract

## Install (local setup only)

`pip install crw` · `npx crw-mcp` · `brew install us/crw/crw` · `curl -fsSL https://raw.githubusercontent.com/us/crw/main/install.sh | sh` · `docker run -p 3000:3000 ghcr.io/us/crw:latest`

## Resources

- OpenAPI spec
- llms.txt
- Context7
- MCP Server (npm)
- Typed Python SDK
- Agent Skills (SKILL.md)
- CLI (Homebrew tap)
- GitHub

# SDK Examples

## TypeScript

    const res = await fetch("https://fastcrw.com/api/v1/scrape", {
      method: "POST",
      headers: {
        Authorization: "Bearer YOUR_API_KEY",
        "Content-Type": "application/json",
      },
      body: JSON.stringify({ url: "https://example.com", formats: ["markdown"] }),
    });

    if (!res.ok) {
      throw new Error(`CRW error: ${res.status}`);
    }

    const payload = await res.json();
    console.log(payload.data?.markdown);

This is enough for most Node.js backends, server actions, and edge handlers.

## Python

    import requests

    res = requests.post(
        "https://fastcrw.com/api/v1/scrape",
        headers={
            "Authorization": "Bearer YOUR_API_KEY",
            "Content-Type": "application/json",
        },
        json={"url": "https://example.com", "formats": ["markdown"]},
    )

    res.raise_for_status()
    payload = res.json()
    print(payload["data"]["markdown"])

## Go

    package main

    import (
      "bytes"
      "fmt"
      "io"
      "net/http"
    )

    func main() {
      body := []byte(`{"url":"https://example.com","formats":["markdown"]}`)
      req, _ := http.NewRequest("POST", "https://fastcrw.com/api/v1/scrape", bytes.NewBuffer(body))
      req.Header.Set("Authorization", "Bearer YOUR_API_KEY")
      req.Header.Set("Content-Type", "application/json")

      res, err := http.DefaultClient.Do(req)
      if err != nil {
        panic(err)
      }
      defer res.Body.Close()

      b, _ := io.ReadAll(res.Body)
      fmt.Println(string(b))
    }

## Search

### TypeScript

    const res = await fetch("https://fastcrw.com/api/v1/search", {
      method: "POST",
      headers: {
        Authorization: "Bearer YOUR_API_KEY",
        "Content-Type": "application/json",
      },
      body: JSON.stringify({ query: "web scraping tools 2026", limit: 5 }),
    });

    if (!res.ok) {
      throw new Error(`CRW error: ${res.status}`);
    }

    const { data } = await res.json();

### Python

    resp = requests.post(
        "https://fastcrw.com/api/v1/search",
        headers={"Authorization": "Bearer YOUR_API_KEY"},
        json={"query": "web scraping tools 2026", "limit": 5},
    )

    resp.raise_for_status()
    data = resp.json()["data"]

### Go

    body := `{"query":"web scraping tools 2026","limit":5}`
    req, _ := http.NewRequest("POST", "https://fastcrw.com/api/v1/search",
        strings.NewReader(body))
    req.Header.Set("Authorization", "Bearer YOUR_API_KEY")
    req.Header.Set("Content-Type", "application/json")

    resp, err := http.DefaultClient.Do(req)
    if err != nil {
        panic(err)
    }
    defer resp.Body.Close()

### Search + Scrape

Add `scrapeOptions` to fetch page content for each result in one call:

    const res = await fetch("https://fastcrw.com/api/v1/search", {
      method: "POST",
      headers: {
        Authorization: "Bearer YOUR_API_KEY",
        "Content-Type": "application/json",
      },
      body: JSON.stringify({
        query: "machine learning papers",
        limit: 3,
        scrapeOptions: { formats: ["markdown"] },
      }),
    });

## Crawl Polling Pattern

Every language follows the same basic loop:

1. `POST /crawl` to start the job.
2. Read the returned crawl id.
3. `GET /crawl/:id` until the job reaches a terminal state.

That is why the examples here stay close to raw HTTP instead of pretending there is an official SDK package.

## When To Use Raw HTTP

Raw HTTP is the right default when you are:

- already inside a backend service that owns retry logic,
- calling CRW from a queue worker or cron job,
- or integrating through an environment where extra SDK abstraction does not buy much.

That includes server actions, background jobs, serverless functions, and internal APIs that just need a predictable request shape. The main advantage is operational clarity: what you send over the wire is exactly what CRW receives.

## Example: Start a Crawl and Poll It

The same shape works in every language:

    const start = await fetch("https://fastcrw.com/api/v1/crawl", {
      method: "POST",
      headers: {
        Authorization: "Bearer YOUR_API_KEY",
        "Content-Type": "application/json",
      },
      body: JSON.stringify({ url: "https://example.com/docs", limit: 10 }),
    });

    const { id } = await start.json();

    const status = await fetch(`https://fastcrw.com/api/v1/crawl/${id}`, {
      headers: { Authorization: "Bearer YOUR_API_KEY" },
    });

If your app already has a standard HTTP client with auth, tracing, and retries, adding a dedicated SDK layer too early usually just hides useful operational details.

## Common Mistakes

- Do not treat `response.ok` as the full success signal. You still need to inspect `warning` and `metadata.statusCode`.
- Do not hardcode retry timing for rate limits. Read `Retry-After` from the API response.
- Do not jump straight into crawl jobs before validating a single page through scrape or getting started.

## What To Read Next

- Use rate limits before adding parallel workers.
- Use error codes when you need machine-readable failure handling.
- Use output formats when you are deciding between markdown, html, and structured extraction output.

## Production Advice

- Load the API key from an environment variable.
- Respect `Retry-After` on `429`.
- Log `warning` and `metadata.statusCode`, not just the HTTP status.
- Start with `scrape` before wiring in `crawl` or extraction.

`esc`