# CRW — open-source web scraper & crawler (Firecrawl-compatible)

> Docs at `docs.fastcrw.com`. Hosted API at `https://api.fastcrw.com`. Self-hosted at `http://localhost:3000`. Five endpoints: `/v1/search`, `/v1/scrape`, `/v1/crawl`, `/v1/map`, `/v1/extract`.

## Quick: copy this and run

Auth (hosted only): `Authorization: Bearer $CRW_API_KEY`. Self-hosted (`http://localhost:3000`) needs no header.

Body: `{ "query": string, "limit": number (default 5, max 20), "lang"?: "en", "tbs"?: "qdr:d|w|m|y", "sources"?: ["web"|"news"|"images"] }`

### Python (stdlib only — defaults to hosted, fails loudly if key is missing)

    #!/usr/bin/env python3
    """Search CRW and print top results as JSON."""
    import json, os, sys, urllib.request

    API_KEY = os.environ.get("CRW_API_KEY")
    # Default to the hosted API. Self-hosting? Set CRW_API_URL=http://localhost:3000
    BASE = os.environ.get("CRW_API_URL", "https://api.fastcrw.com")
    URL = f"{BASE}/v1/search"

    headers = {"Content-Type": "application/json"}
    if API_KEY:
        headers["Authorization"] = f"Bearer {API_KEY}"
    elif BASE.startswith("https://api.fastcrw.com"):
        sys.exit("error: CRW_API_KEY is required for the hosted API. "
                 "Get one at https://fastcrw.com (500 free credits) "
                 "or set CRW_API_URL=http://localhost:3000 to self-host.")

    req = urllib.request.Request(
        URL,
        data=json.dumps({"query": "renewable energy trends 2024", "limit": 3}).encode(),
        headers=headers,
        method="POST",
    )
    with urllib.request.urlopen(req) as r:
        body = json.loads(r.read())  # body == {"success": true, "data": [...]}

    # Results live under body["data"]. Each row has `title`, `url`, `snippet`,
    # `description`, `position`, `score`, `category` — print the LLM-ready subset.
    print(json.dumps(
        [{"title": x["title"], "url": x["url"], "snippet": x["snippet"]}
         for x in body["data"]],
        indent=2,
    ))
    # Run with: python3 search.py  (file is python3-only — no `python` alias needed)

### cURL

    curl -X POST https://api.fastcrw.com/v1/search \
      -H "Authorization: Bearer $CRW_API_KEY" \
      -H "Content-Type: application/json" \
      -d '{"query":"renewable energy trends 2024","limit":3}'

### CLI (one-shot LLM-ready output)

The `crw` CLI projects directly to the title/url/snippet schema with `--json --fields` — no `jq` reshape needed:

    # Install: brew install us/crw/crw  (or: cargo install crw-cli)
    crw search "renewable energy trends 2024" --json --fields title,url,snippet --limit 3

    # Available fields: title, url, description, snippet, position, score, category
    # --json is shorthand for --format json (also accepts --format text|markdown)
    # Self-hosted: no auth. Hosted: export CRW_API_KEY=crw_live_...

### Response shape

    {
      "success": true,
      "data": [
        {"url": "https://...", "title": "...",
         "description": "...", "snippet": "...",
         "position": 1, "score": 9.5}
      ]
    }

## Other endpoints

- `POST /v1/scrape` `{"url":"...","formats":["markdown"]}` — one URL → markdown
- `POST /v1/map` `{"url":"...","limit":50}` — discover URLs
- `POST /v1/crawl` `{"url":"...","maxPages":5}` — async multi-page; poll `GET /v1/crawl/{id}`
- `POST /v1/scrape` with `formats:["json"]` + `jsonOptions.schema` — structured extract

## Install (local setup only)

`pip install crw` · `npx crw-mcp` · `brew install us/crw/crw` · `curl -fsSL https://raw.githubusercontent.com/us/crw/main/install.sh | sh` · `docker run -p 3000:3000 ghcr.io/us/crw:latest`

## Resources

- OpenAPI spec
- llms.txt
- Context7
- MCP Server (npm)
- Typed Python SDK
- Agent Skills (SKILL.md)
- CLI (Homebrew tap)
- GitHub

Integrations

# MCP Client Setup

Add CRW to Claude Code, Codex, Cursor, Windsurf, Cline, Continue, and similar MCP hosts. This page is the copy-paste setup reference for both local embedded mode and fastcrw.com cloud mode.

Best for: host-by-host setup
Local mode: `npx crw-mcp`
Cloud mode: `CRW_API_URL + CRW_API_KEY`
Open MCP Overview Get fastcrw.com API Key

## Start here

Use one of these three patterns:

Pattern | Best when | Copy-paste shape
--- | --- | ---
Embedded local | You want zero server setup and local scrape/map/crawl tools | `command: npx`, `args: ["crw-mcp"]`
fastcrw.com cloud | You want hosted infrastructure and `crw_search` | same as local plus `CRW_API_URL` and `CRW_API_KEY`
HTTP transport | Your host supports HTTP MCP and you already run `crw-server` | point the client at `http://localhost:3000/mcp`
Browser automation | You need a real, stateful browser (click, nav, tree) | `command: crw-browse` (separate MCP server — self-hosted)

## What tools you get

Embedded local mode (`crw-mcp`) exposes:

- `crw_scrape`
- `crw_crawl`
- `crw_check_crawl_status`
- `crw_map`

fastcrw.com cloud mode exposes all of the above plus:

- `crw_search`

Browser automation mode (`crw-browse`, separate server — v0.4.0+) exposes:

- `goto` — navigate the browser to an URL
- `tree` — accessibility snapshot of the current page

:::tip If you only remember one rule, remember this one: local embedded mode is the easiest setup, and fastcrw.com cloud mode is the easiest way to add web search. :::

## Claude Code

### Local embedded

    claude mcp add crw -- npx crw-mcp

### fastcrw.com cloud

    claude mcp add \
      -e CRW_API_URL=https://fastcrw.com/api \
      -e CRW_API_KEY=YOUR_API_KEY \
      crw -- npx crw-mcp

### Local HTTP transport

    claude mcp add --transport http crw http://localhost:3000/mcp

Useful commands:

- `claude mcp list`
- `claude mcp remove crw`

## OpenAI Codex CLI

### Local embedded

    codex mcp add crw -- npx crw-mcp

Or edit `~/.codex/config.toml`:

    [mcp_servers.crw]
    command = "npx"
    args = ["crw-mcp"]

### fastcrw.com cloud

    [mcp_servers.crw]
    command = "npx"
    args = ["crw-mcp"]

    [mcp_servers.crw.env]
    CRW_API_URL = "https://fastcrw.com/api"
    CRW_API_KEY = "YOUR_API_KEY"

:::note Codex uses the same stdio MCP server as Claude Code. The only difference is where you register it. :::

## Claude Desktop

Edit the config file for your OS:

OS | Path
--- | ---
macOS | `~/Library/Application Support/Claude/claude_desktop_config.json`
Windows | `%APPDATA%/Claude/claude_desktop_config.json`
Linux | `~/.config/Claude/claude_desktop_config.json`

### Local embedded

    {
      "mcpServers": {
        "crw": {
          "command": "npx",
          "args": ["crw-mcp"]
        }
      }
    }

### fastcrw.com cloud

    {
      "mcpServers": {
        "crw": {
          "command": "npx",
          "args": ["crw-mcp"],
          "env": {
            "CRW_API_URL": "https://fastcrw.com/api",
            "CRW_API_KEY": "YOUR_API_KEY"
          }
        }
      }
    }

Fully quit and restart Claude Desktop after editing the file.

## Cursor

Create or edit `~/.cursor/mcp.json` for global setup, or `.cursor/mcp.json` inside one project.

### Local embedded

    {
      "mcpServers": {
        "crw": {
          "command": "npx",
          "args": ["crw-mcp"]
        }
      }
    }

### fastcrw.com cloud

    {
      "mcpServers": {
        "crw": {
          "command": "npx",
          "args": ["crw-mcp"],
          "env": {
            "CRW_API_URL": "https://fastcrw.com/api",
            "CRW_API_KEY": "YOUR_API_KEY"
          }
        }
      }
    }

You can also add the same config from Cursor's MCP UI under Settings.

## Windsurf

Edit `~/.codeium/windsurf/mcp_config.json`.

### Local embedded

    {
      "mcpServers": {
        "crw": {
          "command": "npx",
          "args": ["crw-mcp"]
        }
      }
    }

### fastcrw.com cloud

    {
      "mcpServers": {
        "crw": {
          "command": "npx",
          "args": ["crw-mcp"],
          "env": {
            "CRW_API_URL": "https://fastcrw.com/api",
            "CRW_API_KEY": "YOUR_API_KEY"
          }
        }
      }
    }

:::note Windsurf has a total MCP tool limit. CRW stays lightweight: local mode exposes 4 tools, cloud mode exposes 5. :::

## Cline

Edit the Cline MCP settings file:

OS | Path
--- | ---
macOS | `~/Library/Application Support/Code/User/globalStorage/saoudrizwan.claude-dev/settings/cline_mcp_settings.json`
Windows | `%APPDATA%/Code/User/globalStorage/saoudrizwan.claude-dev/settings/cline_mcp_settings.json`
Linux | `~/.config/Code/User/globalStorage/saoudrizwan.claude-dev/settings/cline_mcp_settings.json`

### Local embedded

    {
      "mcpServers": {
        "crw": {
          "command": "npx",
          "args": ["crw-mcp"],
          "alwaysAllow": ["crw_scrape", "crw_map"],
          "disabled": false
        }
      }
    }

### fastcrw.com cloud

    {
      "mcpServers": {
        "crw": {
          "command": "npx",
          "args": ["crw-mcp"],
          "alwaysAllow": ["crw_scrape", "crw_map", "crw_search"],
          "disabled": false,
          "env": {
            "CRW_API_URL": "https://fastcrw.com/api",
            "CRW_API_KEY": "YOUR_API_KEY"
          }
        }
      }
    }

## Continue

Edit `~/.continue/config.yaml`.

### Local embedded

    mcpServers:
      - name: crw
        command: npx
        args:
          - crw-mcp

### fastcrw.com cloud

    mcpServers:
      - name: crw
        command: npx
        args:
          - crw-mcp
        env:
          CRW_API_URL: https://fastcrw.com/api
          CRW_API_KEY: YOUR_API_KEY

MCP tools in Continue only work in Agent mode, not plain chat mode.

## Gemini CLI

### Local embedded

    gemini mcp add crw -- npx crw-mcp

Or edit `~/.gemini/settings.json`:

    {
      "mcpServers": {
        "crw": {
          "command": "npx",
          "args": ["crw-mcp"]
        }
      }
    }

### fastcrw.com cloud

    {
      "mcpServers": {
        "crw": {
          "command": "npx",
          "args": ["crw-mcp"],
          "env": {
            "CRW_API_URL": "https://fastcrw.com/api",
            "CRW_API_KEY": "YOUR_API_KEY"
          }
        }
      }
    }

Useful commands:

- `gemini mcp list`
- `gemini mcp remove crw`

## Any MCP client

If your client accepts the standard JSON `mcpServers` shape, start here:

### Local embedded

    {
      "mcpServers": {
        "crw": {
          "command": "npx",
          "args": ["crw-mcp"]
        }
      }
    }

### fastcrw.com cloud

    {
      "mcpServers": {
        "crw": {
          "command": "npx",
          "args": ["crw-mcp"],
          "env": {
            "CRW_API_URL": "https://fastcrw.com/api",
            "CRW_API_KEY": "YOUR_API_KEY"
          }
        }
      }
    }

## Local rendering and config

Local embedded mode uses the same config chain as `crw-server`:

- `config.default.toml`
- `config.local.toml`
- `CRW_...` environment overrides

If you need JavaScript rendering in local MCP mode, pass a renderer URL:

    claude mcp add \
      -e CRW_RENDERER__LIGHTPANDA__WS_URL=ws://127.0.0.1:9222 \
      crw -- npx crw-mcp

Without a configured renderer, CRW still works in HTTP-only mode.

## How to choose local vs fastcrw.com

Use local embedded mode when:

- you want zero infrastructure,
- scrape/map/crawl are enough,
- or you want everything to stay on your own machine.

Use fastcrw.com when:

- you want managed infrastructure,
- you want the `crw_search` tool,
- or you do not want to manage a local renderer or server.

## What to read next

- MCP Server for the mode overview
- Search to see what cloud mode adds
- JS Rendering if local embedded mode needs browser rendering
- Authentication if you are using a self-hosted CRW server

`esc`