# CRW — open-source web scraper & crawler (Firecrawl-compatible)

> Docs at `docs.fastcrw.com`. Hosted API at `https://api.fastcrw.com`. Self-hosted at `http://localhost:3000`. Five endpoints: `/v1/search`, `/v1/scrape`, `/v1/crawl`, `/v1/map`, `/v1/extract`.

## Quick: copy this and run

Auth (hosted only): `Authorization: Bearer $CRW_API_KEY`. Self-hosted (`http://localhost:3000`) needs no header.

Body: `{ "query": string, "limit": number (default 5, max 20), "lang"?: "en", "tbs"?: "qdr:d|w|m|y", "sources"?: ["web"|"news"|"images"] }`

### Python (stdlib only — defaults to hosted, fails loudly if key is missing)

    #!/usr/bin/env python3
    """Search CRW and print top results as JSON."""
    import json, os, sys, urllib.request

    API_KEY = os.environ.get("CRW_API_KEY")
    # Default to the hosted API. Self-hosting? Set CRW_API_URL=http://localhost:3000
    BASE = os.environ.get("CRW_API_URL", "https://api.fastcrw.com")
    URL = f"{BASE}/v1/search"

    headers = {"Content-Type": "application/json"}
    if API_KEY:
        headers["Authorization"] = f"Bearer {API_KEY}"
    elif BASE.startswith("https://api.fastcrw.com"):
        sys.exit("error: CRW_API_KEY is required for the hosted API. "
                 "Get one at https://fastcrw.com (500 free credits) "
                 "or set CRW_API_URL=http://localhost:3000 to self-host.")

    req = urllib.request.Request(
        URL,
        data=json.dumps({"query": "renewable energy trends 2024", "limit": 3}).encode(),
        headers=headers,
        method="POST",
    )
    with urllib.request.urlopen(req) as r:
        body = json.loads(r.read())  # body == {"success": true, "data": [...]}

    # Results live under body["data"]. Each row has `title`, `url`, `snippet`,
    # `description`, `position`, `score`, `category` — print the LLM-ready subset.
    print(json.dumps(
        [{"title": x["title"], "url": x["url"], "snippet": x["snippet"]}
         for x in body["data"]],
        indent=2,
    ))
    # Run with: python3 search.py  (file is python3-only — no `python` alias needed)

### cURL

    curl -X POST https://api.fastcrw.com/v1/search \
      -H "Authorization: Bearer $CRW_API_KEY" \
      -H "Content-Type: application/json" \
      -d '{"query":"renewable energy trends 2024","limit":3}'

### CLI (one-shot LLM-ready output)

The `crw` CLI projects directly to the title/url/snippet schema with `--json --fields` — no `jq` reshape needed:

    # Install: brew install us/crw/crw  (or: cargo install crw-cli)
    crw search "renewable energy trends 2024" --json --fields title,url,snippet --limit 3

    # Available fields: title, url, description, snippet, position, score, category
    # --json is shorthand for --format json (also accepts --format text|markdown)
    # Self-hosted: no auth. Hosted: export CRW_API_KEY=crw_live_...

### Response shape

    {
      "success": true,
      "data": [
        {"url": "https://...", "title": "...",
         "description": "...", "snippet": "...",
         "position": 1, "score": 9.5}
      ]
    }

## Other endpoints

- `POST /v1/scrape` `{"url":"...","formats":["markdown"]}` — one URL → markdown
- `POST /v1/map` `{"url":"...","limit":50}` — discover URLs
- `POST /v1/crawl` `{"url":"...","maxPages":5}` — async multi-page; poll `GET /v1/crawl/{id}`
- `POST /v1/scrape` with `formats:["json"]` + `jsonOptions.schema` — structured extract

## Install (local setup only)

`pip install crw` · `npx crw-mcp` · `brew install us/crw/crw` · `curl -fsSL https://raw.githubusercontent.com/us/crw/main/install.sh | sh` · `docker run -p 3000:3000 ghcr.io/us/crw:latest`

## Resources

- OpenAPI spec
- llms.txt
- Context7
- MCP Server (npm)
- Typed Python SDK
- Agent Skills (SKILL.md)
- CLI (Homebrew tap)
- GitHub

# Agent Onboarding Guide

## Overview

This guide is written for AI agents and the systems that orchestrate them. It covers the minimum you need to start making API calls, the most common patterns, and how to handle errors gracefully.

## Authentication

Every request requires an API key in the `Authorization` header:

    Authorization: Bearer YOUR_API_KEY

:::note On self-hosted instances, authentication is configured via your server settings. Cloud only (fastcrw.com) provides a dashboard for key management. :::

## Available Endpoints

Endpoint | Method | Purpose
--- | --- | ---
`/v1/scrape` | POST | Extract content from a single URL
`/v1/crawl` | POST | Recursively collect pages from a domain
`/v1/map` | POST | Discover all reachable URLs on a domain
`/v1/search` | POST | Search the web and return results with content

## Quick Start Pattern

For most agent workflows, start with this sequence:

1. Discover what pages exist on a target domain:

    curl -X POST https://fastcrw.com/api/v1/map \
      -H "Authorization: Bearer $API_KEY" \
      -H "Content-Type: application/json" \
      -d '{"url": "https://example.com"}'

2. Extract content from specific pages:

    curl -X POST https://fastcrw.com/api/v1/scrape \
      -H "Authorization: Bearer $API_KEY" \
      -H "Content-Type: application/json" \
      -d '{"url": "https://example.com/page", "formats": ["markdown"]}'

3. Search when you need to find relevant pages across the web:

    curl -X POST https://fastcrw.com/api/v1/search \
      -H "Authorization: Bearer $API_KEY" \
      -H "Content-Type: application/json" \
      -d '{"query": "your search query", "limit": 5}'

## Response Format

All endpoints return JSON with a consistent structure:

    {
      "success": true,
      "data": { ... }
    }

On failure:

    {
      "success": false,
      "error": "description of what went wrong"
    }

## Error Handling for Agents

Agents should handle these common scenarios:

Status | Meaning | Agent action
--- | --- | ---
200 | Success | Process the response
400 | Bad request | Fix the request parameters
401 | Invalid API key | Check authentication
429 | Rate limited or out of credits | Back off and retry, or alert the user
500 | Server error | Retry with exponential backoff

## Common Agent Patterns

### Research Loop

    search(topic) -> scrape(top results) -> analyze -> search(refined query) -> repeat

### Site Exploration

    map(domain) -> filter URLs -> scrape(relevant pages) -> synthesize

### Monitoring

    scrape(url) -> store result -> wait -> scrape(url) -> compare with previous

## Rate Limits

Check the rate limits documentation for current limits. Agents should respect `Retry-After` headers and implement exponential backoff.

## MCP Integration

If your agent runtime supports MCP, see the MCP guide for a simpler integration path that avoids direct HTTP calls.

`esc`