快速爬虫工具

Install:

curl -fsSL https://raw.githubusercontent.com/us/crw/main/install.sh | CRW_BINARY=crw sh

Interactive setup wizard (recommended):

crw setup

Self-hosted (default Docker compose stack)

client = CrwClient(api_url="http://localhost:3000") results = client.search("open source web scraper 2026", limit=10)

📦 Install

One-line install (auto-detects OS & arch):

curl -fsSL https://raw.githubusercontent.com/us/crw/main/install.sh | CRW_BINARY=crw sh

One-line install:

curl -fsSL https://raw.githubusercontent.com/us/crw/main/install.sh | CRW_BINARY=crw-server sh

Docker:

docker run -p 3000:3000 ghcr.io/us/crw

Custom port:

**Docker Compose** ships with `lightpanda` enabled by default; `chrome` is opt-in to keep small VPS deploys lean (~500MB image + 1GB resident):

Setup Wizard (CLI)

The CLI includes an interactive setup wizard for easy configuration:

crw setup

The wizard guides you through: - Cloud vs Local mode selection - Browser engine setup (LightPanda or Chrome for JS rendering) - Search engine setup (SearXNG via Docker) - LLM provider configuration (BYOK for AI features) - Shell configuration (auto-adds to .zshrc/.bashrc)

Options: - crw setup --cloud — skip to cloud setup - crw setup --local — skip to local setup - crw setup --no-color — disable colored output (accessibility) - Press ESC to cancel gracefully at any prompt

See the self-hosting guide for production hardening, auth, reverse proxy, and resource tuning.

---

🚀 Quick Start

```bash

Example Domain

This domain is for use in illustrative examples in documents. You may use this domain in literature without prior coordination.

#### Renderer selection & response metadata

CRW picks between three rendering backends per request:

- **`http`** (1 credit) — plain HTTP fetch. Used for static pages.
- **`lightpanda`** (1 credit) — lightweight JS renderer for most SPAs.
- **`chrome`** (2 credits) — full Chromium for sites where LightPanda's hydration crashes (e.g. some Next.js App Router pages).

By default the engine auto-selects, learns per-host preferences after repeated failures, and falls over chrome → lightpanda → http transparently. Pass `"renderer"` to pin one of `auto | http | lightpanda | chrome` (Firecrawl's `engine` is also accepted as an alias).

Every successful response includes routing metadata so callers can audit and debug:

When you hard-pin a renderer that fails (e.g. "renderer":"lightpanda" on a hydration-crashing page), success stays true for protocol compatibility — but data.warnings[] carries an actionable hint suggesting renderer="chrome" or auto mode. Clients should surface the warnings array.

Configuration

Layered TOML config with environment variable overrides:

1. config.default.toml — built-in defaults
2. config.local.toml — local overrides (or CRW_CONFIG=myconfig)
3. Environment variables — CRW_ prefix, __ separator (e.g. CRW_SERVER__PORT=8080)

```toml [server] host = "0.0.0.0" port = 3000 rate_limit_rps = 10

[renderer] mode = "auto" # auto | lightpanda | playwright | chrome | none

[crawler] max_concurrency = 10 requests_per_second = 10.0 respect_robots_txt = true

[auth]

fastCRW — Open Source Web Scraping API for AI Agents

Power AI agents with clean web data. Single Rust binary, zero config, Firecrawl-compatible API. The open-source Firecrawl alternative you can self-host for free — or use our managed cloud.

Don't want to self-host? Sign up free → — managed cloud with global proxy network, web search, and dashboard. Same API, zero infra. 500 free credits, no credit card required.

---

Web Scraping & Crawling Features

Core

More

Use Cases: RAG pipelines · AI agent web access · content monitoring · data extraction · HTML to markdown conversion · web archiving

---

client = CrwClient(api_url="https://fastcrw.com/api", api_key="YOUR_KEY")

<details>
<summary><b>cURL</b></summary>

API Endpoints

Full API reference →

---

Local (embedded — no server, no API key):

claude mcp add crw -- npx crw-mcp

CLI (`crw`) — scrape URLs from your terminal

```bash brew install us/crw/crw

API Server (`crw-server`) — Firecrawl-compatible REST API

For serving multiple apps, other languages (Node.js, Go, Java), or as a shared microservice.

```bash brew install us/crw/crw-server

SDKs

Community SDKs

• crewai-crw — CRW scraping tools for CrewAI agents
• langchain-crw — CRW document loader for LangChain

Node.js: No official SDK yet — use the REST API directly or npx crw-mcp for MCP. SDK examples →

---

api_keys = ["fc-key-1234"]

```

See full configuration reference.

---

stealth tier — browserless/chromium with anti-fingerprint plugin

Integrations

Frameworks: CrewAI · LangChain · Agno · Dify

Platforms: n8n · Flowise

Missing your favorite tool? Open an issue → · All integrations →

---

[0.10.0](https://github.com/us/crw/compare/v0.9.1...v0.10.0) (2026-05-20)

Why CRW? — Firecrawl & Crawl4AI Alternative

• Single binary, 6 MB RAM — no Redis, no Node.js, no containers. Firecrawl needs 5 containers and 4 GB+. Crawl4AI requires Python + Playwright
• 5.5x faster than Firecrawl — 833ms avg vs 4,600ms (see benchmarks). P50 at 446ms
• 73/100 search win rate — beats Firecrawl (25/100) and Tavily (2/100) in head-to-head benchmarks
• Free self-hosting — $0/1K scrapes vs Firecrawl's $0.83–5.33. No infra, no cold starts (85ms). No API key required for local mode
• Agent ready — add to any MCP client in one command. Embedded mode: no server needed
• Firecrawl-compatible API — drop-in replacement. Same /v1/scrape, /v1/crawl, /v1/map endpoints. HTML to markdown, structured data extraction, website crawler — all built-in
• Built for RAG pipelines — clean LLM-ready markdown output for vector databases and AI data ingestion
• Open source — AGPL-3.0, developed transparently. Join our community

---

Search — CRW vs Firecrawl vs Tavily (100 queries, concurrent)

CRW is 2.3x faster than Tavily and won 73% of latency races. Full search benchmark →

Scrape — CRW vs Firecrawl (1,000 URLs, JS rendering enabled)

Tested on Firecrawl's scrape-content-dataset-v1:

<details> <summary><b>Resource comparison</b></summary>

</details>

Full benchmark details →

Run the benchmark yourself:

pip install datasets aiohttp
python bench/run_bench.py

---

Open Source vs Cloud

Sign up free → — 500 free credits, no credit card required.

---