爬虫工具

Setup

The easy way — npm (installs the crawlie CLI and the crawlie-mcp server):

npm i -g crawlie

The macOS app — grab the signed .dmg from Releases.

From source — needs Rust (engine/CLI/MCP) and, for the desktop app, pnpm + Node:

```bash git clone https://github.com/spronta/crawlie cd crawlie cargo build --release

or install onto your PATH:

cargo install --path crates/crawlie-cli # installs crawlie cargo install --path crates/crawlie-mcp # installs crawlie-mcp ```

How it ships: the CLI + MCP come only through npm — the right native binary installs automatically as a platform package (nothing to download or unblock). The desktop app is the only direct download: a signed, notarized .dmg on Releases.

---

Hosted: the Crawlie Cloud MCP (no install)

Prefer not to install anything, or want crawls to run on our infrastructure? Point any MCP client at the hosted endpoint and authenticate with a Crawlie API key (create one in the dashboard under Settings, API keys). It speaks MCP Streamable HTTP.

Endpoint: https://crawlie.app/mcp

For Claude Code:

claude mcp add --transport http crawlie-cloud https://crawlie.app/mcp \
  --header "Authorization: Bearer crw_your_key"

For Claude Desktop (or any client that takes a JSON config):

{
  "mcpServers": {
    "crawlie-cloud": {
      "type": "http",
      "url": "https://crawlie.app/mcp",
      "headers": { "Authorization": "Bearer crw_your_key" }
    }
  }
}

Hosted crawls run on the same engine as the dashboard, are scoped to your team, and are metered against your plan. The tools mirror the local server (crawl_site, audit_url, top_fixes, geo_gaps, affected_urls, diff_reports, plus crawl_status to poll a long crawl and get_report / list_reports over your saved cloud reports). Every crawl returns a reportId you can re-slice later without re-crawling.

One-step install: the Claude Code plugin

The fastest path. The crawlie plugin bundles the MCP server and a set of skills (audit playbooks) in a single install — the MCP server auto-runs via npx, so you don't even pre-install the binary:

```bash

add this repo as a marketplace, then install the plugin

claude plugin marketplace add spronta/crawlie claude plugin install crawlie@spronta ```

How to use (CLI)

```bash

Example agent prompts

"Crawl crawlie.dev, then give me the top 5 fixes that would most improve my GEO score, with the exact change for each."

"Audit these three landing pages and tell me which is least ready to be cited by AI search, and why."

"Run a crawl with --fail-on error semantics — are there any broken links or 5xx pages blocking launch?"

The agent calls crawl_site, reads the structured issues, and uses explain_issue to turn findings into a prioritized, actionable plan.

---

Use cases

• Pre-launch QA — catch broken links, redirects, 4xx/5xx, and missing metadata before you ship.
• GEO optimization — make pages citable by AI search: structured data, semantic HTML, answer-ready content, authorship/E-E-A-T.
• Agent workflows — let a marketing/SEO agent audit a site and propose fixes autonomously via MCP.
• CI/CD gating — crawlie crawl … --fail-on error in a pipeline to block regressions.
• Client reporting — generate a polished, shareable HTML report in one command.
• Auditing AI-generated sites — verify that the site your agent just built is actually built for search.

---

How it compares

---