SearxNcrawl

Enable CORS for specific origins (required for browser-based MCP clients)

python -m crawler.mcp_server --transport http --cors-origins "http://localhost:3000,https://myapp.com"

Dependencies

Minimal dependencies:

• crawl4ai>=0.7.4 - The underlying crawler engine
• tldextract>=5.1.2 - Domain parsing for site crawls
• playwright>=1.40.0 - Browser automation
• fastmcp>=2.0.0 - MCP server framework
• httpx>=0.27.0 - HTTP client for SearXNG

Installation

```bash cd searxNcrawl python -m venv .venv source .venv/bin/activate pip install -e .

Install playwright browsers (required!)

playwright install chromium ```

Or via installed script

crawl-mcp --transport http --port 8000

Running with Docker Compose

Create a .env file (see .env.example) and run:

docker compose up --build

The MCP HTTP port is configurable via MCP_PORT in .env. Default is 9555, so the server is available at http://localhost:9555/mcp.

To run real‑world checks against the Docker setup (crawl, crawl_site, search), use:

scripts/test-realworld.sh

For extended tests including new features (remove_links, Unicode handling, schema validation):

scripts/test-extended.sh

Option 1: Copy example to user config

mkdir -p ~/.config/searxncrawl cp .env.example ~/.config/searxncrawl/.env

https://example.com/page1

Crawled: 2025-01-09 12:00:00 UTC

[Page content as markdown...]

---

https://example.com/page2

Crawled: 2025-01-09 12:00:01 UTC

[Page content as markdown...] ```

CLI Usage

After installation (pip install -e .), the crawl and search commands are available globally.

Linux example

google-chrome --remote-debugging-port=9222 --user-data-dir="$HOME/.chrome-cdp-searxncrawl"

2) Log in manually to your target app in that browser.

3) List selectable sessions:

4) Export by explicit session index:

Or let CLI selection guide you interactively:

After capture/export, use the file for authenticated crawling:

Explicit outcomes: - success (exit 0): storage state written. - timeout (exit 2): completion condition not reached in time (manual flow only). - abort (exit 130): browser/session closed before completion (manual flow only).

Safety notes: - Keep storage_state files out of version control. - Capture/export is intentionally isolated from standard crawl / crawl_site execution paths. - If multiple tabs share one browser context/profile, they share the same exported session state.

Environment Variables

SearXNG Instance Requirements

SearXNG is a privacy-respecting metasearch engine that aggregates results from multiple search engines without tracking users. To use the search functionality of searxNcrawl, you need access to a SearXNG instance with:

• JSON output format enabled – The instance must have JSON format enabled in its configuration (this is typically set in settings.yml under search.formats).
• Network accessibility – The instance must be reachable from where you run searxNcrawl.

You can either self-host a SearXNG instance or use a public one. For reliable results, self-hosting is recommended as public instances may have rate limits or restricted API access.

Configuration File Search Order

The CLI tools (crawl, search) look for .env files in this order:

1. Current directory - ./.env
2. User config - ~/.config/searxncrawl/.env

If no .env is found and .env.example exists in the package, it will be automatically copied to ~/.config/searxncrawl/.env as a starting point.

Quick setup for global CLI usage:

```bash

Option 2: Export environment variable

export SEARXNG_URL=http://your-searxng:8888

#### CORS Configuration (HTTP Transport)

When using HTTP transport, browser-based MCP clients may need CORS (Cross-Origin Resource Sharing) headers. Use `--cors-origins` to enable them:

MCP Harness Configuration

Add to your MCP client configuration (examples include Zed, opencode, antigravity, VS Code, Claude Code, Codex, OpenClaw, etc.):

{
  "mcpServers": {
    "crawler": {
      "command": "python",
      "args": ["-m", "crawler.mcp_server"],
      "cwd": "/path/to/searxNcrawl",
      "env": {
        "SEARXNG_URL": "http://your-searxng-instance:8888"
      }
    }
  }
}

Or with uv:

{
  "mcpServers": {
    "crawler": {
      "command": "uv",
      "args": ["run", "--directory", "/path/to/searxNcrawl", "python", "-m", "crawler.mcp_server"],
      "env": {
        "SEARXNG_URL": "http://your-searxng-instance:8888"
      }
    }
  }
}

OpenClaw Configuration

OpenClaw is a popular autonomous AI agent (150k+ GitHub stars) that supports MCP natively. To integrate searxNcrawl with OpenClaw, add the following to your OpenClaw MCP config file (~/.clawdbot/mcp.json or openclaw.json):

Python with venv:

{
  "searxNcrawl": {
    "command": "python",
    "args": ["-m", "crawler.mcp_server"],
    "cwd": "/path/to/searxNcrawl",
    "env": {
      "SEARXNG_URL": "http://your-searxng-instance:8888"
    }
  }
}

With uv (no manual venv needed):

{
  "searxNcrawl": {
    "command": "uv",
    "args": ["run", "--directory", "/path/to/searxNcrawl", "python", "-m", "crawler.mcp_server"],
    "env": {
      "SEARXNG_URL": "http://your-searxng-instance:8888"
    }
  }
}

Docker HTTP endpoint:

If you prefer running searxNcrawl via Docker, start the server with:

docker compose up --build

Then configure OpenClaw to connect to the HTTP endpoint at http://localhost:9555/mcp.

Once configured, OpenClaw will have access to the crawl, crawl_site, and search tools.

Configuration

The default configuration is optimized for documentation sites. For advanced customization:

```python from crawler import crawl_page_async, build_markdown_run_config, RunConfigOverrides

Custom configuration

config = build_markdown_run_config( RunConfigOverrides( delay_before_return_html=1.0, # Wait longer for JS mean_delay=1.0, # Delay between requests scan_full_page=True, ) )

doc = await crawl_page_async("https://example.com", config=config) ```

CLI Tools

• crawl - Crawl pages from the command line
• crawl-capture - Manual login capture + CDP session list/select/export
• search - Search the web via SearXNG
• Global installation - Available system-wide after pip install -e .

Python API

Output as JSON (includes metadata and references)

crawl https://example.com --json

STDIO transport (for MCP harnesses such as Zed, opencode, antigravity, VS Code, Claude Code, Codex, OpenClaw, etc.)

python -m crawler.mcp_server