网页爬虫MCP

Prerequisites

• Python 3.13 or higher
• pip or uv package manager
• ScrapeGraph API key

Prerequisites

• Python 3.13 or higher
• Google ADK installed
• ScrapeGraph API key

Prerequisites

• Python 3.13 or higher
• pip or uv package manager
• ScrapeGraph API key

Install dependencies

pip install -e ".[dev]"

2. Install with Smithery (Recommended)

npx -y @smithery/cli install @ScrapeGraphAI/scrapegraph-mcp --client claude

Setup Instructions

To utilize this server, you'll need a ScrapeGraph API key. Follow these steps to obtain one:

1. Navigate to the ScrapeGraph Dashboard
2. Create an account and generate your API key

Automated Installation via Smithery

For automated installation of the ScrapeGraph API Integration Server using Smithery:

npx -y @smithery/cli install @ScrapeGraphAI/scrapegraph-mcp --client claude

Installation

1. Clone the repository (if you haven't already):

git clone https://github.com/ScrapeGraphAI/scrapegraph-mcp
cd scrapegraph-mcp

1. Install the package:

```bash

Using the installed command

scrapegraph-mcp

Installation

1. Install Google ADK (if not already installed):

pip install google-adk

1. Set your API key:

export SGAI_API_KEY=your-api-key-here

Installation from Source

```bash

Deployment

• Smithery - Automated MCP server deployment
• Docker - Container support with Alpine Linux
• stdio transport - Standard MCP communication

Quick Start

Remote Server Usage

Connect to our hosted MCP server - no local installation required!

Local Usage

To run the MCP server locally for development or testing, follow these steps:

Basic Integration Example

Create an agent file (e.g., agent.py) with the following configuration:

```python import os from google.adk.agents import LlmAgent from google.adk.tools.mcp_tool.mcp_toolset import MCPToolset from google.adk.tools.mcp_tool.mcp_session_manager import StdioConnectionParams from mcp import StdioServerParameters

Usage Example

Once configured, your agent can use natural language to interact with web scraping tools:

```python

- "Convert https://example.com to markdown"

Example Use Cases

The server enables sophisticated queries across various scraping scenarios:

Claude Desktop Configuration

Update your Claude Desktop configuration file with the following settings (located on the top rigth of the Cursor page):

(remember to add your API key inside the config)

{
    "mcpServers": {
        "@ScrapeGraphAI-scrapegraph-mcp": {
            "command": "npx",
            "args": [
                "-y",
                "@smithery/cli@latest",
                "run",
                "@ScrapeGraphAI/scrapegraph-mcp",
                "--config",
                "\"{\\\"scrapegraphApiKey\\\":\\\"YOUR-SGAI-API-KEY\\\"}\""
            ]
        }
    }
}

The configuration file is located at: - Windows: %APPDATA%/Claude/claude_desktop_config.json - macOS: ~/Library/Application\ Support/Claude/claude_desktop_config.json

Claude Desktop Configuration (Remote)

Add this to your Claude Desktop config (~/Library/Application Support/Claude/claude_desktop_config.json on macOS):

{
  "mcpServers": {
    "scrapegraph-mcp": {
      "command": "npx",
      "args": [
        "mcp-remote@0.1.25",
        "https://mcp.scrapegraphai.com/mcp",
        "--header",
        "X-API-Key:YOUR_API_KEY"
      ]
    }
  }
}

Cursor Configuration (Remote)

Cursor supports native HTTP MCP connections. Add to your Cursor MCP settings (~/.cursor/mcp.json):

{
  "mcpServers": {
    "scrapegraph-mcp": {
      "url": "https://mcp.scrapegraphai.com/mcp",
      "headers": {
        "X-API-Key": "YOUR_API_KEY"
      }
    }
  }
}

Configuring Claude Desktop for Local Server

To use your locally running server with Claude Desktop, update your configuration file:

macOS/Linux (~/Library/Application Support/Claude/claude_desktop_config.json):

{
    "mcpServers": {
        "scrapegraph-mcp-local": {
            "command": "python",
            "args": [
                "-m",
                "scrapegraph_mcp.server"
            ],
            "env": {
                "SGAI_API_KEY": "your-api-key-here"
            }
        }
    }
}

Windows (%APPDATA%\Claude\claude_desktop_config.json):

{
    "mcpServers": {
        "scrapegraph-mcp-local": {
            "command": "python",
            "args": [
                "-m",
                "scrapegraph_mcp.server"
            ],
            "env": {
                "SGAI_API_KEY": "your-api-key-here"
            }
        }
    }
}

Note: Make sure Python is in your PATH. You can verify by running python --version in your terminal.

Configuring Cursor for Local Server

In Cursor's MCP settings, add a new server with:

• Command: python
• Args: ["-m", "scrapegraph_mcp.server"]
• Environment Variables: {"SGAI_API_KEY": "your-api-key-here"}

Configuration Options

Timeout Settings: - Default timeout is 5 seconds, which may be too short for web scraping operations - Recommended: Set `timeout=300.0 - Adjust based on your use case (crawling operations may need even longer timeouts)

Tool Filtering: - By default, all registered MCP tools are exposed to the agent (see Available Tools) - Use tool_filter to limit which tools are available:

  tool_filter=['scrape', 'extract', 'search']
  

API Key Configuration: - Set via environment variable: export SGAI_API_KEY=your-key - Or pass directly in env dict: 'SGAI_API_KEY': 'your-key-here' - Environment variable approach is recommended for security

API v2

This MCP server targets ScrapeGraph API v2 (https://v2-api.scrapegraphai.com/api), aligned 1:1 with scrapegraph-py PR #84. Auth uses the SGAI-APIKEY header. Environment variables mirror the Python SDK:

• SGAI_API_URL — override the base URL (default https://v2-api.scrapegraphai.com/api)
• SGAI_TIMEOUT — request timeout in seconds (default 120)
• SGAI_API_KEY — API key (can also be passed via MCP scrapegraphApiKey or X-API-Key header)

Legacy aliases (still honored): SCRAPEGRAPH_API_BASE_URL for SGAI_API_URL, SGAI_TIMEOUT_S for SGAI_TIMEOUT.

1. Get Your API Key

Sign up and get your API key from the ScrapeGraph Dashboard

- "Crawl this documentation site and extract all API endpoints"

``` For more information about Google ADK, visit the official documentation.

Single Page Scraping

• Markdownify: "Convert the ScrapeGraph documentation page to markdown"
• Extract: "Extract all product names, prices, and ratings from this e-commerce page"
• Extract with scrolling: "Scrape this infinite scroll page with 5 scrolls and extract all items"
• Basic Scrape: "Fetch the HTML content of this JavaScript-heavy page with full rendering"

Set your API key

export SGAI_API_KEY=your-api-key

API Integration

• ScrapeGraph AI API - Enterprise web scraping service
• Base URL: https://v2-api.scrapegraphai.com/api
• Authentication: API key-based

Cursor Integration

Add the ScrapeGraphAI MCP server on the settings:

Or using Python module

python -m scrapegraph_mcp.server ```

The server will start and communicate via stdio (standard input/output), which is the standard MCP transport method.

Google ADK Integration

The ScrapeGraph MCP server can be integrated with Google ADK (Agent Development Kit) to create AI agents with web scraping capabilities.

Development Workflow

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Make your changes
4. Run linting and type checking
5. Test with MCP Inspector and Claude Desktop
6. Update documentation
7. Commit your changes (git commit -m 'Add amazing feature')
8. Push to the branch (git push origin feature/amazing-feature)
9. Open a Pull Request

AI Assistant Integration

• Claude Desktop - Desktop app with MCP support
• Cursor - AI-powered code editor

Troubleshooting Local Setup

Server not starting: - Verify Python is installed: python --version - Check that the package is installed: pip list | grep scrapegraph-mcp - Ensure API key is set: echo $SGAI_API_KEY (macOS/Linux) or echo %SGAI_API_KEY% (Windows)

Tools not appearing: - Check Claude Desktop logs: - macOS: ~/Library/Logs/Claude/ - Windows: %APPDATA%\Claude\Logs\ - Verify the server starts without errors when run directly - Check that the configuration JSON is valid

Import errors: - Reinstall the package: pip install -e . --force-reinstall - Verify dependencies: pip install -r requirements.txt (if available)