开源AI工具：Go web爬虫

Prerequisites

• Go: Version 1.25 or later
• Git: For cloning the repository
• Disk Space: Sufficient for storing crawled content and state database

Getting Started

Installation

Option 1: Direct Installation (Recommended)

Install the latest version directly from GitHub:

go install github.com/Sriram-PR/doc-scraper/cmd/doc-scraper@latest

This installs the doc-scraper binary to your GOPATH/bin directory (usually ~/go/bin or %USERPROFILE%\go\bin). Make sure this directory is in your PATH.

Option 2: Clone and Build

1. Clone the repository:

   git clone https://github.com/Sriram-PR/doc-scraper.git
   cd doc-scraper
   

1. Install Dependencies:

   go mod tidy
   

1. Build the Binary:

   make build
   # or: go build -o doc-scraper ./cmd/doc-scraper
   

This creates an executable named doc-scraper in the project root.

Quick Start

1. Create a basic config.yaml file (see Configuration section)
2. Run the crawler:

   ./doc-scraper crawl -site your_site_key -loglevel info
   

1. Find your crawled documentation in the ./crawled_docs/ directory

Example Configuration

```yaml

Usage

Execute the compiled binary from the project root directory:

./doc-scraper <command> [options]

Example Usage Scenarios

Basic Crawl:

./doc-scraper crawl -site tensorflow_docs -loglevel info

Resume a Large Crawl:

./doc-scraper resume -site pytorch_docs -loglevel info

Validate Configuration:

./doc-scraper validate -config config.yaml
./doc-scraper validate -site pytorch_docs  # Validate specific site

List Available Sites:

./doc-scraper list-sites

High Performance Crawl with Profiling:

./doc-scraper crawl -site small_docs -loglevel warn -pprof localhost:6060

Debug Mode for Troubleshooting:

./doc-scraper crawl -site test_site -loglevel debug

Parallel Crawl of Multiple Sites:

./doc-scraper crawl -sites pytorch_docs,tensorflow_docs,langchain_docs

Crawl All Configured Sites:

./doc-scraper crawl --all-sites

Start MCP Server for Claude Desktop:

./doc-scraper mcp-server -config config.yaml

Start MCP Server with SSE Transport:

./doc-scraper mcp-server -config config.yaml -transport sse -port 8080

Example Usage

sites:
  pytorch_docs:
    start_urls:
      - "https://pytorch.org/docs/stable/"
    allowed_domain: "pytorch.org"
    allowed_path_prefix: "/docs/stable/"
    content_selector: "auto"  # Auto-detect framework
    max_depth: 0

Usage

```bash

Usage

```bash

Example Output

INFO Starting watch mode for 2 sites with interval 24h0m0s
INFO Watch schedule:
INFO   pytorch_docs: last run 2024-01-15T10:30:00Z (success, 1500 pages), next run 2024-01-16T10:30:00Z
INFO   tensorflow_docs: never run, will run immediately
INFO Running crawl for 1 due sites: [tensorflow_docs]
...
INFO Next crawl: pytorch_docs in 23h45m (at 10:30:00)

Usage

Stdio Transport (for Claude Desktop/Cursor):

./doc-scraper mcp-server -config config.yaml

SSE Transport (HTTP-based):

./doc-scraper mcp-server -config config.yaml -transport sse -port 8080

Tool Examples

List available sites:

Tool: list_sites
Result: Returns all configured sites with their domains and crawl status

Fetch a single page:

Tool: get_page
Arguments: { "url": "https://docs.example.com/guide", "content_selector": "article" }
Result: Returns page content as markdown with metadata

Start a background crawl:

Tool: crawl_site
Arguments: { "site_key": "pytorch_docs", "incremental": true }
Result: Returns job ID for tracking progress

Check crawl progress:

Tool: get_job_status
Arguments: { "job_id": "abc-123-def" }
Result: Returns status, pages processed, and completion info

Search crawled content:

Tool: search_crawled
Arguments: { "query": "neural network", "site_key": "pytorch_docs", "max_results": 10 }
Result: Returns matching pages with snippets

Configuration (`config.yaml`)

A config.yaml file is required to run the crawler. Create this file in the project root or specify its path using the -config flag.

Key Settings for LLM Use

When configuring for LLM documentation processing, pay special attention to these settings:

• sites.<your_site_key>.content_selector: Define precisely to capture only relevant text
• sites.<your_site_key>.allowed_domain / allowed_path_prefix: Define scope accurately
• skip_images: Set to true globally or per-site if images aren't needed for the LLM
• Adjust concurrency/delay settings based on the target site and your resources

Global settings (applied if not overridden by site)

default_delay_per_host: 500ms num_workers: 8 num_image_workers: 8 max_requests: 48 max_requests_per_host: 4 output_base_dir: "./crawled_docs" state_dir: "./crawler_state" max_retries: 4 initial_retry_delay: 1s max_retry_delay: 30s semaphore_acquire_timeout: 30s global_crawl_timeout: 0s skip_images: false # Set to true to skip images globally max_image_size_bytes: 10485760 # 10 MiB enable_output_mapping: true output_mapping_filename: "global_url_map.tsv" enable_metadata_yaml: true metadata_yaml_filename: "crawl_meta.yaml"

HTTP Client Settings

http_client_settings: timeout: 45s max_idle_conns_per_host: 6

Site-specific configurations

sites: # Key used with -site flag pytorch_docs: start_urls: - "https://pytorch.org/docs/stable/" allowed_domain: "pytorch.org" allowed_path_prefix: "/docs/stable/" content_selector: "article.pytorch-article .body" max_depth: 0 # 0 for unlimited depth skip_images: false # Override global mapping filename for this site output_mapping_filename: "pytorch_docs_map.txt" metadata_yaml_filename: "pytorch_metadata_output.yaml" disallowed_path_patterns: - "/docs/stable/./_modules/." - "/docs/stable/.\.html#."

tensorflow_docs: start_urls: - "https://www.tensorflow.org/guide" - "https://www.tensorflow.org/tutorials" allowed_domain: "www.tensorflow.org" allowed_path_prefix: "/" content_selector: ".devsite-article-body" max_depth: 0 delay_per_host: 1s # Site-specific override # Disable mapping for this site, overriding global enable_output_mapping: false enable_metadata_yaml: false disallowed_path_patterns: - "/install/." - "/js/." ```

Full Configuration Options

HTTP Client Settings: (These are global and cannot be overridden per site in the current structure)

• timeout: Overall request timeout (Default in code: 45s)
• max_idle_conns: Total idle connections (Default in code: 100)
• max_idle_conns_per_host: Idle connections per host (Default in code: 2)
• idle_conn_timeout: Timeout for idle connections (Default in code: 90s)
• tls_handshake_timeout: TLS handshake timeout (Default in code: 10s)
• expect_continue_timeout: "100 Continue" timeout (Default in code: 1s)
• force_attempt_http2: null (use Go default), true, or false. (Default in code: null)
• dialer_timeout: TCP connection timeout (Default in code: 15s)
• dialer_keep_alive: TCP keep-alive interval (Default in code: 30s)

Site-Specific Configuration Options:

• start_urls: Array of starting URLs for crawling (Required)
• allowed_domain: Restrict crawling to this domain (Required)
• allowed_path_prefix: Further restrict crawling to URLs with this prefix (Required)
• content_selector: CSS selector for main content extraction, or "auto" for automatic detection (Required)
• max_depth: Maximum crawl depth from start URLs (0 = unlimited)
• delay_per_host: Override global delay setting for this site
• disallowed_path_patterns: Array of regex patterns for URLs to skip
• link_extraction_selectors: Array of CSS selectors for additional link extraction areas
• respect_nofollow: Boolean. Whether to respect rel="nofollow" links
• user_agent: String. Override global user agent for this site
• skip_images: Override global image setting for this site
• max_image_size_bytes: Integer. Override global max image size for this site
• allowed_image_domains: Array of domains from which to download images
• disallowed_image_domains: Array of domains to block image downloads from
• enable_output_mapping: true or false. Override global URL-to-file mapping enablement for this site
• output_mapping_filename: String. Override global URL-to-file mapping filename for this site
• enable_metadata_yaml: true or false. Override global YAML metadata output enablement for this site
• metadata_yaml_filename: String. Override global YAML metadata filename for this site
• enable_jsonl_output: true or false. Override global JSONL output enablement for this site
• jsonl_output_filename: String. Override global JSONL output filename for this site
• chunking.enabled: true or false. Override global chunking enablement for this site
• chunking.max_chunk_size: Integer. Override global max chunk size for this site
• chunking.chunk_overlap: Integer. Override global chunk overlap for this site
• chunking.output_filename: String. Override global chunks output filename for this site

Command Options

crawl / resume:

Note: One of -site, -sites, or --all-sites is required.

validate:

list-sites:

mcp-server:

watch:

Note: One of -site, -sites, or --all-sites is required.

Crawl all configured sites

./doc-scraper crawl --all-sites

Error: site 'langchain_docs' not found in configuration

Watch all configured sites weekly

./doc-scraper watch --all-sites -interval 7d ```

Claude Code Integration

Add to your Claude Code configuration (claude_code_config.json):

{
  "mcpServers": {
    "doc-scraper": {
      "command": "/path/to/doc-scraper",
      "args": ["mcp-server", "-config", "/path/to/config.yaml"]
    }
  }
}