---
name: document-retrieval
description: Build and tune retrieval configs that search, rank, and collect ordinance documents in COMPASS. Use whenever a user asks to improve retrieval precision/recall, tune search queries/keywords, or debug acquisition quality before extraction tuning.
---

# Web Scraper Skill

Use this skill to improve retrieval precision/recall before extraction tuning.
Applies to both one-shot (schema-driven) and legacy decision-tree extraction
pipelines.

## When to use

- Download step returns noisy sources (one-shot extraction).
- Ordinance recall is weak across jurisdictions (one-shot extraction).
- LLM filtering is compensating for poor search quality.

## Do not use

- Schema feature definition or value extraction logic design.
- Post-extraction feature/value debugging when retrieval is already correct.

## Expected assistant output

When using this skill, return:

1. The retrieval axis changed (queries, keyword weights, or heuristics).
2. Evidence from artifacts/logs showing why the change was needed.
3. The next run command against the same jurisdiction sample.

## Canonical reference

Consult example plugin configurations in `examples/`:
- `examples/one_shot_schema_extraction/plugin_config.yaml` — standard one-shot config
- `examples/water_rights_demo/one-shot/plugin_config.yaml` — multi-document edge cases

When creating new tech configs, use `<tech>_plugin_config.yaml` as a recommended
naming convention (e.g. `geothermal_plugin_config.yaml`).

## Scope

- Query-template strategy.
- URL ranking and filtering patterns.
- Heuristic phrase controls before LLM validation.

## Two retrieval phases

COMPASS runs two sequential acquisition passes per jurisdiction:

1. **Search-engine phase** — queries `SerpAPIGoogleSearch` (or configured
   engine) using `query_templates`. This phase is the primary source of
   ordinance documents.
2. **Website crawl phase** — crawls the jurisdiction's official website,
   ranking pages using `website_keywords`. This phase is a secondary pass
   and runs only if the search-engine phase did not yield an ordinance
   context.

Key behaviors:
- Playwright browser errors during the website crawl phase are **non-fatal**.
  COMPASS logs the error and continues.
- `Found 0 potential documents` at the end of the crawl phase is **expected**
  for jurisdictions without relevant online ordinances.
- Disable the crawl phase with `perform_website_search: false` in run config
  when you want faster smoke tests or Playwright is unavailable.

## Key management

For SerpAPI-backed search, keep `api_key` out of committed config and provide
`SERPAPI_KEY` via environment (for example through `.env` loaded in shell).

Recommended shell setup:

```bash
set -a
source .env
set +a
```

Avoid spaces around `=` in `.env` assignments.

## Retrieval design pattern

1. Create 3-7 jurisdiction queries with `{jurisdiction}`.
2. Weight legal document indicators in URL keywords.
3. Apply exclusions for templates/reports/slides.
4. Add focused negative tech terms to reduce false positives.
5. Start with dynamic search, then switch to deterministic known URLs when
  search infrastructure is unstable.

When using `heuristic_keywords`, use these four lists to guide pre-LLM filtering:
- `GOOD_TECH_KEYWORDS` — strong indicators of the target technology
  (e.g., facility types, deployment modes). Documents matching even a
  few keywords are marked as candidates.
- `GOOD_TECH_PHRASES` — multi-word phrases that signal relevant
  ordinance content. Keep specific to avoid false positives.
- `GOOD_TECH_ACRONYMS` — industry-standard abbreviations for the
  technology. Narrow list; include only widely recognized acronyms.
- `NOT_TECH_WORDS` — pre-heuristic filter that rejects documents
  before keyword matching. Use to exclude adjacent technologies and
  irrelevant domains (e.g., residential HVAC, unrelated industries).
  Runs first; prevents wasted keyword evaluation on clearly-wrong
  documents.

If any required list is missing or empty, COMPASS raises a plugin
configuration error and extraction quality should be treated as failed.

For first-pass reliability, test retrieval with deterministic known URLs
before using live web search.

## Technology-specific retrieval controls (template)

- Include target-technology facility/deployment terms.
- Exclude adjacent and non-target terms (residential/HVAC/PV/etc as needed).
- Favor jurisdictional legal-code signals like `land use code`,
  `code of ordinances`, `use table`, and `special use permit`.

## Deterministic smoke-test mode
For this smoke test, at least one of the following documentation sources must be provided:

- **`known_doc_urls`**: A list of URLs pointing to external documentation that the scraper can access and parse
- **`known_local_docs`**: A collection of local documentation files available in the repository or system

Use run-config controls to bypass flaky search while tuning:

- supply `known_doc_urls` or `known_local_docs`,
- set `perform_se_search: false`,
- set `perform_website_search: false`.

Then validate:

- download artifacts exist,
- cleaned text exists,
- ordinance DB rows are non-empty.

## Tuning loop

1. Run SE-search phase on small sample.
2. Inspect kept vs discarded PDFs (`ordinance_files/`).
3. Run heuristic filter and review false rejects/accepts (`cleaned_text/`).
4. Check website crawl phase independently if needed (enable, run, inspect logs).
5. Update one axis only:
   - query templates (affects SE phase),
   - URL weights (affects both phases),
   - include/exclude heuristic patterns (pre-LLM filter),
   - `NOT_TECH_WORDS` (upstream document rejection).
6. Re-run same sample and compare.

## Cross-tech onboarding

When reusing this workflow for any technology:

- keep legal retrieval tokens (`ordinance`, `zoning`, `code`),
- replace all technology terms in `query_templates`, `website_keywords`,
  and `heuristic_keywords`,
- seed `known_doc_urls` with authoritative regulatory documents for smoke
  testing,
- avoid copying negatives from previous technologies into the new tech config,
- verify `NOT_TECH_WORDS` excludes adjacent technologies for your domain.

## Phase gates

- **3 jurisdictions**: ensure major source classes are found.
- **10 jurisdictions**: verify stability across regions.


## Guardrails

- Keep feature extraction logic out of retrieval config.
- Do not overfit to one county's document style.
- Preserve auditable rationale for each retrieval change.
- Keep one canonical retrieval config per active technology.
- Ensure each run uses a unique `out_dir` to avoid COMPASS aborting early.

