# Session Handoff — 2026-05-17

**Branch:** `main` (33 commits pushed in this session)
**Test count:** 1583 → **1610 passed** (+27 new) | 1 skipped (unchanged)
**Operator guide:** `docs/google-scholar-abstract-only-2026-05-17.md`

---

## What was shipped in this session

### 1. Search source expansion (commits `dc52868` → `5021e29`)

Full new `DomainAwareAggregator` stack with 6 new search providers:

| Provider | Key | Notes |
|----------|-----|-------|
| EuropePMC | `europepmc` | Biomedical REST, free |
| PubChem | `pubchem` | Compound → PMID → Paper |
| CORE | `core` | Open-access aggregator; optional key |
| INSPIRE-HEP | `inspire` | Physics bibliography |
| NASA ADS | `ads` | Requires `ads_api_key` |
| Google Scholar | `google_scholar` | Playwright/Chromium; see below |

`DomainClassifier` routes queries to the right tier based on regex patterns. `ProviderHealthTracker` records per-provider failures so the aggregator skips unreachable providers.

**Configure:** `config.yml → search.enabled_providers`

---

### 2. Cite-graph OpenCitations arm (commit `fc9a24c`)

Third citation source (COCI / OpenCitations) alongside OpenAlex `cited_by` + Semantic Scholar. Papers seen in ≥ 2 sources get a `multi_source_bonus` score boost.

---

### 3. GitHub skill-bundle ingest (commits `2450d51` → `a91d690`)

New `pipeline/github/` package and `pipeline/github_kb.py` orchestrator.

```bash
# CLI
uv run perspicacite ingest-github-repo https://github.com/org/repo --kb my-kb

# MCP tools
ingest_github_repo(source=url, kb_name=...)
ingest_skill_bundle(source=url, kb_name=...)
```

Parses `bundle.yml`, extracts DOI/arXiv/PMC links, ingests repo code + docs as KB chunks. Falls back to README-only mode when no `bundle.yml` is present.

**Operator guide:** `docs/github-skill-bundle-ingest-2026-05-15.md`

---

### 4. Google Scholar Playwright provider (commits `a7ec795` → `0d7e510`)

Headless Chromium search using `playwright`. Single `_render_and_extract_cards` seam makes it fully testable without a browser.

#### Activation

```bash
uv pip install -e "[browser]"
playwright install chromium
```

```yaml
# config.yml
google_scholar:
  enabled: true
  delay_seconds: 2.0   # keep ≥ 2.0 to avoid CAPTCHA
  headless: true

search:
  enabled_providers:
    - scilex
    - europepmc
    - google_scholar
```

#### Tier
Uses `tier = "flaky"` (45 s timeout, 0 retries). Won't block other providers on failure.

---

### 5. Abstract-only KB ingest mode (commit `a9d03f2`)

Skip PDF download for fast large-corpus population (~80% faster).

```yaml
knowledge_base:
  ingest_mode: "abstract_only"   # "auto" | "full_text" | "abstract_only"
```

`retrieve_paper_content(abstract_only=True)` returns after OpenAlex/Crossref discovery with `content_type="abstract"`. Papers without an abstract count as failures; papers with abstract-only count as success.

---

### 6. `search_literature` KB dedup (commit `3ee9928`)

```python
# MCP call — skip papers already in KB
await client.call_tool("search_literature", {
    "query": "CRISPR base editing",
    "max_results": 20,
    "exclude_kb": "my-crispr-kb",
})
```

Papers whose DOI exists in the named KB are stripped before returning. Best-effort (errors silently pass papers through).

---

## Key files touched

| File | What changed |
|------|-------------|
| `src/perspicacite/config/schema.py` | `GoogleScholarConfig`, `KnowledgeBaseConfig.ingest_mode`, `GitHubConfig`, `BundlesConfig`, `SearchConfig`, `PaperSource` additions |
| `src/perspicacite/search/google_scholar_playwright.py` | New — Playwright provider |
| `src/perspicacite/search/domain_aggregator.py` | `DomainAwareAggregator`, `build_aggregator`, Scholar block |
| `src/perspicacite/pipeline/download/unified.py` | `abstract_only` parameter |
| `src/perspicacite/pipeline/search_to_kb.py` | `abstract_only` pass-through, success counting |
| `src/perspicacite/mcp/server.py` | `exclude_kb` on `search_literature`, `ingest_github_repo`, `ingest_skill_bundle` tools |
| `src/perspicacite/pipeline/github/` | New package (fetcher, bundle, walk, chunk_producer) |
| `src/perspicacite/pipeline/github_kb.py` | New — GitHub ingest orchestrator |
| `config.example.yml` | `google_scholar`, `ingest_mode`, `enabled_providers` comment |

---

## Known limitations / next steps

1. **`recency_weight` not wired into `advanced` / `profound` RAG modes** — only `basic` and `contradiction` honor it currently. Known follow-up.
2. **Multi-KB query not wired into `advanced`, `profound`, `agentic`, `literature_survey`** — only `basic` and `contradiction` support `kb_names`. Known follow-up.
3. **Google Scholar CAPTCHA** — at scale (> 50 queries/day) Scholar will show CAPTCHA. Provider returns `[]` and logs `google_scholar_captcha_detected`. Mitigation: increase `delay_seconds`, rotate IP.
4. **Scholar DOI gaps** — Scholar doesn't always link to `doi.org`. DOI-less papers get a synthetic `scholar:XXXXXX` id and can't be auto-ingested via `add_dois_to_kb`.
5. **Search → KB auto-fallback** — when top KB retrieval score < threshold, fall back to live search. Designed but not implemented. See brainstorming notes in session context.
6. **Session domain routing / learning** — per-session DomainClassifier profile that adapts to user queries. Future work.
7. **`abstract_only` CLI flag** — `ingest_mode` is only settable via `config.yml`; a `--ingest-mode` CLI flag on ingest commands would be ergonomic.

---

## Test inventory for this session

| Test file | Coverage |
|-----------|----------|
| `test_config_scholar_fields.py` | `GoogleScholarConfig`, `KnowledgeBaseConfig.ingest_mode` |
| `test_google_scholar_playwright.py` | URL builder, meta parser, DOI extractor, provider convert/filter/error/year |
| `test_abstract_only_kb.py` | `retrieve_paper_content(abstract_only=True)`, ingest_mode pass-through, success counting |
| `test_search_literature_dedup.py` | `exclude_kb` filter and passthrough |
| `test_build_aggregator_scholar.py` | Scholar excluded when disabled, included when enabled |

---

## Quick health check

```bash
uv run pytest tests/unit/ -q   # should be 1610 passed, 1 skipped
uv run ruff check src/ --select I001,E501,RUF 2>&1 | grep -v "^Found\|^All\|schema.py:56\|schema.py:61\|models/papers.py" | head -20
```
