# Egress Policy

A self-contained guardrail that constrains **where a research run's traffic
may go** — which search engines run, which LLM/embeddings endpoints are
reached, and which URLs may be fetched.

> ⚠️ **Experimental (first release).** Surfaced in the UI as
> *Egress Scope (Experimental)*. It blocks the known egress paths under the
> selected scope and has been through two adversarial review rounds, but it is
> early — treat it as defense-in-depth, not an absolute guarantee, and don't
> make it the sole protection for highly sensitive data.

> **This is an in-process correctness guardrail, NOT a hard security
> boundary.** It defends against honest misconfiguration, prompt-injection-
> induced fetches, accidental egress, and the LangGraph silent-expansion
> class of bug. It does **not** defend against a determined adversary with
> code execution inside the LDR process. For a hard boundary, layer OS-level
> controls (network namespaces, firewall egress rules, restricted
> containers). See the root `SECURITY.md` for the full threat statement.

---

## 1. Mental model: egress scopes

Every run has an **egress scope** (`policy.egress_scope`). It decides which
engines/destinations are permitted.

| Scope | Meaning |
|---|---|
| `adaptive` *(default)* | **Follow the primary engine.** A private primary → behaves PRIVATE_ONLY; a public primary → PUBLIC_ONLY; an unclassifiable primary → BOTH. Resolved to a concrete scope at run start. Classification uses the engine's **static class flags** (e.g. SearXNG is always public because it queries the internet), not the network location of its server — with one asymmetric **fail-up** exception: a local-nature engine (Elasticsearch, Paperless) whose configured URL resolves to a *public* host is reclassified public, because querying it sends data off the box. The override only ever tightens. |
| `both` | Any classified engine (the pre-policy behaviour). |
| `public_only` | Only public web/academic engines. Local collections excluded. |
| `private_only` | Only local engines (library, collections, Ollama). **Forces local LLM + embeddings inference** so nothing leaves the box. Note: a locally-hosted SearXNG is still treated as *public* because it proxies to internet search engines — only its *connection target* is local, not its *data source*. |
| `strict` | Only the user's *primary* engine; no expansion at all. |

Two independent toggles refine inference locality (and are **implied** under
PRIVATE_ONLY / Adaptive-private):

- `llm.require_local_endpoint` — refuse cloud LLM providers / non-local URLs.
- `embeddings.require_local` — refuse cloud embeddings (e.g. OpenAI), and
  refuse a HuggingFace download for an uncached sentence-transformers model.

**Per-collection classification.** Each RAG collection carries an `is_public`
flag (default **private**). A private collection is excluded under
PUBLIC_ONLY / Adaptive-public and, when it is the primary under Adaptive,
pulls the whole run to PRIVATE_ONLY (→ local inference). To process a
collection with a cloud model, mark it **public** — the explicit opt-in.

---

## 2. Architecture: PDP + PEPs

Vocabulary borrowed from XACML / zero-trust:

- **PDP — Policy Decision Point.** The pure decision functions. *They live in
  this package* (`policy.py`).
- **PEP — Policy Enforcement Point.** The call sites that consult the PDP and
  act on the decision. *They are distributed across the codebase* at the
  places where engines/LLMs/URLs are actually used — a PEP can't be moved
  here because it lives in its domain (search factory, LLM config, fetcher,
  …). This package is the brain; the PEPs are the hands.

### Files in this package

| File | Role |
|---|---|
| `policy.py` | The PDP. `EgressScope`, `EgressContext`, `Decision`, `PolicyDeniedError`, `context_from_snapshot()` (builds the frozen per-run context + resolves Adaptive), and the `evaluate_*()` decision functions. Also host classification (DNS + private-IP, NAT64 metadata, percent-decode), the run-scoped DNS cache, and the denied-fetch quota. |
| `audit_hook.py` | A process-wide **PEP-578 `sys.audit` hook** on `socket.connect`. The *secondary* net: catches outbound connections from code paths the explicit PEPs don't cover (third-party libs, new contributors using raw `requests`). Inactive until a worker calls `set_active_context(ctx)`; only gates PRIVATE_ONLY / STRICT. |
| `fetch.py` | `policy_aware_validate_url()` — the egress-scope-aware SSRF wrapper (lets PRIVATE_ONLY reach private lab hosts without disabling SSRF globally; metadata IPs stay blocked). Depends on `../ssrf_validator.py`, not the other way around. |
| `warnings.py` | The three UI banner checks (`check_public_egress_enabled`, `check_cloud_llm_enabled`, `check_cloud_embeddings_enabled`), each with its own dismiss key. Pure functions; the `web/warning_checks` orchestrator calls them. |
| `validators.py` | Settings-save cross-field validators (`validate_allowed_local_hostnames`). Pure functions; the settings write routes call them. |
| `__init__.py` | Re-exports the public PDP / audit-hook API. |

### Public API

```python
from local_deep_research.security.egress import (
    EgressScope, EgressContext, Decision, PolicyDeniedError,
    context_from_snapshot,        # build the per-run context
    evaluate_engine,              # may engine X run under this scope?
    evaluate_retriever,           # … for a registered retriever
    evaluate_url,                 # may this URL be fetched?
    evaluate_llm_endpoint,        # is this LLM endpoint allowed?
    evaluate_embeddings,          # is this embeddings provider allowed?
    set_active_context,           # arm the audit-hook net for this thread
    clear_active_context,
    active_egress_context,        # context manager form
    install_audit_hook,
)
```

The decision functions return a `Decision(allowed: bool, reason: str)` where
`reason` is a short machine code (`"scope_mismatch_public_only"`,
`"blocked_metadata_ip"`, …) — never user content. PEPs raise
`PolicyDeniedError(decision)` on a hard denial.

---

## 3. Where the PEPs live (enforcement points)

The PDP is consulted at these call sites. This table is the map for "what
actually enforces the policy":

| Domain | PEP location | Consults |
|---|---|---|
| Search engine creation | `web_search_engines/search_engine_factory.py` (`create_search_engine`) | `evaluate_engine` / `evaluate_retriever` |
| LangGraph agent tools | `advanced_search_system/strategies/langgraph_agent_strategy.py` (`_build_tools`) | `evaluate_engine` / `evaluate_retriever` (tool-list filter) |
| Full-content fetch | `web_search_engines/engines/full_search.py` | `evaluate_url` |
| Content fetcher tool | `content_fetcher/fetcher.py` | `policy_aware_validate_url` + `evaluate_url` |
| Library downloads | `research_library/services/download_service.py` | `evaluate_url` at the fetch fire-points |
| LLM construction | `config/llm_config.py` (`get_llm`) | `evaluate_llm_endpoint` (+ snapshot-less allow-list) |
| Embeddings | `embeddings/embeddings_config.py`, `…/implementations/sentence_transformers.py`, `web_search_engines/engines/local_embedding_manager.py` | `evaluate_embeddings` / scope-coupled require-local |
| Journal reputation fetch | `advanced_search_system/filters/journal_reputation_filter.py` | scope skip |
| Run-start precheck | `web/routes/research_routes.py` (`_precheck_engine_policy`) | `evaluate_engine` (clean 400 at the API boundary) |
| Settings validation | `web/routes/settings_routes.py` (calls `egress/validators.py`) | `validate_allowed_local_hostnames` |
| Secondary net (all sockets) | `audit_hook.py` (installed at `security/__init__`) | `evaluate_url` on every `socket.connect` |

Adjacent (general security utils, not egress-specific — used here but live one level up):

- `security/ssrf_validator.py` — SSRF IP-class validation (`validate_url`, `is_ip_blocked`, metadata blocklist). The egress-scope-aware wrapper that *uses* it now lives in `fetch.py` here.
- `security/network_utils.py` — `is_private_ip`.

The UI orchestrator that calls `warnings.py` lives at `web/warning_checks/__init__.py` (it also handles non-egress warnings like backups/context, so it stays in the web layer).

---

## 4. Settings keys

| Key | Default | Effect |
|---|---|---|
| `policy.egress_scope` | `adaptive` | The scope (table above). |
| `llm.require_local_endpoint` | `false` | Refuse non-local LLM endpoints. |
| `embeddings.require_local` | `false` | Refuse cloud embeddings / uncached HF download. |
| `llm.allowed_local_hostnames` | `[]` | Hostnames treated as local by the classifier (public hosts rejected at save time). |
| `collections.is_public` (per collection) | `false` (private) | Collection egress classification. |
| `app.warnings.dismiss_egress_policy` / `…dismiss_cloud_llm` / `…dismiss_cloud_embeddings` | `false` | Per-banner dismiss flags (separate so dismissing one never hides the critical others). |

---

## 5. Threat model & caveats

**Covered:** forgotten/unreviewed code paths (via the audit hook), cloud LLM/
embeddings under a "local-only" claim, prompt-injection URL fetches, cloud-
metadata fetches under any scope, NAT64-wrapped metadata, percent-encoded
host bypass, DNS-timeout hangs, cache-hit policy bypass, and the LangGraph/MCP
silent-expansion timing leak.

**Caveats (important):**

- **Guardrail, not a boundary.** Code running inside the process can clear the
  context, monkey-patch the hook, or open a socket out-of-band.
- **DNS-rebinding TOCTOU.** The host is classified at evaluation time; the HTTP
  client resolves again at connect time. (Accepted risk; see `SECURITY.md`.)
- **LLM/embeddings endpoint locality is best-effort.** *Strong* for named
  providers (OpenAI/Anthropic/… blocked; Ollama/LM Studio/llama.cpp allowed);
  *weaker* for configurable-URL providers (`openai_endpoint`) classified by
  DNS — a private-looking URL that tunnels to the cloud is trusted as local.
  The policy stops *accidental/silent* egress; a user who deliberately points
  "local" inference at a cloud endpoint, or who marks a collection **public**,
  is making an informed choice.
- **Settings tampering.** An attacker who can write a user's per-user
  (SQLCipher) settings DB can flip the scope to `both` and disable enforcement.
  Policy-key changes emit `policy_audit=True` audit log lines.

---

## 6. Extending it — adding a PEP

1. Build the per-run context once: `ctx = context_from_snapshot(snapshot, primary, username=...)`.
   (At run start this is also armed for the audit hook via `set_active_context(ctx)`.)
2. At the point where you reach a destination, call the matching
   `evaluate_*` function.
3. On `decision.allowed is False`, raise `PolicyDeniedError(decision)` (hard
   stop) **or** log-and-skip if multi-source resilience matters — and emit a
   `logger.bind(policy_audit=True)` line. **Re-raise `PolicyDeniedError`
   before any broad `except Exception`** so a denial is never downgraded to a
   generic error (fail-open of the audit trail).
4. Prefer `is None` over truthiness when guarding on the snapshot — an empty
   dict is a real snapshot.

---

## 7. Tests

- `tests/security/test_egress_policy.py` — the PDP (scopes, adaptive
  resolution, classification, quota, DNS timeout).
- `tests/security/test_egress_audit_hook.py` — the PEP-578 hook (context
  lifecycle, re-entrancy, scope gating).
- `tests/security/test_egress_pep_coverage.py` — the distributed PEPs.
- Plus per-domain tests (factory, langgraph strategy, embeddings,
  download service, settings validators, warning banners, collections API).

## Data-flow reference (private vs public)

### Private-vs-Public Egress Data Flow (per scope)

```mermaid
flowchart TD
    Run([Research run starts]) --> Snap["context_from_snapshot()\nreads policy.egress_scope"]
    Snap --> Scope{egress_scope?}

    Scope -->|adaptive| Adapt{"_resolve_adaptive_scope()\nclassify PRIMARY engine"}
    Adapt -->|private primary| PRIV
    Adapt -->|public primary| PUB
    Adapt -->|unknown| BOTH

    Scope -->|private_only| PRIV["PRIVATE_ONLY\n(forces require_local_llm\n+ require_local_embeddings)"]
    Scope -->|public_only| PUB[PUBLIC_ONLY]
    Scope -->|both| BOTH[BOTH]
    Scope -->|strict| STRICT["STRICT\n(primary engine only)"]

    %% PRIVATE_ONLY: nothing leaves the box
    PRIV --> P_eng["evaluate_engine:\nlocal engines only\n(public BLOCKED)"]
    PRIV --> P_llm["evaluate_llm_endpoint:\ncloud providers BLOCKED\nlocal forced"]
    PRIV --> P_emb["evaluate_embeddings:\ncloud BLOCKED\nlocal forced"]
    PRIV --> P_url["evaluate_url:\nprivate host ALLOWED\npublic host BLOCKED"]
    PRIV --> P_hook["audit hook ARMED:\nany public socket BLOCKED"]
    P_eng --> NoEgress([No external data egress])
    P_llm --> NoEgress
    P_emb --> NoEgress
    P_url --> NoEgress
    P_hook --> NoEgress

    %% PUBLIC_ONLY: open to public sources only
    PUB --> U_eng["evaluate_engine:\npublic engines only\n(private collections BLOCKED)"]
    PUB --> U_url["evaluate_url:\npublic host ALLOWED\nprivate host BLOCKED"]
    PUB --> U_inf["LLM/embeddings:\nnot scope-forced\n(toggles still apply)"]
    PUB --> U_hook["audit hook INACTIVE\n(passthrough)"]
    U_eng --> PubEgress([Public-source egress permitted])
    U_url --> PubEgress
    U_inf --> PubEgress

    %% BOTH: any classified destination
    BOTH --> B_any["evaluate_*:\nany classified engine/host ALLOWED\nunclassified fails closed"]
    B_any --> AllEgress([Public + private egress])

    %% STRICT: primary engine only
    STRICT --> S_eng["evaluate_engine:\nONLY primary engine\nno expansion"]
    STRICT --> S_url["evaluate_url:\nprivate host ALLOWED\npublic host BLOCKED"]
    STRICT --> S_hook["audit hook ARMED:\nany public socket BLOCKED"]

    %% Metadata always blocked everywhere
    Meta["Cloud-metadata IPs\n(169.254.169.254, NAT64 wraps)"] -.->|BLOCKED under ALL scopes| Scope
```

### Egress vector x scope decision matrix

Each cell is the decision produced by the real `evaluate_*` logic / PEP for that vector. ADAPTIVE has no cell of its own: at run start it is resolved (`_resolve_adaptive_scope`) to PRIVATE_ONLY (private primary), PUBLIC_ONLY (public primary), or BOTH (unclassifiable), so it inherits that column. STRICT additionally requires the engine to BE the primary.

| Egress vector | adaptive (resolves to →) | both | public_only | private_only | strict |
|---|---|---|---|---|---|
| **Search engine selection** (`evaluate_engine`) | PRIVATE_ONLY \| PUBLIC_ONLY \| BOTH | any classified engine ALLOWED; unclassified BLOCKED | public engines ALLOWED; private collections BLOCKED | local engines ALLOWED; public BLOCKED | only PRIMARY engine ALLOWED; all others BLOCKED |
| **LLM inference** (`evaluate_llm_endpoint` + `get_llm`) | private→forced-local; public→cloud ALLOWED (unless toggle) | ALLOWED unless `llm.require_local_endpoint` set | ALLOWED unless toggle set (not scope-forced) | **forced-local** — cloud providers BLOCKED (`require_local_llm` implied) | ALLOWED unless toggle set (STRICT is orthogonal to inference) |
| **Embeddings / index** (`evaluate_embeddings`) | private→forced-local; public→cloud ALLOWED (unless toggle) | ALLOWED unless `embeddings.require_local` set | ALLOWED unless toggle set | **forced-local** — cloud embedders BLOCKED (`require_local_embeddings` implied) | ALLOWED unless toggle set |
| **Arbitrary URL fetch** (`evaluate_url`, content fetcher / full_search) | private→private-host only; public→public-host only | any classified host ALLOWED; unclassified BLOCKED | public host ALLOWED; private host BLOCKED | private host ALLOWED; **public host BLOCKED** | private host ALLOWED; public host BLOCKED |
| **Library download** (`evaluate_url`, download_service) | private→private-host only; public→public-host only | any classified host ALLOWED | public host ALLOWED; private BLOCKED | private host ALLOWED; **public BLOCKED** | private host ALLOWED; public BLOCKED |
| **Notification webhook** (`evaluate_url`, notifications/manager) | private→private-host only; public→public-host only | any classified http/https host ALLOWED | public webhook ALLOWED; private BLOCKED | private webhook ALLOWED; **public webhook BLOCKED** | private webhook ALLOWED; public BLOCKED |
| **Raw socket** (audit hook → `evaluate_url`) | private→ARMED (public BLOCKED); public→INACTIVE | INACTIVE (passthrough) | INACTIVE (passthrough) | **ARMED** — every public socket.connect BLOCKED | ARMED — every public socket.connect BLOCKED |
| **Cloud-metadata IP** (169.254.169.254 / NAT64) | BLOCKED | BLOCKED | BLOCKED | BLOCKED | BLOCKED |

### Key invariants (the crystal-clear takeaways)

| Claim | Where it is enforced |
|---|---|
| **private_only blocks ALL external data egress** | engine selection (`evaluate_engine` → `scope_mismatch_private_only`), URL/library/webhook fetch (`evaluate_url` PRIVATE_ONLY branch), LLM (`require_local_llm` forced in `context_from_snapshot` → `provider_cloud_only`), embeddings (`require_local_embeddings` forced → `provider_cloud`), and the raw-socket audit-hook net armed for PRIVATE_ONLY (`audit_hook.py`). No vector permits a public destination. |
| **public_only stays open to public sources** | public engines pass `evaluate_engine` (private collections BLOCKED via `scope_mismatch_public_only`); `evaluate_url` allows public hosts, blocks private; inference is NOT scope-forced (toggles still apply); the audit hook stays INACTIVE (only PRIVATE_ONLY/STRICT arm it) so legitimate local infra traffic — local Ollama, settings DB — is never falsely blocked. |
| **adaptive just matches the primary engine** | `_resolve_adaptive_scope`: private primary → PRIVATE_ONLY, public primary → PUBLIC_ONLY, unknown → BOTH. A registered local retriever primary also resolves PRIVATE_ONLY. Uses the engine's **static class flags** (`is_public`/`is_local` on the Python class), not the network location of the engine's server — so a locally-hosted SearXNG (class `is_public=True`) resolves to PUBLIC_ONLY. Resolved once at run start; the stored `EgressContext` carries the concrete scope, not ADAPTIVE. |
| **A private RAG collection as ADAPTIVE primary pulls the whole run local** | `_resolve_collection_is_public` defaults private → `_engine_bucket` returns local → adaptive resolves PRIVATE_ONLY → forces local LLM + embeddings. Mark the collection `is_public` to opt into cloud inference. |
| **Cloud-metadata is never reachable, under any scope** | `evaluate_url` runs `is_ip_blocked(..., allow_private_ips=True)` before scope logic; `_classify_host` applies the same metadata block on BOTH the literal-IP and DNS-resolved paths; NAT64-wrapped metadata is reclassified PUBLIC. |
| **An engine self-checks scope at run time, even off the factory path** | `BaseSearchEngine._verify_egress_scope()` runs at the top of `run()` (and inside `CollectionSearchEngine.search()` / `LibraryRAGSearchEngine.search()`, which bypass `run()`), re-evaluating `evaluate_engine` against the engine's stored snapshot and **raising `PolicyDeniedError`** on a mismatch. Memoized per snapshot identity + (scope, primary) values so the hot path pays the evaluation once. A defense-in-depth backstop behind the factory PEP for engines built by direct instantiation; it cannot deny anything the factory would have allowed. |
| **Direct MCP searches still arm the socket-level net** | `mcp/server.py::_egress_audit_net()` builds the run's `EgressContext` from the request snapshot and arms the PEP-578 audit hook around `engine.run()`, so a direct MCP search (which never goes through `AdvancedSearchSystem`) still gets the raw-socket backstop under PRIVATE_ONLY/STRICT. Fails open to a no-op when the policy is unevaluable — the factory PEP remains primary. |

### Advisory candidate pre-filter

LLM-selection callers can use `filter_candidates_by_egress(names,
snapshot)`, which handles the snapshot plumbing (scope/primary extraction,
context build, fail-open on any error) and delegates to
`filter_engines_by_egress(names, ctx, snapshot)`, which strips engines the scope
**actively denies** (scope mismatches, STRICT violations, unclassified
registry engines) from LLM candidate lists, so selection
slots aren't wasted on engines the factory would refuse. It is advisory,
not enforcement: names unknown to the static registry are KEPT (they may
be retriever-backed or dynamically injected engines the factory evaluates
via its own path) — the pre-filter is never stricter than the factory PEP
it fronts.

> **Notification footnote.** The notification PEP (`notifications/manager.py` `_filter_urls_by_egress_policy`) gates **http/https** webhook URLs through `evaluate_url`. Apprise vendor schemes (`slack://`, `discord://`, `telegram://`, `mailto://`, …) dispatch to external vendor APIs: under **PRIVATE_ONLY** (incl. Adaptive-private) they are **refused outright** (we can't verify a vendor token is local — fail closed; address a self-hosted notifier by its `http(s)://` URL, which `evaluate_url` allows as a private host). Under the other scopes they pass the URL gate (the modeled threat there is internal-http SSRF, not vendor APIs); the audit-hook net is an additional backstop for the actual connection when armed.
