# Contributing

Thanks for your interest in contributing to recon. Read the section below
first: it is the project's single non-negotiable rule.

## No real company data, ever

**Never commit real-company apex domains as examples, targets, test
corpora, regression fixtures, or anywhere else in this repository.**
This applies to commits, PRs, issue comments, code comments, YAML
fingerprints, test fixtures, snapshots, and discussion threads.

Use Microsoft's fictional-company names: `contoso.com`,
`northwindtraders.com`, `fabrikam.com`, `tailspintoys.com`,
`wingtiptoys.com`, `adventure-works.com`, `wideworldimporters.com`.
Or IETF reserved placeholders (`example.com`, `example.org`,
`example.net`).

It does **not** apply to:

- Vendor / product names recon detects as part of a tech stack
  (Cloudflare, Okta, Microsoft 365, Google Workspace, etc.).
  These are detection classes, not targeting examples.
- Upstream service hostnames recon itself queries
  (`login.microsoftonline.com`, `crt.sh`, `api.certspotter.com`,
  `dns.google`, etc.).

Accuracy reports and live-validation work use real apexes locally;
that's why `validation/corpus-private/` and `validation/runs-private/`
are permanently gitignored. The committed validation tooling (the
runner, gap finder, triage script, synthetic-corpus generator, the
synthetic corpus itself) is allowed; real-apex inputs and per-domain
outputs are not. Aggregate counts produced by the corpus aggregator
may be committed; per-domain results never.

If you're writing a bug report that needs a real domain to reproduce,
file it privately or describe the behaviour using the fictional
examples whenever the real name isn't load-bearing. The rationale: no
upside, and accumulating reputational and legal downside over the
lifetime of the repository.

Reviewers actively check incoming PRs for real-apex strings before
merge. A PR found to contain real-company data will be rejected and
the contributor asked to refactor it against the fictional brands.

---

This doc answers:

- [What kinds of contributions we're looking for](#what-were-looking-for)
- [What's out of scope](#whats-out-of-scope)
- [Local customization vs upstream contribution](#local-customization-vs-upstream-contribution)
- [How to contribute a fingerprint](#adding-fingerprints)
- [How to contribute a signal or profile](#adding-signals-and-profiles)
- [How to contribute code](#code-changes)
- [How to report a bug or suggest an idea](#bug-reports-and-ideas)

## Quick Start

```bash
git clone https://github.com/blisspixel/recon.git
cd recon
uv sync                    # installs the dev group (pip: pip install -e . --group dev, pip 25.1+)
pre-commit install                     # activate pre-commit hooks
uv run pytest tests/                   # or: pytest tests/
```

---

## What we're looking for

recon is a **narrow tool** focused on one thing: passive, zero-creds
domain intelligence. Contributions that tighten what the tool already does
are welcome:

| Contribution type | Bar | Examples |
|---|---|---|
| **New fingerprints** | Must use public DNS/CT only. Specific pattern, not generic substring. Tested against a real customer domain. | "Add Statsig TXT verification", "Add Workspace ONE CNAME" |
| **Refined fingerprints** | `match_mode: all` to eliminate false positives, improved regex, broader selector coverage. | Converting ambiguous slug to a chained pattern |
| **New signals** | Must derive from existing evidence. Hedged language. | "AI tooling indicators observed alongside identity-provider signals" |
| **New profiles** | Must reweight existing observations; cannot invent new ones. | "Retail / e-commerce" profile |
| **Bug reports** | Reproducible + output of `recon <domain> --json --explain`. | Wrong provider classification, insight wording, display glitch |
| **Accuracy reports** | Domain you know ground truth for, plus what recon got wrong. | "Our org uses M365 primary; recon says Exchange on-prem" (omit real names in public reports) |
| **Documentation fixes** | Typos, clarifications, better examples. | README, CONTRIBUTING, docs/* |
| **Performance improvements** | With before/after measurement. Hot paths: `detect_provider`, `_curate_insights`, DNS query batching. | Reducing average lookup time, eliminating redundant DNS queries |
| **Test additions** | Sparse-data fixtures, adversarial inputs, corner cases. | Domain with zero MX + no tenant, IDN / punycode, wildcard DNS |

The **engine stays lean; the data grows**. Most contributions should be
YAML additions or refinements; that is the intended scaling path for the
tool. Code changes earn their place against the post-1.0 ethos: correctness,
reliability, explainability, composability, then new features, in that
order.

---

## What's out of scope

These are **intentional rejections**, not gaps waiting to be filled. Don't
spend effort on PRs that will be closed:

### Violates invariants (hard no)

- **Active scanning of any kind.** No HTTP probes against target domains
  (even "light"), no port scans, no TLS handshakes, no brute-forcing.
- **Anything requiring credentials or API keys.** No OAuth, no PAT, no
  paid APIs (SecurityTrails, Censys, Shodan, etc.).
- **Aggregated local database.** Per-domain JSON files only. No SQLite/DuckDB
  store.
- **Bundled ML models, embeddings, or large data files.** ASN / GeoIP /
  fingerprint embeddings are all out.
- **Plugin systems that run user code.** YAML-only extensibility.

### Not this tool (soft rejections)

- HTML / PDF output, web dashboard, TUI, `recon serve`, interactive REPL
- STIX / MISP / Maltego / Excel / SBOM / Sigstore exports
- Scheduled / daemon mode with alerts
- Generic subdomain brute-forcing (DNS wordlist attacks)
- People-search / email-harvesting OSINT
- Docker image, Homebrew tap, PyInstaller single-binary

If you want any of these, **pipe `--json` into the right tool**. recon is a
CLI + a local-stdio MCP server; the integration surface is JSON, pipe it
wherever you need.

If unsure whether your idea fits, open an **idea** issue (template below)
and ask before writing code.

---

## Local customization vs upstream contribution

recon supports both.

**Local customization** (for your own use, never pushed):
- `~/.recon/fingerprints.yaml`: custom fingerprints (additive only; cannot override built-ins)
- `~/.recon/signals.yaml`: custom signal rules
- `~/.recon/profiles/*.yaml`: custom posture profiles
- `RECON_CONFIG_DIR`: override the default `~/.recon` location

Anything that's specific to your organization, your vendors, or your
internal naming should stay local.

**Upstream contribution** (shared with everyone via a PR):
- Detections that apply to at least one publicly-known service
- At least one real customer domain you can point to as evidence
  (kept locally and described in the PR body, not committed)
- General-purpose (not a specific org's internal pattern)

---

## Validation strategy

The committed corpus is synthetic-only. The 79-fixture synthetic
corpus lives at `validation/synthetic_corpus/` along with its
generator and the per-run aggregate output. Both are safe to commit
and reproducible by anyone who clones the repo.

Real-apex validation is maintainer-side. Real corpora land in
`validation/corpus-private/` (gitignored). Per-run scans land in
`validation/runs-private/<UTC-stamp>/` (gitignored). The corpus
aggregator (`validation/corpus_aggregator.py`) emits aggregate
counts only and is safe to run locally; per-domain results never
leave the maintainer's machine.

Release notes default to citing synthetic numbers. Real-corpus
aggregate counts may be cited in release docs when they are
load-bearing (the v2.0 corpus-run runbook in
`validation/v2.0-corpus-run-runbook.md` documents this path), but
the per-domain corpus stays local.

---

## Adding fingerprints

The most common contribution. From v1.1 onward, fingerprints live under
`recon_tool/data/fingerprints/`, one YAML file per category.

### 1. Find the right file

```bash
recon fingerprints list --category ai       # where do AI tools live?
recon fingerprints show openai              # what does the existing pattern look like?
```

Current layout:

```
recon_tool/data/fingerprints/
├── ai.yaml             AI / LLM providers, agent frameworks
├── email.yaml          Email providers, gateways, DMARC / DKIM tooling
├── security.yaml       EDR, SIEM, IdP, zero-trust access
├── infrastructure.yaml Cloud, CDN, DNS, CAs, CI/CD
├── productivity.yaml   Suite tools, helpdesk, HR
├── crm-marketing.yaml  CRM, sales intel, ad platforms
├── data-analytics.yaml Warehouses, BI, observability
└── verticals.yaml      Education, nonprofit, payments
```

Pick the file that matches your service's primary category. If nothing
fits, open an issue before adding a new category file; the split is
intentionally coarse.

### 2. Add your entry

Use the scaffolding command when possible:

```bash
recon fingerprints new service-slug
```

It walks through the stable schema and specificity checks before emitting YAML.

```yaml
- name: Service Name
  slug: service-slug          # lowercase, unique across ALL files
  category: Category Name     # use an existing category if possible
  confidence: high             # high, medium, or low
  detections:
    - type: txt                # txt, spf, mx, ns, cname, subdomain_txt, caa, srv, dmarc_rua
      pattern: "^service-domain-verification="
      description: What this record means
      reference: https://vendor.example/docs/domain-verification
```

`description` should explain the observable record. `reference` is optional,
but preferred when public vendor documentation exists. Do not add fields that
are not in the stable data-file schema unless a separate schema change has
already been accepted.

### 3. Validate locally before opening a PR

```bash
recon fingerprints check                    # validates the built-in catalog
recon fingerprints check path/to/custom.yaml  # validate a candidate file
```

The `check` command runs the same validation recon uses at runtime
(regex safety via a ReDoS heuristic, required fields, known detection
types, weight range 0.0 to 1.0, `match_mode` value), **plus** a cross-file
duplicate-slug check. Exits 0 on success, 1 on failure with per-entry
error messages.

Under the hood this uses the packaged `recon_tool.fingerprint_validator`
module, so it works from installed wheels as well as source checkouts.
Contributors who prefer invoking the wrapper script directly can still run
`python scripts/validate_fingerprint.py <path>`.

### Chained patterns (`match_mode: all`)

For high-confidence attribution where a single record could be a false
positive, use `match_mode: all`: the fingerprint only fires when every
listed detection matches. See [docs/fingerprints.md](docs/fingerprints.md#chained-patterns-match_mode-all) for details.

For multi-detection changes, run:

```bash
python -m validation.audit_fingerprints
```

The audit is advisory. Use it to document whether the entry should remain
`any`, move to `all`, or be tightened before changing behavior.

### Fingerprint PR checklist

- [ ] Validates locally with `recon fingerprints check`
- [ ] `recon fingerprints show <slug>` displays your new entry after load
- [ ] Slug does not collide with an existing one (`recon fingerprints list | grep <slug>`)
- [ ] At least one detection pattern uses a service-specific token (not a generic substring)
- [ ] Detection metadata includes `description`, plus `reference` when public vendor docs exist
- [ ] Tested against a real public domain you know uses the service
- [ ] The service's DNS footprint is documented or publicly-observable (not leaked from a customer engagement)
- [ ] Multi-detection entries include a `match_mode` rationale from the audit output or PR notes
- [ ] Common legitimate false-negative cases are captured in the PR body, or in `docs/weak-areas.md` when broadly useful
- [ ] `pytest tests/` passes; property tests must still hold on the sparse-data corpus
- [ ] If the service could trip false positives on domains that merely *visited* the vendor's marketing site, used `match_mode: all`

### Detection description rubric (v1.9.7+)

Every detection rule in `recon_tool/data/fingerprints/*.yaml` must
carry a non-empty `description` field. The metadata-coverage gate
(`scripts/check_metadata_coverage.py`) enforces presence at commit
time via pre-commit and at release time via CI. There is no
percentage threshold. Every detection, every category.

The gate enforces *presence*, not quality. The rubric below raises
the bar for new contributions; reviewer judgement is the enforcer
of quality.

**Three-part rubric:**

1. **What the slug detects.** Vendor and the specific evidence
   pattern the detection matches (TXT verification token, MX
   record terminus, CNAME chain, NS pattern, certificate SAN,
   and so on).
2. **What it doesn't detect.** When the slug fires, what claim is
   *not* being made. Common framings: "fires on `X`, not on `Y`";
   "indicates administrative binding to `TENANT`, not active use";
   "evidence of `PRODUCT`, not of `RELATED_PRODUCT` in the same
   vendor family."
3. **Common false positives if known.** Patterns where the
   detection fires correctly but the operator-facing inference
   would be misleading. Skip if no known FPs. Do not invent them.

**Tone:** humble, factual, no overclaim. recon's catalog reads
like an analyst's careful notes, not marketing copy. Avoid
adjectives like "strong", "robust", "cleanly", "the best", and
similar self-congratulatory framings. State what the evidence
shows and what it does not show. Acknowledge limits.

**Style:** no em-dashes (use commas, periods, or parentheses
instead). No emojis. Length target: 1 to 3 sentences. Terser
than docstrings, longer than slug names. The goal is that a
future contributor (or an AI agent reading the YAML) understands
the claim's edge cases without grepping source.

**Worked examples:**

Good (`microsoft365` TXT detection):

```yaml
detections:
  - type: txt
    pattern: ^MS=ms\d{8}$
    description: >-
      Microsoft 365 domain-verification TXT token. Indicates the
      domain is administratively bound to a Microsoft 365 tenant.
      Does not prove the tenant is actively used for email. Some
      orgs verify the domain to claim the namespace but route
      mail elsewhere, so corroborate with MX evidence before
      reading this as "they run M365 email."
```

Good (`cloudflare` NS detection):

```yaml
detections:
  - type: ns
    pattern: \.cloudflare\.com\.$
    description: >-
      Apex NS records resolve to Cloudflare nameservers,
      indicating Cloudflare is the authoritative DNS provider
      for the domain. Does not prove the apex sits behind
      Cloudflare's CDN or WAF (those are separate products on
      the same vendor). Corroborate via Server headers or
      CDN-specific chain motifs for the front-the-origin claim.
```

Placeholder (rejected):

```yaml
detections:
  - type: txt
    pattern: ^MS=ms\d{8}$
    description: "Detects Microsoft 365."
```

Rejection reason: does not explain what the pattern matches, or
what claim is or is not being made. Empty-string presence is the
floor; the rubric is the bar.

### Vendor-doc-sourced `cname_target` rules

`cname_target` rules in `recon_tool/data/fingerprints/surface.yaml`
can be added two ways:

1. **Corpus-observed.** The default historical path: a private-corpus
   scan surfaces an unclassified CNAME terminus, the
   `/recon-fingerprint-triage` skill proposes a slug, and the rule
   ships in a release-window catalog-growth pass.
2. **Vendor-doc-sourced** *(v1.9.3.9+)*. A cloud vendor (GCP, AWS,
   Azure, Oracle, IBM, Alibaba, SSE/SASE platforms, PaaS providers,
   etc.) documents customer-facing custom-domain CNAME targets in
   their own product docs. Reading those docs and seeding fingerprints
   from them closes coverage blindspots BEFORE a customer of that
   vendor lands in our private corpus. Corpus-observed alone has a
   built-in bias toward the segments our corpus already represents.

Both paths are encouraged. The **`reference`** field on each
detection makes the source explicit:

- Vendor-doc-sourced: `reference` MUST point at the canonical vendor
  documentation URL that names the CNAME pattern (e.g.
  `https://firebase.google.com/docs/hosting/custom-domain` for
  Firebase Hosting). The PR description should quote the relevant
  doc excerpt so review can verify the pattern is documented as
  stable, not a transient internal endpoint.
- Corpus-observed: `reference` SHOULD point at the vendor's
  marketing page or main docs index, and the PR description should
  cite the private-corpus delta count (e.g. "fired on 23/4270
  domains in the v1.9.2-pre-release scan").

Either way, `reference` exists so a future maintainer can re-verify
the pattern. A rule shipped without a `reference` is allowed but
will be flagged by the advisory metadata-richness audit
(`scripts/check_metadata_coverage.py --report-richness`, v1.9.8+).

Use the fingerprint PR template; GitHub surfaces it automatically.

---

## Adding signals and profiles

> **Heads up: engine changes go through a design doc.** Fingerprints
> are data; contributors can iterate on them freely. The signal,
> fusion, and absence engines are inference code; bad changes there
> affect every domain recon analyses, not just the one you tested.
> Before PRing a change to `recon_tool/signals.py`, `merger.py`,
> `absence.py`, `fusion.py`, or the two-pass evaluator, please open
> an issue with:
>
> 1. What inference pattern you want to add or change.
> 2. Which domains in the validation corpus would fire differently.
> 3. How you'd guard against false positives on sparse-signal domains.
>
> Data-only contributions (new signals in `signals.yaml` that reuse
> the existing engine) don't need this; they follow the fingerprint
> PR workflow.

### Signals

Custom signal rules derive insights from fingerprint matches. Go in
`~/.recon/signals.yaml` (local) or `recon_tool/data/signals.yaml`
(upstream):

```yaml
signals:
  - name: My Custom Signal
    category: Custom
    confidence: medium
    description: Hedged explanation of what this signal suggests
    requires:
      any: [slug-a, slug-b, slug-c]    # fingerprint slugs to match
    min_matches: 2                       # how many must be present
```

Signals must use **hedged language** in their `description` (observed,
likely, fits a pattern) unless the evidence density genuinely justifies a
firmer claim. Sparse-data targets should never see confident verdicts.

See [docs/signals.md](docs/signals.md) for the full schema, including
`metadata` conditions, `contradicts`, `requires_signals`, and
`positive_when_absent`.

### Profiles

Profiles reweight existing observations for a specific audience (fintech,
healthcare, higher-ed, etc.). Go in `~/.recon/profiles/{name}.yaml` (local)
or `recon_tool/data/profiles/{name}.yaml` (upstream):

```yaml
name: retail
description: E-commerce / retail posture lens
category_boost:
  email: 1.5          # email security matters more for retail
  saas_footprint: 1.2
signal_boost:
  "DMARC Governance Investment": 1.5
focus_categories: [email, identity, consistency]
```

Profiles are **additive only**: they cannot introduce new observations,
only reweight existing ones, and they cannot create false confidence (caps
at "high" salience).

---

## Code changes

- Run `pre-commit run --all-files` or `ruff check recon_tool/` and `pyright recon_tool/` before submitting.
- Run `pytest tests/ --cov=recon_tool --cov-branch`; branch coverage must stay above 82%.
- Integration tests (`pytest -m integration`) require network access and are skipped by default.
- Keep PRs focused: one concern per PR.

### Post-1.0 bar for code changes

From 1.0 onward, `docs/stability.md` defines the stable public surface.
Any change that touches a stable surface needs:

- A clear reason the change can't be done as data-only (YAML).
- An explanation of backward compatibility (or a deprecation plan if the
  change is breaking, which requires a major version bump).
- Tests covering both existing consumers and the new behavior.

The bar is deliberately high. Most "I want recon to do X" ideas can be
solved with a new fingerprint, signal, or profile rather than new code.

---

## CPT-change discipline (v1.9.6+)

The v1.9 Bayesian layer (`recon_tool/data/bayesian_network.yaml`) is a
**data file with semantic content**, not a free parameter surface to
tune against the corpus. Every CPT entry encodes a claim about how
evidence should move the posterior. Changing a number changes a
claim.

The discipline is one rule: **corpus runs are mirrors, not fitters.**
When the corpus disagrees with the network, the first question is
*is this node asking the right question?*, not *what number gets
the disagreement number down?* Only after the topology is clean do
CPT numbers get re-examined, and only with explicit reasoning
written in the YAML alongside the change.

The deep-dive reference, with worked examples (v1.9.3 topology
surgery on `email_security_strong`, v1.9.6 binding removal on
`email_security_policy_enforcing`), the anti-pattern catalog, and
the concept-comment requirement, lives at
[`docs/bayesian-cpt-discipline.md`](docs/bayesian-cpt-discipline.md).
Read it before changing any CPT entry.

---

## Bug reports and ideas

Three issue templates:

- **Fingerprint request**: you want recon to detect a service it
  currently misses.
- **Bug report**: recon got something wrong, crashed, or behaved
  unexpectedly.
- **Idea**: you want to propose something larger (new flag, new MCP
  tool, new CLI command, etc.).

For ideas, the template asks you to confirm the proposal fits the
project invariants (passive, zero-creds, no aggregated DB, no active
scanning). Ideas that don't fit will be politely closed with a pointer
to `--json` + pipe to the right tool.

---

## Pull request etiquette

- Keep PRs focused: one feature or fix per PR.
- Fingerprint-only PRs are welcome and easy to review.
- Include a brief description of what you changed and why.
- Reference the issue number if there is one.
- Be patient: this is a small project with a single maintainer.
