# Phase 166.1: Contentful Source Registration Default - Research

**Researched:** 2026-04-13
**Domain:** arXiv registration default semantics, discovery forwarding parity, and fresh-topic source-registration proof
**Confidence:** HIGH

<user_constraints>
## User Constraints (from CONTEXT.md)

### Locked Decisions
- `register_arxiv_source.py` must become contentful-by-default rather than
  metadata-only-by-default.
- `--metadata-only` must be the explicit lightweight opt-out.
- `--download-source` must remain available as a compatibility-positive alias.
- `discover_and_register.py` must inherit the same default behavior and expose
  the same `--metadata-only` opt-out.
- the intake compatibility wrapper must keep routing through the same Layer 0
  helper rather than forking behavior.
- operator docs must stop teaching `--download-source` as the normal path.
- the phase must leave one bounded fresh-topic proof of
  `bootstrap -> handoff -> registration` on the real repository kernel root.
- the fallback honesty rule stays unchanged: if source download or extraction
  fails, the paper still registers and `snapshot.md` must show the failure.

### the agent's Discretion
- exact parser mechanics between `--download-source` and `--metadata-only`
- whether compatibility help text calls `--download-source` deprecated or just
  keeps it as a harmless synonym
- exact proof artifact naming and evidence layout for the fresh-topic lane

### Deferred Ideas (OUT OF SCOPE)
- non-arXiv provider defaults
- automatic registration from the runtime queue
- wider retry/caching redesign for source acquisition

</user_constraints>

<research_summary>
## Summary

This phase is a narrow semantic correction, not a new workflow. The repository
already has the right durable fallback behavior for partial acquisition:
`register_arxiv_source.py` writes registration artifacts even when bundle
download or extraction fails, and `discover_and_register.py` already routes
through that helper. The main gap is that both surfaces still default to
`download_source=False`, so the new concrete `L0` handoff from Phase `166`
still drops the operator into metadata-only behavior unless they remember
`--download-source`.

**Primary recommendation:** move the default truth into the shared helper
signature (`download_source=True`), make CLI flags override that default with a
new explicit `--metadata-only` opt-out, preserve `--download-source` as an
explicit compatibility-positive flag, then align the discovery bridge and docs
to the same story. Close the phase with one bounded fresh-topic proof that
starts from the already-shipped front door, follows the new handoff, and lands
one real registration artifact set on the repository kernel root without
pretending download success when content acquisition fails.
</research_summary>

<standard_stack>
## Standard Stack

The right implementation surface is entirely repo-local.

### Core
| Library | Version | Purpose | Why Standard |
|---------|---------|---------|--------------|
| `research/knowledge-hub/source-layer/scripts/register_arxiv_source.py` | repo-local | Canonical Layer 0 arXiv registration helper | This is the source of truth for download default semantics and fallback reporting |
| `research/knowledge-hub/source-layer/scripts/discover_and_register.py` | repo-local | Query-first discovery bridge into registration | Phase `166` made this the primary operator lane, so default parity must land here too |
| `research/knowledge-hub/tests/test_source_discovery_contracts.py` | repo-local | Direct registration + discovery-to-registration regression surface | Already covers both entry routes and is the main place to pin the new semantics |
| `research/knowledge-hub/intake/ARXIV_FIRST_SOURCE_INTAKE.md` | repo-local | Operator-facing runbook | Must stop teaching `--download-source` as the normal path |

### Supporting
| Library | Version | Purpose | When to Use |
|---------|---------|---------|-------------|
| `research/knowledge-hub/L0_SOURCE_LAYER.md` | repo-local | Canonical Layer 0 acquisition policy | Use to keep docs aligned with the arXiv-first contentful story |
| `research/knowledge-hub/README.md` | repo-local | Top-level operator references | Update if the normal registration example still implies metadata-only |
| `research/knowledge-hub/runtime/scripts/run_first_run_topic_acceptance.py` | repo-local | Existing first-run front-door proof lane | Reuse or extend this surface for the bounded handoff-to-registration proof |
| `research/knowledge-hub/tests/test_aitp_cli_e2e.py` | repo-local | Subprocess CLI regression surface | Use if the proof lane is exposed through CLI or acceptance-script behavior |

### Alternatives Considered
| Instead of | Could Use | Tradeoff |
|------------|-----------|----------|
| helper-level default flip | CLI-only default flip | Leaves Python callers and bridge logic semantically behind the CLI story |
| explicit `--metadata-only` opt-out | remove `--download-source` immediately | Cleaner CLI, but breaks compatibility too abruptly for current docs and wrappers |
| one shared proof lane | docs-only claim that the path works | Leaves `REQ-VERIFY-01` open and invites drift between docs and reality |

**Installation:**
```bash
# No new dependencies required for Phase 166.1.
```
</standard_stack>

<architecture_patterns>
## Architecture Patterns

### Pattern 1: Function defaults are the truth; CLI flags only override
**What:** set `download_source=True` in the shared registration helper so both
direct Python callers and CLI entrypoints become contentful-by-default.
**When to use:** when multiple entry surfaces ultimately depend on one helper.
**Example:**
```python
def register_arxiv_source(..., download_source: bool = True, ...) -> dict[str, Any]:
    ...
```

### Pattern 2: Mutually exclusive explicit override flags
**What:** expose one explicit negative override (`--metadata-only`) and keep
the existing positive form (`--download-source`) as a compatibility synonym.
**When to use:** when behavior is changing but old command forms should stay
accepted.
**Example:**
```python
group = parser.add_mutually_exclusive_group()
group.add_argument("--download-source", dest="download_source", action="store_true")
group.add_argument("--metadata-only", dest="download_source", action="store_false")
parser.set_defaults(download_source=True)
```

### Pattern 3: Reuse the front-door proof lane rather than inventing a second entry story
**What:** prove registration by continuing from the same fresh-topic bootstrap
lane already used for first-run acceptance, then following the concrete handoff
into one bounded registration call.
**When to use:** when the phase requirement explicitly targets
`bootstrap -> handoff -> registration`.
**Example:**
```text
bootstrap -> status confirms concrete L0 handoff ->
register_arxiv_source.py or discover_and_register.py ->
check source-layer/topics/<slug>/sources/<source_slug>/
```

### Anti-Patterns to Avoid
- **CLI-only behavior flip:** if `register_arxiv_source()` stays default-false,
  the discovery bridge and tests can silently drift behind the docs.
- **Discovery/direct divergence:** making only direct registration contentful by
  default leaves the new Phase `166` primary handoff semantically wrong.
- **Pretend-content success:** download/extraction failure must remain visible
  in `snapshot.md` and result payloads.
- **Synthetic proof claims:** `REQ-VERIFY-01` is specifically about a fresh
  front-door topic path, not just isolated helper-unit coverage.

</architecture_patterns>

<dont_hand_roll>
## Don't Hand-Roll

| Problem | Don't Build | Use Instead | Why |
|---------|-------------|-------------|-----|
| new download state model | fresh schema or new acquisition status tree | existing `download_status` / `extraction_status` contract | the fallback surface already expresses success vs failure honestly |
| separate discovery semantics | a special-case default only inside `discover_and_register.py` | one shared helper default plus aligned parser override | direct and query-first registration must stay equivalent |
| front-door proof | an unrelated synthetic fixture-only check | the existing first-run acceptance lane plus one bounded registration continuation | requirement explicitly references bootstrap/handoff/registration continuity |

**Key insight:** the repository already knows how to register a paper and how
to prove first-run front-door behavior. Phase `166.1` should connect those two
surfaces cleanly rather than inventing a new orchestration layer.
</dont_hand_roll>

<common_pitfalls>
## Common Pitfalls

### Pitfall 1: Flipping only the parser default
**What goes wrong:** CLI examples look correct, but direct helper callers and
bridge code still default to metadata-only behavior.
**How to avoid:** change both the function default and the parser default.

### Pitfall 2: Forgetting discovery-to-registration parity
**What goes wrong:** the new Phase `166` handoff points to
`discover_and_register.py`, but the bridge still forwards `download_source=False`.
**How to avoid:** treat the bridge as part of the same semantic change, not as
an afterthought.

### Pitfall 3: Treating `--download-source` as the new normal in docs
**What goes wrong:** shipped behavior is improved, but operator docs continue
to teach an obsolete explicit flag.
**How to avoid:** update examples to show the normal command without
`--download-source` and document `--metadata-only` as the exceptional path.

### Pitfall 4: Closing the milestone without a fresh-topic proof
**What goes wrong:** unit regressions are green, but `REQ-VERIFY-01` remains
open because the full handoff path was never exercised.
**How to avoid:** leave one bounded evidence lane that starts from bootstrap,
observes the concrete handoff, and creates real registration artifacts for a
fresh topic.

</common_pitfalls>

<code_examples>
## Code Examples

Verified patterns from the current repository:

### Registration already has honest fallback reporting
```python
# Source: research/knowledge-hub/source-layer/scripts/register_arxiv_source.py
download_status = "not_requested"
extraction_status = "not_requested"
...
except (urllib.error.URLError, TimeoutError, OSError) as exc:
    download_status = "failed"
    extraction_status = "skipped"
    download_error = str(exc)
```

### Discovery already forwards into the canonical registration helper
```python
# Source: research/knowledge-hub/source-layer/scripts/discover_and_register.py
registration = register_module.register_arxiv_source(
    ...,
    download_source=download_source,
    ...
)
```

### The first-run front door already has a bounded acceptance harness
```python
# Source: research/knowledge-hub/runtime/scripts/run_first_run_topic_acceptance.py
bootstrap_payload = run_cli_json(...)
loop_payload = run_cli_json(...)
status_payload = run_cli_json(...)
```
</code_examples>

<sota_updates>
## State of the Art (2026 Repository State)

| Old Approach | Current Repository Baseline | Target for Phase 166.1 |
|--------------|-----------------------------|------------------------|
| metadata-only default | `download_source=False` in direct registration and discovery bridge | contentful-by-default with explicit `--metadata-only` |
| positive explicit flag only | `--download-source` required in docs and CLI | `--download-source` remains accepted but no longer required |
| front-door proof stops at handoff | Phase `166` proves concrete `L0` handoff only | milestone closes with one bounded registration continuation |

**Deprecated/outdated after this phase:**
- teaching `--download-source` as the normal arXiv registration path
- assuming the new handoff is sufficient without a registration proof lane

</sota_updates>

<open_questions>
## Open Questions

1. **Should the fresh-topic proof use direct registration or discovery-first registration?**
   - What we know: Phase `166` primary handoff is discovery-first, but
     `register_arxiv_source.py` is the narrower surface for asserting default
     semantics directly.
   - Recommendation: prove the direct registration default mechanically in unit
     coverage, then let the bounded fresh-topic lane continue from the handoff
     into the simplest deterministic registration step that still satisfies the
     milestone narrative.

2. **Should the proof lane hit the network?**
   - What we know: arXiv metadata/source downloads are live surfaces, but the
     repo already supports metadata overrides and offline search fixtures.
   - Recommendation: keep regression tests deterministic with overrides; leave
     the bounded fresh-topic repo-root proof free to record honest failure if a
     live download cannot complete, as long as registration artifacts still
     land and the failure is visible.

</open_questions>
