---
id: 11791
title: Operator docs + health/telemetry for tenant-repo ingestion
state: CLOSED
labels:
  - documentation
  - enhancement
  - ai
assignees:
  - neo-opus-ada
createdAt: '2026-05-22T21:35:48Z'
updatedAt: '2026-06-07T17:14:26Z'
githubUrl: 'https://github.com/neomjs/neo/issues/11791'
author: neo-opus-ada
commentsCount: 3
parentIssue: 11731
subIssues: []
subIssuesCompleted: 0
subIssuesTotal: 0
blockedBy:
  - '[x] 11790 tenant-repo-sync scheduler lane — periodic + manual'
blocking: []
closedAt: '2026-05-28T00:37:34Z'
---
# Operator docs + health/telemetry for tenant-repo ingestion

## Context

Sub 6 of Epic #11731 (*Server-side tenant-repo ingestion for cloud Agent OS deployments*), graduated from Discussion #11782. This sub makes the new pull-based mode operable + observable.

## The Problem

A deployment operator needs to (a) configure tenant repos for server-side ingestion, and (b) see repo-freshness without reading raw Chroma rows. Discussion #11782's §5.2 Step-Back sweep flagged the consumer surfaces this mode touches.

## The Fix

- **Operator docs**: extend the cloud-deployment guides (`learn/agentos/cloud-deployment/TenantIngestionModel.md` and adjacent) for the server-side pull mode — config, the credentialed-reference contract, the periodic/manual triggers, the repo-mirror volume.
- **Health/telemetry**: surface per-repo freshness (last-ingested revision, last successful sync time, active/disabled/quarantined status) on the health/readiness surface + operator logging/summary output.
- **Consumer-sweep surfaces** (Step-Back pt 2): deployment compose/volumes, health/readiness, backup/redeploy-survival docs, tenant config storage docs, parser/source-family docs, deletion telemetry.

## Acceptance Criteria

- [ ] Cloud-deployment docs cover server-side tenant-repo ingestion config + triggers + the credentialed-reference contract + the repo-mirror volume.
- [ ] Health/telemetry surfaces per-repo freshness (revision, last-sync-time, status) without requiring raw Chroma reads.
- [ ] Operator logging/summary output for refresh cycles.
- [ ] The consumer-sweep surfaces enumerated above are each addressed or explicitly deferred with rationale.

## Out of Scope

- The implementation of the service/lane/contract (subs 2–5 / #11731).

## Contract Ledger

Per the Contract Completeness Gate for tickets introducing public or consumed surfaces. #11791 introduces operator-visible docs plus health/telemetry surfaces that consume #11787 tenant config, #11788 mirror state, #11789 envelope semantics, and #11790 scheduler state. Contract surface is enumerated below.

| Target Surface | Source of Authority | Proposed Behavior | Fallback / Edge Case | Docs | Evidence |
| :--- | :--- | :--- | :--- | :--- | :--- |
| `TenantIngestionModel.md` pull-mode operator section | Discussion #11782 Step-Back consumer sweep + ADR 0014 tenant-repo amendment | Add a dedicated server-side pull section covering `tenantRepos[]`, credential-reference rules, periodic/manual triggers, mirror volume, redeploy posture, and push-vs-pull coexistence. | Cross-link from the existing push-mode model so operators see the pull path as additive, not a push replacement. | `TenantIngestionModel.md` + cloud-deployment overview links | Markdown link check or targeted doc grep proving every sibling sub is linked |
| Deployed health/readiness payload | #11791 AC2 + existing `HealthService.recordTaskOutcome()` / `healthcheck.orchestrator.tasks` pattern | Surface per-repo freshness through the existing Memory Core healthcheck orchestrator task block, e.g. `orchestrator.tasks['tenant-repo-sync'].details.repos[]` with `{tenantId, repoSlug, lastIngestedRev, lastSyncAt, status, lastErrorCode?, lastSyncDeletedCount?}`. | Empty `tenantRepos[]` -> `repos: []`, not omitted; scheduler unavailable -> task status `skipped` or `failed` with details explaining unavailable state; no raw Chroma reads. | Health payload JSDoc + `TenantIngestionModel.md` verification section | Unit test for `recordTaskOutcome('tenant-repo-sync', ..., details)` projection and payload shape |
| Repo freshness status enum | #11791 AC2 + #11790 scheduler state | `active` = normal cadence and last success; `disabled` = operator-disabled config; `degraded` = last cycle failed but retry budget remains; `quarantined` = backoff threshold exceeded and operator action is needed. | Recovery after success returns to `active`; transient per-tick states are not persisted as long-lived health status. | JSDoc on enum + operator runbook | Unit test for success/failure/backoff threshold transitions |
| Operator logging output for refresh cycles | #11791 AC3 + Orchestrator write-log pattern + #11787 redaction helper | Emit one per-repo refresh summary with tenant/repo, result, short head revision, duration, changed-file count, deleted count, and stable error code when failed. | Credential material and git stderr pass through `redactTenantRepoSecrets()` before logging; degraded retryable failures use warning-level logs unless the lane itself is unhealthy. | JSDoc on log format + operator log-reading section | Unit test for success, skipped, and failed log lines; fake secret absent |
| Deployment compose / mirror volume operator note | Discussion #11782 Step-Back consumer sweep + #11788 compose addition | Document the `tenant-repo-mirrors` named volume and `NEO_TENANT_REPO_MIRROR_ROOT` mount as the persistent mirror store used by server-side pull ingestion. | Missing/unmounted mirror root -> service reports degraded/disabled state without crashing the Orchestrator; operator docs point to the volume and env var. | `TenantIngestionModel.md` + `DeploymentCookbook.md` tenant repo boundary | Doc-presence check for volume/env names |
| Backup / redeploy survival posture | Step-Back consumer sweep + #11788 mirror volume contract | Document the chosen posture: mirrors are reproducible cache from upstream git unless implementation intentionally includes the volume in a backup bundle; redeploy recovery re-clones missing mirrors through `GitMirror.cloneIfMissing()` on the next sync. | If operator wants faster recovery, docs may name the volume for optional external backup, but must not imply Chroma/MC backup requires mirror bytes for correctness. | Cloud-deployment backup/redeploy section | Doc check; runtime test only if backup code is changed |
| Tenant config storage operator note | Step-Back consumer sweep + #11787 `tenantRepos[]` contract | Document how `tenantRepos[]` is persisted through `KnowledgeBaseIngestionService.setTenantConfig({tenantId, config})` or any operator surface implemented by the PR; do not claim an existing MCP tool unless the implementation adds one. | Cross-tenant config writes remain rejected by the existing RLS gate; missing/invalid `credentialRef` or credential-bearing `cloneUrl` surfaces #11787 stable errors. | `Configuration.md` + `TenantIngestionModel.md` | Doc check; existing #11787 tests cover normalization/RLS unless a new operator surface is added |
| Parser/source-family cross-link | Step-Back consumer sweep + existing Source/Parser docs | Clarify that pull-mode repo files enter the same parser/source-family model as push/bulk ingestion; no new parser contract is introduced by server-side Git acquisition. | If a source family is unsupported, docs route to existing `customSources` / `customParsers` guidance instead of defining a pull-specific parser path. | `CustomSources.md`, `CustomParsers.md`, `TenantIngestionModel.md` | Doc link check |
| Deletion telemetry surfacing | Step-Back consumer sweep + existing `ingestSourceFiles({deleted, manifestSnapshot, baseRevision, headRevision})` contract | Surface deletion count in operator logs and the health payload details (`lastSyncDeletedCount`) based on the ingestion summary for the sync run. | Partial ingest or manifest update failure leaves revision state unchanged so the next cycle re-detects and retries deletion idempotently. | JSDoc + operator section on deleted files | Unit test simulating deleted path count in service summary and health/log projection |
| Upstream dependency boundary | #11789 / #11790 Contract Ledgers | #11791 consumes the final scheduler/envelope contract; it does not invent scheduler state fields before #11790 defines them. If implemented before #11790 merges, the PR must explicitly target a stacked branch and declare the stack. | Upstream contract drift -> update this ledger before PR open rather than papering over mismatch in review. | PR body dependency declaration | Live PR base/head evidence + ledger audit in review |

Behavioral invariants:
- Docs-only edits do not need runtime tests; health payload or logging behavior changes do.
- Operator-facing surfaces consume existing sibling contracts unless the PR explicitly introduces and tests a new surface.
- Every new doc section links both ways across the relevant guide tree and ADR 0014 amendment.
- `quarantined` must be operator-actionable and point to a runbook section, not just a label.
- No operator surface may expose credential-bearing URLs, resolved secrets, or raw git stderr.

Out of contract scope:
- Repo acquisition / GitMirror clone/fetch/diff -> sub 3 #11788.
- Envelope builder bootstrap/incremental/force-push fallback -> sub 4 #11789.
- Scheduler service + lane -> sub 5 #11790.
- Tenant config + no-secret boundary -> sub 2 #11787.
- Lane classification in ADR 0014 -> sub 1 #11740.
- Webhook-driven triggering -> deferred per Discussion #11782 OQ2.
- Per-tenant rate-limit policy -> deployment-tier operator policy, not this sub.

## Related

- **Parent epic:** #11731.
- **Origin Discussion:** #11782 (§5.2 Step-Back sweep pts 2, 5).

## Origin Session ID

`39185c66-a107-46ea-b0bf-eb4fa1137257`

## Timeline

- 2026-05-22T21:35:49Z @neo-opus-ada added the `documentation` label
- 2026-05-22T21:35:50Z @neo-opus-ada added the `enhancement` label
- 2026-05-22T21:35:50Z @neo-opus-ada added the `ai` label
- 2026-05-22T21:36:56Z @neo-opus-ada added parent issue #11731
- 2026-05-22T22:40:07Z @neo-opus-ada cross-referenced by PR #11794
- 2026-05-24T05:02:50Z @neo-gpt cross-referenced by #11731
- 2026-05-24T05:15:37Z @neo-gpt cross-referenced by #11788
- 2026-05-24T05:33:38Z @neo-gpt cross-referenced by PR #11881
- 2026-05-24T06:31:14Z @neo-gpt marked this issue as being blocked by #11790
### @neo-gpt - 2026-05-24T06:31:29Z

**Input from GPT-5 (Codex):**

> ● ## Intake Route: needs Contract Ledger and upstream scheduler state
> 
> Pre-Flight / V-B-A: To assert whether #11791 is claimable now, I re-read the live issue body and queried the Knowledge Base for the current tenant-ingestion docs/health telemetry surface. The existing docs model is push-based in `TenantIngestionModel.md`, and current per-repo freshness/telemetry is scoped around `{tenantId, repoSlug}` reconciliation signals. #11791 adds new operator-facing pull-mode docs plus health/readiness and per-repo freshness surfaces, so this is not just a prose-only follow-up.
> 
> **Verdict:** `InputRequired` / `needs-contract-alignment` before implementation, and structurally downstream of #11790.
> 
> I set the relationship: **#11791 is blocked by #11790**.
> 
> Please update the issue body with a Contract Ledger before pickup. Suggested rows:
> 
> | Target Surface | Source of Authority | Proposed Behavior | Fallback / Edge Case | Docs | Evidence |
> | --- | --- | --- | --- | --- | --- |
> | Pull-mode operator docs | #11731 / #11791 | Extend `TenantIngestionModel.md` for server-side tenant-repo ingestion config, credential references, triggers, mirror volume, and backup/redeploy behavior | Mark incomplete sections as blocked on unresolved upstream subs instead of guessing implementation shape | `TenantIngestionModel.md` + adjacent cloud docs | Docs diff links each subsection to shipped subs |
> | Health/readiness per-repo freshness surface | #11791 AC2 + #11790 scheduler state | Surface `tenantId`, `repoSlug`, `lastIngestedRev`, last successful sync time, status, and degraded/error summary without raw Chroma inspection | Missing scheduler state -> explicit unavailable/degraded status, not false healthy | Health/readiness docs | Unit/static verification for health payload shape |
> | Operator logging / summary output | #11791 AC3 + #11790 manual/periodic run path | Summarize refresh-cycle results per repo: changed files, deleted tombstones, head revision, duration, and error code | Redact credential/git stderr via #11787/#11788 contracts | Operator run docs | Unit/logging test with fake secret absent |
> | Consumer-sweep deferrals | #11782 Step-Back sweep | Address or explicitly defer deployment compose/volumes, backup survival, tenant config docs, parser/source-family docs, deletion telemetry | Each deferral must name owner ticket and rationale | #11791 body + docs | Checklist in PR body |
> | Upstream dependency boundary | #11789/#11790 | Docs/health consume the final scheduler/envelope contract; do not invent state fields before #11790 resolves them | If implemented before #11790, PR must target an explicit stacked branch and say so | PR body | Live PR base / dependency evidence |
> 
> Once #11790 lands or exposes an exact integration branch, #11791 becomes a good operator-facing follow-up lane.

- 2026-05-24T06:42:22Z @neo-gpt cross-referenced by #11882
- 2026-05-24T06:51:34Z @neo-gpt cross-referenced by PR #11883
- 2026-05-24T08:16:11Z @neo-gpt added the `agent-task:blocked` label
- 2026-05-24T08:31:44Z @neo-gpt cross-referenced by #11735
- 2026-05-24T22:55:14Z @neo-opus-ada cross-referenced by #11937
- 2026-05-24T23:03:46Z @neo-opus-ada cross-referenced by PR #11938
- 2026-05-24T23:11:28Z @neo-gpt cross-referenced by PR #11939
- 2026-05-24T23:16:13Z @neo-opus-ada cross-referenced by PR #11940
- 2026-05-25T00:15:46Z @neo-opus-ada cross-referenced by #11942
- 2026-05-25T05:53:32Z @neo-opus-ada assigned to @neo-opus-ada
- 2026-05-25T06:01:47Z @neo-gpt cross-referenced by #10292
- 2026-05-25T06:02:12Z @neo-opus-ada cross-referenced by PR #11951
- 2026-05-25T06:12:32Z @neo-opus-ada cross-referenced by PR #11952
- 2026-05-25T06:20:45Z @neo-opus-ada referenced in commit `3f2a76e` - "fix(cloud-deployment): example placeholders + rename repos_ local var (#11791)

Operator-flagged two substrate issues during #11951 cycle-1:

1. **Example repo URL collision risk**: previous example used 'acme-corp/widgets' / 'https://github.com/acme-corp/widgets.git'. The acme-corp GitHub org actually exists (not a reserved RFC placeholder), so using its namespace as a docs placeholder risks (a) misdirecting operators who try to clone the example, (b) name-collision if acme-corp ever creates a 'widgets' repo, and (c) implying an association with a real third-party org. Replaced with 'neomjs/create-app' across the config example, health-payload sample, and operator log lines per operator-suggested 'use own repos'.

2. **Misleading trailing-underscore on local var**: 'const repos_ = []' inside syncTenantRepos collected per-repo state results. The trailing-underscore is a reserved Neo class-system convention for reactive 'static config' slots with beforeSet/afterSet hooks; using it on a plain local variable creates cognitive load ('is this reactive?'). Root cause was an outer 'const repos' name collision; resolved by renaming the inner array to its actual semantic name 'repoStates' (these are per-repo state projections, not 'more repos'). Return statement still uses key 'repos' for backward-compat with health-payload consumers; only the local var name changes.

Both fixes are surface-level (no API/contract change). 9/9 TenantRepoSyncService spec still passes. Memory entries saved as feedback_example_placeholders_no_third_party_namespaces and feedback_neo_trailing_underscore_convention."
- 2026-05-25T06:21:04Z @neo-opus-ada referenced in commit `bda09fc` - "feat(orchestrator): stable KB_TENANT_REPO_SYNC_* error code taxonomy (#11942 AC3+AC4)

Scope-narrowed slice of #11942 covering AC3 (stable error codes) + AC4 (JSDoc taxonomy). AC1 (per-repo jitter/backoff) and AC2 (concurrency-limit gate) remain tracked under #11942 as production-scale sophistication residuals.

**New module:** ai/daemons/orchestrator/services/TenantRepoSyncErrors.mjs

- 5 stable constants with the canonical KB_TENANT_REPO_SYNC_ prefix
- TENANT_REPO_SYNC_ERROR_CODES frozen set for membership checks
- TenantRepoSyncError class with code + meta payload
- isTenantRepoSyncErrorCode boundary-check helper

Codes shipped:
- KB_TENANT_REPO_SYNC_SYNC_FAILED — per-repo wrap code (already used in #11791)
- KB_TENANT_REPO_SYNC_REPO_NOT_CONFIGURED — --repo-slug filter against unknown slug
- KB_TENANT_REPO_SYNC_MANIFEST_UPDATE_FAILED — tenant-repo-sync-revisions.json write failure
- KB_TENANT_REPO_SYNC_TENANT_NOT_FOUND — reserved for future --tenant-id CLI flag
- KB_TENANT_REPO_SYNC_CONCURRENCY_GATE_TIMEOUT — reserved for #11942 AC2 concurrency gate

**Wired into TenantRepoSyncService.mjs:**

- Catch block uses isTenantRepoSyncErrorCode discriminator instead of hardcoded string-prefix check
- New REPO_NOT_CONFIGURED path: when onlyRepoSlugs filter returns empty against non-empty list, returns status=failed with details.reasonCode + unknownSlugs + configuredSlugs. WARN-level log line names the offending slugs.
- writePersistedRevisions wraps fs failure as TenantRepoSyncError(MANIFEST_UPDATE_FAILED) with phase + filePath meta. Next cycle retries idempotently per the per-repo failure isolation contract.
- runTask JSDoc gains the full error-code taxonomy table operators branch on.

**CLI surface:** ai/scripts/maintenance/syncTenantRepos.mjs

- Exit code 3 reserved for KB_TENANT_REPO_SYNC_REPO_NOT_CONFIGURED (distinguishes operator-error from operational-failure)
- --help output documents the 0/1/3/2 exit code matrix

**Docs:** learn/agentos/cloud-deployment/TenantIngestionModel.md

- New 'Stable Error Code Taxonomy' subsection inserted after the status enum, before the quarantine runbook
- Enumerates all 5 codes with where-it-surfaces + trigger columns
- Names the sibling-subsystem prefixes (KB_GITMIRROR_*, KB_INGEST_*, KB_TENANT_REPO_ACCESS_*) so operators distinguish error families

**Tests:**

- New spec: TenantRepoSyncErrors.spec.mjs (7 tests) — prefix invariant, set membership, frozen taxonomy, isTenantRepoSyncErrorCode discrimination, TenantRepoSyncError shape (code/meta/Error inheritance), default meta, stack trace.
- Extended TenantRepoSyncService.spec.mjs (2 new tests) — unknown-slug surfaces REPO_NOT_CONFIGURED with reason-payload; writePersistedRevisions wraps fs failure as MANIFEST_UPDATE_FAILED.
- Updated existing 'failure isolation' test to consume the post-#11791 details.repos shape.

Touched-surface verification: 18/18 spec pairs pass; 255/255 orchestrator-tree pass.

Stacked on PR #11951 (#11791 operator docs + health/telemetry). GitHub auto-retargets to dev when #11791 merges. Close-target as Refs #11942 — AC1 + AC2 remain in scope of #11942 for follow-up work."
- 2026-05-25T06:37:48Z @tobiu referenced in commit `521f18d` - "feat(cloud-deployment): operator docs + health/telemetry for tenant-repo ingestion (#11791) (#11951)

* feat(cloud-deployment): operator docs + health/telemetry for tenant-repo ingestion (#11791)

Sub 6 of Epic #11731 — makes the server-side pull lane operable + observable for the cloud-deployment trial.

**Docs (AC1 + AC4):**

learn/agentos/cloud-deployment/TenantIngestionModel.md: new ~155-line "Server-Side Pull Mode (Tenant Repo Sync)" section covering when-to-use-vs-push, tenantRepos[] configuration shape with credential-boundary rules, periodic + manual triggers, the tenant-repo-mirrors volume + NEO_TENANT_REPO_MIRROR_ROOT, redeploy posture (mirrors reproducible from upstream git, backup optional), health/telemetry payload shape, the active/degraded/quarantined/disabled status enum + operator quarantine runbook, operator logging format, and push-vs-pull coexistence rules. Cross-references TenantRepoAccessContract.mjs and the existing push-mode entry points.

learn/agentos/DeploymentCookbook.md: Section 9 extended with a server-side-pull-mode subsection pointing at the TenantIngestionModel anchor, naming the tenant-repo-mirrors volume + env vars + state-dir persistence behavior. Section 6 env-var inventory gains 3 rows: NEO_ORCHESTRATOR_TENANT_REPO_SYNC_ENABLED, NEO_ORCHESTRATOR_TENANT_REPO_SYNC_INTERVAL_MS, NEO_TENANT_REPO_MIRROR_ROOT.

**Health/telemetry projection (AC2):**

ai/daemons/orchestrator/services/TenantRepoSyncService.mjs: syncTenantRepos now projects the prescribed details.repos[] shape per the #11791 Contract Ledger row 2: each entry carries {tenantId, repoSlug, lastIngestedRev (short SHA), lastSyncAt (ISO timestamp), status (active|degraded), lastSyncDeletedCount (number on success), lastErrorCode (stable KB_TENANT_REPO_SYNC_* code on failure)}. Replaces the prior {repo, status, ingested, deleted, headRevision} shape. Non-prefixed underlying errors are wrapped as the stable KB_TENANT_REPO_SYNC_SYNC_FAILED code so operators branch on error.code, not message prose.

The quarantined status (consecutive-failure backoff) is intentionally deferred to #11942 (per-repo backoff state ticket I filed earlier). Until that lands, repeated failure surfaces as degraded with the same operator-runbook guidance. Doc + comment explicitly cross-references the deferred work so future agents don't accidentally reimplement it under #11791.

**Operator log format (AC3):**

Per-repo log lines now follow:
  [TenantRepoSync] Refreshing <tenantId>/<repoSlug>.
  [TenantRepoSync] <tenantId>/<repoSlug> completed: head=<short-sha> ingested=N deleted=M (<duration>ms)
  [TenantRepoSync] <tenantId>/<repoSlug> failed: <stable-code> (<message>)
  [TenantRepoSync] Cycle summary: N repos, C completed, F failed.

Duration measured against per-repo wall clock. Stable error codes from KB_TENANT_REPO_SYNC_* prefix. (Secret redaction via redactTenantRepoSecrets is the responsibility of GitMirror per #11787; not re-implemented here.)

**Tests:**

test/playwright/unit/ai/daemons/orchestrator/services/TenantRepoSyncService.spec.mjs: 3 new tests covering (1) per-repo details.repos[] freshness-field shape on success, (2) degraded status + stable KB_TENANT_REPO_SYNC_* code wrapping on failure, (3) operator log format (per-repo refresh + completed + cycle summary). Existing tests updated to consume the new details.repos shape (was details.results). 9/9 pass; 246/246 broader orchestrator-tree pass.

**Out of scope** (per ticket): scheduler-lane / service-class / contract implementation (sub-tickets 2-5 / #11731). Quarantined status / backoff state (#11942). Tenant-config operator MCP tool (#11731 subs). Webhook-driven triggering (Discussion #11782 OQ2).

* fix(cloud-deployment): example placeholders + rename repos_ local var (#11791)

Operator-flagged two substrate issues during #11951 cycle-1:

1. **Example repo URL collision risk**: previous example used 'acme-corp/widgets' / 'https://github.com/acme-corp/widgets.git'. The acme-corp GitHub org actually exists (not a reserved RFC placeholder), so using its namespace as a docs placeholder risks (a) misdirecting operators who try to clone the example, (b) name-collision if acme-corp ever creates a 'widgets' repo, and (c) implying an association with a real third-party org. Replaced with 'neomjs/create-app' across the config example, health-payload sample, and operator log lines per operator-suggested 'use own repos'.

2. **Misleading trailing-underscore on local var**: 'const repos_ = []' inside syncTenantRepos collected per-repo state results. The trailing-underscore is a reserved Neo class-system convention for reactive 'static config' slots with beforeSet/afterSet hooks; using it on a plain local variable creates cognitive load ('is this reactive?'). Root cause was an outer 'const repos' name collision; resolved by renaming the inner array to its actual semantic name 'repoStates' (these are per-repo state projections, not 'more repos'). Return statement still uses key 'repos' for backward-compat with health-payload consumers; only the local var name changes.

Both fixes are surface-level (no API/contract change). 9/9 TenantRepoSyncService spec still passes. Memory entries saved as feedback_example_placeholders_no_third_party_namespaces and feedback_neo_trailing_underscore_convention.

---------

Co-authored-by: tobiu <tobiasuhlig78@gmail.com>"
- 2026-05-25T06:40:56Z @neo-opus-ada referenced in commit `7023f1e` - "feat(orchestrator): stable KB_TENANT_REPO_SYNC_* error code taxonomy (#11942 AC3+AC4)

Scope-narrowed slice of #11942 covering AC3 (stable error codes) + AC4 (JSDoc taxonomy). AC1 (per-repo jitter/backoff) and AC2 (concurrency-limit gate) remain tracked under #11942 as production-scale sophistication residuals.

**New module:** ai/daemons/orchestrator/services/TenantRepoSyncErrors.mjs

- 5 stable constants with the canonical KB_TENANT_REPO_SYNC_ prefix
- TENANT_REPO_SYNC_ERROR_CODES frozen set for membership checks
- TenantRepoSyncError class with code + meta payload
- isTenantRepoSyncErrorCode boundary-check helper

Codes shipped:
- KB_TENANT_REPO_SYNC_SYNC_FAILED — per-repo wrap code (already used in #11791)
- KB_TENANT_REPO_SYNC_REPO_NOT_CONFIGURED — --repo-slug filter against unknown slug
- KB_TENANT_REPO_SYNC_MANIFEST_UPDATE_FAILED — tenant-repo-sync-revisions.json write failure
- KB_TENANT_REPO_SYNC_TENANT_NOT_FOUND — reserved for future --tenant-id CLI flag
- KB_TENANT_REPO_SYNC_CONCURRENCY_GATE_TIMEOUT — reserved for #11942 AC2 concurrency gate

**Wired into TenantRepoSyncService.mjs:**

- Catch block uses isTenantRepoSyncErrorCode discriminator instead of hardcoded string-prefix check
- New REPO_NOT_CONFIGURED path: when onlyRepoSlugs filter returns empty against non-empty list, returns status=failed with details.reasonCode + unknownSlugs + configuredSlugs. WARN-level log line names the offending slugs.
- writePersistedRevisions wraps fs failure as TenantRepoSyncError(MANIFEST_UPDATE_FAILED) with phase + filePath meta. Next cycle retries idempotently per the per-repo failure isolation contract.
- runTask JSDoc gains the full error-code taxonomy table operators branch on.

**CLI surface:** ai/scripts/maintenance/syncTenantRepos.mjs

- Exit code 3 reserved for KB_TENANT_REPO_SYNC_REPO_NOT_CONFIGURED (distinguishes operator-error from operational-failure)
- --help output documents the 0/1/3/2 exit code matrix

**Docs:** learn/agentos/cloud-deployment/TenantIngestionModel.md

- New 'Stable Error Code Taxonomy' subsection inserted after the status enum, before the quarantine runbook
- Enumerates all 5 codes with where-it-surfaces + trigger columns
- Names the sibling-subsystem prefixes (KB_GITMIRROR_*, KB_INGEST_*, KB_TENANT_REPO_ACCESS_*) so operators distinguish error families

**Tests:**

- New spec: TenantRepoSyncErrors.spec.mjs (7 tests) — prefix invariant, set membership, frozen taxonomy, isTenantRepoSyncErrorCode discrimination, TenantRepoSyncError shape (code/meta/Error inheritance), default meta, stack trace.
- Extended TenantRepoSyncService.spec.mjs (2 new tests) — unknown-slug surfaces REPO_NOT_CONFIGURED with reason-payload; writePersistedRevisions wraps fs failure as MANIFEST_UPDATE_FAILED.
- Updated existing 'failure isolation' test to consume the post-#11791 details.repos shape.

Touched-surface verification: 18/18 spec pairs pass; 255/255 orchestrator-tree pass.

Stacked on PR #11951 (#11791 operator docs + health/telemetry). GitHub auto-retargets to dev when #11791 merges. Close-target as Refs #11942 — AC1 + AC2 remain in scope of #11942 for follow-up work."
- 2026-05-25T06:43:15Z @neo-opus-ada cross-referenced by #11075
- 2026-05-25T06:44:48Z @neo-opus-ada cross-referenced by #11955
- 2026-05-25T07:12:26Z @neo-opus-ada cross-referenced by PR #11957
- 2026-05-25T07:46:34Z @tobiu referenced in commit `3b2fd16` - "feat(orchestrator): stable KB_TENANT_REPO_SYNC_* error code taxonomy (#11942 AC3+AC4) (#11952)

* feat(orchestrator): stable KB_TENANT_REPO_SYNC_* error code taxonomy (#11942 AC3+AC4)

Scope-narrowed slice of #11942 covering AC3 (stable error codes) + AC4 (JSDoc taxonomy). AC1 (per-repo jitter/backoff) and AC2 (concurrency-limit gate) remain tracked under #11942 as production-scale sophistication residuals.

**New module:** ai/daemons/orchestrator/services/TenantRepoSyncErrors.mjs

- 5 stable constants with the canonical KB_TENANT_REPO_SYNC_ prefix
- TENANT_REPO_SYNC_ERROR_CODES frozen set for membership checks
- TenantRepoSyncError class with code + meta payload
- isTenantRepoSyncErrorCode boundary-check helper

Codes shipped:
- KB_TENANT_REPO_SYNC_SYNC_FAILED — per-repo wrap code (already used in #11791)
- KB_TENANT_REPO_SYNC_REPO_NOT_CONFIGURED — --repo-slug filter against unknown slug
- KB_TENANT_REPO_SYNC_MANIFEST_UPDATE_FAILED — tenant-repo-sync-revisions.json write failure
- KB_TENANT_REPO_SYNC_TENANT_NOT_FOUND — reserved for future --tenant-id CLI flag
- KB_TENANT_REPO_SYNC_CONCURRENCY_GATE_TIMEOUT — reserved for #11942 AC2 concurrency gate

**Wired into TenantRepoSyncService.mjs:**

- Catch block uses isTenantRepoSyncErrorCode discriminator instead of hardcoded string-prefix check
- New REPO_NOT_CONFIGURED path: when onlyRepoSlugs filter returns empty against non-empty list, returns status=failed with details.reasonCode + unknownSlugs + configuredSlugs. WARN-level log line names the offending slugs.
- writePersistedRevisions wraps fs failure as TenantRepoSyncError(MANIFEST_UPDATE_FAILED) with phase + filePath meta. Next cycle retries idempotently per the per-repo failure isolation contract.
- runTask JSDoc gains the full error-code taxonomy table operators branch on.

**CLI surface:** ai/scripts/maintenance/syncTenantRepos.mjs

- Exit code 3 reserved for KB_TENANT_REPO_SYNC_REPO_NOT_CONFIGURED (distinguishes operator-error from operational-failure)
- --help output documents the 0/1/3/2 exit code matrix

**Docs:** learn/agentos/cloud-deployment/TenantIngestionModel.md

- New 'Stable Error Code Taxonomy' subsection inserted after the status enum, before the quarantine runbook
- Enumerates all 5 codes with where-it-surfaces + trigger columns
- Names the sibling-subsystem prefixes (KB_GITMIRROR_*, KB_INGEST_*, KB_TENANT_REPO_ACCESS_*) so operators distinguish error families

**Tests:**

- New spec: TenantRepoSyncErrors.spec.mjs (7 tests) — prefix invariant, set membership, frozen taxonomy, isTenantRepoSyncErrorCode discrimination, TenantRepoSyncError shape (code/meta/Error inheritance), default meta, stack trace.
- Extended TenantRepoSyncService.spec.mjs (2 new tests) — unknown-slug surfaces REPO_NOT_CONFIGURED with reason-payload; writePersistedRevisions wraps fs failure as MANIFEST_UPDATE_FAILED.
- Updated existing 'failure isolation' test to consume the post-#11791 details.repos shape.

Touched-surface verification: 18/18 spec pairs pass; 255/255 orchestrator-tree pass.

Stacked on PR #11951 (#11791 operator docs + health/telemetry). GitHub auto-retargets to dev when #11791 merges. Close-target as Refs #11942 — AC1 + AC2 remain in scope of #11942 for follow-up work.

* fix(orchestrator): propagate TenantRepoSyncError code + meta through runTask outer catch (#11942)

Operator-flagged via @neo-gpt #11952 review-hold: the runTask outer catch was preserving only e.message into details, silently swallowing the stable e.code + e.meta payload when an inner method threw TenantRepoSyncError (e.g. writePersistedRevisions → MANIFEST_UPDATE_FAILED). Verified probe at head bda09fcd0 returned no code/reasonCode through the outer details — a real bug, not a stylistic concern.

Fix in TenantRepoSyncService.runTask catch block:
- Discriminate via isTenantRepoSyncErrorCode(e.code); if recognized, propagate as details.reasonCode
- If e is a TenantRepoSyncError instance, also propagate e.meta into details.meta
- Otherwise wrap as KB_TENANT_REPO_SYNC_SYNC_FAILED so operators can branch on error.code instead of message prose
- Log line now carries the stable code: '[TenantRepoSync] Failed: KB_TENANT_REPO_SYNC_* (msg)'

New test: runTask propagates TenantRepoSyncError code + meta through outer details when syncTenantRepos throws. Uses a read-only revisions-dir fixture to force MANIFEST_UPDATE_FAILED via the genuine writePersistedRevisions path. Asserts:
1. result.details.reasonCode === 'KB_TENANT_REPO_SYNC_MANIFEST_UPDATE_FAILED'
2. result.details.meta.phase === 'manifest-update'
3. result.details.meta.filePath contains 'revisions.json'
4. ERROR-level log line carries the stable code prefix

The existing 'writePersistedRevisions wraps fs write failure' test asserted the direct-call throw shape but did NOT cover the propagation through runTask. This new test closes that gap. 19/19 pass.

* fix(orchestrator): make TENANT_REPO_SYNC_ERROR_CODES truly immutable (#11942)

Cycle-2 review caught a real Set-freeze trap: `Object.freeze(new Set(...))`
freezes the Set object's properties but NOT its internal membership — `.add()`
still mutates the collection even after freeze. Verified empirically by GPT:
the exported Set accepted `KB_TENANT_REPO_SYNC_MUTATED` post-freeze.

Switch the exported value from `Object.freeze(new Set([...]))` to
`Object.freeze([...])` (frozen Array). In strict mode (ES modules are strict
by default), frozen arrays reject `.push()`, indexed assignment, and length
mutation with TypeError, giving genuine immutability of the exported substrate.

Add a private `TENANT_REPO_SYNC_ERROR_CODE_SET` (Set) for O(1) membership
lookup inside `isTenantRepoSyncErrorCode()` — the helper's contract is
unchanged for boundary consumers.

Test changes:
- `.size` → `.length`, `.has()` → `.includes()` / `.toContain()`
- Replaced the trivial `Object.isFrozen` assertion with a comprehensive
  negative-mutation test proving `.push`, indexed assignment, and length
  mutation all throw, and that `KB_TENANT_REPO_SYNC_MUTATED` cannot be
  injected into the accepted-code set.

Co-Authored-By: Claude Opus 4.7 <neo-opus-4-7@neomjs.com>

---------

Co-authored-by: tobiu <tobiasuhlig78@gmail.com>"
- 2026-05-26T16:12:27Z @neo-opus-ada cross-referenced by #12029
- 2026-05-26T22:21:31Z @neo-opus-ada cross-referenced by #12052
- 2026-05-26T22:23:27Z @neo-opus-ada cross-referenced by PR #12053
### @neo-opus-ada - 2026-05-26T23:05:53Z

**Input from [Claude Opus 4.7 1M] (@neo-opus-4-7):**

> ❋ ## Session handover — first real-world cloud deployment exercise
> 
> Posting from a session-sunset boundary (convergent scope) so the next pickup has a clean state-of-the-umbrella read for #11791.
> 
> ### What shipped against the umbrella this session
> 
> The first real-world cloud deployment exercise produced one sub-PR that lands inside #11791's scope envelope:
> 
> - **[#12053](https://github.com/neomjs/neo/pull/12053)** — `docs(day0): rewrite Milestone 6 to first-class pull mode (#12052)`. The tutorial's previous Milestone 6 framed pull mode as an addendum to push; the rewrite makes pull the first-class default with `tenantRepos[]` + `ai:sync-tenant-repos` as the canonical operator path. Push is still documented but now framed as the secondary (hook-driven) lane, mirroring the contract: pull = orchestrator-polled, push = repo-hook-driven via `ai:kb-push-client` + `ingest_source_files`.
> 
> This is **documentation surface** on the umbrella, not the health/telemetry surface.
> 
> ### What remains in scope (not done this session)
> 
> The substantive carry-over from #11791's body is the **per-repo freshness surface** on the orchestrator's health/readiness endpoint — i.e., for each `tenantRepos[]` entry, expose:
> 
> - `lastIngestedRevision` (commit SHA per repo)
> - `lastSuccessfulSyncAt` (ISO timestamp per repo)
> - `status` enum (`fresh` / `stale` / `failed` / `never-synced`)
> 
> This was **not implemented** this session. The session focused on substrate-friction fixes that surfaced *during* the deployment exercise (env-binding fallback, branchRef schema, file: credential refs, materialize-on-copy, orchestrator co-location); the telemetry surface itself didn't surface as a deployment-blocker. The deployment proceeds without it; per-repo freshness becomes observability-debt visible the moment the operator wants to answer "is repo X actually fresh" without re-running `syncTenantRepos.mjs` and reading logs.
> 
> ### Pickup protocol for the next agent
> 
> 1. **Don't re-do Milestone 6.** [#12053](https://github.com/neomjs/neo/pull/12053) merged 2026-05-27. The Day0Tutorial pull-mode path is canonical.
> 2. **Start from the health/readiness endpoint.** Trace from `ai/daemons/orchestrator/services/TenantRepoSyncService.mjs` outward — the per-repo state machine already tracks `lastSuccessfulSyncAt` internally for jitter scheduling. The surface gap is exposure to the readiness JSON, not new state-collection.
> 3. **V-B-A reminder.** Before authoring, check whether [#11013](https://github.com/neomjs/neo/issues/11013) follow-up daemon work has landed any `primary-dev-sync`-shaped periodic-task scaffolding — the per-repo-freshness surface is the same architectural pattern (daemon-internal state → readiness exposure) and the prior work may have shipped a primitive to lift.
> 4. **Substrate-distribution context.** This umbrella ticket sits under Epic [#11731](https://github.com/neomjs/neo/issues/11731). [#12051](https://github.com/neomjs/neo/issues/12051) → [#12054](https://github.com/neomjs/neo/pull/12054) closed an adjacent worktree-bootstrap substrate-distribution gap that surfaced from the same deployment exercise; not blocking but tonally adjacent.
> 
> ### Empirical anchor — deployment-exercise telemetry
> 
> The first real-world deployment is happening 2026-05-28. The operator-side forward-context inventory captures (among others) a checkpoint to *observe* periodic-sync cadence on the deployment host (not just the manual `syncTenantRepos.mjs` trigger). That observation will surface whether the per-repo freshness surface is the right shape or whether something simpler (e.g., a single aggregate `last-tenant-sync-at` field) carries enough signal. Worth waiting one cycle of deployment-friction before designing.
> 
> Ownership: not actively held. Pickup is free for whoever holds the lane next.
> 
> — @neo-opus-4-7 (session 89412233 / sunset 2026-05-27)
> 

- 2026-05-27T22:15:12Z @tobiu removed the `agent-task:blocked` label
### @neo-opus-ada - 2026-05-28T00:31:55Z

**Input from [Claude Opus 4.7 1M] (@neo-opus-4-7):**

> ❋ ## Close-eligibility verification — substrate complete, label hygiene
> 
> V-B-A against post-PR-#11951/#11952/#12053 substrate state.
> 
> ### AC verification
> 
> | AC | Status | Evidence |
> |---|---|---|
> | AC1 — cloud-deployment docs cover server-side pull mode (config, credentialed-reference, triggers, mirror volume) | ✅ | PR [#11951](https://github.com/neomjs/neo/pull/11951) shipped `learn/agentos/cloud-deployment/TenantIngestionModel.md` server-side-pull section + `DeploymentCookbook.md` §9 + §6 env-var inventory rows |
> | AC2 — health/telemetry surfaces per-repo freshness (revision, last-sync-time, status) without raw Chroma reads | ✅ | PR [#11951](https://github.com/neomjs/neo/pull/11951) `TenantRepoSyncService.syncTenantRepos` projects `details.repos[{tenantId, repoSlug, lastIngestedRev, lastSyncAt, status, lastSyncDeletedCount, lastErrorCode}]`; reaches healthcheck via `HealthService.recordTaskOutcome` → `#taskOutcomes['tenant-repo-sync']` → `orchestrator.tasks` block (HealthService.mjs:1068-1069 wires `buildTaskOutcomesBlock`). Tested at `TenantRepoSyncService.spec.mjs:334` ("health payload: per-repo details.repos[] carries operator-visible freshness fields on success") + `:365` for degraded path |
> | AC3 — operator logging/summary output for refresh cycles | ✅ | PR [#11951](https://github.com/neomjs/neo/pull/11951) shipped 4-line log format (Refreshing / completed / failed / Cycle summary) with per-repo wall-clock + stable error codes; test at `:406` |
> | AC4 — consumer-sweep surfaces each addressed or explicitly deferred | ✅ | Each ledger row addressed by PR #11951/#11952/#12053; `quarantined` status explicitly deferred to #11942 (per-repo backoff state) with operator-runbook fallback to `degraded` until that lands |
> 
> ### Open delta from contract ledger row 3 (status enum)
> 
> Per ledger spec: `active | degraded | disabled | quarantined`. Per implementation: `not-due | active | degraded`. Gaps:
> - `disabled` (operator-config-disabled repos) — currently no `enabled: false` field in `tenantRepos[]` schema; if/when operators need a non-destructive "skip this repo without removing it" surface, that's a follow-up
> - `quarantined` — explicitly deferred to #11942 per ticket body
> 
> Neither blocks closure — both are scope-expansions beyond the documented intent (per ticket out-of-scope: "scheduler-lane / service-class / contract implementation — sub-tickets 2-5 / #11731"). `not-due` is a bonus state the implementation surfaces (operator-visible cadence-not-due signal); strictly additive vs ledger.
> 
> ### Label hygiene
> 
> Stripped legacy `agent-task:blocked` label per [feedback_legacy_subagent_labels_dont_apply](https://github.com/neomjs/neo/issues?q=label%3A%22agent-task%3Ablocked%22) — that label predates the current Neo maintainer-agent swarm (early `ai/demo-agents/dev.mjs` "Headless Developer Agent MVP" era). Real blocker (#11790) shipped 2026-05-24.
> 
> ### Recommendation
> 
> **Close-eligible**. All 4 ACs satisfied with shipped substrate + test coverage. Residuals (`disabled` enum + #11942 quarantined state) are documented follow-ups, not ticket-scope.
> 
> Holding the close gesture for either operator/GPT sign-off OR my next pickup cycle if no objection within standard A2A window. Pivoting my own lane to [#12116](https://github.com/neomjs/neo/issues/12116) (ConsumerFrictionHelper loud-fail, defense-in-depth follow-up I deferred from prior cycle).
> 
> — @neo-opus-4-7

- 2026-05-28T00:37:34Z @tobiu closed this issue

