---
id: 11942
title: 'Sub-of-#11790: tenant-repo-sync residuals (jitter/backoff + concurrency-gate + KB_TENANT_REPO_SYNC_ error codes)'
state: CLOSED
labels:
  - enhancement
  - ai
assignees:
  - neo-opus-ada
createdAt: '2026-05-25T00:15:45Z'
updatedAt: '2026-05-25T11:20:19Z'
githubUrl: 'https://github.com/neomjs/neo/issues/11942'
author: neo-opus-ada
commentsCount: 1
parentIssue: null
subIssues: []
subIssuesCompleted: 0
subIssuesTotal: 0
blockedBy: []
blocking: []
closedAt: '2026-05-25T11:20:19Z'
---
# Sub-of-#11790: tenant-repo-sync residuals (jitter/backoff + concurrency-gate + KB_TENANT_REPO_SYNC_ error codes)

## Parent Issue

#11790 — TenantRepoSyncService scheduler lane MVP. PR #11940 shipped the load-bearing surface (service + scheduling trigger + Orchestrator.poll() wiring + deployment-aware cloud-toggle + CLI operator path). This sub-ticket tracks the Contract Ledger residuals that #11940 explicitly deferred for post-MVP delivery.

## Scope (Contract Ledger residuals from #11790)

### 1. Per-repo cadence + deterministic jitter/backoff (Contract Ledger row 5)

Replace the current single global `tenantRepoSyncIntervalMs` with per-repo cadence + deterministic jitter:

- Pure scheduling function computes next-due state from `lastRunAttemptAt`, per-repo configured cadence (default = global interval), deterministic `repoSlug + tenantId` salt jitter (to prevent thundering-herd on first bootstrap), and failure backoff (exponential growing on consecutive failures, resetting on success).
- Per-repo failure backoff state persists alongside `lastIngestedRev` in the existing `tenant-repo-sync-revisions.json`.
- Jitter must be deterministic per repoSlug/tenantId so test runs and operator diagnosis are reproducible.

### 2. Concurrency-limit gate (Contract Ledger row 7)

Reactive `concurrencyLimit` config on `TenantRepoSyncService` caps simultaneous git/ingest work:

- Default conservative limit (e.g., 2) to avoid multi-tenant CPU/network exhaustion.
- `concurrencyLimit: 1` serializes (current MVP shape).
- Per-repo work acquires a semaphore slot before `cloneIfMissing/fetch/buildIngestEnvelope/ingestSourceFiles`; releases after completion (success or failure).
- Stable error code `KB_TENANT_REPO_SYNC_CONCURRENCY_GATE_TIMEOUT` for slot-acquisition timeout.

### 3. Stable `KB_TENANT_REPO_SYNC_*` error codes (Contract Ledger row 9)

Replace ad-hoc error throws with stable constants per the #11787/#11788 redaction-path discipline:

- `KB_TENANT_REPO_SYNC_TENANT_NOT_FOUND` — manual CLI path with unknown tenantId
- `KB_TENANT_REPO_SYNC_REPO_NOT_CONFIGURED` — manual CLI path with unknown repoSlug
- `KB_TENANT_REPO_SYNC_SYNC_FAILED` — per-repo sync failure (wraps the underlying GitMirror/envelope/ingest error with redacted message)
- `KB_TENANT_REPO_SYNC_MANIFEST_UPDATE_FAILED` — `tenant-repo-sync-revisions.json` write failure
- `KB_TENANT_REPO_SYNC_CONCURRENCY_GATE_TIMEOUT` — (from sub-item 2 above)
- All surfaced errors route via the #11787/#11788 redaction path for credential-bearing hints or git stderr before logging/returning.
- Callers branch on `error.code`, not message prose.

## Acceptance Criteria

- [ ] AC1 — Per-repo cadence stored in `tenant-repo-sync-revisions.json` alongside `lastIngestedRev`; deterministic jitter via `repoSlug + tenantId` hash; jitter window ≤ 20% of cadence; backoff doubles on consecutive failures and resets on success.
- [ ] AC2 — `concurrencyLimit` reactive config on `TenantRepoSyncService`; semaphore implementation; CONCURRENCY_GATE_TIMEOUT error code; tests covering serialized (`1`), bounded-parallel (`2`), and gate-timeout paths.
- [ ] AC3 — All `TenantRepoSyncService.runTask` error paths return `{status: 'failed', details: {code: 'KB_TENANT_REPO_SYNC_*'}}` per the enumerated taxonomy. Manual CLI surface returns matching exit codes. Tests for each error path including secret-redaction.
- [ ] AC4 — JSDoc updated for `runTask` to document the error-code taxonomy as part of the contract.

## Dependencies

- **Parent:** #11790 (MVP scheduler lane already shipped in PR #11940)
- **Substrate:** #11787 (TenantRepoAccessContract redaction path), #11788 (GitMirror error surfaces)

## Out of Scope

- Webhook-driven triggering — deferred per Discussion #11782 OQ2 (different lane).
- Cross-deployment KB-collection conflict resolution — out of scope; tenant/repo scoping handles isolation.
- Per-repo health dashboard / operator-visible telemetry — owned by #11791.

## Why deferred (PR #11940 cycle-1 scope decision)

Per @neo-gpt #11940 cycle-1 review: the MVP poll-wiring + cloud-toggle gap was the load-bearing blocker for the 40h cloud trial. Jitter/backoff and concurrency-gate are sophistication beyond MVP — useful for production scale but not gating trial start (small tenant count, low concurrent traffic). Stable error codes were similarly deferred to keep PR #11940 scoped to load-bearing integration. This ticket tracks the residuals so the original #11790 Contract Ledger commitments are honored as follow-up work.

**Authored by:** [Claude Opus 4.7] (Claude Code)

## Timeline

- 2026-05-25T00:15:45Z @neo-opus-ada assigned to @neo-opus-ada
- 2026-05-25T00:15:46Z @neo-opus-ada added the `enhancement` label
- 2026-05-25T00:15:46Z @neo-opus-ada added the `ai` label
- 2026-05-25T00:16:40Z @neo-opus-ada referenced in commit `c573089` - "feat(orchestrator): wire tenant-repo-sync lane into Orchestrator.poll() + cloud-deployment toggle (#11790)

Closes the load-bearing Contract Ledger gap from PR #11940 cycle-1 review: the tenant-repo-sync lane was registered in TaskDefinitions but never scheduled by Orchestrator.poll(). Future-agent KB grep would see the lane as wired even though no periodic tick ever invoked it.

Changes:
- AiConfig.template.mjs: add intervals.tenantRepoSyncMs (30min default) + new orchestrator.cloudOnly collection with tenantRepoSyncEnabled (null = deployment-default: cloud enables, local disables, mirrors localOnly semantics inverted per ADR 0014 lane classification)
- Orchestrator.mjs: import TenantRepoSyncService + scheduling getDueTask + TENANT_REPO_SYNC_TASK_NAME constant; add tenantRepoSyncService + tenantRepoSyncGetDueTask instance-field DI seams; add resolveCloudOnlyEnabled helper (inverse of resolveDeploymentEnabled); add tenantRepoSyncIntervalMs + tenantRepoSyncEnabled lazy getters with NEO_ORCHESTRATOR_TENANT_REPO_SYNC_* env precedence; add cadenceEngine.runIfDue(TENANT_REPO_SYNC_TASK_NAME, ...) branch matching the PRIMARY_DEV_SYNC pattern (trigger + executor split)
- TenantRepoSyncService.mjs JSDoc lines 32-35: fix the rhetorical-drift mismatch GPT flagged; class-comment claimed all-or-fail semantics while line 213 implements partial-success (status='completed' when no repos failed OR at least one succeeded). Updated comment to document the correct partial-success contract
- Orchestrator.spec.mjs: add tenantRepoSyncEnabled/IntervalMs/Service/GetDueTask DI seam defaults in createTestOrchestrator helper (interval defaults to MAX_SAFE_INTEGER + enabled defaults to false so existing tests don't accidentally trigger); add savedCloudOnly restore in afterEach; add 2 new tests covering the load-bearing wiring: 'routes tenant-repo-sync through Orchestrator.poll() when enabled + interval elapsed (#11790 AC2 poll-wiring)' + 'skips tenant-repo-sync when disabled (local deployment default — Contract Ledger row 2)'

Deferred residuals tracked in follow-up #11942 (per-repo cadence + deterministic jitter/backoff + concurrency-limit gate + stable KB_TENANT_REPO_SYNC_* error codes). PR body for #11940 is being narrowed from `Resolves #11790` to `Refs #11790` since the original Contract Ledger has 9 surfaces and this PR delivers ~5.

Touched-spec verification: 139/139 unit tests pass (Orchestrator.spec + daemon.spec + scheduling/* + TenantRepoSyncService.spec + PrimaryRepoSyncService.spec)."
- 2026-05-25T00:19:29Z @neo-opus-ada cross-referenced by PR #11940
- 2026-05-25T05:48:20Z @tobiu referenced in commit `41a93c4` - "feat(orchestrator): cloud-deployable tenant-repo-sync scheduler lane (#11790) (#11940)

* feat(kb): add tenant repo ingest envelope builder (#11789)

* feat(orchestrator): cloud-deployable tenant-repo-sync scheduler lane (#11790)

Resolves #11790

Implements the trigger-layer scheduler for server-side tenant-repo ingestion under Epic #11731. Consumes the merged #11788 GitMirror primitive + the in-flight #11789 TenantRepoIngestEnvelopeBuilder adapter. The MVP push-based KB ingestion path is unchanged; this is the additive PULL-side complement per Discussion #11782 Option A.

**Files (NEW + modified):**

- `ai/daemons/orchestrator/scheduling/tenantRepoSync.mjs` (NEW) — pure-function trigger module mirroring `primaryDevSync.mjs`. `buildTenantRepoSyncTrigger` + `getDueTask` exports; no class, no Neo machinery.
- `ai/daemons/orchestrator/services/TenantRepoSyncService.mjs` (NEW) — Neo class singleton extending Base. `runTask({...})` iterates configured `tenantRepos[]` (normalized via `TenantRepoAccessContract.normalizeTenantRepoConfig` #11787) → `GitMirror.cloneIfMissing` + `GitMirror.fetch` → `buildIngestEnvelope({...})` → `KnowledgeBaseIngestionService.ingestSourceFiles({...envelope, viaMcp: false})` → persist per-repo `lastIngestedRev` for next cycle. Per-repo failure isolation: one failing tenantRepo does NOT halt remaining repos; outer task reports `completed` if any repo succeeded, `failed` if all failed.
- `ai/daemons/orchestrator/TaskDefinitions.mjs` — adds `TENANT_REPO_SYNC_TASK_NAME = 'tenant-repo-sync'` constant + lane entry (`serviceTask: true`, `expectedCommand: 'TenantRepoSyncService'`, `pidFileName: 'tenant-repo-sync.pid'`, label `'tenant repo sync (cloud)'`).
- `ai/scripts/maintenance/syncTenantRepos.mjs` (NEW) — operator/CI-facing CLI for manual one-shot sweep. Supports `--repo-slug <slug>` (repeatable) for scoped subset, `--help`, and an in-memory `TaskStateService` stand-in so the CLI works without orchestrator booted state.
- `test/playwright/unit/ai/daemons/orchestrator/scheduling/tenantRepoSync.spec.mjs` (NEW) — trigger purity unit tests (6 cases: disabled/zero-interval/not-elapsed/enabled-elapsed/state-derived/bootstrap).
- `test/playwright/unit/ai/daemons/orchestrator/services/TenantRepoSyncService.spec.mjs` (NEW) — service-behavior tests (6 cases: no-config skip, re-entrancy guard, happy-path iteration, per-repo failure isolation, onlyRepoSlugs scoping, persisted-revisions round-trip).

**Persistence shape:** per-repo `lastIngestedRev` lives in a dedicated sibling JSON file (`<DEFAULT_DATA_DIR>/tenant-repo-sync-revisions.json`) — separate from TaskStateService's state.json to prevent `markCompleted/markFailed` task-lifecycle writes from racing with revision-map writes. Bootstrap: missing file = empty map.

**Test seams** (per `feedback_truth_in_code` + sibling pattern): `runTask` accepts `gitMirror`, `knowledgeBaseIngestionService`, `envelopeBuilder`, `aiConfig`, `tenantReposConfig`, `revisionsFilePath`, `onlyRepoSlugs` overrides. Production callers omit; unit tests stub.

**Stacked PR:** this PR is stacked on top of `codex/11789-ingest-envelope-builder` (the in-flight #11938 envelope-builder PR). My commits target that branch as base; once #11938 merges into `dev`, GitHub auto-retargets this PR to `dev` and the diff shows only my surface.

Test plan:
- `npm run test-unit -- test/playwright/unit/ai/daemons/orchestrator/scheduling/tenantRepoSync.spec.mjs` → 6/6 PASS (512ms)
- `npm run test-unit -- test/playwright/unit/ai/daemons/orchestrator/services/TenantRepoSyncService.spec.mjs` → 6/6 PASS (597ms)

Avoided traps (per Epic #11731 + ticket #11790):
- NOT touching `primary-dev-sync` lane — ADR 0014 §5.2 anti-pattern against re-pointing `kbSync` at tenant content preserved
- NOT inlining persistence into TaskStateService's state.json — race risk on shared file
- NOT halting on per-repo failure — isolation per ticket prescription; operator sees partial-completion in healthService payload

* chore(ci): retrigger PR #11940 checks (#11790)

* fix(test): include TENANT_REPO_SYNC_TASK_NAME in initial-task-state envelope assertion (#11790)

The createInitialTaskState envelope assertion at Orchestrator.spec.mjs:127 used exact-equality on the full ordered key list. Adding the tenant-repo-sync lane to buildTaskDefinitions (this PR's #11940 change) silently broke it post-rebase to dev. Test was green when stacked on #11789 because the integration boundary happened to land on the matching snapshot; the dev rebase exposed the gap.

Imports TENANT_REPO_SYNC_TASK_NAME alongside the other lane name imports and inserts it between PRIMARY_DEV_SYNC_TASK_NAME and DREAM_TASK_NAME, matching the lane order in buildTaskDefinitions.

* feat(orchestrator): wire tenant-repo-sync lane into Orchestrator.poll() + cloud-deployment toggle (#11790)

Closes the load-bearing Contract Ledger gap from PR #11940 cycle-1 review: the tenant-repo-sync lane was registered in TaskDefinitions but never scheduled by Orchestrator.poll(). Future-agent KB grep would see the lane as wired even though no periodic tick ever invoked it.

Changes:
- AiConfig.template.mjs: add intervals.tenantRepoSyncMs (30min default) + new orchestrator.cloudOnly collection with tenantRepoSyncEnabled (null = deployment-default: cloud enables, local disables, mirrors localOnly semantics inverted per ADR 0014 lane classification)
- Orchestrator.mjs: import TenantRepoSyncService + scheduling getDueTask + TENANT_REPO_SYNC_TASK_NAME constant; add tenantRepoSyncService + tenantRepoSyncGetDueTask instance-field DI seams; add resolveCloudOnlyEnabled helper (inverse of resolveDeploymentEnabled); add tenantRepoSyncIntervalMs + tenantRepoSyncEnabled lazy getters with NEO_ORCHESTRATOR_TENANT_REPO_SYNC_* env precedence; add cadenceEngine.runIfDue(TENANT_REPO_SYNC_TASK_NAME, ...) branch matching the PRIMARY_DEV_SYNC pattern (trigger + executor split)
- TenantRepoSyncService.mjs JSDoc lines 32-35: fix the rhetorical-drift mismatch GPT flagged; class-comment claimed all-or-fail semantics while line 213 implements partial-success (status='completed' when no repos failed OR at least one succeeded). Updated comment to document the correct partial-success contract
- Orchestrator.spec.mjs: add tenantRepoSyncEnabled/IntervalMs/Service/GetDueTask DI seam defaults in createTestOrchestrator helper (interval defaults to MAX_SAFE_INTEGER + enabled defaults to false so existing tests don't accidentally trigger); add savedCloudOnly restore in afterEach; add 2 new tests covering the load-bearing wiring: 'routes tenant-repo-sync through Orchestrator.poll() when enabled + interval elapsed (#11790 AC2 poll-wiring)' + 'skips tenant-repo-sync when disabled (local deployment default — Contract Ledger row 2)'

Deferred residuals tracked in follow-up #11942 (per-repo cadence + deterministic jitter/backoff + concurrency-limit gate + stable KB_TENANT_REPO_SYNC_* error codes). PR body for #11940 is being narrowed from `Resolves #11790` to `Refs #11790` since the original Contract Ledger has 9 surfaces and this PR delivers ~5.

Touched-spec verification: 139/139 unit tests pass (Orchestrator.spec + daemon.spec + scheduling/* + TenantRepoSyncService.spec + PrimaryRepoSyncService.spec).

* docs(orchestrator): remove ticket/Discussion/ADR-section archaeology from tenant-repo-sync JSDoc (#11790)

Operator-flagged substrate failure: I was reviewing and approving GPT's archaeology cleanup PRs (#11943/#11944/#11947/#11949) while still authoring the same decay-prone ticket/PR/Epic/Discussion/ADR-section anchor pattern in my own new JSDoc. Reviewing-removal while authoring-the-pattern is incoherent and creates future cleanup work.

Cleaned 3 files:

- ai/daemons/orchestrator/services/TenantRepoSyncService.mjs: removed (#11790) from @summary, removed (#11787/#11788/#11789) from per-step ASCII-flow refs, removed 'Epic #11731 + Discussion #11782 Option A' provenance, removed 'ADR 0014 §5.2' citation. Operational content preserved: the lane purpose, the per-repo loop steps, the local-only lane separation invariant, and the partial-success failure-isolation contract.
- ai/daemons/orchestrator/scheduling/tenantRepoSync.mjs: removed (#11790) from @summary.
- ai/scripts/maintenance/syncTenantRepos.mjs: removed (#11790) from @summary.

Tests in #11945 and #11946 use ticket-IDs only in test.describe labels (test-grep-traceability convention shared across the codebase, not archaeology in implementation comments) — verified clean and left as-is.

Substantive learning saved as memory entry feedback_jsdoc_archaeology_self_audit.

---------

Co-authored-by: tobiu <tobiasuhlig78@gmail.com>
Co-authored-by: Neo GPT <neo-gpt@users.noreply.github.com>"
- 2026-05-25T06:02:12Z @neo-opus-ada cross-referenced by PR #11951
- 2026-05-25T06:12:32Z @neo-opus-ada cross-referenced by PR #11952
- 2026-05-25T06:21:04Z @neo-opus-ada referenced in commit `bda09fc` - "feat(orchestrator): stable KB_TENANT_REPO_SYNC_* error code taxonomy (#11942 AC3+AC4)

Scope-narrowed slice of #11942 covering AC3 (stable error codes) + AC4 (JSDoc taxonomy). AC1 (per-repo jitter/backoff) and AC2 (concurrency-limit gate) remain tracked under #11942 as production-scale sophistication residuals.

**New module:** ai/daemons/orchestrator/services/TenantRepoSyncErrors.mjs

- 5 stable constants with the canonical KB_TENANT_REPO_SYNC_ prefix
- TENANT_REPO_SYNC_ERROR_CODES frozen set for membership checks
- TenantRepoSyncError class with code + meta payload
- isTenantRepoSyncErrorCode boundary-check helper

Codes shipped:
- KB_TENANT_REPO_SYNC_SYNC_FAILED — per-repo wrap code (already used in #11791)
- KB_TENANT_REPO_SYNC_REPO_NOT_CONFIGURED — --repo-slug filter against unknown slug
- KB_TENANT_REPO_SYNC_MANIFEST_UPDATE_FAILED — tenant-repo-sync-revisions.json write failure
- KB_TENANT_REPO_SYNC_TENANT_NOT_FOUND — reserved for future --tenant-id CLI flag
- KB_TENANT_REPO_SYNC_CONCURRENCY_GATE_TIMEOUT — reserved for #11942 AC2 concurrency gate

**Wired into TenantRepoSyncService.mjs:**

- Catch block uses isTenantRepoSyncErrorCode discriminator instead of hardcoded string-prefix check
- New REPO_NOT_CONFIGURED path: when onlyRepoSlugs filter returns empty against non-empty list, returns status=failed with details.reasonCode + unknownSlugs + configuredSlugs. WARN-level log line names the offending slugs.
- writePersistedRevisions wraps fs failure as TenantRepoSyncError(MANIFEST_UPDATE_FAILED) with phase + filePath meta. Next cycle retries idempotently per the per-repo failure isolation contract.
- runTask JSDoc gains the full error-code taxonomy table operators branch on.

**CLI surface:** ai/scripts/maintenance/syncTenantRepos.mjs

- Exit code 3 reserved for KB_TENANT_REPO_SYNC_REPO_NOT_CONFIGURED (distinguishes operator-error from operational-failure)
- --help output documents the 0/1/3/2 exit code matrix

**Docs:** learn/agentos/cloud-deployment/TenantIngestionModel.md

- New 'Stable Error Code Taxonomy' subsection inserted after the status enum, before the quarantine runbook
- Enumerates all 5 codes with where-it-surfaces + trigger columns
- Names the sibling-subsystem prefixes (KB_GITMIRROR_*, KB_INGEST_*, KB_TENANT_REPO_ACCESS_*) so operators distinguish error families

**Tests:**

- New spec: TenantRepoSyncErrors.spec.mjs (7 tests) — prefix invariant, set membership, frozen taxonomy, isTenantRepoSyncErrorCode discrimination, TenantRepoSyncError shape (code/meta/Error inheritance), default meta, stack trace.
- Extended TenantRepoSyncService.spec.mjs (2 new tests) — unknown-slug surfaces REPO_NOT_CONFIGURED with reason-payload; writePersistedRevisions wraps fs failure as MANIFEST_UPDATE_FAILED.
- Updated existing 'failure isolation' test to consume the post-#11791 details.repos shape.

Touched-surface verification: 18/18 spec pairs pass; 255/255 orchestrator-tree pass.

Stacked on PR #11951 (#11791 operator docs + health/telemetry). GitHub auto-retargets to dev when #11791 merges. Close-target as Refs #11942 — AC1 + AC2 remain in scope of #11942 for follow-up work."
- 2026-05-25T06:29:48Z @neo-opus-ada referenced in commit `7952a64` - "fix(orchestrator): propagate TenantRepoSyncError code + meta through runTask outer catch (#11942)

Operator-flagged via @neo-gpt #11952 review-hold: the runTask outer catch was preserving only e.message into details, silently swallowing the stable e.code + e.meta payload when an inner method threw TenantRepoSyncError (e.g. writePersistedRevisions → MANIFEST_UPDATE_FAILED). Verified probe at head bda09fcd0 returned no code/reasonCode through the outer details — a real bug, not a stylistic concern.

Fix in TenantRepoSyncService.runTask catch block:
- Discriminate via isTenantRepoSyncErrorCode(e.code); if recognized, propagate as details.reasonCode
- If e is a TenantRepoSyncError instance, also propagate e.meta into details.meta
- Otherwise wrap as KB_TENANT_REPO_SYNC_SYNC_FAILED so operators can branch on error.code instead of message prose
- Log line now carries the stable code: '[TenantRepoSync] Failed: KB_TENANT_REPO_SYNC_* (msg)'

New test: runTask propagates TenantRepoSyncError code + meta through outer details when syncTenantRepos throws. Uses a read-only revisions-dir fixture to force MANIFEST_UPDATE_FAILED via the genuine writePersistedRevisions path. Asserts:
1. result.details.reasonCode === 'KB_TENANT_REPO_SYNC_MANIFEST_UPDATE_FAILED'
2. result.details.meta.phase === 'manifest-update'
3. result.details.meta.filePath contains 'revisions.json'
4. ERROR-level log line carries the stable code prefix

The existing 'writePersistedRevisions wraps fs write failure' test asserted the direct-call throw shape but did NOT cover the propagation through runTask. This new test closes that gap. 19/19 pass."
- 2026-05-25T06:37:49Z @tobiu referenced in commit `521f18d` - "feat(cloud-deployment): operator docs + health/telemetry for tenant-repo ingestion (#11791) (#11951)

* feat(cloud-deployment): operator docs + health/telemetry for tenant-repo ingestion (#11791)

Sub 6 of Epic #11731 — makes the server-side pull lane operable + observable for the cloud-deployment trial.

**Docs (AC1 + AC4):**

learn/agentos/cloud-deployment/TenantIngestionModel.md: new ~155-line "Server-Side Pull Mode (Tenant Repo Sync)" section covering when-to-use-vs-push, tenantRepos[] configuration shape with credential-boundary rules, periodic + manual triggers, the tenant-repo-mirrors volume + NEO_TENANT_REPO_MIRROR_ROOT, redeploy posture (mirrors reproducible from upstream git, backup optional), health/telemetry payload shape, the active/degraded/quarantined/disabled status enum + operator quarantine runbook, operator logging format, and push-vs-pull coexistence rules. Cross-references TenantRepoAccessContract.mjs and the existing push-mode entry points.

learn/agentos/DeploymentCookbook.md: Section 9 extended with a server-side-pull-mode subsection pointing at the TenantIngestionModel anchor, naming the tenant-repo-mirrors volume + env vars + state-dir persistence behavior. Section 6 env-var inventory gains 3 rows: NEO_ORCHESTRATOR_TENANT_REPO_SYNC_ENABLED, NEO_ORCHESTRATOR_TENANT_REPO_SYNC_INTERVAL_MS, NEO_TENANT_REPO_MIRROR_ROOT.

**Health/telemetry projection (AC2):**

ai/daemons/orchestrator/services/TenantRepoSyncService.mjs: syncTenantRepos now projects the prescribed details.repos[] shape per the #11791 Contract Ledger row 2: each entry carries {tenantId, repoSlug, lastIngestedRev (short SHA), lastSyncAt (ISO timestamp), status (active|degraded), lastSyncDeletedCount (number on success), lastErrorCode (stable KB_TENANT_REPO_SYNC_* code on failure)}. Replaces the prior {repo, status, ingested, deleted, headRevision} shape. Non-prefixed underlying errors are wrapped as the stable KB_TENANT_REPO_SYNC_SYNC_FAILED code so operators branch on error.code, not message prose.

The quarantined status (consecutive-failure backoff) is intentionally deferred to #11942 (per-repo backoff state ticket I filed earlier). Until that lands, repeated failure surfaces as degraded with the same operator-runbook guidance. Doc + comment explicitly cross-references the deferred work so future agents don't accidentally reimplement it under #11791.

**Operator log format (AC3):**

Per-repo log lines now follow:
  [TenantRepoSync] Refreshing <tenantId>/<repoSlug>.
  [TenantRepoSync] <tenantId>/<repoSlug> completed: head=<short-sha> ingested=N deleted=M (<duration>ms)
  [TenantRepoSync] <tenantId>/<repoSlug> failed: <stable-code> (<message>)
  [TenantRepoSync] Cycle summary: N repos, C completed, F failed.

Duration measured against per-repo wall clock. Stable error codes from KB_TENANT_REPO_SYNC_* prefix. (Secret redaction via redactTenantRepoSecrets is the responsibility of GitMirror per #11787; not re-implemented here.)

**Tests:**

test/playwright/unit/ai/daemons/orchestrator/services/TenantRepoSyncService.spec.mjs: 3 new tests covering (1) per-repo details.repos[] freshness-field shape on success, (2) degraded status + stable KB_TENANT_REPO_SYNC_* code wrapping on failure, (3) operator log format (per-repo refresh + completed + cycle summary). Existing tests updated to consume the new details.repos shape (was details.results). 9/9 pass; 246/246 broader orchestrator-tree pass.

**Out of scope** (per ticket): scheduler-lane / service-class / contract implementation (sub-tickets 2-5 / #11731). Quarantined status / backoff state (#11942). Tenant-config operator MCP tool (#11731 subs). Webhook-driven triggering (Discussion #11782 OQ2).

* fix(cloud-deployment): example placeholders + rename repos_ local var (#11791)

Operator-flagged two substrate issues during #11951 cycle-1:

1. **Example repo URL collision risk**: previous example used 'acme-corp/widgets' / 'https://github.com/acme-corp/widgets.git'. The acme-corp GitHub org actually exists (not a reserved RFC placeholder), so using its namespace as a docs placeholder risks (a) misdirecting operators who try to clone the example, (b) name-collision if acme-corp ever creates a 'widgets' repo, and (c) implying an association with a real third-party org. Replaced with 'neomjs/create-app' across the config example, health-payload sample, and operator log lines per operator-suggested 'use own repos'.

2. **Misleading trailing-underscore on local var**: 'const repos_ = []' inside syncTenantRepos collected per-repo state results. The trailing-underscore is a reserved Neo class-system convention for reactive 'static config' slots with beforeSet/afterSet hooks; using it on a plain local variable creates cognitive load ('is this reactive?'). Root cause was an outer 'const repos' name collision; resolved by renaming the inner array to its actual semantic name 'repoStates' (these are per-repo state projections, not 'more repos'). Return statement still uses key 'repos' for backward-compat with health-payload consumers; only the local var name changes.

Both fixes are surface-level (no API/contract change). 9/9 TenantRepoSyncService spec still passes. Memory entries saved as feedback_example_placeholders_no_third_party_namespaces and feedback_neo_trailing_underscore_convention.

---------

Co-authored-by: tobiu <tobiasuhlig78@gmail.com>"
- 2026-05-25T06:40:56Z @neo-opus-ada referenced in commit `7023f1e` - "feat(orchestrator): stable KB_TENANT_REPO_SYNC_* error code taxonomy (#11942 AC3+AC4)

Scope-narrowed slice of #11942 covering AC3 (stable error codes) + AC4 (JSDoc taxonomy). AC1 (per-repo jitter/backoff) and AC2 (concurrency-limit gate) remain tracked under #11942 as production-scale sophistication residuals.

**New module:** ai/daemons/orchestrator/services/TenantRepoSyncErrors.mjs

- 5 stable constants with the canonical KB_TENANT_REPO_SYNC_ prefix
- TENANT_REPO_SYNC_ERROR_CODES frozen set for membership checks
- TenantRepoSyncError class with code + meta payload
- isTenantRepoSyncErrorCode boundary-check helper

Codes shipped:
- KB_TENANT_REPO_SYNC_SYNC_FAILED — per-repo wrap code (already used in #11791)
- KB_TENANT_REPO_SYNC_REPO_NOT_CONFIGURED — --repo-slug filter against unknown slug
- KB_TENANT_REPO_SYNC_MANIFEST_UPDATE_FAILED — tenant-repo-sync-revisions.json write failure
- KB_TENANT_REPO_SYNC_TENANT_NOT_FOUND — reserved for future --tenant-id CLI flag
- KB_TENANT_REPO_SYNC_CONCURRENCY_GATE_TIMEOUT — reserved for #11942 AC2 concurrency gate

**Wired into TenantRepoSyncService.mjs:**

- Catch block uses isTenantRepoSyncErrorCode discriminator instead of hardcoded string-prefix check
- New REPO_NOT_CONFIGURED path: when onlyRepoSlugs filter returns empty against non-empty list, returns status=failed with details.reasonCode + unknownSlugs + configuredSlugs. WARN-level log line names the offending slugs.
- writePersistedRevisions wraps fs failure as TenantRepoSyncError(MANIFEST_UPDATE_FAILED) with phase + filePath meta. Next cycle retries idempotently per the per-repo failure isolation contract.
- runTask JSDoc gains the full error-code taxonomy table operators branch on.

**CLI surface:** ai/scripts/maintenance/syncTenantRepos.mjs

- Exit code 3 reserved for KB_TENANT_REPO_SYNC_REPO_NOT_CONFIGURED (distinguishes operator-error from operational-failure)
- --help output documents the 0/1/3/2 exit code matrix

**Docs:** learn/agentos/cloud-deployment/TenantIngestionModel.md

- New 'Stable Error Code Taxonomy' subsection inserted after the status enum, before the quarantine runbook
- Enumerates all 5 codes with where-it-surfaces + trigger columns
- Names the sibling-subsystem prefixes (KB_GITMIRROR_*, KB_INGEST_*, KB_TENANT_REPO_ACCESS_*) so operators distinguish error families

**Tests:**

- New spec: TenantRepoSyncErrors.spec.mjs (7 tests) — prefix invariant, set membership, frozen taxonomy, isTenantRepoSyncErrorCode discrimination, TenantRepoSyncError shape (code/meta/Error inheritance), default meta, stack trace.
- Extended TenantRepoSyncService.spec.mjs (2 new tests) — unknown-slug surfaces REPO_NOT_CONFIGURED with reason-payload; writePersistedRevisions wraps fs failure as MANIFEST_UPDATE_FAILED.
- Updated existing 'failure isolation' test to consume the post-#11791 details.repos shape.

Touched-surface verification: 18/18 spec pairs pass; 255/255 orchestrator-tree pass.

Stacked on PR #11951 (#11791 operator docs + health/telemetry). GitHub auto-retargets to dev when #11791 merges. Close-target as Refs #11942 — AC1 + AC2 remain in scope of #11942 for follow-up work."
- 2026-05-25T06:40:56Z @neo-opus-ada referenced in commit `a076b07` - "fix(orchestrator): propagate TenantRepoSyncError code + meta through runTask outer catch (#11942)

Operator-flagged via @neo-gpt #11952 review-hold: the runTask outer catch was preserving only e.message into details, silently swallowing the stable e.code + e.meta payload when an inner method threw TenantRepoSyncError (e.g. writePersistedRevisions → MANIFEST_UPDATE_FAILED). Verified probe at head bda09fcd0 returned no code/reasonCode through the outer details — a real bug, not a stylistic concern.

Fix in TenantRepoSyncService.runTask catch block:
- Discriminate via isTenantRepoSyncErrorCode(e.code); if recognized, propagate as details.reasonCode
- If e is a TenantRepoSyncError instance, also propagate e.meta into details.meta
- Otherwise wrap as KB_TENANT_REPO_SYNC_SYNC_FAILED so operators can branch on error.code instead of message prose
- Log line now carries the stable code: '[TenantRepoSync] Failed: KB_TENANT_REPO_SYNC_* (msg)'

New test: runTask propagates TenantRepoSyncError code + meta through outer details when syncTenantRepos throws. Uses a read-only revisions-dir fixture to force MANIFEST_UPDATE_FAILED via the genuine writePersistedRevisions path. Asserts:
1. result.details.reasonCode === 'KB_TENANT_REPO_SYNC_MANIFEST_UPDATE_FAILED'
2. result.details.meta.phase === 'manifest-update'
3. result.details.meta.filePath contains 'revisions.json'
4. ERROR-level log line carries the stable code prefix

The existing 'writePersistedRevisions wraps fs write failure' test asserted the direct-call throw shape but did NOT cover the propagation through runTask. This new test closes that gap. 19/19 pass."
- 2026-05-25T06:43:15Z @neo-opus-ada cross-referenced by #11075
- 2026-05-25T06:44:48Z @neo-opus-ada cross-referenced by #11955
- 2026-05-25T06:56:00Z @neo-opus-ada referenced in commit `dff727c` - "fix(orchestrator): make TENANT_REPO_SYNC_ERROR_CODES truly immutable (#11942)

Cycle-2 review caught a real Set-freeze trap: `Object.freeze(new Set(...))`
freezes the Set object's properties but NOT its internal membership — `.add()`
still mutates the collection even after freeze. Verified empirically by GPT:
the exported Set accepted `KB_TENANT_REPO_SYNC_MUTATED` post-freeze.

Switch the exported value from `Object.freeze(new Set([...]))` to
`Object.freeze([...])` (frozen Array). In strict mode (ES modules are strict
by default), frozen arrays reject `.push()`, indexed assignment, and length
mutation with TypeError, giving genuine immutability of the exported substrate.

Add a private `TENANT_REPO_SYNC_ERROR_CODE_SET` (Set) for O(1) membership
lookup inside `isTenantRepoSyncErrorCode()` — the helper's contract is
unchanged for boundary consumers.

Test changes:
- `.size` → `.length`, `.has()` → `.includes()` / `.toContain()`
- Replaced the trivial `Object.isFrozen` assertion with a comprehensive
  negative-mutation test proving `.push`, indexed assignment, and length
  mutation all throw, and that `KB_TENANT_REPO_SYNC_MUTATED` cannot be
  injected into the accepted-code set.

Co-Authored-By: Claude Opus 4.7 <neo-opus-ada@neomjs.com>"
- 2026-05-25T07:12:26Z @neo-opus-ada cross-referenced by PR #11957
- 2026-05-25T07:46:33Z @tobiu referenced in commit `3b2fd16` - "feat(orchestrator): stable KB_TENANT_REPO_SYNC_* error code taxonomy (#11942 AC3+AC4) (#11952)

* feat(orchestrator): stable KB_TENANT_REPO_SYNC_* error code taxonomy (#11942 AC3+AC4)

Scope-narrowed slice of #11942 covering AC3 (stable error codes) + AC4 (JSDoc taxonomy). AC1 (per-repo jitter/backoff) and AC2 (concurrency-limit gate) remain tracked under #11942 as production-scale sophistication residuals.

**New module:** ai/daemons/orchestrator/services/TenantRepoSyncErrors.mjs

- 5 stable constants with the canonical KB_TENANT_REPO_SYNC_ prefix
- TENANT_REPO_SYNC_ERROR_CODES frozen set for membership checks
- TenantRepoSyncError class with code + meta payload
- isTenantRepoSyncErrorCode boundary-check helper

Codes shipped:
- KB_TENANT_REPO_SYNC_SYNC_FAILED — per-repo wrap code (already used in #11791)
- KB_TENANT_REPO_SYNC_REPO_NOT_CONFIGURED — --repo-slug filter against unknown slug
- KB_TENANT_REPO_SYNC_MANIFEST_UPDATE_FAILED — tenant-repo-sync-revisions.json write failure
- KB_TENANT_REPO_SYNC_TENANT_NOT_FOUND — reserved for future --tenant-id CLI flag
- KB_TENANT_REPO_SYNC_CONCURRENCY_GATE_TIMEOUT — reserved for #11942 AC2 concurrency gate

**Wired into TenantRepoSyncService.mjs:**

- Catch block uses isTenantRepoSyncErrorCode discriminator instead of hardcoded string-prefix check
- New REPO_NOT_CONFIGURED path: when onlyRepoSlugs filter returns empty against non-empty list, returns status=failed with details.reasonCode + unknownSlugs + configuredSlugs. WARN-level log line names the offending slugs.
- writePersistedRevisions wraps fs failure as TenantRepoSyncError(MANIFEST_UPDATE_FAILED) with phase + filePath meta. Next cycle retries idempotently per the per-repo failure isolation contract.
- runTask JSDoc gains the full error-code taxonomy table operators branch on.

**CLI surface:** ai/scripts/maintenance/syncTenantRepos.mjs

- Exit code 3 reserved for KB_TENANT_REPO_SYNC_REPO_NOT_CONFIGURED (distinguishes operator-error from operational-failure)
- --help output documents the 0/1/3/2 exit code matrix

**Docs:** learn/agentos/cloud-deployment/TenantIngestionModel.md

- New 'Stable Error Code Taxonomy' subsection inserted after the status enum, before the quarantine runbook
- Enumerates all 5 codes with where-it-surfaces + trigger columns
- Names the sibling-subsystem prefixes (KB_GITMIRROR_*, KB_INGEST_*, KB_TENANT_REPO_ACCESS_*) so operators distinguish error families

**Tests:**

- New spec: TenantRepoSyncErrors.spec.mjs (7 tests) — prefix invariant, set membership, frozen taxonomy, isTenantRepoSyncErrorCode discrimination, TenantRepoSyncError shape (code/meta/Error inheritance), default meta, stack trace.
- Extended TenantRepoSyncService.spec.mjs (2 new tests) — unknown-slug surfaces REPO_NOT_CONFIGURED with reason-payload; writePersistedRevisions wraps fs failure as MANIFEST_UPDATE_FAILED.
- Updated existing 'failure isolation' test to consume the post-#11791 details.repos shape.

Touched-surface verification: 18/18 spec pairs pass; 255/255 orchestrator-tree pass.

Stacked on PR #11951 (#11791 operator docs + health/telemetry). GitHub auto-retargets to dev when #11791 merges. Close-target as Refs #11942 — AC1 + AC2 remain in scope of #11942 for follow-up work.

* fix(orchestrator): propagate TenantRepoSyncError code + meta through runTask outer catch (#11942)

Operator-flagged via @neo-gpt #11952 review-hold: the runTask outer catch was preserving only e.message into details, silently swallowing the stable e.code + e.meta payload when an inner method threw TenantRepoSyncError (e.g. writePersistedRevisions → MANIFEST_UPDATE_FAILED). Verified probe at head bda09fcd0 returned no code/reasonCode through the outer details — a real bug, not a stylistic concern.

Fix in TenantRepoSyncService.runTask catch block:
- Discriminate via isTenantRepoSyncErrorCode(e.code); if recognized, propagate as details.reasonCode
- If e is a TenantRepoSyncError instance, also propagate e.meta into details.meta
- Otherwise wrap as KB_TENANT_REPO_SYNC_SYNC_FAILED so operators can branch on error.code instead of message prose
- Log line now carries the stable code: '[TenantRepoSync] Failed: KB_TENANT_REPO_SYNC_* (msg)'

New test: runTask propagates TenantRepoSyncError code + meta through outer details when syncTenantRepos throws. Uses a read-only revisions-dir fixture to force MANIFEST_UPDATE_FAILED via the genuine writePersistedRevisions path. Asserts:
1. result.details.reasonCode === 'KB_TENANT_REPO_SYNC_MANIFEST_UPDATE_FAILED'
2. result.details.meta.phase === 'manifest-update'
3. result.details.meta.filePath contains 'revisions.json'
4. ERROR-level log line carries the stable code prefix

The existing 'writePersistedRevisions wraps fs write failure' test asserted the direct-call throw shape but did NOT cover the propagation through runTask. This new test closes that gap. 19/19 pass.

* fix(orchestrator): make TENANT_REPO_SYNC_ERROR_CODES truly immutable (#11942)

Cycle-2 review caught a real Set-freeze trap: `Object.freeze(new Set(...))`
freezes the Set object's properties but NOT its internal membership — `.add()`
still mutates the collection even after freeze. Verified empirically by GPT:
the exported Set accepted `KB_TENANT_REPO_SYNC_MUTATED` post-freeze.

Switch the exported value from `Object.freeze(new Set([...]))` to
`Object.freeze([...])` (frozen Array). In strict mode (ES modules are strict
by default), frozen arrays reject `.push()`, indexed assignment, and length
mutation with TypeError, giving genuine immutability of the exported substrate.

Add a private `TENANT_REPO_SYNC_ERROR_CODE_SET` (Set) for O(1) membership
lookup inside `isTenantRepoSyncErrorCode()` — the helper's contract is
unchanged for boundary consumers.

Test changes:
- `.size` → `.length`, `.has()` → `.includes()` / `.toContain()`
- Replaced the trivial `Object.isFrozen` assertion with a comprehensive
  negative-mutation test proving `.push`, indexed assignment, and length
  mutation all throw, and that `KB_TENANT_REPO_SYNC_MUTATED` cannot be
  injected into the accepted-code set.

Co-Authored-By: Claude Opus 4.7 <neo-opus-ada@neomjs.com>

---------

Co-authored-by: tobiu <tobiasuhlig78@gmail.com>"
- 2026-05-25T08:17:55Z @neo-opus-ada cross-referenced by PR #11960
- 2026-05-25T08:25:10Z @neo-opus-ada referenced in commit `2d261f9` - "chore(orchestrator): use neomjs/* cloneUrl placeholders instead of example.com in tenant-repo-sync specs (#11942)

Operator-flagged 2026-05-25 (same day as PR #11951 acme cleanup): "not cool.
again an existing url, instead of using a neo repo url (e.g. neomjs/create-app).
we have been there within the last 5 PRs."

The pattern was inherited from the original #11790 spec fixtures and propagated
into the AC2 concurrency-gate tests added in this PR. operator's substrate-quality
discipline refinement: example.com is RESTRICTED to legitimate URL-shape cases
(secret-redaction tests etc.), NOT the default for placeholder URLs. Default is
always neomjs-owned URLs.

Replace `https://example.com/<slug>.git` → `https://github.com/neomjs/<slug>.git`
across all 18 spec fixture cloneUrl values. Test fakes don't read cloneUrl so
no behavioral change; this is pure substrate-quality polish.

15/15 PASS post-fix.

Refs #11942

Co-Authored-By: Claude Opus 4.7 <neo-opus-ada@neomjs.com>"
- 2026-05-25T08:29:17Z @neo-opus-ada referenced in commit `d4337b7` - "fix(orchestrator): guard concurrencyLimit + concurrencyGateTimeoutMs reactive configs against invalid values (#11942)

Cycle-1 review caught a real boundary-guard gap. The newly-added reactive
configs accepted any value:

- concurrencyLimit = 0 → never-acquirable semaphore (active < 0 always false)
- concurrencyLimit = 1.5 → admits 2 simultaneous slots (active < 1.5 true at 1)
- concurrencyLimit = NaN/Infinity/string → ambiguous semaphore state
- concurrencyGateTimeoutMs = -100/NaN/Infinity → broken setTimeout semantics

Add Neo class-system beforeSet hooks:
- beforeSetConcurrencyLimit: Number.isInteger(value) && value >= 1 → accept;
  else fall back to oldValue ?? 2 (template default)
- beforeSetConcurrencyGateTimeoutMs: Number.isFinite(value) && value >= 0
  → accept (0 is the valid no-timeout sentinel); else fall back to
  oldValue ?? 30000

Two new boundary-guard tests covering:
- concurrencyLimit invalid values: 0, -1, 1.5, NaN, Infinity, '3', null,
  undefined, {}, []  → all rejected; valid integers 1 and 10 accepted
- concurrencyGateTimeoutMs invalid: -1, -100, NaN, Infinity, -Infinity, '5000',
  null, undefined → all rejected; 0 (no-timeout sentinel) and 60000 accepted

17/17 PASS (15 prior + 2 new boundary-guard tests).

Refs #11942

Co-Authored-By: Claude Opus 4.7 <neo-opus-ada@neomjs.com>"
- 2026-05-25T08:45:14Z @tobiu referenced in commit `d21b41b` - "feat(orchestrator): concurrency-limit gate for tenant-repo-sync (#11942 AC2) (#11960)

* feat(orchestrator): concurrency-limit gate for tenant-repo-sync (#11942 AC2)

Reactive `concurrencyLimit` + `concurrencyGateTimeoutMs` config on
TenantRepoSyncService. Per-repo git/ingest work acquires a semaphore slot
before clone/fetch/envelope/ingest; releases on completion (success OR
failure). Slot-acquisition timeout surfaces stable
KB_TENANT_REPO_SYNC_CONCURRENCY_GATE_TIMEOUT error code per the existing
taxonomy.

Architectural shift:
- Sequential `for (const repo of repos)` loop converted to
  `Promise.all(repos.map(async ...))` with semaphore-guarded entry.
- In-memory async-semaphore primitive created per runTask invocation
  from current reactive config values, so live edits take effect on the
  next cycle.
- Default concurrencyLimit: 2 (conservative for multi-tenant cloud
  deployment). Set to 1 to serialize (pre-AC2 behavior).
- Default concurrencyGateTimeoutMs: 30000 (30s).

JSDoc taxonomy table updated:
KB_TENANT_REPO_SYNC_CONCURRENCY_GATE_TIMEOUT transitions from
"reserved" to "per-repo lastErrorCode" with active emitter.

Tests (3 new):
- concurrencyLimit=1 serializes (in-flight count never exceeds 1)
- concurrencyLimit=2 caps parallel at 2 (4 repos, peak in-flight = 2)
- gate-timeout surfaces KB_TENANT_REPO_SYNC_CONCURRENCY_GATE_TIMEOUT
  on queued repo when slot wait exceeds concurrencyGateTimeoutMs

afterEach restores reactive config defaults to prevent leakage between
sibling tests in fully-parallel mode.

262/262 PASS across orchestrator test tree post-feature.

Refs #11942 (AC2 closed; AC1 jitter/backoff remains open as separate PR)

Co-Authored-By: Claude Opus 4.7 <neo-opus-ada@neomjs.com>

* chore(orchestrator): use neomjs/* cloneUrl placeholders instead of example.com in tenant-repo-sync specs (#11942)

Operator-flagged 2026-05-25 (same day as PR #11951 acme cleanup): "not cool.
again an existing url, instead of using a neo repo url (e.g. neomjs/create-app).
we have been there within the last 5 PRs."

The pattern was inherited from the original #11790 spec fixtures and propagated
into the AC2 concurrency-gate tests added in this PR. operator's substrate-quality
discipline refinement: example.com is RESTRICTED to legitimate URL-shape cases
(secret-redaction tests etc.), NOT the default for placeholder URLs. Default is
always neomjs-owned URLs.

Replace `https://example.com/<slug>.git` → `https://github.com/neomjs/<slug>.git`
across all 18 spec fixture cloneUrl values. Test fakes don't read cloneUrl so
no behavioral change; this is pure substrate-quality polish.

15/15 PASS post-fix.

Refs #11942

Co-Authored-By: Claude Opus 4.7 <neo-opus-ada@neomjs.com>

* fix(orchestrator): guard concurrencyLimit + concurrencyGateTimeoutMs reactive configs against invalid values (#11942)

Cycle-1 review caught a real boundary-guard gap. The newly-added reactive
configs accepted any value:

- concurrencyLimit = 0 → never-acquirable semaphore (active < 0 always false)
- concurrencyLimit = 1.5 → admits 2 simultaneous slots (active < 1.5 true at 1)
- concurrencyLimit = NaN/Infinity/string → ambiguous semaphore state
- concurrencyGateTimeoutMs = -100/NaN/Infinity → broken setTimeout semantics

Add Neo class-system beforeSet hooks:
- beforeSetConcurrencyLimit: Number.isInteger(value) && value >= 1 → accept;
  else fall back to oldValue ?? 2 (template default)
- beforeSetConcurrencyGateTimeoutMs: Number.isFinite(value) && value >= 0
  → accept (0 is the valid no-timeout sentinel); else fall back to
  oldValue ?? 30000

Two new boundary-guard tests covering:
- concurrencyLimit invalid values: 0, -1, 1.5, NaN, Infinity, '3', null,
  undefined, {}, []  → all rejected; valid integers 1 and 10 accepted
- concurrencyGateTimeoutMs invalid: -1, -100, NaN, Infinity, -Infinity, '5000',
  null, undefined → all rejected; 0 (no-timeout sentinel) and 60000 accepted

17/17 PASS (15 prior + 2 new boundary-guard tests).

Refs #11942

Co-Authored-By: Claude Opus 4.7 <neo-opus-ada@neomjs.com>

---------

Co-authored-by: tobiu <tobiasuhlig78@gmail.com>"
### @neo-gpt - 2026-05-25T08:53:51Z

Post-merge reconciliation after #11960:

- #11960 merged at `d21b41bc95c705f581ddc817fe2c651920d747fd` on 2026-05-25.
- AC2 is shipped: `TenantRepoSyncService.concurrencyLimit`, semaphore-gated per-repo work, `KB_TENANT_REPO_SYNC_CONCURRENCY_GATE_TIMEOUT`, and serialized/bounded/timeout + invalid-config guard tests landed.
- #11952 had already covered the stable error-code/JSDoc taxonomy slice for AC3/AC4, with AC2 activating the previously reserved timeout emitter.
- Do not close #11942 yet: AC1 remains the live residual (per-repo cadence + deterministic jitter/backoff state stored alongside `lastIngestedRev`).

Operational next slice is AC1 only, unless the owner wants to split AC1 into a narrower sub-ticket before implementation.

- 2026-05-25T09:33:07Z @neo-opus-ada cross-referenced by PR #11962
- 2026-05-25T09:43:17Z @neo-opus-ada referenced in commit `5d374c7` - "chore(orchestrator): remove hardcoded magic numbers in AC1 jitter+backoff (#11942)

Operator-flagged 2026-05-25 same-day: the AC1 PR introduced TWO hardcoded
magic numbers on a branch where the active substrate-evolution discussion
(#11961 Agent OS config ownership lift) is literally about removing magic
numbers:

- DEFAULT_JITTER_RATIO = 0.20 in ai/daemons/orchestrator/scheduling/tenantRepoSync.mjs
- DEFAULT_GLOBAL_CADENCE_MS = 30 * 60 * 1000 in ai/daemons/orchestrator/services/TenantRepoSyncService.mjs

Self-inconsistency caught mid-cycle-1 review. Both constants now move to AiConfig:

- aiConfig.orchestrator.intervals.tenantRepoSyncMs (already exists; default 30min) —
  used directly as the global cadence default via TenantRepoSyncService.runTask
  signature default
- aiConfig.orchestrator.tenantRepoSync.jitterRatio (NEW; default 0.20) — used
  directly as the jitter ratio default via same signature

Pure functions stay pure: computeDeterministicJitter and isRepoDue default to
jitterRatio=0 (no jitter) when caller omits the value. Production caller
(TenantRepoSyncService) always passes the AiConfig value through. This keeps
the scheduling primitives config-free for testability while routing all
operator-tunable values through AiConfig.

Tests adjusted to pass jitterRatio: 0.20 explicitly when asserting bounded/
distinct-jitter behavior (the pure-function contract). AiConfig-canonical-value
assertion belongs in service-level integration tests, not pure-function tests.

40/40 PASS post-fix (18 scheduling + 22 service).

Refs #11942

Co-Authored-By: Claude Opus 4.7 <neo-opus-ada@neomjs.com>"
- 2026-05-25T09:44:49Z @neo-gpt cross-referenced by #10103
- 2026-05-25T10:44:33Z @neo-opus-ada referenced in commit `cb010c1` - "fix(orchestrator): decouple sweep cadence + bootstrap-seed for actual jitter spread (#11942 AC1 cycle-2)

Resolves @neo-gpt cycle-1 review CHANGES_REQUESTED on PR #11962 — Required
Actions RA1 (scheduler composition) + RA2 (cadence pass-through) + RA3
(integration test).

The original AC1 implementation computed per-repo deterministic jitter
correctly but the production scheduler invoked `tenant-repo-sync` on the same
cadence used as the per-repo base cadence. With `lastRunAttemptAt=0` for
fresh repos and `Date.now()` as `now`, all repos became due on cycle 1 →
thundering herd despite jitter. After cycle 1, non-zero jitter caused all
repos to miss the next sweep and become due together on the cycle after →
thundering herd cycle 2.

Two architectural decoupling moves:

1. Sweep cadence vs per-repo cadence are now distinct configs. New
   `AiConfig.orchestrator.tenantRepoSync.sweepCadenceMs` (default 1min)
   drives Orchestrator scheduling; existing
   `AiConfig.orchestrator.intervals.tenantRepoSyncMs` (default 30min) is now
   strictly the per-repo default cadence read by `isRepoDue()`.

2. Bootstrap seeding: `syncTenantRepos` pre-seeds new repos with
   `lastRunAttemptAt = now - baseCadenceMs`, making effective due-time
   `now + jitterMs`. Subsequent short-cadence sweeps within the jitter
   window pick up repos at staggered times — actual spread, not synchronized
   thundering herd. Seeded state persists across restarts.

Plus cadence-passthrough: Orchestrator now explicitly passes
`globalCadenceMs: this.tenantRepoSyncIntervalMs` AND
`jitterRatio: this.tenantRepoSyncJitterRatio` into `runTask()`, eliminating
the split source of truth between `Orchestrator.mjs` env-resolved cadence
and `TenantRepoSyncService.runTask()` AiConfig-default fallback.

New getters: `tenantRepoSyncSweepCadenceMs` + `tenantRepoSyncJitterRatio`
with env precedence (`NEO_ORCHESTRATOR_TENANT_REPO_SYNC_SWEEP_CADENCE_MS`,
`NEO_ORCHESTRATOR_TENANT_REPO_SYNC_JITTER_RATIO`).

Test seam: `seedBootstrap` parameter (default `true` in production) lets
spec files that simulate "first cycle fires all repos" opt out of seeding.

Test evidence: 286/286 PASS across orchestrator/. 5 new cycle-2 tests
(bootstrap seeding persistence, multi-sweep spread, orchestrator-resolved
globalCadenceMs pass-through to isRepoDue, jitterRatio pass-through, manual
CLI bootstrap-bypass)."
- 2026-05-25T11:20:19Z @tobiu referenced in commit `323a8d5` - "feat(orchestrator): per-repo jitter + exponential backoff for tenant-repo-sync (#11942 AC1) (#11962)

* feat(orchestrator): pure functions for #11942 AC1 jitter+backoff (foundation)

Foundation commit for #11942 AC1 (per-repo jitter/backoff). Adds two pure
functions to ai/daemons/orchestrator/scheduling/tenantRepoSync.mjs:

1. computeDeterministicJitter({tenantId, repoSlug, baseCadenceMs, jitterRatio})
   - FNV-1a hash on `${tenantId}/${repoSlug}` seed → offset in [0, jitterRatio*baseCadenceMs)
   - Default jitterRatio=0.20 (per AC1 prescription: jitter window ≤ 20% of cadence)
   - Deterministic across daemon restarts + HA failovers (operator reproducibility)
   - Anti-thundering-herd on first-bootstrap multi-tenant deployments

2. isRepoDue({repo, persistedRepoState, now, globalCadenceMs, jitterRatio})
   - Effective cadence = (configuredCadence + jitter) * 2^consecutiveFailures
   - Per-repo cadence override via repo.cadenceMs; falls back to globalCadenceMs
   - Backoff: doubles per consecutive failure; reset on success
   - Returns full envelope {due, effectiveCadenceMs, jitterMs, backoffMultiplier, lastRunAttemptAt}

12 new unit tests covering:
- Invalid-input guards on computeDeterministicJitter
- Determinism (same inputs → same jitter)
- Bounded output (jitter < ratio * cadence)
- Distinct-repos-distinct-jitter (anti-thundering-herd proven)
- jitterRatio scaling cap behavior
- Bootstrap due-state (lastRunAttemptAt=0 + now=0 → not due)
- Bootstrap due once now exceeds effectiveCadence
- Not-yet-due (interval not elapsed)
- Per-repo cadence override beats global
- Backoff multiplier doubles per consecutive failure
- Backoff reset on consecutiveFailures: 0
- Deterministic due-state envelope (idempotent computation)

18/18 PASS (6 existing trigger tests + 12 new AC1 tests).

Next: persistence-schema migration + syncTenantRepos integration.

Refs #11942 AC1

Co-Authored-By: Claude Opus 4.7 <neo-opus-ada@neomjs.com>

* feat(orchestrator): per-repo jitter + exponential backoff for tenant-repo-sync (#11942 AC1)

Integrate isRepoDue + per-repo state mutation into TenantRepoSyncService.

Architectural shift:
- Per-repo persistence schema extended from bare SHA strings to full state objects:
  {lastIngestedRev, lastRunAttemptAt, consecutiveFailures}
- Backward-compatible read: pre-AC1 string-shaped entries auto-migrate to the
  new object shape on read (lastIngestedRev = old-string; lastRunAttemptAt = 0;
  consecutiveFailures = 0). Next write persists new shape, completing in-place
  migration.
- Per-repo due-check via isRepoDue before semaphore.acquire: not-due repos skip
  cleanly with status='not-due' health-payload entry + nextDueAt + effectiveCadenceMs
  (operator visibility), no semaphore slot consumed, no state mutation.
- Manual CLI invocation (onlyRepoSlugs) bypasses due-check — operator-initiated
  syncs always fire for the requested repos regardless of cadence/backoff.
- On success: lastRunAttemptAt advances + consecutiveFailures reset to 0.
- On failure: lastRunAttemptAt advances + consecutiveFailures increments (drives
  next cycle's backoff multiplier).
- Status logic: not-due cycles still 'completed' (cycle ran successfully; each
  repo's decision was honored). 'failed' only when attempted work failed and
  none succeeded.

New runTask + syncTenantRepos parameter:
- globalCadenceMs (default 30min) — feeds into per-repo isRepoDue decisions.
  Caller (Orchestrator) should pass the actual configured tenantRepoSyncMs.

5 new integration tests (TenantRepoSyncService.spec.mjs):
1. Skip not-due repo with prior recent lastRunAttemptAt → status='not-due'
2. Persists consecutiveFailures increment on failure
3. Persists consecutiveFailures reset on subsequent success
4. Backward-compatible read of pre-AC1 string-shaped persistence
5. onlyRepoSlugs (manual CLI path) bypasses due-check

Plus 12 pure-function tests from prior commit covering computeDeterministicJitter
(determinism, bounded output, anti-thundering-herd, jitterRatio scaling) + isRepoDue
(bootstrap, cadence override, backoff multiplier doubling, success reset).

281/281 PASS across orchestrator test tree (40/40 in #11942-touched specs).

Refs #11942 AC1 (jitter/backoff). AC1 + AC2 (#11960) + AC3+AC4 (#11952) all
shipped — #11942 epic complete.

Co-Authored-By: Claude Opus 4.7 <neo-opus-ada@neomjs.com>

* chore(orchestrator): remove hardcoded magic numbers in AC1 jitter+backoff (#11942)

Operator-flagged 2026-05-25 same-day: the AC1 PR introduced TWO hardcoded
magic numbers on a branch where the active substrate-evolution discussion
(#11961 Agent OS config ownership lift) is literally about removing magic
numbers:

- DEFAULT_JITTER_RATIO = 0.20 in ai/daemons/orchestrator/scheduling/tenantRepoSync.mjs
- DEFAULT_GLOBAL_CADENCE_MS = 30 * 60 * 1000 in ai/daemons/orchestrator/services/TenantRepoSyncService.mjs

Self-inconsistency caught mid-cycle-1 review. Both constants now move to AiConfig:

- aiConfig.orchestrator.intervals.tenantRepoSyncMs (already exists; default 30min) —
  used directly as the global cadence default via TenantRepoSyncService.runTask
  signature default
- aiConfig.orchestrator.tenantRepoSync.jitterRatio (NEW; default 0.20) — used
  directly as the jitter ratio default via same signature

Pure functions stay pure: computeDeterministicJitter and isRepoDue default to
jitterRatio=0 (no jitter) when caller omits the value. Production caller
(TenantRepoSyncService) always passes the AiConfig value through. This keeps
the scheduling primitives config-free for testability while routing all
operator-tunable values through AiConfig.

Tests adjusted to pass jitterRatio: 0.20 explicitly when asserting bounded/
distinct-jitter behavior (the pure-function contract). AiConfig-canonical-value
assertion belongs in service-level integration tests, not pure-function tests.

40/40 PASS post-fix (18 scheduling + 22 service).

Refs #11942

Co-Authored-By: Claude Opus 4.7 <neo-opus-ada@neomjs.com>

* fix(orchestrator): decouple sweep cadence + bootstrap-seed for actual jitter spread (#11942 AC1 cycle-2)

Resolves @neo-gpt cycle-1 review CHANGES_REQUESTED on PR #11962 — Required
Actions RA1 (scheduler composition) + RA2 (cadence pass-through) + RA3
(integration test).

The original AC1 implementation computed per-repo deterministic jitter
correctly but the production scheduler invoked `tenant-repo-sync` on the same
cadence used as the per-repo base cadence. With `lastRunAttemptAt=0` for
fresh repos and `Date.now()` as `now`, all repos became due on cycle 1 →
thundering herd despite jitter. After cycle 1, non-zero jitter caused all
repos to miss the next sweep and become due together on the cycle after →
thundering herd cycle 2.

Two architectural decoupling moves:

1. Sweep cadence vs per-repo cadence are now distinct configs. New
   `AiConfig.orchestrator.tenantRepoSync.sweepCadenceMs` (default 1min)
   drives Orchestrator scheduling; existing
   `AiConfig.orchestrator.intervals.tenantRepoSyncMs` (default 30min) is now
   strictly the per-repo default cadence read by `isRepoDue()`.

2. Bootstrap seeding: `syncTenantRepos` pre-seeds new repos with
   `lastRunAttemptAt = now - baseCadenceMs`, making effective due-time
   `now + jitterMs`. Subsequent short-cadence sweeps within the jitter
   window pick up repos at staggered times — actual spread, not synchronized
   thundering herd. Seeded state persists across restarts.

Plus cadence-passthrough: Orchestrator now explicitly passes
`globalCadenceMs: this.tenantRepoSyncIntervalMs` AND
`jitterRatio: this.tenantRepoSyncJitterRatio` into `runTask()`, eliminating
the split source of truth between `Orchestrator.mjs` env-resolved cadence
and `TenantRepoSyncService.runTask()` AiConfig-default fallback.

New getters: `tenantRepoSyncSweepCadenceMs` + `tenantRepoSyncJitterRatio`
with env precedence (`NEO_ORCHESTRATOR_TENANT_REPO_SYNC_SWEEP_CADENCE_MS`,
`NEO_ORCHESTRATOR_TENANT_REPO_SYNC_JITTER_RATIO`).

Test seam: `seedBootstrap` parameter (default `true` in production) lets
spec files that simulate "first cycle fires all repos" opt out of seeding.

Test evidence: 286/286 PASS across orchestrator/. 5 new cycle-2 tests
(bootstrap seeding persistence, multi-sweep spread, orchestrator-resolved
globalCadenceMs pass-through to isRepoDue, jitterRatio pass-through, manual
CLI bootstrap-bypass).

---------

Co-authored-by: tobiu <tobiasuhlig78@gmail.com>"
- 2026-05-25T11:20:19Z @tobiu closed this issue
- 2026-05-25T13:20:36Z @neo-opus-ada cross-referenced by PR #11971

