---
id: 11507
title: 'Lane C of #11503 — wrap manual heavy maintenance scripts with HeavyMaintenanceLeaseService'
state: CLOSED
labels:
  - enhancement
  - ai
  - model-experience
assignees:
  - neo-opus-ada
createdAt: '2026-05-16T23:29:29Z'
updatedAt: '2026-05-17T01:08:33Z'
githubUrl: 'https://github.com/neomjs/neo/issues/11507'
author: neo-opus-ada
commentsCount: 0
parentIssue: null
subIssues: []
subIssuesCompleted: 0
subIssuesTotal: 0
blockedBy: []
blocking: []
closedAt: '2026-05-17T01:08:33Z'
---
# Lane C of #11503 — wrap manual heavy maintenance scripts with HeavyMaintenanceLeaseService

Resolves Lane C of #11503 umbrella. Builds on the lease primitive shipped by PR #11506 (#11505).

FAIR-band: in-band [10/30] — prio-0 per operator elevation 2026-05-16T22:55Z; closes the gh-CLI bypass surface at the maintenance-script layer (sibling to PR #11502 closing the same pattern at the pr-review surface).

## Premise

PR #11506 / #11505 shipped `HeavyMaintenanceLeaseService` with `withHeavyMaintenanceLease(task, {owner, ...})` wrapper. Lane C wires manual CLI scripts into that lease so they cannot collide with orchestrator-owned heavy work OR with each other.

Empirical anchor (today's wedge at 19:27Z): orchestrator's kbSync subprocess wedged while operator's manual `npm run ai:sync-kb` would have collided in parallel had operator triggered it. The substrate gap is real and exactly the failure class Lane C closes.

## Scope (per #11503 ACs 4-7 and lead-sync from @neo-gpt MESSAGE:20f48f73)

| Script | AC | Lease owner string |
|---|---|---|
| `buildScripts/ai/runSandman.mjs` | AC4 | `sandman` |
| `buildScripts/ai/syncKnowledgeBase.mjs` | AC5 | `kbSync` |
| `buildScripts/ai/backup.mjs` | AC6 | `backup` |
| `buildScripts/ai/syncGithubWorkflow.mjs` | AC7 | `syncGithubWorkflow` (whole-run guarded per GPT's call; Stage 1/Stage 2 split deferred) |

## Prescription

For each script, wrap the existing `main()` (or equivalent top-level async work) with `withHeavyMaintenanceLease`:

```js
import {withHeavyMaintenanceLease} from '../../ai/services/github-workflow/...' // path TBD per services.mjs barrel

const result = await withHeavyMaintenanceLease(
    async (lease) => {
        // ... existing script body ...
    },
    {owner: 'sandman', reason: 'manual-cli'}
);

if (result.status === 'held') {
    console.log(`⏸️  Deferred: heavy-maintenance lease held by '${result.lease.owner}' (reason='${result.lease.reason}', pid=${result.lease.pid}, acquiredAt=${result.lease.acquiredAt}).`);
    console.log(`   This script will not run while another heavy-maintenance task is active.`);
    process.exit(0);
}
```

`result.status === 'held'` is a non-error exit (matches GPT's AC9 "deferred due to active heavy-maintenance lease" log shape).

## Acceptance Criteria

- [ ] **AC1**: All 4 scripts (`runSandman.mjs`, `syncKnowledgeBase.mjs`, `backup.mjs`, `syncGithubWorkflow.mjs`) wrap their main body with `withHeavyMaintenanceLease`
- [ ] **AC2**: When lease is held by another owner, scripts exit 0 with a clear stdout message naming the active owner + acquisition timestamp
- [ ] **AC3**: When lease is acquired, scripts execute existing behavior unchanged + auto-release on completion (via `withHeavyMaintenanceLease` finally block)
- [ ] **AC4**: Spec coverage matches sibling-script-test precedent: per-script test asserting both code paths (acquired + held) — at least 1 test per script
- [ ] **AC5**: No regression in non-lease behavior (existing tests + smoke runs continue to work)

## Avoided Traps

- **Splitting `syncGithubWorkflow` into Stage 1 (GitHub IO, light) vs Stage 2 (Chroma writes, heavy)**: rejected per GPT's lead-sync (MESSAGE:20f48f73). Whole-run guard for v1; Stage-split refinement is a separate future ticket if lease hold-time becomes empirical pain.
- **Adding lease semantics to `npm run ai:check-substrate-size` / `lint-skill-manifest` / `check-retired-primitives`**: rejected — these are pure-static-validation scripts (no Chroma/SQLite/LLM writes); not in the heavy-maintenance set.
- **Building a script-specific lock primitive in each script**: rejected — that's exactly the "per-script private locks" trap #11503 explicitly named. The shared lease primitive (PR #11506) is the substrate.
- **Blocking the script on held-lease (wait-loop)**: rejected — `withHeavyMaintenanceLease` returns `status: 'held'` immediately; scripts should defer-and-exit, not wait. Operator can re-invoke later. Matches non-error deferral semantics from AC9.

## Related

- **Parent umbrella**: #11503 (Enforce heavy-maintenance mutex across Agent OS tasks)
- **Substrate dependency**: PR #11506 / #11505 (shared lease primitive)
- **Conceptual sibling at different surface**: PR #11502 / #11495 (CI lint for gh-pr-review CLI bypass) — same Helpful-Assistant CLI-bypass failure pattern, different substrate surface
- **Empirical anchor**: today's wedge cascade at 19:27Z (manual kb-sync + orchestrator kb-sync collision the lease primitive + this Lane C close together)

## Out of Scope

- Lane A — adding `backup` to `DEFAULT_HEAVY_MAINTENANCE_TASK_NAMES` in `Orchestrator.mjs` (folded by GPT into Lane B per their MESSAGE:20f48f73; will be handled separately if not part of PR #11506)
- Lane D — `PrimaryRepoSyncService.runKbSync()` nested cascade lease wiring (GPT considering during Lane B API design)
- Lane E — observability/stale-adoption surfacing (separate ticket if needed)

## Timeline

- 2026-05-16T23:29:30Z @neo-opus-ada assigned to @neo-opus-ada
- 2026-05-16T23:29:30Z @neo-opus-ada added the `enhancement` label
- 2026-05-16T23:29:30Z @neo-opus-ada added the `ai` label
- 2026-05-16T23:29:31Z @neo-opus-ada added the `model-experience` label
- 2026-05-16T23:37:50Z @neo-opus-ada cross-referenced by PR #11509
- 2026-05-16T23:50:33Z @neo-gpt cross-referenced by #11511
- 2026-05-16T23:55:52Z @neo-gpt cross-referenced by PR #11512
- 2026-05-17T00:28:00Z @neo-opus-ada referenced in commit `375a1e9` - "fix(buildScripts): runSandman.mjs fail-closed on lease-acquisition error (PR #11509 cycle 1 per GPT review)

GPT's cycle-1 review (PRR_kwDODSospM8AAAABAJIdTg on PR #11509) caught a fail-closed
gap: when withHeavyMaintenanceLease itself failed (e.g., lease-write IO error), the
prior code set process.exitCode=1 in the catch but FELL THROUGH to
GraphService.decayGlobalTopology() which mutates SQLite graph edges and
_SYSTEM_STATE. That defeats the substrate protection PR #11509 ships — the one
path where the lease was NOT acquired could still mutate the graph outside the
mutex contract.

Fix: process.exit(1) directly in the lease-acquisition catch block, short-circuiting
before the decay call. The decay-on-completion path is now the only branch where
graph mutation runs, and it's guaranteed lease-protected.

Added regression spec asserting:
- The acquisition-failure catch block calls process.exit(1) (not just sets exitCode)
- Defensive: if any process.exitCode=1 assignment exists in the catch, it MUST be
  paired with a process.exit call (prevents future regression to the original buggy
  pattern)

14/14 Lane C content-verification spec tests pass in 553ms (13 prior + 1 new).

Refs #11507 (Lane C of #11503)"
- 2026-05-17T00:45:05Z @neo-opus-ada referenced in commit `3631fb3` - "fix(buildScripts): runSandman.mjs decay inside lease window (PR #11509 cycle 2 per GPT review)

GPT's cycle-2 review (PRR_kwDODSospM8AAAABAJKF8w) caught: `withHeavyMaintenanceLease`
releases the lease in its own `finally` BEFORE returning, so the post-wrapper
`GraphService.decayGlobalTopology()` call ran OUTSIDE the lease window — defeating
the substrate protection this PR is shipping (the one path where the lease was
ACQUIRED but the graph mutation still happened unprotected).

Fix:
- Add a `finally` block to the inner try/catch inside the async task passed to
  `withHeavyMaintenanceLease`. JS guarantees the inner finally completes before
  the awaited task settles, so the wrapper's release runs strictly after decay.
- Wrap decay in its own try/catch inside the finally to preserve prior
  graceful-fail semantics: a decay failure must NOT throw out of the lease-wrapped
  task (which would suppress the inner return value AND propagate to the outer
  catch reserved for lease-acquisition fail-closed semantics).
- Remove the now-redundant post-wrapper decay block.

Spec coverage:
- New test pins the release-timing invariant via three independent assertions:
  (1) decay offset < wrapper-options-trailer offset (decay is inside the wrapped task);
  (2) decay offset < post-wrapper held-handling offset (decay is not after the await);
  (3) decay is invoked from inside a `finally {` block (pins structural mechanism,
      not just offset ordering — guards against accidental refactor into outer catch
      which only fires on lease-acquisition failure).
- All 15 tests pass (13 baseline + cycle-1 fail-closed + cycle-2 release-timing).

(#11507)"
- 2026-05-17T01:08:33Z @tobiu referenced in commit `a5c6380` - "feat(buildScripts): wrap manual heavy-maintenance scripts with shared lease — Lane C of #11503 (#11507) (#11509)

* feat(buildScripts): wrap manual heavy-maintenance scripts with shared lease — Lane C of #11503 (#11507)

Wires the four operator-runnable heavy-maintenance CLI scripts into the shared
HeavyMaintenanceLeaseService primitive shipped by PR #11506 / #11505. Closes Lane C
of the #11503 umbrella: the gh-CLI bypass surface at the maintenance-script layer
(parallel to PR #11502 closing the same Helpful-Assistant CLI-bypass pattern at the
pr-review surface).

Empirical anchor: today's wedge cascade at 2026-05-16T19:27Z had the orchestrator's
own kbSync subprocess wedged while a manual npm run ai:sync-kb would have collided
in parallel. The substrate gap is exactly the failure class this PR closes.

Scripts wrapped (owner string aligns with Orchestrator.mjs DEFAULT_HEAVY_MAINTENANCE_TASK_NAMES):

- buildScripts/ai/runSandman.mjs           → owner: 'sandman'
- buildScripts/ai/syncKnowledgeBase.mjs    → owner: 'kbSync'
- buildScripts/ai/backup.mjs               → owner: 'backup'
- buildScripts/ai/syncGithubWorkflow.mjs   → owner: 'syncGithubWorkflow' (whole-run guarded per AC7 / GPT lead-sync)

Wrapper pattern (all 4 scripts):
1. Import withHeavyMaintenanceLease from ai/daemons/services/HeavyMaintenanceLeaseService.mjs
2. Wrap existing main-work async function inside withHeavyMaintenanceLease({owner, reason: 'manual-cli', metadata})
3. On outcome.status === 'held' → print deferred message naming active lease owner + reason + pid + acquiredAt → process.exit(0) (non-error semantics per #11503 AC9)
4. On outcome.status === 'completed' → existing behavior unchanged + auto-release via withHeavyMaintenanceLease's try/finally

runSandman.mjs preserves its existing graceful-fail semantics for provider-readiness:
inner try/catch returns return-value to wrapper (no throw); decay-topology finally step
still runs on completed (skipped on held since no graph mutation occurred).

backup.mjs wraps only the auto-run-when-invoked-directly block (NOT the exported
runBackup function itself) so test harnesses + other module-level callers retain full
control of their own concurrency context.

Spec coverage: 13/13 tests pass in 548ms via content-verification across all 4 scripts.
Asserts (per script): import-presence, wrapper-invocation with correct owner string,
held-outcome handling with diagnostic message + process.exit(0). Plus one cross-script
assertion that no per-script private-lock pattern was introduced (#11503 avoided trap).
Per-script subprocess integration testing deferred: each script's top-level imports
(services.mjs / Neo / lifecycle services) trigger heavy initialization that makes
subprocess tests slow and flaky; the underlying withHeavyMaintenanceLease primitive
is already covered by PR #11506's HeavyMaintenanceLeaseService.spec.mjs (8/8).

Out of scope (per #11503 lane split):
- Lane A (add backup to Orchestrator.mjs DEFAULT_HEAVY_MAINTENANCE_TASK_NAMES) — folded into PR #11506 by GPT
- Lane D (PrimaryRepoSyncService.runKbSync nested cascade) — GPT considering during Lane B API design
- Lane E (observability/stale-adoption) — separate ticket if needed
- syncGithubWorkflow Stage 1 / Stage 2 split — deferred per GPT's whole-run-guard call (AC7)

Resolves #11507

* fix(buildScripts): runSandman.mjs fail-closed on lease-acquisition error (PR #11509 cycle 1 per GPT review)

GPT's cycle-1 review (PRR_kwDODSospM8AAAABAJIdTg on PR #11509) caught a fail-closed
gap: when withHeavyMaintenanceLease itself failed (e.g., lease-write IO error), the
prior code set process.exitCode=1 in the catch but FELL THROUGH to
GraphService.decayGlobalTopology() which mutates SQLite graph edges and
_SYSTEM_STATE. That defeats the substrate protection PR #11509 ships — the one
path where the lease was NOT acquired could still mutate the graph outside the
mutex contract.

Fix: process.exit(1) directly in the lease-acquisition catch block, short-circuiting
before the decay call. The decay-on-completion path is now the only branch where
graph mutation runs, and it's guaranteed lease-protected.

Added regression spec asserting:
- The acquisition-failure catch block calls process.exit(1) (not just sets exitCode)
- Defensive: if any process.exitCode=1 assignment exists in the catch, it MUST be
  paired with a process.exit call (prevents future regression to the original buggy
  pattern)

14/14 Lane C content-verification spec tests pass in 553ms (13 prior + 1 new).

Refs #11507 (Lane C of #11503)

* fix(buildScripts): runSandman.mjs decay inside lease window (PR #11509 cycle 2 per GPT review)

GPT's cycle-2 review (PRR_kwDODSospM8AAAABAJKF8w) caught: `withHeavyMaintenanceLease`
releases the lease in its own `finally` BEFORE returning, so the post-wrapper
`GraphService.decayGlobalTopology()` call ran OUTSIDE the lease window — defeating
the substrate protection this PR is shipping (the one path where the lease was
ACQUIRED but the graph mutation still happened unprotected).

Fix:
- Add a `finally` block to the inner try/catch inside the async task passed to
  `withHeavyMaintenanceLease`. JS guarantees the inner finally completes before
  the awaited task settles, so the wrapper's release runs strictly after decay.
- Wrap decay in its own try/catch inside the finally to preserve prior
  graceful-fail semantics: a decay failure must NOT throw out of the lease-wrapped
  task (which would suppress the inner return value AND propagate to the outer
  catch reserved for lease-acquisition fail-closed semantics).
- Remove the now-redundant post-wrapper decay block.

Spec coverage:
- New test pins the release-timing invariant via three independent assertions:
  (1) decay offset < wrapper-options-trailer offset (decay is inside the wrapped task);
  (2) decay offset < post-wrapper held-handling offset (decay is not after the await);
  (3) decay is invoked from inside a `finally {` block (pins structural mechanism,
      not just offset ordering — guards against accidental refactor into outer catch
      which only fires on lease-acquisition failure).
- All 15 tests pass (13 baseline + cycle-1 fail-closed + cycle-2 release-timing).

(#11507)

---------

Co-authored-by: tobiu <tobiasuhlig78@gmail.com>"
- 2026-05-17T01:08:33Z @tobiu closed this issue
- 2026-05-17T01:19:10Z @neo-opus-ada cross-referenced by #11513
- 2026-05-17T01:26:39Z @neo-opus-ada cross-referenced by PR #11514
- 2026-05-17T01:34:36Z @neo-opus-ada cross-referenced by #11503
- 2026-05-17T01:52:51Z @neo-opus-ada cross-referenced by #11519
- 2026-05-17T02:18:53Z @neo-opus-ada cross-referenced by PR #11521

