CI coverage papercuts
Two gaps found while draining an 8-PR dependabot pile. Both are "the gate does not cover what you think" — the family that produced four separate incidents this week.
① 284 tests that ran nowhere
orchestkit-demos/
├── 8 vitest files · 284 assertions · environment: "jsdom"
└── demos-validate.yml ran: npx remotion compositions ← bundle only
never `npm test`
Caught reviewing #3259: jsdom 29 → 30, a MAJOR version bump. The PR carried 43 green checks and not one executed a line of jsdom. Verified by hand instead — 284 tests passed, so the bump shipped safely. The next one gets caught by CI.
+ - name: Run demo unit tests
+ working-directory: orchestkit-demos
+ run: npm test # vitest run -c vitest.demos.config.ts
Gotcha encoded in the comment: the bare vitest resolves the
repo-ROOT config and dies on its src/hooks project reference. That detour cost a
wrong diagnosis during the review — it looked like the jsdom bump had broken the suite.
② chmod churn on files that never needed it
run-tests.sh discovered a test ──▶ chmod +x ──▶ ran it
.sh → invoked as `bash file` → exec bit irrelevant, but harmless
.mjs → invoked as `node file` → exec bit NEVER consulted
…yet still flipped 644 → 755
Result: a dirty working tree after every local suite run.
test-docs-site-drift.mjs was discarded three times in two days. #3235 made the
LOCAL runner glob these directories too, so the churn graduated from CI-only to
every developer machine.
The A/B that actually proves ②
| engine | probe test-probe.mjs | probe test-probe.sh |
|---|---|---|
| before | 644 | 644 |
| OLD (origin/main) | 644 → 755 ✗ | 644 → 755 |
| NEW (this branch) | 644 → 644 ✓ | 644 → 755 ✓ |
Why the A/B and not a simple run: the first verification ran the
engine over tests/manifests, whose .mjs files are already tracked 755 —
so it "passed" while demonstrating nothing. Same shape as the other self-caught measurement
errors this week: check the thing at the layer where it can actually differ.
Verification
| check | result |
|---|---|
| actionlint on demos-validate.yml | exit 0 |
| demos suite at main's deps | 8 files / 284 tests pass |
| chmod A/B against origin/main engine | fix demonstrated |