Same code, 48x apart
One assertion times five node spawns with a stopwatch and fails
above 200ms. On the same commit, the same day: 42ms on a CI runner,
2003ms on a workstation running eight concurrent agent sessions. The code under test
never changed. What changed was who else wanted the CPU.
Move the load, watch the verdict flip
before this change
after this change
What the fix actually does
if avg < 200ms -> PASS (everywhere) else if CI is unset -> REPORT-ONLY (print the number, exit 0) else -> FAIL (CI enforces the budget)
The budget is unchanged and still gates. It gates where the number is stable, and reports where it is not. CI has 4.7x headroom at 42ms, so a real regression still trips it; a busy laptop no longer manufactures a red suite that means nothing.
The claim this PR refutes
The issue reported that the suite "is not wired into CI" and "gates nothing",
from a grep for test:agents across .github/workflows/ that returned
nothing. The grep is accurate and the conclusion is wrong: CI does not invoke the npm script,
it invokes the same runner directly, bash scripts/ci/run-tests.sh tests/agents
at ci.yml:622, inside the required Agents & Skills check.
The control that catches this: the identical grep for
test:skills and run-security-tests also returns zero, and both of
those are definitely gated. When a probe returns zero for a case you know is non-zero, the
probe is measuring the wrong channel. The proof is in the log of this PR's own predecessor:
[RUN] test-agent-skill-validation.sh then
Test 6: ... PASS (42ms avg).