---
id: 10113
title: Reduce DevIndex Spider CI loop from 3x to 1x per pipeline invocation
state: CLOSED
labels:
  - ai
  - performance
assignees:
  - tobiu
createdAt: '2026-04-19T22:59:36Z'
updatedAt: '2026-04-19T23:18:12Z'
githubUrl: 'https://github.com/neomjs/neo/issues/10113'
author: tobiu
commentsCount: 0
parentIssue: null
subIssues: []
subIssuesCompleted: 0
subIssuesTotal: 0
blockedBy: []
blocking: []
closedAt: '2026-04-19T23:18:12Z'
---
# Reduce DevIndex Spider CI loop from 3x to 1x per pipeline invocation

## The Problem

`.github/workflows/data-sync-pipeline.yml` currently runs the DevIndex Spider three times per CI invocation:

```yaml
for i in 1 2 3; do
    echo "Starting Spider Run $i/3..."
    npm run devindex:spider -- --strategy random
done
```

This was valuable during the bootstrap phase when discovery throughput was the bottleneck. At current saturation, it wastes the GitHub GraphQL rate-limit budget and contributes to downstream failures in the same pipeline (see companion ticket for `LabelService` diagnostic restoration).

## The Architectural Reality

The DevIndex is now at the saturation state the architecture was designed around:

- `users.jsonl` is at the `maxUsers` cap (50,000)
- `threshold.json` has raised the contribution bar accordingly (`DevIndex.services.Updater`'s Meritocracy filter aggressively prunes below-threshold candidates per `learn/guides/devindex/data-factory/Updater.md`)
- `tracker.json` sits near cap (~49,649), kept in sync with `users.jsonl` by `DevIndex.services.Cleanup` per `learn/guides/devindex/data-factory/DataHygiene.md` §1

The Spider has a **Backpressure Valve** (`Spider.mjs:97`, `config.spider.maxPendingUsers: 2000`) that halts a run when the Pending backlog exceeds 2000. But the valve triggers *after* a run's state loading — each spider invocation still consumes some API budget even when backpressure aborts it, and the 2nd and 3rd loop iterations may fire a full discovery run if the Updater has drained Pending below 2000 between iterations.

## Why 3x No Longer Earns Its Cost

- **Low Pending backlog (< 2000):** all 3 runs proceed fully → 3× the rate-limit burn on core + search buckets for marginal additional discovery
- **High Pending backlog (≥ 2000):** runs 2 and 3 abort via backpressure, but still cost state-loading cycles
- In **both** regimes, the Updater's throughput (~800/hour per `learn/guides/devindex/data-factory/DataHygiene.md`) is the actual bottleneck. Discovery oversupply yields new Pending entries that wait in queue, most of which then get threshold-rejected when Updater reaches them. Net new qualified users from runs 2-3 is low relative to their API cost.

## Downstream Damage

The companion ticket (LabelService error propagation) is triggered by `data-sync-pipeline` failures where `labels.mjs` gets an opaque "Invalid response from GH_LabelService" error. Hypothesis: the spider loop's 3× GraphQL burn pushes subsequent GraphQL calls in the same workflow (labels, release, tickets index generation) into rate-limit territory. This ticket addresses the root cause of that pressure.

## Fix

Change the workflow step from the 3-iteration bash loop to a single invocation:

```yaml
- name: Run DevIndex Spider
  env:
    GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
  run: npm run devindex:spider -- --strategy random
```

## Avoided Traps

- **Touching `config.spider.maxPendingUsers` or Backpressure Valve logic.** The architectural primitives at `Spider.mjs:97` and `DevIndex.services.Cleanup` handle saturation correctly; they don't need new mechanisms. This ticket is purely a CI frequency tuning.
- **Reducing spider schedule frequency.** The CI workflow cadence (i.e. how often the pipeline itself runs) is orthogonal; tuning it is a separate concern if needed after observing post-merge behavior.
- **Modifying `Updater` throughput or Meritocracy threshold.** Increasing Updater drain rate would be another way to let 3x-discovery land useful candidates, but that's a significantly larger change with its own rate-limit exposure. Out of scope.

## Acceptance Criteria

- [ ] `.github/workflows/data-sync-pipeline.yml` runs `npm run devindex:spider -- --strategy random` once per pipeline invocation
- [ ] No other changes to spider invocation, strategy, env, or step ordering

## Post-Merge Validation

- Observe next `data-sync-pipeline` run: total workflow duration, GraphQL rate-limit remaining at end-of-run, whether the `labels.mjs` step completes cleanly
- If LabelService companion ticket has landed: its new clear-error logging will confirm whether rate-limit 429 was the actual prior failure mode

## Related

- Companion: `LabelService` error propagation ticket (diagnostic restoration; the two drain independent channels — this removes pressure, that restores visibility)
- Future candidate if invariant check flags a real issue: `tracker.json` (~49,649) vs `users.jsonl` (~50,000) count gap — 351 users exist without tracker entries, which would prevent re-update. Deliberately NOT scoped here; needs confirmation before filing

## Origin Session ID

`07f601dc-353a-44d2-a373-18da2a0d305a`


## Timeline

- 2026-04-19T22:59:37Z @tobiu added the `ai` label
- 2026-04-19T22:59:37Z @tobiu added the `performance` label
- 2026-04-19T22:59:43Z @tobiu assigned to @tobiu
- 2026-04-19T23:02:04Z @tobiu cross-referenced by PR #10114
- 2026-04-19T23:15:01Z @tobiu referenced in commit `ee66671` - "chore(data-sync-pipeline): reduce DevIndex Spider loop from 3x to 1x (#10113)"
- 2026-04-19T23:15:38Z @tobiu cross-referenced by PR #10115
- 2026-04-19T23:18:12Z @tobiu referenced in commit `9415aa6` - "chore(data-sync-pipeline): reduce DevIndex Spider loop from 3x to 1x (#10113) (#10115)"
- 2026-04-19T23:18:12Z @tobiu closed this issue
- 2026-04-20T00:06:05Z @tobiu cross-referenced by #10117

