# Test Generator Agent Memory

## OrchestKit Test Architecture
- Test runner: `tests/run-all-tests.sh` runs all suites sequentially
- TypeScript hooks tested via vitest in `src/hooks/`
- Shell-based tests in `tests/` organized by category: unit, security, integration, e2e, etc.
- CI uses `scripts/ci/run-tests.sh <dir>` for category-based test discovery
- Skill validation: `bash tests/skills/run-skill-tests.sh` (slow; use `npm run test:skills`)

## Known Test Issues (2026-02-07)
- `tests/skills/structure/test-skill-md.sh:511` — `EXPECTED_INTERNAL` count drifts when skills added/removed. Currently 175 (HeyGen removed). Check this whenever skills change.
- `plugins/.composite-agent-index.md` — Must be generated by build. If missing, 37 tests fail (all agents missing from index).
- `tests/unit/test-hooks-unit.sh` — All 6 hook unit tests SKIP because they can't find compiled bundles. Tests look for hooks in wrong location.
- `session-env-setup.test.ts:289` — Pre-existing: asserts `agent_type === 'env-agent'` but gets empty string.
- `memory-writer.test.ts` — Pre-existing: fails on Windows due to path separators.
- Local vitest skip: `@rollup/rollup-darwin-x64` missing → fix with `cd src/hooks && rm -rf node_modules && npm i`

## CI Workflow Patterns
- Pinned workflows: ci.yml, build.yml, plugin-validation.yml, release-please.yml, release-video.yml, mem0-visualization.yml, version-check.yml, reusable-test-runner.yml
- Unpinned workflows (need fixing): eval-index-effectiveness.yml, ci-report.yml — use `@v4`/`@v7` tags instead of SHA pins
- Composite action: `.github/actions/setup/action.yml` handles cross-platform deps + node setup + plugin build
- Artifact sharing: build step uploads `plugins/`, downstream jobs download via `actions/download-artifact`

## Count Drift Pattern
- Correct counts (as of 2026-03-05): 71 skills, 30 agents, 81 hooks (30 global + 44 agent-scoped + 7 skill-scoped)
- `split-bundles.test.ts` asserts `totalHooks === 167` (bundle export count, not hook entry count) — update this comment chain when hook counts change
- Files to update when counts change: `test-skill-md.sh` EXPECTED_INTERNAL, `validate-counts.sh` (auto from filesystem), plugin.json description

## plugins/ vs src/ Tests
- The 4 "deleted" test files in the #960 question (intent-classifier, orchestration-integration, calibration-tracker, calibration-persist) lived ONLY in `plugins/` (generated dir), never in `src/hooks/src/__tests__/`
- Matching source code (intent-classifier.ts, calibration-tracker.ts, calibration-persist.ts) was also deleted as part of v7.0.0 / #897 hook slimming
- `orchestration-state.ts` and `pipeline-detector.ts` STILL EXIST in src/ and have test coverage in `src/hooks/src/__tests__/orchestration-state.test.ts` and `src/hooks/src/__tests__/prompt/pipeline-detector.test.ts`

## Vitest Mock Patterns (learned 2026-03-08)
- `additionalContext` lives at `result.hookSpecificOutput?.additionalContext` (CC 2.1.9 format), NOT `result.additionalContext`
- `vi.mock()` for `node:fs` must spread actual: `const actual = await importOriginal(); return { ...actual, existsSync: mockFn }`
- When mocking `node:fs`, `basename` from `node:path` runs normally in source — but in certain test environments `basename('/project/.claude/rules/a.md')` may produce `.claude/rules/a.md` instead of `a.md`. Use `keys.some(k => k.includes('a.md'))` for robust basename assertions.
- Top-level `await` in a sync `test()` body is rejected by esbuild — mark the `test()` callback `async` when using dynamic `await import()`

## Branch refactor/split-testing-patterns-669 Findings (2026-03-10)
- 189 vitest test files, 5952 tests — all passing
- security: 11/11, manifests: all passed, skills-structure: 89 skills, 12/12 checks passed
- testing-patterns split: 5 new evals exist (testing-unit/e2e/integration/llm/perf); old testing-patterns.eval.json cleanly removed
- testing-patterns skill kept as redirect stub (`disable-model-invocation: true`) — correct pattern
- Rule-to-eval ID mismatches (naming only, content covered): a11y-testing→a11y-jest-axe+a11y-ci-gates; verification-techniques→verification-property; llm-evaluation→3 evals; pytest-execution→pytest-markers+pytest-xdist
- All 33 testing sub-skill evals tagged `happy-path` only — zero `error-path` / `edge-case` tags
- Verify test-cases.json: 9 cases; server-fails-to-start scenario missing (dev server crash during visual capture)
- instructions-loaded hooks: 8 source files, 8 test files — full parity (types.ts excluded)
- `@vitest/coverage-v8` not installed — cannot compute line coverage %

## Prompt Dispatcher (#960 changes)
- `skill-nudge-prompt` was removed from the prompt unified-dispatcher in #960 — replaced by CC native skill matching
- Current prompt dispatcher HOOKS array (4 entries): handoff-injector, agentation-context, context-exhaustion-warner, antipattern-warning
- No dedicated unified-dispatcher.test.ts for prompt — coverage comes from `e2e/hook-execution-e2e.test.ts` (execution tests) and individual hook test files
- The e2e test asserts `registeredHookNames` is non-empty and contains `antipattern-warning` — it does NOT assert exact count 4

## Sprint 2 Wave A Eval Framework (2026-03-13)
- 17 skill eval YAMLs: 5 existing (api-design, assess, commit, explore, implement) + 12 new
- Quality runner: `run-quality-eval.sh` — A/B skill vs baseline, LLM-graded assertions
- Trigger runner: `run-trigger-eval.sh` — precision/recall with N reps
- Dry-run mode validates YAML structure only (no Claude CLI needed) — used in CI
- Results: `tests/evals/results/skills/*.json` (gitignored). Exit: 0=pass, 1=regression
- Dry-run does NOT produce JSON output — gap for CI parseable results

## B-Enhancements Applied to All 17 Evals (2026-03-13)
- B1: Added confusion pair annotations to 5 original evals (assess, commit, explore, implement, api-design)
- B2: Fixed non-reciprocal confusion pairs: assess<->verify, assess<->review-pr, brainstorm<->visualize-plan, implement<->fix-issue
- B3: Added 2nd quality eval to doctor.eval.yaml (hooks-only scoped check)
- B4: Added adversarial triggers to ALL 17 files (terse 1-2 word, typo-laden, multi-intent negatives)
- B5: Added 1-2 negative assertions to EVERY quality eval across all 17 files
- All 17 files pass yq/yaml parsing and dry-run validation (17/17 passed)
