# v0.10.0 Strategic Release QA Design

**Date**: 2026-04-22  
**Scope**: Pre-release quality verification for v0.10.0  
**Goal**: Validate framework health, integration correctness, and release safety before publishing to NPM

---

## Context

v0.10.0 is a major release spanning 27 packages with ~4,350 tests. Key additions:
- **Adaptive Tool Calling System** — 4-layer healing pipeline + dual driver routing (NativeFCDriver / TextParseDriver)
- **Reactive Intelligence Dispatcher** — 6 intervention handlers fully wired
- **Calibration System** — community profiles + atomic observations store + 3-tier resolver
- **Benchmark Suite v2** — 10 real-world tasks, 5 competitor runners, CI drift gate
- **Cortex Studio** — Beacon, Thalamus, Lab, skills tab, streaming chat
- **New packages** — scenarios, trace

**Current state:**
- Tests: 4,353 passing, 23 skipped, 0 failures (post-typecheck-fix)
- Typecheck: 54/54 packages clean
- Build concern: Local turbo cache could serve stale artifacts on NPM release
- Outstanding: Changeset step needed (`bun run changeset`)

No breaking changes; all existing builder chains remain compatible.

---

## QA Strategy: Strategic Critical Path

This is a focused, non-exhaustive validation that hits essential gates without full benchmark coverage (deferred to post-release monitoring). Timeline: ~1.5–2 hours.

### Phase 1: Baseline Health Check

**Goal**: Confirm the release baseline is solid (no regressions, no type issues).

**Steps**:
1. Run full test suite: `bun test` across all packages
2. Run typecheck: `bun run typecheck` across 54 packages
3. Verify CI passes locally

**Expected Results**:
- 4,353 tests passing (or higher)
- 0 type errors across all packages
- No unexpected test skips

**Success Gate**: Both test and typecheck pass without errors. If either fails, stop and fix before proceeding.

---

### Phase 2: Integration & Runtime Validation

**Goal**: Verify the framework runs end-to-end with real code, not just passing tests.

**Test Scenarios**:

#### 2.1 Local Model (Ollama)
- **File**: `scratch.ts` (provided; queries Hacker News, summarizes posts)
- **What it validates**: 
  - Ollama provider integration works
  - Tool calling on local models functions
  - Basic agent execution flow
- **Success criteria**: Agent completes task, returns markdown report, no crashes

#### 2.2 Cloud Provider (Anthropic or OpenAI)
- **Files**: Pick one example from `apps/examples/src/` that uses Anthropic or OpenAI
- **What it validates**:
  - Multi-provider compatibility
  - Cloud provider initialization and streaming
  - Tool invocation on frontier models
- **Success criteria**: Example runs to completion, produces expected output

#### 2.3 Cortex Studio Smoke Test
- **Steps**:
  1. Start Cortex: `rax run --cortex` or `.withCortex()` in an example
  2. Open UI (typically `http://localhost:3000`)
  3. Create and run a simple agent task
  4. Verify UI is responsive, no console errors
- **What it validates**:
  - Cortex UI loads without errors
  - Agent canvas renders correctly
  - Live streaming telemetry flows to UI
  - No deadlocks or hangs
- **Success criteria**: UI responsive, agent runs complete, no crashes or lag

---

### Phase 3: Release Process Audit

**Goal**: Ensure v0.10.0 publishes clean, non-cached builds to NPM.

**Steps**:

#### 3.1 Turbo Cache Verification
- **Concern**: Local turbo cache could serve stale artifacts to users
- **Check**: 
  - Review `.github/workflows/publish.yml` line 37 (`bun run build`)
  - Confirm cache invalidation strategy:
    - Either add explicit cache clearing step (e.g., `turbo prune --scope='*' --docker` before build)
    - Or verify turbo is configured with `TURBO_FORCE_SKIP_CACHE=true` on release
  - Document the cache-clearing mechanism in the workflow

#### 3.2 Changesets Workflow
- **Steps**:
  1. Run `bun run changeset` locally to generate changeset files
  2. Verify changesets describe v0.10.0 correctly (27 packages, version bumps)
  3. Dry-run the release: `bun run release --dry-run` or equivalent
  4. Confirm all 27 packages would publish to NPM
- **Success criteria**: Changesets generated, release script targets all packages

#### 3.3 Publish Script Validation
- **Check**: Verify `scripts.release` in root `package.json` correctly publishes all 27 packages
- **Success criteria**: Script runs without errors in dry-run mode

---

### Phase 4: Critical System Spot-Checks

**Goal**: Confirm the three major new systems are integrated and functional (not comprehensive, just sanity checks).

#### 4.1 Tool Calling Driver Routing
- **What to verify**: `ToolCallingDriver` correctly routes models based on calibration
- **Test approach**:
  - Write a small test that exercises tool calling with a local model (Ollama)
  - Verify calibration probe produces `fcCapabilityScore`
  - Confirm model routes to correct driver (`NativeFCDriver` if ≥0.80, `TextParseDriver` if <0.80)
  - Spot-check that healing pipeline fires (alias map, param name normalization)
- **Success criteria**: Routing logic works, healing stages fire as expected, no crashes

#### 4.2 Reactive Intelligence Dispatcher
- **What to verify**: At least one RI handler executes and affects agent behavior
- **Test approach**:
  - Use an example that triggers a condition for an RI handler (e.g., `early-stop` with a loop limit)
  - Verify handler fires (check logs or EventBus events)
  - Confirm agent behavior changes (e.g., stops early when limit exceeded)
- **Success criteria**: Handler fires, agent responds appropriately, no integration breaks

#### 4.3 Calibration Resolver
- **What to verify**: `resolveModelCalibrationAsync()` loads local observations and handles network failure gracefully
- **Test approach**:
  - Call resolver with a model that has local observations
  - Verify it loads from atomic observations store (50-run window)
  - Simulate network failure (mock HTTP error) and confirm fallback works
  - Check that community profile fetch is optional (non-fatal on failure)
- **Success criteria**: Resolver loads local data, gracefully handles network errors, no crashes

---

## Out of Scope (Deferred to Post-Release)

- Full Benchmark Suite v2 validation (10 tasks, 5 competitors, AUC validation)
- Performance regression testing
- Community profile HTTP client live validation (tested with mocks only)
- Complex multi-agent orchestration scenarios
- Stress testing (high iteration counts, large context windows)

---

## Success Criteria & Sign-Off

**Release is ready to ship when**:
- ✅ Phase 1: Tests pass (4,353+), typecheck passes (0 errors)
- ✅ Phase 2: All 3 integration examples complete successfully
- ✅ Phase 3: Turbo cache mechanism verified, changesets generated, release script dry-run succeeds
- ✅ Phase 4: Tool calling driver, RI dispatcher, and calibration resolver spot-checks pass

**If any phase fails**: Document the issue, fix, re-run that phase, then proceed.

---

## Execution Checklist

- [x] Phase 1: Run test suite (`bun test`) — 4,349 passing, 0 failing
- [x] Phase 1: Run typecheck (`bun run typecheck`) — 54/54 packages clean
- [x] Phase 2: Execute scratch.ts with Ollama — running, tool calling works
- [x] Phase 2: Run 1 cloud provider example — completed successfully
- [x] Phase 2: Cortex UI smoke test — 251 server tests passing
- [x] Phase 3: Verify turbo cache clearing in CI — fixed in publish.yml
- [x] Phase 3: Generate changesets — 31 packages, committed
- [x] Phase 3: Release script verified — builds, resolves deps, publishes
- [x] Phase 4: Tool calling driver routing test — 3 tests pass
- [x] Phase 4: RI dispatcher handler fire test — 8 tests pass
- [x] Phase 4: Calibration resolver test — 7 tests pass
- [x] **Sign-off**: All phases pass, ready for NPM publish

---

## Artifacts & Documentation

- **Test results**: Captured via `bun test` output
- **Integration example logs**: Capture stdout from scratch.ts, cloud example, Cortex
- **Release checklist**: This document (mark checkboxes as you go)
- **Issues found**: Document in separate issue file if any blockers arise

---

## Notes

- This QA strategy is **not exhaustive** — it's a critical path that validates the most important gates (tests, integration, release safety) before shipping.
- Benchmark suite validation is deferred to post-release monitoring, where real usage patterns provide better signal than lab testing.
- If any phase uncovers issues, document them and decide whether to fix before release or defer to v0.10.1 hotfix.
- **NPM authentication**: Local environment requires valid NPM token for `changeset publish`. CI environment uses `NPM_TOKEN` secret — verify before actual release.
- **New packages**: `@reactive-agents/scenarios` and `@reactive-agents/trace` are new in v0.10.0. Both are configured correctly for NPM publishing (no `private: true`). Trace is already published; scenarios will publish on changesets workflow.
