# Skill Studio — Architecture & Testing Guide

**Last updated**: 2026-04-23
**Target audience**: developers running or extending the studio
**Live package**: [`vskill` on npm](https://www.npmjs.com/package/vskill) (current: `v0.5.84`)

---

## 1. Quick start

```bash
npx vskill@latest studio
```

The `@latest` tag is important — plain `npx vskill` caches aggressively and may launch a stale version. The `@latest` pin forces npx to check the registry first.

Default: opens the studio at `http://localhost:3162`.

Also valid:
```bash
# If you've installed globally
npm i -g vskill
vskill studio

# Dev mode from source (hot reload)
cd repositories/anton-abyzov/vskill
npm run dev:eval-ui   # UI at localhost:3078
# In a second terminal:
node dist/index.js eval serve --root .   # API at localhost:3077
```

---

## 2. Architecture diagrams

Three views of the same system. All diagrams are [Mermaid](https://mermaid.js.org/) — they render natively on GitHub, in Docusaurus, and in most modern Markdown viewers. Source-controlled, diff-friendly, and exportable to SVG via `mmdc -i diagram.mmd -o diagram.svg` if a binary is needed.

### 2.1 Where skills live on disk

A skill exists in **one of two states**: *source* (an authored repo on your machine or GitHub) or *installed* (a copy materialized into a per-agent directory by `vskill install`). The studio's sidebar reflects both states; `classifyOrigin()` in `src/eval/skill-scanner.ts` is the SSoT.

```mermaid
flowchart LR
    subgraph Source["Source — authored once"]
        direction TB
        A1["Local repo<br/>vskill/skills/&lt;skill&gt;/SKILL.md"]
        A2["Local repo<br/>vskill/plugins/&lt;plugin&gt;/skills/&lt;skill&gt;/"]
        A3["GitHub<br/>github.com/&lt;owner&gt;/&lt;repo&gt;"]
        A1 -.git push.-> A3
        A2 -.git push.-> A3
    end

    subgraph Installed["Installed — materialized per-agent by vskill install"]
        direction TB
        B1["~/.claude/skills/&lt;skill&gt;/"]
        B2["~/.cursor/skills/&lt;skill&gt;/"]
        B3["~/.codex/skills/&lt;skill&gt;/"]
        B4["~/.gemini/skills/&lt;skill&gt;/"]
        B5["~/.windsurf/skills/&lt;skill&gt;/"]
        B6["+30 more agents<br/>see src/agents/agents-registry.ts"]
        B7["Project-scoped<br/>&lt;project&gt;/.&lt;agent&gt;/skills/&lt;skill&gt;/"]
    end

    Source ==vskill install==> Installed
    Installed -.scanned by.-> S["eval-server<br/>port 3077"]
    Source -.scanned by.-> S
    S --> UI["Studio sidebar<br/>port 3162"]
```

**Read this diagram when**: you can't find a skill you just edited, you're confused why the same skill appears twice in the sidebar (source + installed), or you're adding support for a new agent.

### 2.2 Publish / submit flow

"Submit" = make a local edit visible to *other* users. Two paths in parallel — git is the durable record, the internal broadcast is a latency optimization.

```mermaid
sequenceDiagram
    autonumber
    participant U as Author<br/>(in Studio)
    participant S as eval-server<br/>(localhost:3077)
    participant FS as Local FS
    participant GH as GitHub
    participant VS as verified-skill.com<br/>(production worker)
    participant Other as Other user

    U->>S: Apply improvement (UI button)
    S->>FS: Write SKILL.md + tests, bump version
    par Durable path (always)
        S->>GH: POST /api/git/publish<br/>(git commit + push)
        Note over GH,VS: Scanner polls GitHub<br/>every ~10 min
        GH-->>VS: New version detected
        VS->>VS: Catalog version<br/>in registry
    and Optimization (when INTERNAL_BROADCAST_KEY set)
        S->>VS: POST /api/v1/internal/skills/publish<br/>(submitSkillUpdateEvent)
        VS-->>VS: UpdateHub fans out<br/>via SSE immediately
    end
    VS-->>Other: SSE notification<br/>"new version available"
    Other->>VS: vskill install &lt;skill&gt;
    VS-->>Other: Tarball + metadata
    Other->>Other: Materialize into<br/>~/.&lt;agent&gt;/skills/
```

**Key invariants**:
- The browser **never** calls `verified-skill.com` directly — all upstream traffic goes through the eval-server's `platform-proxy.ts`. This keeps the studio CORS-free and lets `VSKILL_PLATFORM_URL` redirect to a local platform for dev.
- If `INTERNAL_BROADCAST_KEY` is missing (the default in shipped builds), the broadcast path no-ops and the user is told to `git push` so the scanner picks the change up.
- `git push` alone is sufficient — the broadcast is an "instant notify" optimization, not a requirement for publication.

### 2.3 Runtime topology

`npx vskill studio` boots one Node process serving two roles: the static SPA (port 3162) and the REST/SSE API (port 3077, internally proxied at `/api`).

```mermaid
flowchart TB
    Browser["Browser<br/>localhost:3162<br/>Vite-built React 19 SPA"]

    subgraph Node["vskill studio (single Node process)"]
        direction TB
        Static["Static dist/eval-ui/<br/>served at :3162"]
        Server["eval-server<br/>:3077 — REST + SSE"]
        Static -.proxy /api.-> Server
        Scanner["skill-scanner<br/>classifyOrigin()"]
        Server --> Scanner
    end

    Browser -- HTTP / EventSource --> Static

    subgraph FS["Local filesystem"]
        SrcFS["Source skills<br/>vskill/skills, vskill/plugins/"]
        InstFS["Installed skills<br/>~/.&lt;agent&gt;/skills/"]
    end

    Scanner --> SrcFS
    Scanner --> InstFS

    subgraph LLMs["LLM providers (src/eval/llm.ts)"]
        Claude["claude CLI"]
        Anthropic["Anthropic API"]
        OpenRouter["OpenRouter"]
        Ollama["Ollama / LM Studio<br/>(local)"]
    end

    Server -- shells out --> Claude
    Server -- HTTPS --> Anthropic
    Server -- HTTPS --> OpenRouter
    Server -- HTTP --> Ollama

    Server -- "/api/v1/skills/*<br/>(platform-proxy.ts)" --> Platform["verified-skill.com<br/>(or VSKILL_PLATFORM_URL)"]
```

**Why one process serves two ports**: in production the studio is deployed behind the platform on the same origin, so `/api/v1/skills/*` resolves to the platform without a proxy. In local dev they're separate processes, and `platform-proxy.ts` closes the gap so the same SPA code works in both environments.

---

## 2.4 Source layout

Key paths inside the `vskill` repo:
- `src/eval/` — evaluation engine: comparator (A/B skill-on/off), benchmark runner, LLM provider adapters, skill scanner (`classifyOrigin()` is the Own-vs-Installed SSoT).
- `src/eval-server/` — HTTP server. Routes in `api-routes.ts`, skill creation in `skill-create-routes.ts`, skill improvement in `improve-routes.ts`, platform proxy in `platform-proxy.ts`, git publish in `git-routes.ts`, SSE helpers in `sse-helpers.ts`.
- `src/eval-ui/` — Vite SPA. Shell in `App.tsx`, sidebar at `src/components/Sidebar.tsx`, detail panel at `src/components/RightPanel.tsx`, update management UI at `src/pages/UpdatesPanel.tsx`.
- `src/agents/agents-registry.ts` — list of supported agents and their `globalSkillsDir` values (the `~/.<agent>/skills/` paths in diagram 2.1).
- `src/bin.ts` — CLI entry (`vskill` binary).
- `scripts/` — CI scripts: `check-no-shimmer.ts`, `check-serif-scope.ts`, `check-bundle-size.ts`, `check-strings-voice.ts`.

---

## 3. REST endpoints (port 3077)

All routes prefixed with `/api`. The UI talks to them via `src/eval-ui/src/api.ts`.

### Skills — metadata & listing
| Method | Path | Purpose |
|---|---|---|
| `GET` | `/api/skills` | List all skills with `SkillInfo` (origin, frontmatter, filesystem stats, deps, source agent). Primary data source for the sidebar + detail panel. |
| `GET` | `/api/config` | Project root, active model, health status. Used by `ConfigContext`. |

### Skills — version & update management
| Method | Path | Purpose |
|---|---|---|
| `GET` | `/api/skills/updates` | Returns `SkillUpdateInfo[]` — wraps `vskill outdated --json`. Each entry has `installed`, `latest`, `updateAvailable`, `pinned`, `changelog` fields. **Drives the notification badge proposed in 0681.** |
| `GET` | `/api/skills/:plugin/:skill/versions` | Full version history for one skill. |
| `POST` | `/api/skills/:plugin/:skill/update` | **SSE stream** — updates a single installed skill. Emits `progress`, `done`, `error` events. |
| `POST` | `/api/skills/batch-update` | SSE stream — updates multiple skills in parallel. |
| `GET` | `/api/skills/:plugin/:skill/version-diff` | Diff between two versions. |

### Evaluation & benchmarking
| Method | Path | Purpose |
|---|---|---|
| `POST` | `/api/skills/:plugin/:skill/generate-evals` | AI-generate test cases. Body: `{ provider, model, prompt }`. Synchronous. |
| `POST` | `/api/skills/:plugin/:skill/baseline` | Run prompts without the skill. |
| `POST` | `/api/skills/:plugin/:skill/compare` | **The A/B primitive.** Runs the same prompts with AND without the skill, returns `winner: "skill"\|"baseline"\|"tie"` plus per-case pass rates and rubric scores. Called by `ComparisonPage.tsx`. |
| `POST` | `/api/skills/:plugin/:skill/benchmark/case/:evalId` | Run one test case. |
| `GET` | `/api/skills/:plugin/:skill/history` | Benchmark history. |
| `GET` | `/api/skills/:plugin/:skill/history-compare` | Compare two historical runs. |

### Skill improvement
| Method | Path | Purpose |
|---|---|---|
| `POST` | `/api/skills/:plugin/:skill/improve` | **SSE** — AI-suggested improvements to SKILL.md body + test cases. |
| `POST` | `/api/skills/:plugin/:skill/apply-improvement` | Commits an improvement to disk. |

### Skill creation
| Method | Path | Purpose |
|---|---|---|
| `POST` | `/api/skills` | Create a new skill (body: `{ name, description, provider, model }`). Planned: expose source-model picker in UI (increment 0678). |

---

## 4. Server-Sent Events (SSE)

vskill uses **SSE, not WebSockets**. The client consumes events via the browser-native `EventSource` API.

### Endpoints that stream

| Endpoint | Event types emitted |
|---|---|
| `GET /api/events` | `benchmark:complete`, `history:written`, `leaderboard:updated` — global data-event bus. |
| `POST /api/skills/:plugin/:skill/compare` (when `?sse=1`) | `progress`, `case:start`, `case:done`, `comparison:done`, `error` |
| `POST /api/skills/:plugin/:skill/benchmark/...` | Similar per-case progress stream |
| `POST /api/skills/:plugin/:skill/update` | `progress` (status: `updating` / `scanning` / `installing`), `done`, `error` |
| `POST /api/skills/:plugin/:skill/improve` | `progress`, `patch`, `done`, `error` |
| `POST /api/skills/batch-update` | Per-skill progress frames |

### Testing an SSE endpoint from the terminal

```bash
# 1. Boot the studio (or just the backend)
node dist/index.js eval serve --root .

# 2. In another shell, subscribe to global events
curl -N http://localhost:3077/api/events

# 3. Trigger a skill update — watch events flow in the first shell
curl -N -X POST \
  http://localhost:3077/api/skills/personal/tax-filing/update \
  -H "Content-Type: application/json" \
  -d '{}'

# 4. Simulate comparison streaming (returns plain JSON by default; add ?sse=1 for streaming)
curl -N -X POST 'http://localhost:3077/api/skills/personal/tax-filing/compare?sse=1' \
  -H "Content-Type: application/json" \
  -d '{"provider":"claude-cli","model":"sonnet","prompts":["hello"]}'
```

`-N` disables curl's output buffering so SSE frames appear as they arrive.

### Frame shape
```
event: progress
data: {"status":"updating","skill":"tax-filing","pct":12}

event: done
data: {"installed":"1.2.3","newVersion":"1.3.0","filesChanged":14}

event: error
data: {"message":"network timeout","retriable":true}
```

### Heartbeats
Long streams include heartbeat comments every ~30s (`: heartbeat\n\n`) to keep proxies from dropping the connection. Heartbeats are ignored by `EventSource`.

---

## 5. LLM provider adapters

Configured in `src/eval/llm.ts`. Current providers:

| Provider | Adapter | Env var | Notes |
|---|---|---|---|
| `claude-cli` | shells out to `claude` binary | — (uses Claude Code session) | Default — zero config inside Claude Code. **April 2026: Third-party OAuth token scraping banned; official CLI delegation is still allowed.** |
| `anthropic` | Anthropic SDK | `ANTHROPIC_API_KEY` | Direct API access. Requires key from [platform.claude.com/settings/keys](https://platform.claude.com/settings/keys). |
| `openrouter` | OpenAI-compat SDK | `OPENROUTER_API_KEY` | 300+ models. |
| `ollama` | REST to `localhost:11434` | `OLLAMA_HOST` (primary) / `OLLAMA_BASE_URL` (deprecated) | Detection probes `GET /api/tags` with 30s cache. Uses `/api/chat` for system+user messages. Helper: `src/eval/env.ts::resolveOllamaBaseUrl()`. |
| `codex-cli` | shells out to OpenAI Codex CLI | — | |
| `gemini-cli` | shells out to Google Gemini CLI | — | |
| `lmstudio` *(planned 0677)* | OpenAI-compat via SDK | `LM_STUDIO_BASE_URL` (default `http://localhost:1234/v1`) | Detection probes `GET /v1/models`, 500ms timeout. |

### Verified model IDs (April 2026)

| Model ID | Context | $/MTok in-out |
|---|---|---|
| `claude-opus-4-7` | 1M | 5 / 25 |
| `claude-opus-4-6` | 1M | 5 / 25 |
| `claude-sonnet-4-6` | 1M | 3 / 15 |
| `claude-haiku-4-5-20251001` | ~200K | 1 / 5 |

*Sonnet 4.7 and Opus 4.7 — Opus 4.7 confirmed April 16, 2026; Sonnet 4.7 not currently available.*

### 5.1 Environment variables

All env vars Studio reads. Every `Read at` file:line is grep-verified against source. Precedence rules are authoritative — conflicting assignments emit one deprecation line via `warnOnce()` (see `src/eval/env.ts`).

| Variable | Default | Required | Read at | Purpose |
|---|---|---|---|---|
| `ANTHROPIC_API_KEY` | — | Only for `anthropic` provider | `src/eval/llm.ts::createAnthropicClient`, `src/eval-server/api-routes.ts::detectAvailableProviders` | Direct Anthropic API calls. |
| `OPENROUTER_API_KEY` | — | Only for `openrouter` provider | `src/eval-server/api-routes.ts` (`/api/openrouter/models` handler) | OpenRouter catalog + completions. Stored key (via Settings modal) takes precedence when set. |
| `GOOGLE_API_KEY` | — | Optional for `gemini-cli` | `src/eval/llm.ts::createGeminiCliClient` | Gemini CLI auth passthrough. |
| `OLLAMA_HOST` | `http://localhost:11434` | No | `src/eval/env.ts::resolveOllamaBaseUrl` (single call site) | Ollama server URL (primary). Bare `host:port` auto-prefixes `http://`. |
| `OLLAMA_BASE_URL` | `http://localhost:11434` | No (deprecated — use `OLLAMA_HOST`) | `src/eval/env.ts::resolveOllamaBaseUrl` | Legacy fallback; one-shot warn-once log when both `OLLAMA_HOST` and `OLLAMA_BASE_URL` disagree. |
| `LM_STUDIO_BASE_URL` | `http://localhost:1234/v1` | No | `src/eval-server/api-routes.ts::probeLmStudio`, `src/eval/llm.ts::createLmStudioClient` | LM Studio OpenAI-compat endpoint. |
| `VSKILL_EVAL_PROVIDER` | `claude-cli` | No | `src/eval/llm.ts::createLlmClient` | Provider override for headless eval runs. |
| `VSKILL_EVAL_MODEL` | provider default | No | `src/eval/llm.ts::createLlmClient` | Model override for headless eval runs. |
| `VSKILL_EVAL_TIMEOUT` | `300` (seconds) | No | `src/eval/llm.ts::getTimeoutMs` | Per-call timeout for CLI providers. |
| `CODEX_API_KEY` | — | Optional for `codex-cli` | `src/eval/llm.ts::createCodexCliClient` (via CLI env passthrough) | Codex CLI credential. |

Examples:

```bash
# Point Ollama at a remote GPU workstation (OLLAMA_HOST is primary).
OLLAMA_HOST=http://gpu-server:11434 npx vskill@latest studio

# Bare host:port is auto-prefixed with http://.
OLLAMA_HOST=gpu.local:11434 npx vskill@latest studio

# Both set — OLLAMA_HOST wins and a deprecation line is logged once.
OLLAMA_HOST=http://gpu:11434 OLLAMA_BASE_URL=http://legacy:11434 npx vskill@latest studio
```

### 5.2 Claude Code session delegation — ToS compliance (April 2026)

Anthropic's April 4, 2026 Terms of Service update prohibits third-party tools from consuming Max/Pro quota via OAuth token scraping. **Skill Studio does not consume your Max/Pro subscription quota directly.** It delegates to the official [Claude Code CLI](https://docs.claude.com/en/docs/claude-code), which is the sanctioned consumer. The adapter in `src/eval/llm.ts::createClaudeCliClient()` shells out to the `claude` binary and never reads any file under `~/.claude/credentials*`, `~/.claude/auth*`, or `~/.claude/token*`.

This contract is enforced by two blocking gates:

1. **Unit test** — `src/eval/__tests__/claude-cli-compliance.test.ts` mocks every fs read API and asserts zero observed paths match `/\.claude\/(credentials|auth|token)/i` while the adapter runs against a fake `claude` shim.
2. **Bundle grep** — `scripts/check-bundle-compliance.sh` runs after `npm run build` and fails CI on any credential-path literal in `dist/`.

If the `claude` binary is missing, the UI tells you to install it (`npm install -g @anthropic-ai/claude-code`) and never suggests `ANTHROPIC_API_KEY` as a substitute — that would require a different (paid) provider.

---

## 6. Test procedure

### Unit + integration (Vitest)
```bash
cd repositories/anton-abyzov/vskill
npx vitest run src/eval-ui        # 988+ tests across ~103 files
npx vitest run src/eval-server    # server-side tests
npx vitest run src/eval           # engine tests
```
Target coverage: 90% (per `.specweave/config.json`).

### Type-check
```bash
npx tsc --noEmit
```

### Playwright E2E
```bash
# One-time
npx playwright install chromium

# Run all specs
npx playwright test

# Single spec
npx playwright test e2e/theme-persistence.spec.ts

# UI mode for interactive debugging
npx playwright test --ui
```
Specs live in `e2e/`. The config boots the studio on port 3077 via `webServer` directive. See `playwright.config.ts`.

### CI gate scripts
```bash
npm run lint:shimmer       # no shimmer animations anywhere in eval-ui
npm run lint:serif-scope   # serif typeface only in allowed selectors
npm run lint:bundle-size   # initial chunk ≤ 250KB gzipped
npm run lint:voice         # no "oops"/"awesome"/celebration emoji in strings.ts
```

### Manual smoke
```bash
# 1. Backend only
node dist/index.js eval serve --root .
curl http://localhost:3077/api/skills | jq '.[0]'
curl http://localhost:3077/api/skills/updates | jq

# 2. Full studio
npx vskill@latest studio
# → browse http://localhost:3162, click a skill, switch Overview ↔ Versions, press / to search, Cmd+K for palette, ? for shortcuts
```

---

## 7. Skill update detection (status + planned 0681 surface)

**Backend is already built**:
- `vskill outdated --json` scans installed skills against their remote source.
- `GET /api/skills/updates` wraps that and returns structured `SkillUpdateInfo[]`.
- `POST /api/skills/:plugin/:skill/update` updates a single skill via SSE.
- `src/eval-ui/src/pages/UpdatesPanel.tsx` renders a full updates page with bulk actions + changelog preview.

**UI surfacing is the gap — planned in increment `0681-studio-update-notifications`**:
- Count badge on the Installed section header: `Installed · Claude · 3 updates ▾`.
- Small `↑` indicator glyph on each outdated `SkillRow` (amber `--color-own`).
- Notification icon in `TopRail` when updates are available, opens a dropdown summary.
- "Update to X.Y.Z" button in the detail panel `RightPanel` when a skill is outdated.
- Icons generated via Gemini 3 Pro Image (Nano Banana Pro).

---

## 8. Release process

Patch release is automated via the `sw:npm` skill. Manual steps:
```bash
git commit -m "feat(studio): <change>"
npm version patch
npm run build && npm run build:eval-ui
npm publish --registry https://registry.npmjs.org
git push origin main --follow-tags
```

Version bumps follow semver:
- `patch` (0.5.83 → 0.5.84) — bug fixes, cosmetic updates
- `minor` (0.5.x → 0.6.0) — new features, new providers
- `major` (0.x.x → 1.0.0) — breaking API or CLI changes

Recent releases:
- `v0.5.83` — warm-neutral UI redesign (increment 0674)
- `v0.5.84` — hotfix: restore `.skeleton` as static alias so loading skeletons render again

---

## 9. Where to look next

- **Increment specs** (source of truth for what's planned / built): `.specweave/increments/` at the umbrella root `specweave-umb/`.
- **ADRs**: `.specweave/docs/internal/architecture/adr/0674-0{1,2,3,4}-*.md` (theme tokens, sidebar classification, typography, Tailwind 4 `@theme`).
- **Test reports**: `e2e/playwright-verification-report.md`, `e2e/llm-integration-report.md` (generated by verifier agents).
- **Shipped increments touching this studio**: `0674-vskill-studio-redesign` (closed), `0675-skill-creator-detection-hotfix`, planned `0677-lm-studio-provider`, `0678-skill-gen-source-model-picker`, `0679-skills-spec-compliance`, `0680-studio-model-access-reform`, `0681-studio-update-notifications`.

---

## 10. Contributing

Follow the SpecWeave loop for any non-trivial change: `sw:increment` → `sw:do` → `sw:done`. Every task has a Test Plan. Every AC is traceable.

Questions? Open an issue on [`anton-abyzov/vskill`](https://github.com/anton-abyzov/vskill) or check the active increments at the umbrella root.
