Three evaluation capabilities

## One pipeline across a skill's whole life

doctor, eval, and observe aren't three tools — they're one measurement discipline at three points in a skill's lifecycle, each answering a different question.

🩺

### doctor pre-ship

Is the skill itself written healthily? 7 built-in dimensions scored independently — static rules at zero cost, LLM audit for depth, plus endpoint-driven custom dimensions.

`$ omk doctor my-skill --dimensions audit.yaml`

- Triggers / docs / instructions / deps / tools / safety / examples
- --static-only: offline, no LLM calls
- Custom endpoint dimensions: call an API for deep review

📊

### eval on release

Is v2 really better than v1? A controlled A/B — same model, same cases, only the knowledge changes. Six dimensions scored independently, one-line verdict with a ship recommendation.

`$ omk eval --control v1 --treatment v2`

- Bootstrap CI / length-debias / saturation curves on by default
- Krippendorff α: judge↔human agreement on a gold set
- Blind A/B · judge ensemble · multi-run variance

🔭

### observe in prod

How is it doing in production? Parse real session JSONL to measure each skill's failure rate, latency, and token cost, and surface severity-weighted knowledge-gap signals.

`$ omk observe ~/.claude/sessions`

- Failure rate / latency / cost broken down per skill
- Knowledge-gap detection: quantify risk exposure
- Feeds production signal into sample / evolve iteration

Moat · measurement credibility

## Rigor is the foundation, not an add-on

Five often-overlooked distortions decide whether a comparison is trustworthy. omk builds every defense into the foundation, so you don't have to enable them one by one.

01
A point estimate mistakes sampling noise for a real gain
Bootstrap confidence intervals built-in
Reports an interval, not a point — significance is read off directly.
02
A composite average hides a regression in a single dimension
Three-layer independent scoring · pass-all gate built-in
Fail any of fact / behavior / judge and it doesn't pass.
03
The control group reads the very carrier under test
construct validity breaks down
strict-baseline isolation built-in
Closes three leaks: skill self-discovery, the Skill tool, and the cwd bypass.
04
Judges systematically prefer longer answers
Length-debiased judging built-in
Scoring removes the length covariate — verbosity no longer buys points.
05
The reliability of the judge's own scoring is unmeasurable
Krippendorff α on with a gold set
Anchored on human labels, it quantifies judge↔human agreement.

Peer tools typically cover only one or two of these. omk's choice: build credibility into the foundation rather than leave it optional.

Tool comparison

## A side-by-side under one set of criteria

Criteria come from common LLMOps selection axes (metric library / judge / CI / observability / collaboration) plus measurement validity & reliability — not rules tailored to omk. On several axes omk doesn't win, and we mark that honestly.

Capability | omk | promptfoo | DeepEval | LangSmith
--- | --- | --- | --- | ---
Measurement credibility · validity / reliability
Statistical significance (CI / tests) | ✓ Bootstrap | — | — | —
Judge ↔ human reliability (agreement) | ✓ Krippendorff α | — | — | —
Evaluation bias control (length-debias) | ✓ default | — | — | —
Evaluation capability
Assertion / metric library breadth | ✓ 30+ | ✓ | ✓ | ◑
RAG-specific metrics | ◑ 3 | ◑ | ✓ rich | ◑
LLM-as-judge | ✓ | ✓ | ✓ | ✓
Engineering & collaboration
CI/CD integration (exit-code routing) | ✓ | ✓ | ✓ | ◑
Onboarding speed / config simplicity | ◑ | ✓ very fast | ◑ | ◑
Experiment tracking / tracing | — | — | ◑ | ✓ strong
Hosted SaaS dashboard / team collab | — | — | ✓ | ✓
Ecosystem & integration
Community size (GitHub stars, 2026-04) | nascent | 9k+ | 12k+ | commercial
Native Claude Code skill | ✓ | — | — | —

✓ Native◑ Partial / needs config— Not supported

Full comparison (8 tools × 30+ dimensions, incl. RAGAS / OpenAI Evals / lm-eval-harness / inspect-ai) is in the comparison doc, current as of 2026-04 — spot something stale? send a PR. Takeaway: no silver bullet — omk's tradeoff is "statistical credibility by default"; want a SaaS dashboard, pick LangSmith; want academic benchmarks, pick lm-eval-harness.

$ omk install omk-agent-skill # one-time install
✓ Wrote to detected Claude Code / Codex
/omk eval # evaluate this project's artifact
/omk evolve # one shot: checkup → samples → self-iterate
/omk sample # generate or top up eval cases
> Compare v1 and v2 for me
↳ infer intent → omk eval --control v1 --treatment v2 …
Agent integration

### Use it right inside your coding agent

One line — `omk install omk-agent-skill` — installs the official Agent Skill into the Claude Code / Codex it detects locally (`--to all` writes to all). After that, `/omk` works out of the box in Claude Code; in Codex and others, just run the `omk` CLI.

No commands to memorize — state your goal in plain words, and the agent locates the skill from context and picks the right command.

You
Make this skill more robust, then compare it against the last version

Before your next release, let the data speak first.

Run your first eval in 5 minutes

## From a bare number to a conclusion that holds up

No files to touch — `omk init` scaffolds two skill versions and three cases, and `omk eval` produces an HTML report plus a one-line verdict in under 5 minutes.

$ omk init demo && cd demo && omk eval Copy ⧉