# Design: MuJoCo Alarmed Grasp Loop

Generated by /office-hours on 2026-04-14
Branch: dongxu/dev-0412-02
Repo: MiaoDX/roboharness
Status: APPROVED
Mode: Builder
Source: mirrored from `~/.gstack/projects/MiaoDX-roboharness/mi-dongxu/dev-0412-02-design-20260414-153259.md`

## Problem Statement

Roboharness already captures screenshots, reports metrics, and integrates with several
simulation stacks, but the current "AI agent closes the loop by itself" experience is
still too implicit and too manual for broad reuse.

The concrete pain came from a real GR00T WBC + SONIC grasp workflow. Before the harness
idea was split into this repo, one grasp-debug cycle required multiple shells, robot
init and safety prep, keyboard-triggered bottle placement / grasp / home / placement,
then manual copy-paste of filenames back into Claude Code or Codex. One cycle took
roughly 2-3 minutes, and the agent still depended on a human to babysit the ritual.

The product wedge is to replace that human glue with one autonomous MuJoCo grasp loop:
run the task, raise metric alarms, package visual evidence by semantic phase, and tell
the agent where to look next. The goal is not "more screenshots." The goal is that
CC/Codex can debug robot behavior from a compact artifact pack and rerun without a human
replaying the task.

## What Makes This Cool

The "whoa" moment is not a prettier report. It is collapsing a miserable robotics debug
ritual into something the agent can run in about 10 seconds on its own.

The delightful part is that the output is meaningful to both humans and agents:
- metric alarms catch obvious regressions
- phase-level screenshots catch the behavior regressions scalar metrics hide
- the same report points to the failed phase and the likely next edit target

This is the first thing that can make robotics dev feel like modern agent-assisted
software development instead of supervised puppet theater.

## Constraints

- Start with one stack: MuJoCo.
- Start with one task: grasp.
- Start with one loop: alarmed autonomous retry for CC/Codex.
- Preserve headless-server usability. The report and artifacts must work over SSH/CI.
- Use semantic task phases (`plan_start`, `pre_grasp`, `approach`, `contact`, `lift`)
  instead of flat recordings.
- Package both metrics and visual evidence. Either one alone is insufficient.
- Avoid introducing a second judge model. The coding agent itself is the evaluator.
- Reuse existing roboharness primitives where possible instead of inventing a second
  reporting / storage stack.
- Leave one seed for the near-future platform: a machine-readable phase manifest that
  can later generalize to Isaac Lab, ManiSkill, LeRobot, and real-hardware workflows.

## Premises

1. The correct first product is not "robot eval in general." It is one autonomous debug
   loop for MuJoCo grasp where scalar metrics and visual checkpoints travel together.
2. Existing tools are complements, not enemies. Roboharness should sit above evaluation,
   recording, and visualization stacks as the glue layer for AI-agent diagnosis and
   iteration.
3. The first thing to spread is the 10-second loop that replaces the 2-3 minute human
   babysitting ritual.
4. The long game is larger: give AI agents enough metric and visual evidence that they
   can iteratively make robots do the actual task, not just pass one scalar check.

## Approaches Considered

### Approach A: Alarmed Grasp Loop

Productize one gold workflow end to end, MuJoCo grasp only. One command runs the task,
emits metric alarms, produces a phase-by-phase multi-angle visual diff report, and
writes a compact machine-readable manifest so the agent knows what failed and where to
look next.

Why it works:
- matches the original pain exactly
- fastest path to something another team can copy
- easiest way to produce a real "whoa" demo

Tradeoffs:
- narrow enough that some people will initially mistake it for "just a grasp demo"
- does not by itself prove the full cross-ecosystem thesis

### Approach B: PhasePack Platform

Define the long-term contract now: phases, alarms, camera sets, diff artifacts, verdicts,
rerun hints, and agent-facing manifests as first-class concepts, then apply that contract
across MuJoCo, LeRobot, Isaac Lab, and ManiSkill.

Why it works:
- strongest long-term architecture
- clear product boundary
- integrations become systematic, not ad hoc

Tradeoffs:
- bigger design surface before users feel magic
- easy to spend time polishing abstractions before the wedge becomes indispensable

### Approach C: Adapter/Judge Layer

Keep existing execution stacks intact and focus on ingesting outputs from LeRobot,
Isaac Lab, Rerun, or custom scripts, then compile them into agent-readable debug packets
with phases, alarms, and summaries.

Why it works:
- fast path to ecosystem compatibility
- attractive to teams already committed to another stack

Tradeoffs:
- weaker control over resets, checkpoints, and rerun flow
- risk of becoming a report converter instead of the autonomous loop-closer

## Recommended Approach

Build **Approach A now**, but make one deliberate extraction from Approach B:
ship a **machine-readable phase manifest** from day one.

That means the first product surface should look like this:
- one command runs the MuJoCo grasp example and compares against a known-good baseline
- output includes `alarms.json`, `phase_manifest.json`, per-phase screenshot folders,
  and `report.html`
- the HTML report prioritizes:
  - top-level alarm cards
  - phase timeline with the first divergent phase highlighted
  - multi-angle visual diff grid for the failing phase
  - "Agent Next Action" panel that tells the model which evidence to inspect first
- the phase manifest tells the agent:
  - failed phase
  - suspected root-cause class
  - primary views to inspect
  - which metrics regressed
  - rerun hint (`restore:pre_grasp`, etc.)

This keeps the wedge honest. It solves the real workflow now, not the hypothetical
future platform. But it also avoids painting the repo into a dead-end corner.

## Open Questions

- What is the minimum useful alarm set for the first release?
  Likely: grasp accuracy drift, task runtime drift, and one path-quality / collision clue.
- How should baseline comparison work?
  Single blessed baseline first, then multiple saved baselines later.
- How opinionated should the root-cause hint be?
  Strong enough to focus the agent, not so magical that it becomes fake certainty.
- Which phase names should be standardized now vs left task-specific?
- How much of the "Agent Next Action" panel is deterministic rules vs derived summary?

## Success Criteria

- A single MuJoCo grasp command generates an artifact pack containing:
  `report.html`, `alarms.json`, `phase_manifest.json`, and per-phase screenshots.
- A known bad regression is localized to a specific phase, not just marked "failed."
- At least one failure mode that scalar success metrics miss is obvious from the report
  (example: mid-path collision or path ugliness despite final task success).
- The artifact pack removes the need for manual replay and manual screenshot copy-paste.
- The total grasp-debug loop is close to the user's current target, roughly 10 seconds in sim.
- Another developer can understand the failure without opening multiple shells or replaying
  the trajectory manually.

## Distribution Plan

The first delivery channel is the existing `roboharness` Python package plus one example
command that produces a report locally and in CI.

Distribution shape:
- package install via `pip install roboharness[demo]`
- canonical example entrypoint in `examples/demos/mujoco/grasp.py`
- generated `report.html` and machine-readable manifests in the output directory
- optional publication of the report through the existing GitHub Pages workflow

CI/CD:
- keep using the repo's current GitHub Actions pipeline
- add one CI-visible artifact path for the alarmed grasp loop output
- later, once verdict confidence is strong enough, gate PRs on explicit task regression

## Next Steps

1. Extend the MuJoCo grasp example so one run emits an alarmed artifact pack, not just a
   generic report.
2. Add `phase_manifest.json` with failed phase, primary views, regressed metrics, and rerun hint.
3. Redesign the HTML report around alarm-first reading order:
   alarms -> phase timeline -> failing-phase diff grid -> next action.
4. Add one baseline comparison path that proves visual phase diffs matter even when scalar
   success looks acceptable.
5. Add focused tests around manifest generation, alarm logic, and report rendering.
6. Re-apply the wedge back into the original GR00T WBC + SONIC grasp workflow to verify
   that the abstraction still matches the real pain.

## What I noticed about how you think

- You did not start from a market category. You started from a lived pain: "after the
  visual harness is done, all previous works are automatially done by cc/codex themselves."
- You are optimizing for removal of human glue work, not just more instrumentation:
  "It will take around 10s for one grasp in sim."
- You care about the failure mode that metrics hide, not the vanity metric itself:
  "the same grasp accuray can be very different if the middle waypoints in the planner
  will collision with the robot or env."
- Your product instinct is to extract a real working loop and make it reusable for
  "more people/teams like me," which is the right open-source origin pattern.
