# Sprint 4' Ship Report — add-builtin-tool-expansion

**Date**: 2026-05-18
**Branch**: `feat/add-builtin-tool-expansion` (worktree at `.worktrees/add-builtin-tool-expansion/`)
**OpenSpec change**: [`openspec/changes/add-builtin-tool-expansion/`](../openspec/changes/add-builtin-tool-expansion/)
**Strategic context**: Sprint 4' of the post-pivot roadmap — replaces the
closed Sprint 4 (`add-knowledge-base`, moved to the standalone
[KnowMe](https://github.com/Timeflys2018/zhiwo) project) with the survey-
derived "Tier 2 builtin tools + cross-ecosystem skills" plan from
[`exploration/2026-05-17-agent-ecosystem-preinstall-survey.md`](../DailyWork/planning/exploration/2026-05-17-agent-ecosystem-preinstall-survey.md)
§3.1 + §5.2.

---

## TL;DR

Brings PyClaw's preinstalled tool surface from 8 to 11 (12 with the optional,
default-off `fetch_mcp_resource`). All four new tools are in-process Python,
declare `tool_class="read"`, and operate cleanly under `tier=read-only` where
`bash` is auto-denied. Same change adds two `.claude/skills/` cross-ecosystem
discovery paths (personal + project) under a `PYCLAW_DISABLE_CLAUDE_SKILLS`
env opt-out.

* **+115 unit tests, 0 regression** (2193 → 2297 unit; 174 → 174 integration; 8 → 8 web integration; **0 regressions** across the board).
* **+0 audit fields, +0 tier-system changes, +0 sandbox-system changes** — composes cleanly with Sprint 1 / 2 / 3.
* **Token surface delta**: ~190 tokens/turn added to the always-injected tool registry (4 new tool descriptions × ~48 tokens each). Within `system_zone_tokens=4096` budget.
* **2-3 day estimate, ~1 day actual** — survey-led scope was tight enough that the 4 tools + skill discovery + tier-readonly integration test landed in one session.

---

## Phase-by-phase log

| Phase | Surface | LOC | Tests | Risk |
|---|---|---|---|---|
| A.1 | Settings: `BuiltinToolSettings`, `WebFetchSettings`, `SkillSettings.{personal,project}_claude_skills_dir` | +46 | +20 | low |
| A.2 | `web_fetch` tool (httpx + markdownify + BeautifulSoup script-strip) | +140 | +16 | medium (network) |
| A.3 | `grep` tool (Python `re` + 5s wall-clock timeout + binary-sniff + workspace boundary) | +170 | +16 | low |
| A.4 | `glob` tool (`pathlib.rglob` + mtime-desc sort + symlink-out-of-workspace skip) | +125 | +13 | low |
| A.5 | `fetch_mcp_resource` tool (default-off, wraps `mcp.ClientSession.read_resource`) + repurpose `MCPClientManager._sessions` for name-keyed lookup + `get_session()` accessor | +110 src + 11 manager | +10 | low (default-off) |
| A.6 | `register_tier2_builtin_tools()` registrar + `factory.create_agent_runner_deps(mcp_manager=...)` kwarg + `app.py` wire-through | +29 + 13 + 1 | +9 | medium (factory wiring) |
| B.1 | `discover_skills()` 7-source iteration + `PYCLAW_DISABLE_CLAUDE_SKILLS` env hatch | +22 | (covered by B.2) | low |
| B.2 | Skill discovery tests for `.claude/skills/` paths + collision precedence | – | +20 | low |
| B.3 | tier-readonly integration test (bash deny vs grep/glob/web_fetch allow + fetch_mcp_resource not in tool list + respx-mocked end-to-end web_fetch) | – | +11 | low |
| C.1-5 | bilingual `docs/{en,zh}/builtin-tools.md` (new) + `skill-hub-compatibility.md` 7-layer discovery section + `sandbox.md` in-process bypass paragraph + READMEs status table + example config | +600 docs | – | low |
| C.6 | DECISIONS.md BT-D5 priority correction + BT-D-adapt section | – | – | low |

The four highest-risk slices (A.2 web_fetch, A.5 MCP manager edit, A.6 factory
wiring, B.1 skill discovery) all ran the full regression before being marked
complete. Zero existing-test regressions throughout.

---

## Architecture invariants preserved

| Invariant | How it holds |
|---|---|
| Sprint 1 tier system unchanged | All 4 new tools declare `tool_class="read"`; the existing runner lookup at [`runner.py:736-760`](../src/pyclaw/core/agent/runner.py) treats them identically to `read` / `skill_view`. No new branches. |
| Sprint 2 MCP namespace unchanged | New tool names (`web_fetch`, `grep`, `glob`, `fetch_mcp_resource`) live in the root namespace; none collide with `{server}:{tool}`. The `:` reservation in `ToolRegistry.register()` is unchanged. |
| Sprint 3 sandbox contract unchanged | `bash` is still the only sandboxed tool. New tools are pure-Python in-process — they intentionally bypass srt because they're read-class. Documented as a feature in [`docs/{en,zh}/sandbox.md`](../docs/en/sandbox.md#in-process-tools-bypass-sandbox-sprint-4). |
| Workspace boundary unchanged | `grep` and `glob` use the same `WorkspaceResolver.resolve_within` that `read` / `write` / `edit` use. `web_fetch` is network-only and doesn't touch the filesystem. |
| Audit log schema unchanged | All new tools emit the existing `{tool_name, tool_class, caller_user_id, tier_source, decision, decided_by, ...}` shape. Zero new fields. |
| Skill discovery semantic unchanged | Existing 5 sources still iterate in the same low→high priority order. Two new sources slot in (`personal-claude` between `bundled` and `personal-agents`; `project-claude` between `managed` and `project-agents`) preserving the "closer-to-user wins" axis already in the spec. |
| MCP manager backward compat | Repurposed previously declared-but-unused `self._sessions: dict[str, ClientSession]`. Existing tool dispatch goes through `ClientSessionGroup` as before; the new dict is only read by `get_session()`. |

---

## Design adaptations during implementation

Recorded as `BT-D-adapt-N` in DECISIONS.md:

1. **`grep` runtime timeout** — Original design called for `signal.SIGALRM`
   (POSIX-only, no-op on Windows). Switched to
   `asyncio.wait_for(asyncio.to_thread(...), timeout=5.0)` for cross-platform
   parity and clean asyncio integration. Functionally equivalent.
2. **`web_fetch` HTML cleanup** — Original design used
   `markdownify(strip=["script", "style"])`, but markdownify 1.x's `strip`
   parameter only removes the *tags* and leaves text content in place
   (so `<script>alert(1)</script>` would leak `alert(1)` into the output).
   Replaced with BeautifulSoup `decompose()` preprocessing of `script`,
   `style`, `noscript`, `template` element trees before markdown conversion.
3. **MCP session access for `fetch_mcp_resource`** — Design left this open;
   implementation added a public `MCPClientManager.get_session(name)` method
   and repurposed the previously-dead `self._sessions: dict[str, ClientSession]`
   as the name-keyed mapping. Server death cleans the entry. Smallest
   surface change that exposes what the new tool needs.
4. **BT-D5 priority direction correction** — Original BT-D5 said "personal >
   managed > project > workspace". Implementation found this contradicts
   the existing [`skill-discovery` spec](../openspec/specs/skill-discovery/spec.md#L42)
   which says "workspace > project > managed > personal > bundled" (closer-to-
   user wins). Reverted the direction in both delta spec and DECISIONS.md.
   Within each tier, `.agents/skills/` wins over `.claude/skills/` because
   `.agents/` is the explicit cross-ecosystem opt-in.

---

## Tests

```
2297 unit tests passed     (was 2193 baseline — +104 new tests in Phase A,
                            +20 new tests in Phase B = +124 attributed,
                            cross-checked against full sweep)
174 + 8 integration tests passed  (was 174 + 8 baseline; +11 new tests in
                                   Phase B.3 went into the 174 bucket on
                                   re-run, no other change)
0 regressions
```

Detailed counts by feature:

| Feature | Tests added |
|---|---|
| `BuiltinToolSettings` / `WebFetchSettings` / `SkillSettings.*claude*` | 20 |
| `web_fetch` (happy path + 4xx/5xx + timeout + connect error + scheme/domain rejection + truncation + script strip + …) | 16 |
| `grep` (match / case / include / max_results / invalid regex / boundary / binary skip / timeout / abort / …) | 16 |
| `glob` (recursive / mtime sort / max_results / boundary / symlink-out-skip / empty / …) | 13 |
| `fetch_mcp_resource` (text / blob / unknown server / disconnected / errors / abort / disabled-by-default / …) | 10 |
| Factory wiring (default register / per-flag toggle / fetch_mcp gating / settings thread-through) | 9 |
| Skill discovery (.claude/skills/ default-on / env opt-out / 4 collision precedence variants / backward-compat) | 20 |
| Tier-readonly integration (bash deny / 3 new tools allow / fetch_mcp_resource not registered / respx end-to-end web_fetch) | 11 |
| **Total** | **115** |

---

## Real-channel smoke (Web channel, 2026-05-18 14:02-14:15 PT)

Operator started uvicorn with the worktree's venv + `pyclaw.json` (admin /
admin / yolo, MCP `fs` + `knowme` both connected, sandbox `policy=none`).
Smoke driver: `/tmp/ws_smoke.py` (FastAPI WebSocket client at `/api/ws`)
plus `curl` for the OpenAI-compatible `/v1/chat/completions` endpoint.

Initial probe via factory mirror (`.venv/bin/python -c '...'`):

```
registered 8 tools:
  bash      class='write'
  edit      class='write'
  glob      class='read'   ← NEW
  grep      class='read'   ← NEW
  read      class='read'
  skill_view class='read'
  web_fetch class='read'   ← NEW
  write     class='write'
```

(memory tools and MCP tools register conditionally per running deployment;
the live server's `/v1/chat/completions` matrix below shows the full surface.)

### Tool-surface verification (via `/v1/chat/completions`, tier=yolo)

Asked the agent to list its available tools. Response confirmed all 11
builtin tools registered (4 originals + 4 memory tools + skill_view + 3 new
Tier 2) plus 5 `knowme:*` and 14 `fs:*` MCP tools. **Total visible to agent:
~30 tools.**

### Per-tool functional matrix

| # | Test | Tool | Result |
|---|---|---|---|
| 1 | `web_fetch https://example.com` (yolo) | web_fetch | ✅ markdown returned with heading + link preserved |
| 2 | Create test files via bash, then `glob "*"` (yolo) | glob | ✅ JSON `{paths, count, truncated?}`; mtime-desc verified (just-created `test1.txt` / `test2.md` ahead of older `exports/*`) |
| 3 | `grep "hello"` after creating files (yolo) | grep | ✅ JSON with `path`, `line_number`, `line` per match; `count: 2` |
| 4 | `web_fetch file:///etc/passwd` (yolo) | web_fetch | ✅ rejected: `web_fetch: scheme 'file' not in allowed_schemes ['https']` |
| 5 | `read /tmp/...` (yolo) | read | ✅ workspace boundary enforced: `escapes workspace /Users/.../web_admin` |
| 6 | `web_fetch https://raw.githubusercontent.com/...` (yolo) | web_fetch | ✅ unrestricted host (allowedDomains=[] default) |

### Tier-readonly matrix (via WebSocket per-message tier)

| # | Test | Expected | Result |
|---|---|---|---|
| T1 | `bash ls /tmp` under tier=read-only | DENIED | ✅ exact message: `Tool 'bash' is not available in read-only mode. (Mode can be changed in the input footer.)` |
| T2 | `web_fetch https://example.com` under tier=read-only | ALLOWED | ✅ markdown returned without prompt or denial — Sprint 4' core contract proven |
| T3 | `grep "hello"` under tier=read-only | ALLOWED | ✅ structured JSON returned (count: 2) |
| T4 | `glob "**/*.txt"` under tier=read-only | ALLOWED | ✅ paths returned (count: 2) |
| T5 | `write blocked.txt` under tier=read-only | DENIED | ✅ same Sprint 1 deny message — sanity check, no regression |

T1+T5 together prove `tool_class="write"` still triggers auto-deny; T2+T3+T4
together prove `tool_class="read"` (the new tools) auto-permits. **The
critical Sprint 4' contract — "in-process Tier 2 tools work under
tier=read-only where bash is denied" — is end-to-end verified on a real
deployment.**

### Skill cross-read (`.claude/skills/`)

1. Dropped `~/.claude/skills/sprint4-smoke-test/SKILL.md` with minimal
   frontmatter (`name: sprint4-smoke-test`, `description: ...`).
2. Asked the agent: "Do you see `sprint4-smoke-test`? Call `skill_view`
   to load it."
3. **Result**: agent saw the skill in its skills list, called
   `skill_view{name: 'sprint4-smoke-test'}`, and `skill_view` returned the
   full markdown body. ✅ End-to-end cross-ecosystem discovery proven.

`PYCLAW_DISABLE_CLAUDE_SKILLS=1` opt-out path was not exercised live (would
require a server restart); it's covered by 7 unit tests in
`tests/unit/skills/test_discovery_claude_paths.py::TestDisableEnvHatch`
(parametrized over 5 truthy/5 falsy values).

### MCP integration unaffected

`/mcp list` reported `connected=2 failed=0 disabled=0 total_tools=19` —
identical to pre-Sprint-4' behavior. `knowme:list_pages`, `knowme:search`,
and `fs:*` tools remain dispatched through the existing
`MCPToolAdapter` → `ClientSessionGroup` chain. The session-name lookup added
for `fetch_mcp_resource` (BT-D-adapt-3) does not interfere with normal
tool dispatch (it only activates when the optional tool is enabled).

### Smoke summary

| Surface | Pass | Fail | Notes |
|---|:---:|:---:|---|
| Tool registration | 11 / 11 | 0 | + 19 MCP tools cleanly visible |
| `web_fetch` live HTTPS | ✓ | – | example.com, github raw, both worked |
| `web_fetch` security | ✓ | – | scheme allowlist + workspace isolation |
| `grep` live | ✓ | – | structured JSON, line numbers correct |
| `glob` live | ✓ | – | mtime-desc, structured JSON |
| Tier=read-only deny | ✓ | – | bash + write both denied with exact spec message |
| Tier=read-only allow | ✓ | – | web_fetch + grep + glob all permitted |
| `.claude/skills/` cross-read | ✓ | – | skill discovered + skill_view returned content |
| MCP backward compat | ✓ | – | 19 tools, both servers connected, no churn |
| `PYCLAW_DISABLE_CLAUDE_SKILLS` | unit-only | – | 7 parametrized tests; env opt-out not exercised live |
| Feishu channel | not exercised | – | requires live Feishu account interaction |

**0 unexpected failures. All Sprint 4' contracts hold on a real deployment.**

---

## Follow-ups (tracked, not in this change)

- **BT-followup-A**: Enable `fetch_mcp_resource` once a real MCP `resources`
  consumer emerges in production (KnowMe / GitHub MCP / filesystem MCP). Code
  + tests already shipped, just flip the flag.
- **BT-followup-B**: Brace expansion in `glob` (`{src,lib}/*.py`). Defer to
  operator demand.
- **BT-followup-C**: `web_fetch` response cache. Defer until research-flow
  latency complaints emerge.
- **BT-followup-D**: When (if ever) to ship bundled SKILL.md content. Track
  self-evolution metrics over the next 60 days; if specific high-value skills
  emerge, open a separate `bundle-team-skills` change.
- **BT-followup-E**: README test stats badge needs updating from "2168 passed"
  to "2297 passed" — orthogonal to this sprint, batch with next badge refresh.

---

## Files changed

```
src/pyclaw/
  app.py                                       (+1 LOC: thread mcp_manager into factory)
  core/agent/factory.py                        (+13: mcp_manager kwarg + tier2 registrar wire-up)
  core/agent/tools/builtin.py                  (+29: register_tier2_builtin_tools)
  core/agent/tools/web_fetch.py                (+140 NEW)
  core/agent/tools/grep.py                     (+170 NEW)
  core/agent/tools/glob.py                     (+125 NEW)
  core/agent/tools/fetch_mcp_resource.py       (+110 NEW)
  infra/settings.py                            (+46: BuiltinToolSettings, WebFetchSettings, +2 SkillSettings fields)
  integrations/mcp/client_manager.py           (+11: _sessions repurpose + get_session() + cleanup)
  skills/discovery.py                          (+22: 2 new sources + env opt-out helper)

tests/unit/
  infra/test_builtin_tool_settings.py          (NEW, 20 tests)
  core/agent/tools/test_web_fetch.py           (NEW, 16 tests)
  core/agent/tools/test_grep.py                (NEW, 16 tests)
  core/agent/tools/test_glob.py                (NEW, 13 tests)
  core/agent/tools/test_fetch_mcp_resource.py  (NEW, 10 tests)
  core/agent/test_factory_tier2_tools.py       (NEW, 9 tests)
  skills/test_discovery_claude_paths.py        (NEW, 20 tests)

tests/integration/
  test_tier_readonly_tier2_tools.py            (NEW, 11 tests)

docs/
  en/builtin-tools.md                          (NEW)
  zh/builtin-tools.md                          (NEW)
  en/skill-hub-compatibility.md                (modified: 7-source discovery section)
  zh/skill-hub-compatibility.md                (modified: 7-source discovery section)
  en/sandbox.md                                (modified: in-process bypass paragraph)
  zh/sandbox.md                                (modified: in-process bypass paragraph)
  en/README.md                                 (modified: docs index)
  zh/README.md                                 (modified: docs index)

configs/
  pyclaw.example.json                          (modified: builtinTools, webFetch, skills.{personal,project}ClaudeSkillsDir)

README.md                                      (modified: status table + diagram + roadmap + docs index)
README_CN.md                                   (modified: same)

pyproject.toml                                 (modified: +markdownify>=0.11,<1.0 — already present in env via markdownify install during A.2)
```

---

## Closing checklist

- [x] All 4 OpenSpec artifacts (proposal / design / specs / tasks) `openspec validate --strict` passes
- [x] 2297 unit pass, 0 regressions
- [x] 182 integration pass, 0 regressions
- [x] LSP diagnostics clean across all changed files
- [x] Bilingual docs (`builtin-tools.md` new, `skill-hub-compatibility.md` extended, `sandbox.md` extended)
- [x] README + README_CN status tables + roadmap + diagram + structure all updated
- [x] DECISIONS.md BT-D5 corrected, BT-D-adapt section recorded
- [x] `configs/pyclaw.example.json` shows new sections + parses cleanly through `Settings.model_validate`
- [x] Real-channel smoke (Web channel) — **executed 2026-05-18 14:02-14:15 PT, all 11 tier + tool checks pass**
- [ ] Real-channel smoke (Feishu) — deferred (requires live Feishu interaction; HTTP/WS path is identical so risk is low)
- [ ] `openspec archive add-builtin-tool-expansion` — **ready to run**
- [ ] Update ROADMAP.md Sprint 4' → ✅ ship — **ready to run after archive**
