docs: codebase_research evidenced — 5 of 12, and its citations check out
deploy / test (push) Successful in 5m15s
deploy / build (push) Successful in 1m27s

Run against the real clawmates repo with a task that cannot be faked
from filenames: trace every hop of a mission agent's tool call from the
guest hook to the recorded event, naming file, function, and guest vs
host for each.

It produced a 189-line research/GATE-MAP.md describing code committed
the SAME DAY — the four-field NODE_EXTRACT including agent_type, the
shadow-mode would-deny.jsonl semantics, install_with — and all six of
its line citations verify exactly. Nothing hallucinated. That is a
stronger result than the judge's verdict, which could only check that
the structure was present.

Also recorded: agents reach for Bash. 74 Bash calls against 34 Read, 3
Glob, 1 Edit, 1 Agent — a mapping mission that could have used Grep used
Bash 33 times. So the allowlist's dedicated-tool entries may never be
exercised, and the surface that actually needs governing is Bash, which
the floor rules already cover.

Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01WZb5A2kfVfjpdwSochkuHz
This commit is contained in:
Omar Sobh
2026-09-22 15:09:22 -05:00
co-authored by Claude Opus 5
parent b1a23727da
commit a88d51b078
+26 -3
View File
@@ -62,12 +62,12 @@ holding (it was 55 of 85 dangling once).
| `mobile` | designer coder tester committer | coding_readwrite | 11 | no | | `mobile` | designer coder tester committer | coding_readwrite | 11 | no |
| `gpu` | arch_analyst kernel_author bench_engineer coder committer | coding_readwrite | 13 | no | | `gpu` | arch_analyst kernel_author bench_engineer coder committer | coding_readwrite | 13 | no |
| `threejs` | scene_designer coder shader_author perf_engineer committer | coding_readwrite | 13 | no | | `threejs` | scene_designer coder shader_author perf_engineer committer | coding_readwrite | 13 | no |
| `codebase_research` | code_archeologist architecture_mapper flow_tracer vault_scribe | research_readonly | 11 | no | | `codebase_research` | code_archeologist architecture_mapper flow_tracer vault_scribe | research_readonly | 11 | **yes** — first run 2026-09-22 |
| `papers_research` | domain_scout paper_reader library_curator | research_web_readonly | 8 | no | | `papers_research` | domain_scout paper_reader library_curator | research_web_readonly | 8 | no |
| `insight_research` | implementation_tracker novelty_hunter publication_drafter | research_readonly | 8 | no | | `insight_research` | implementation_tracker novelty_hunter publication_drafter | research_readonly | 8 | no |
| `continuous_improvement` | brain_inspector improvement_proposer improvement_evaluator | research_readonly | 7 | no | | `continuous_improvement` | brain_inspector improvement_proposer improvement_evaluator | research_readonly | 7 | no |
**Four of twelve are evidenced.** `backend` was exercised on 2026-09-22, **Five of twelve are evidenced.** `backend` was exercised on 2026-09-22,
the first time anything had been staffed from it: all five roles the first time anything had been staffed from it: all five roles
provisioned with real agents (api_designer, db_engineer, coder, tester, provisioned with real agents (api_designer, db_engineer, coder, tester,
committer), and the mission delivered cursor pagination — `Paged<T>`, committer), and the mission delivered cursor pagination — `Paged<T>`,
@@ -75,7 +75,19 @@ committer), and the mission delivered cursor pagination — `Paged<T>`,
**met on the first pass**, 2 files pushed. It also gave the task-permission **met on the first pass**, 2 files pushed. It also gave the task-permission
shadow its first confirmed `Edit` call. shadow its first confirmed `Edit` call.
**Three of the remaining eight are still unevidenced.** The other nine are well-formed scaffolding: `codebase_research` was exercised the same day, against the real
`clawmates` repository rather than the toy scratch crate, with a task that
cannot be faked from filenames: trace every hop of a mission agent's tool
call from the guest hook to the recorded event, naming file, function and
whether each hop runs in the guest or on the host. It produced a 189-line
`research/GATE-MAP.md` describing code committed **the same day** — the
four-field `NODE_EXTRACT` including `agent_type`, the shadow-mode
`would-deny.jsonl` semantics, `install_with` — and **all six of its line
citations verify exactly** (`hook_script_with` 562, `NODE_EXTRACT` 789,
`TaskPolicy` 297, `ROLE_POLICIES` 261, `settings_hook` 809,
`install_command_with` 819). Nothing hallucinated. **Five of twelve.**
**Three of the remaining seven are still unevidenced.** The other nine are well-formed scaffolding:
roles, prompts, brain seeds and resolving skills, and no run behind any of roles, prompts, brain seeds and resolving skills, and no run behind any of
them. They will probably work — they are structurally identical to the three them. They will probably work — they are structurally identical to the three
that do — but "probably" is the word, and this codebase has a name for the gap that do — but "probably" is the word, and this codebase has a name for the gap
@@ -88,6 +100,17 @@ blocked on a target, not on the template. The remaining four —
`codebase_research`, `papers_research`, `continuous_improvement`, `backend` `codebase_research`, `papers_research`, `continuous_improvement`, `backend`
could be exercised against repositories we already have. could be exercised against repositories we already have.
## One thing both runs showed: agents reach for Bash
Cumulative tool calls across every mission since the policy shipped:
**Bash 74, Read 34, Write 15, Glob 3, Edit 1, Agent 1**. A read-only
mapping mission that could have used `Grep` and `Glob` used `Bash` 33
times and `Read` 5. That matters for task permission: the allowlist's
dedicated-tool entries (`Grep`, `ToolSearch`, `WebFetch`, `TodoWrite`)
may simply never be exercised, so "unconfirmed by a real run" will not
converge for them — and it means the surface that actually needs
governing is `Bash`, which the floor rules already cover.
## What this means we can ask for today ## What this means we can ask for today
**With confidence:** a repo-backed research→code loop on a Rust project, with **With confidence:** a repo-backed research→code loop on a Rust project, with