From a88d51b0786b3baba0b73f6fdea19b2a6d88b19e Mon Sep 17 00:00:00 2001 From: Omar Sobh Date: Tue, 22 Sep 2026 15:09:22 -0500 Subject: [PATCH] =?UTF-8?q?docs:=20codebase=5Fresearch=20evidenced=20?= =?UTF-8?q?=E2=80=94=205=20of=2012,=20and=20its=20citations=20check=20out?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Run against the real clawmates repo with a task that cannot be faked from filenames: trace every hop of a mission agent's tool call from the guest hook to the recorded event, naming file, function, and guest vs host for each. It produced a 189-line research/GATE-MAP.md describing code committed the SAME DAY — the four-field NODE_EXTRACT including agent_type, the shadow-mode would-deny.jsonl semantics, install_with — and all six of its line citations verify exactly. Nothing hallucinated. That is a stronger result than the judge's verdict, which could only check that the structure was present. Also recorded: agents reach for Bash. 74 Bash calls against 34 Read, 3 Glob, 1 Edit, 1 Agent — a mapping mission that could have used Grep used Bash 33 times. So the allowlist's dedicated-tool entries may never be exercised, and the surface that actually needs governing is Bash, which the floor rules already cover. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01WZb5A2kfVfjpdwSochkuHz --- docs/TEMPLATE-MATURITY.md | 29 ++++++++++++++++++++++++++--- 1 file changed, 26 insertions(+), 3 deletions(-) diff --git a/docs/TEMPLATE-MATURITY.md b/docs/TEMPLATE-MATURITY.md index 51d1323..b00f8b0 100644 --- a/docs/TEMPLATE-MATURITY.md +++ b/docs/TEMPLATE-MATURITY.md @@ -62,12 +62,12 @@ holding (it was 55 of 85 dangling once). | `mobile` | designer coder tester committer | coding_readwrite | 11 | no | | `gpu` | arch_analyst kernel_author bench_engineer coder committer | coding_readwrite | 13 | no | | `threejs` | scene_designer coder shader_author perf_engineer committer | coding_readwrite | 13 | no | -| `codebase_research` | code_archeologist architecture_mapper flow_tracer vault_scribe | research_readonly | 11 | no | +| `codebase_research` | code_archeologist architecture_mapper flow_tracer vault_scribe | research_readonly | 11 | **yes** — first run 2026-09-22 | | `papers_research` | domain_scout paper_reader library_curator | research_web_readonly | 8 | no | | `insight_research` | implementation_tracker novelty_hunter publication_drafter | research_readonly | 8 | no | | `continuous_improvement` | brain_inspector improvement_proposer improvement_evaluator | research_readonly | 7 | no | -**Four of twelve are evidenced.** `backend` was exercised on 2026-09-22, +**Five of twelve are evidenced.** `backend` was exercised on 2026-09-22, the first time anything had been staffed from it: all five roles provisioned with real agents (api_designer, db_engineer, coder, tester, committer), and the mission delivered cursor pagination — `Paged`, @@ -75,7 +75,19 @@ committer), and the mission delivered cursor pagination — `Paged`, **met on the first pass**, 2 files pushed. It also gave the task-permission shadow its first confirmed `Edit` call. -**Three of the remaining eight are still unevidenced.** The other nine are well-formed scaffolding: +`codebase_research` was exercised the same day, against the real +`clawmates` repository rather than the toy scratch crate, with a task that +cannot be faked from filenames: trace every hop of a mission agent's tool +call from the guest hook to the recorded event, naming file, function and +whether each hop runs in the guest or on the host. It produced a 189-line +`research/GATE-MAP.md` describing code committed **the same day** — the +four-field `NODE_EXTRACT` including `agent_type`, the shadow-mode +`would-deny.jsonl` semantics, `install_with` — and **all six of its line +citations verify exactly** (`hook_script_with` 562, `NODE_EXTRACT` 789, +`TaskPolicy` 297, `ROLE_POLICIES` 261, `settings_hook` 809, +`install_command_with` 819). Nothing hallucinated. **Five of twelve.** + +**Three of the remaining seven are still unevidenced.** The other nine are well-formed scaffolding: roles, prompts, brain seeds and resolving skills, and no run behind any of them. They will probably work — they are structurally identical to the three that do — but "probably" is the word, and this codebase has a name for the gap @@ -88,6 +100,17 @@ blocked on a target, not on the template. The remaining four — `codebase_research`, `papers_research`, `continuous_improvement`, `backend` — could be exercised against repositories we already have. +## One thing both runs showed: agents reach for Bash + +Cumulative tool calls across every mission since the policy shipped: +**Bash 74, Read 34, Write 15, Glob 3, Edit 1, Agent 1**. A read-only +mapping mission that could have used `Grep` and `Glob` used `Bash` 33 +times and `Read` 5. That matters for task permission: the allowlist's +dedicated-tool entries (`Grep`, `ToolSearch`, `WebFetch`, `TodoWrite`) +may simply never be exercised, so "unconfirmed by a real run" will not +converge for them — and it means the surface that actually needs +governing is `Bash`, which the floor rules already cover. + ## What this means we can ask for today **With confidence:** a repo-backed research→code loop on a Rust project, with