docs: codebase_research evidenced — 5 of 12, and its citations check out
Run against the real clawmates repo with a task that cannot be faked from filenames: trace every hop of a mission agent's tool call from the guest hook to the recorded event, naming file, function, and guest vs host for each. It produced a 189-line research/GATE-MAP.md describing code committed the SAME DAY — the four-field NODE_EXTRACT including agent_type, the shadow-mode would-deny.jsonl semantics, install_with — and all six of its line citations verify exactly. Nothing hallucinated. That is a stronger result than the judge's verdict, which could only check that the structure was present. Also recorded: agents reach for Bash. 74 Bash calls against 34 Read, 3 Glob, 1 Edit, 1 Agent — a mapping mission that could have used Grep used Bash 33 times. So the allowlist's dedicated-tool entries may never be exercised, and the surface that actually needs governing is Bash, which the floor rules already cover. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01WZb5A2kfVfjpdwSochkuHz
This commit is contained in:
co-authored by
Claude Opus 5
parent
b1a23727da
commit
a88d51b078
@@ -62,12 +62,12 @@ holding (it was 55 of 85 dangling once).
|
|||||||
| `mobile` | designer coder tester committer | coding_readwrite | 11 | no |
|
| `mobile` | designer coder tester committer | coding_readwrite | 11 | no |
|
||||||
| `gpu` | arch_analyst kernel_author bench_engineer coder committer | coding_readwrite | 13 | no |
|
| `gpu` | arch_analyst kernel_author bench_engineer coder committer | coding_readwrite | 13 | no |
|
||||||
| `threejs` | scene_designer coder shader_author perf_engineer committer | coding_readwrite | 13 | no |
|
| `threejs` | scene_designer coder shader_author perf_engineer committer | coding_readwrite | 13 | no |
|
||||||
| `codebase_research` | code_archeologist architecture_mapper flow_tracer vault_scribe | research_readonly | 11 | no |
|
| `codebase_research` | code_archeologist architecture_mapper flow_tracer vault_scribe | research_readonly | 11 | **yes** — first run 2026-09-22 |
|
||||||
| `papers_research` | domain_scout paper_reader library_curator | research_web_readonly | 8 | no |
|
| `papers_research` | domain_scout paper_reader library_curator | research_web_readonly | 8 | no |
|
||||||
| `insight_research` | implementation_tracker novelty_hunter publication_drafter | research_readonly | 8 | no |
|
| `insight_research` | implementation_tracker novelty_hunter publication_drafter | research_readonly | 8 | no |
|
||||||
| `continuous_improvement` | brain_inspector improvement_proposer improvement_evaluator | research_readonly | 7 | no |
|
| `continuous_improvement` | brain_inspector improvement_proposer improvement_evaluator | research_readonly | 7 | no |
|
||||||
|
|
||||||
**Four of twelve are evidenced.** `backend` was exercised on 2026-09-22,
|
**Five of twelve are evidenced.** `backend` was exercised on 2026-09-22,
|
||||||
the first time anything had been staffed from it: all five roles
|
the first time anything had been staffed from it: all five roles
|
||||||
provisioned with real agents (api_designer, db_engineer, coder, tester,
|
provisioned with real agents (api_designer, db_engineer, coder, tester,
|
||||||
committer), and the mission delivered cursor pagination — `Paged<T>`,
|
committer), and the mission delivered cursor pagination — `Paged<T>`,
|
||||||
@@ -75,7 +75,19 @@ committer), and the mission delivered cursor pagination — `Paged<T>`,
|
|||||||
**met on the first pass**, 2 files pushed. It also gave the task-permission
|
**met on the first pass**, 2 files pushed. It also gave the task-permission
|
||||||
shadow its first confirmed `Edit` call.
|
shadow its first confirmed `Edit` call.
|
||||||
|
|
||||||
**Three of the remaining eight are still unevidenced.** The other nine are well-formed scaffolding:
|
`codebase_research` was exercised the same day, against the real
|
||||||
|
`clawmates` repository rather than the toy scratch crate, with a task that
|
||||||
|
cannot be faked from filenames: trace every hop of a mission agent's tool
|
||||||
|
call from the guest hook to the recorded event, naming file, function and
|
||||||
|
whether each hop runs in the guest or on the host. It produced a 189-line
|
||||||
|
`research/GATE-MAP.md` describing code committed **the same day** — the
|
||||||
|
four-field `NODE_EXTRACT` including `agent_type`, the shadow-mode
|
||||||
|
`would-deny.jsonl` semantics, `install_with` — and **all six of its line
|
||||||
|
citations verify exactly** (`hook_script_with` 562, `NODE_EXTRACT` 789,
|
||||||
|
`TaskPolicy` 297, `ROLE_POLICIES` 261, `settings_hook` 809,
|
||||||
|
`install_command_with` 819). Nothing hallucinated. **Five of twelve.**
|
||||||
|
|
||||||
|
**Three of the remaining seven are still unevidenced.** The other nine are well-formed scaffolding:
|
||||||
roles, prompts, brain seeds and resolving skills, and no run behind any of
|
roles, prompts, brain seeds and resolving skills, and no run behind any of
|
||||||
them. They will probably work — they are structurally identical to the three
|
them. They will probably work — they are structurally identical to the three
|
||||||
that do — but "probably" is the word, and this codebase has a name for the gap
|
that do — but "probably" is the word, and this codebase has a name for the gap
|
||||||
@@ -88,6 +100,17 @@ blocked on a target, not on the template. The remaining four —
|
|||||||
`codebase_research`, `papers_research`, `continuous_improvement`, `backend` —
|
`codebase_research`, `papers_research`, `continuous_improvement`, `backend` —
|
||||||
could be exercised against repositories we already have.
|
could be exercised against repositories we already have.
|
||||||
|
|
||||||
|
## One thing both runs showed: agents reach for Bash
|
||||||
|
|
||||||
|
Cumulative tool calls across every mission since the policy shipped:
|
||||||
|
**Bash 74, Read 34, Write 15, Glob 3, Edit 1, Agent 1**. A read-only
|
||||||
|
mapping mission that could have used `Grep` and `Glob` used `Bash` 33
|
||||||
|
times and `Read` 5. That matters for task permission: the allowlist's
|
||||||
|
dedicated-tool entries (`Grep`, `ToolSearch`, `WebFetch`, `TodoWrite`)
|
||||||
|
may simply never be exercised, so "unconfirmed by a real run" will not
|
||||||
|
converge for them — and it means the surface that actually needs
|
||||||
|
governing is `Bash`, which the floor rules already cover.
|
||||||
|
|
||||||
## What this means we can ask for today
|
## What this means we can ask for today
|
||||||
|
|
||||||
**With confidence:** a repo-backed research→code loop on a Rust project, with
|
**With confidence:** a repo-backed research→code loop on a Rust project, with
|
||||||
|
|||||||
Reference in New Issue
Block a user