osobh/clawmates
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(skill-use): a research phase writing markdown is not a TDD failure

Browse Source 
The first live scoring of run 3 reported `cargo-test-driven-development`
and `tdd-red-green-refactor` as compliance=FAIL: files were written and no
test ever ran.

Wrong, and wrong in the way this module exists to prevent. The phase wrote
fifteen markdown notes and a helper script; there was no code to
test-drive. Reporting it as an agent failure is a system defect wearing an
agent's name — and it would have buried the actual finding, which is that
a repo-less `research_only` mission is staffed with a Rust SDLC crew whose
coder, tester, reviewer and committer have nothing to do.

The check is now scoped to files with a source extension in the languages
the skill itself names. Shell is deliberately excluded: a helper script
written during a research turn is not behaviour-adding code, and the false
failure costs more than the missed one.

Recorded in SKILL-USE-BASELINE.md as finding 8 rather than quietly
corrected. A measurement that hides its own false positives cannot be
trusted about anyone else's.

Also in the doc: the Trigger reason is half false now (the transport can
surface a tool call; we simply still inline), and the architecture doc's
observe/gate table said the container tier was ungated and unobserved,
which shipped work has made wrong.

Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_018i9Ten1LU4jUr5d7TAWda9
This commit is contained in:
Omar Sobh
2026-08-21 08:37:15 -07:00
co-authored by Claude Opus 5
parent 4a6d0dfe01
commit c209e654d9
4 changed files with 259 additions and 126 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/cm-api/src/skill_use.rs
+52 -7
View File
@@ -420,10 +420,9 @@ fn red_before_green(ev: &Evidence<'_>) -> Verdict {
.into(), .into(),
); );
} }
let first_write = ev let first_write = ev.tools.iter().position(|t| {
.tools WRITE_TOOLS.contains(&t.tool.as_str()) && t.path.as_deref().is_some_and(is_source)
.iter() });
.position(|t| WRITE_TOOLS.contains(&t.tool.as_str()) && t.path.is_some());
let first_test = ev let first_test = ev
.tools .tools
.iter() .iter()
@@ -432,9 +431,11 @@ fn red_before_green(ev: &Evidence<'_>) -> Verdict {
// Nothing was written, so there was no implementation to test-drive. // Nothing was written, so there was no implementation to test-drive.
(None, _) => Verdict::NotApplicable, (None, _) => Verdict::NotApplicable,
(Some(_), None) => Verdict::Fail(format!( (Some(_), None) => Verdict::Fail(format!(
"wrote {} file(s) and never ran a test — the loop is red, green, \ "wrote {} source file(s) and never ran a test — the loop is red, \
refactor, and a test that never ran cannot have been red", green, refactor, and a test that never ran cannot have been red",
ev.writes().count() ev.writes()
.filter(|w| w.path.as_deref().is_some_and(is_source))
.count()
)), )),
(Some(w), Some(t)) if t < w => Verdict::Pass, (Some(w), Some(t)) if t < w => Verdict::Pass,
_ => Verdict::NotObservable( _ => Verdict::NotObservable(
@@ -446,6 +447,28 @@ fn red_before_green(ev: &Evidence<'_>) -> Verdict {
} }
} }
/// Is this file behaviour-adding code, in the languages the skill names?
///
/// The check is scoped to source because the first live run scored `fail`
/// against a **research** phase: its agents wrote fifteen markdown notes and
/// never ran a test, which is not a violation of anything — there was no code
/// to test-drive. Reporting that as an agent failure would be a system defect
/// wearing an agent's name, and the real finding it obscures is that a research
/// role is pinned TDD skills at all.
///
/// Extensions the skill itself names — "Rust, TypeScript, Python, anywhere
/// tests can run cheap" — plus their immediate neighbours. Prose, config and
/// data are excluded. Shell is excluded deliberately: a helper script written
/// during a research turn is not the behaviour-adding code this loop is about,
/// and the false failure costs more than the missed one.
fn is_source(path: &str) -> bool {
const SOURCE: [&str; 10] = [
".rs", ".ts", ".tsx", ".py", ".js", ".jsx", ".go", ".java", ".rb", ".kt",
];
let p = path.to_ascii_lowercase();
SOURCE.iter().any(|e| p.ends_with(e))
}
/// Commands that run a test suite, across the languages the skills name. /// Commands that run a test suite, across the languages the skills name.
fn is_test_command(cmd: &str) -> bool { fn is_test_command(cmd: &str) -> bool {
const RUNNERS: [&str; 9] = [ const RUNNERS: [&str; 9] = [
@@ -991,6 +1014,28 @@ mod tests {
} }
/// A phase that wrote nothing had no implementation to test-drive. /// A phase that wrote nothing had no implementation to test-drive.
/// The verdict the first live run got wrong.
///
/// A research phase wrote fifteen markdown notes and ran no test, and the
/// check called it a TDD failure. There was no code to test-drive. The
/// real finding is that a research role is pinned TDD skills at all, and
/// scoring the agent for it would have buried that.
#[test]
fn writing_prose_is_not_a_tdd_failure() {
let prompt = rendered(&[("tdd-red-green-refactor", "Red, green, refactor.")]);
let tools = acted(&[
("Write", Some("/mission/repo/research/notes.md"), json!({"file_path": "/mission/repo/research/notes.md"})),
("Edit", Some("/mission/repo/research/notes.md"), json!({"file_path": "/mission/repo/research/notes.md"})),
("Write", Some("/mission/repo/research/check.sh"), json!({"file_path": "/mission/repo/research/check.sh"})),
]);
assert_eq!(
score(&prompt, &Evidence::new("", &tools), &builtin)[0].compliance,
Verdict::NotApplicable,
"markdown and a helper script are not the behaviour-adding code \
this loop is about"
);
}
#[test] #[test]
fn a_phase_that_wrote_nothing_is_not_a_tdd_failure() { fn a_phase_that_wrote_nothing_is_not_a_tdd_failure() {
let prompt = rendered(&[("tdd-red-green-refactor", "Red, green, refactor.")]); let prompt = rendered(&[("tdd-red-green-refactor", "Red, green, refactor.")]);
docs/SKILL-USE-BASELINE.md
+179 -112
View File
@@ -1,7 +1,7 @@
# Skill-Use baseline # Skill-Use baseline
*First measurement of whether ClawMates' skills change what agents do. *Whether ClawMates' skills change what agents do. First measured 2026-08-19;
2026-08-19.* re-measured 2026-08-21 against tool evidence rather than agent prose.*
Scored on the three axes from `Skill-Use` (arXiv, 2026-08-05): **Trigger** (did Scored on the three axes from `Skill-Use` (arXiv, 2026-08-05): **Trigger** (did
the agent reach for the skill), **Compliance** (did it follow the procedure), the agent reach for the skill), **Compliance** (did it follow the procedure),
@@ -10,165 +10,232 @@ the agent reach for the skill), **Compliance** (did it follow the procedure),
Read the method before the numbers. A measurement whose limits are not stated Read the method before the numbers. A measurement whose limits are not stated
is worse than none, because it gets quoted without them. is worse than none, because it gets quoted without them.
## Why there was no baseline before today ## What changed on 2026-08-21
Not because nobody ran it. Because **it could not have returned anything but The 2026-08-19 measurement scored Compliance and Boundary from the
zero**, for two structural reasons that had nothing to do with agent behaviour: `reasoning` events — the agent's own account of its turn. Since then the
container tier records what agents actually **do**
(`container_tool_hooks`, `PostToolUse`), and `vm_tool_tap` stopped throwing the
tool's arguments away, so `Bash` commands and `Write` paths are on the record.
1. 55 of 85 role skill bindings resolved to skills that were never authored. `skill_use` now reads those. The difference is not cosmetic:
2. Even resolved skills had no delivery channel to a mission agent — the
catalogue's only route was an MCP server that mission claws cannot reach.
Both were fixed in the two commits preceding this document. Anyone who had run - `workspace-repo-commit-protocol`'s Boundary was a substring search for
this measurement in July would have concluded "our agents ignore their skills", `/workspace/repo` in the narrative. **An agent that wrote to the wrong root
which would have been false and expensive. without narrating it scored a clean pass.** It now reads the write paths.
- `arxiv-daily`'s Boundary read a URL in prose, which may be the agent
explaining that it did *not* fetch it. It now reads the `curl` that ran.
- `tdd-red-green-refactor`, `cargo-test-driven-development` and
`small-focused-commits` gained their first checks at all.
Two verdicts changed for honesty rather than coverage. **Silence used to score
`Pass`** — a mission with no evidence scored identically to one checked and
found clean; it is now `NotObservable`. And a test that ran *after* the first
write is `NotObservable`, not a failure, because a Rust unit test lives in the
file under test.
### Trigger: the reason changed, and only half of it went away
The 2026-08-19 document said Trigger was unobservable because `claude_cli`
"cannot surface a tool call — there is nothing to retrieve *with*."
**That half is now false.** Mission tool calls are recorded on both tiers; a
retrieval would be as visible as any other call.
The other half still holds and is the one that decides the verdict: **we still
inline**. `pinned_skills_text` puts full skill bodies in the prompt, so the
agent never reaches for anything — it is simply holding one. Trigger is now
*instrumentable* and still not *observable*, and the blocker has moved from the
transport to the delivery model.
Making it real is one change and no scorer work: serve skills through the door
(`TOOL-CALL-ARCHITECTURE.md` §3) so retrieval becomes a tool call.
## Method, and what it cannot see ## Method, and what it cannot see
Scored by `cm_api::skill_use` from what the platform records: the Scored by `cm_api::skill_use` from what the platform records: `prompt.composed`
`prompt.composed` event (the exact bytes an agent received) and the `reasoning` (the exact bytes an agent received), `reasoning` (what it said it did), and
events (what it said it did). No re-derivation from the catalogue — the `tool.call` (what it did). No re-derivation from the catalogue — the catalogue
catalogue changes, and now that agents author their own skills it changes by changes, and now that agents author their own skills it changes by itself.
itself.
### Trigger is not observable here, and that is a finding **Every tool-backed check is one-sided.** It reports a violation it can see and
never infers compliance from silence: the recorded stream is capped per phase
The paper measures agents under **progressive disclosure**: the agent sees a (`PER_PHASE_CAP = 400`), so an absent call is not proof of an absent action.
name and description and must decide to retrieve the body. That retrieval is a
tool call, which makes Trigger observable.
We do not deliver skills that way. `pinned_skills_text` inlines full bodies into
the prompt, because mission claws run on `claude_cli`, which cannot surface a
tool call — there is nothing to retrieve *with*. The agent never reaches for a
skill; it is simply holding one.
So Trigger is reported as `not_observable` with the reason attached, **never as
zero**. Scoring it zero would report a delivery-model property as an agent
failure — the same confusion that kept 55 empty bindings invisible.
### Compliance and Boundary are checked mechanically, or not at all
Only skills whose procedure has a machine-checkable consequence are scored. Only skills whose procedure has a machine-checkable consequence are scored.
Everything else returns `not_applicable` rather than a guess: a heuristic that Everything else returns `not_applicable` rather than a guess — a heuristic that
scores prose by keyword overlap produces a number that looks like a measurement scores prose by keyword overlap produces a number that looks like a measurement
and is not one. and is not one.
Compliance for `int-xx-marker-protocol` is checked by running the **real** Compliance for `int-xx-marker-protocol` is checked by running the **real**
`task_card_parser`, not a copy of its rules — a second implementation would `task_card_parser`, not a copy of its rules; a second implementation would
drift, and then the score would pass while the mission loop still stalled. drift, and then the score would pass while the mission loop still stalled.
## The runs ## The runs
Two missions on the container/ZeroClaw tier, local stack, `research_only`. Three missions on the container/ZeroClaw tier, local stack, `research_only`.
Run 3 uses the **same task text as run 2**, so the only variable is the scorer.
| | run 1 | run 2 | | | run 1 | run 2 | run 3 |
|---|---|---| |---|---|---|---|
| distinct skills delivered | 3 | **9** | | date | 08-19 | 08-19 | 08-21 |
| total deliveries (per role prompt) | 3 | 14 | | distinct skills delivered | 3 | 9 | 9 |
| phantom "skills" scored | **2** | 0 | | total deliveries (per role prompt) | 3 | 14 | 14 |
| phantom "skills" scored | **2** | 0 | 0 |
| tool calls recorded | 0 | 0 | **49** |
| axes scored from actions | 0 | 0 | **3 skills** |
Run 1's phantom entries are the finding of the run, described below. Run 3, per skill (all `source_kind=builtin`; no agent-authored skill has been
Run 2, per skill (all `source_kind=builtin`; no agent-authored skill has been
delivered yet): delivered yet):
| skill | deliveries | compliance | boundary | | skill | deliveries | compliance | boundary |
|---|---|---|---| |---|---|---|---|
| `int-xx-marker-protocol` | 1 | **pass** | n/a | | `int-xx-marker-protocol` | 1 | **pass** | n/a |
| `small-focused-commits` | 4 | n/a | n/a | | `workspace-repo-commit-protocol` | 2 | n/a | **pass** *(from 12 write paths)* |
| `small-focused-commits` | 4 | n/a | n/a *(no commit ran)* |
| `cargo-test-driven-development` | 2 | n/a | n/a | | `cargo-test-driven-development` | 2 | n/a | n/a |
| `workspace-repo-commit-protocol` | 2 | n/a | n/a | | `tdd-red-green-refactor` | 1 | n/a | n/a |
| `decompose-int-items` | 1 | n/a | n/a | | `decompose-int-items` | 1 | n/a | n/a |
| `write-rust-current-edition` | 1 | n/a | n/a | | `write-rust-current-edition` | 1 | n/a | n/a |
| `code-review-checklist` | 1 | n/a | n/a | | `code-review-checklist` | 1 | n/a | n/a |
| `criterion-benchmarking` | 1 | n/a | n/a | | `criterion-benchmarking` | 1 | n/a | n/a |
| `tdd-red-green-refactor` | 1 | n/a | n/a |
**n = 2 runs. No spread is reported because two runs cannot establish one.** **n = 3 runs. No spread is reported because three cannot establish one.** This
This is a baseline in the sense of "the first honest number", not in the sense is a baseline in the sense of "the first honest number", not in the sense of
of `metrics-baseline-comparison.md`, which requires enough runs to see the noise `metrics-baseline-comparison.md`, which requires enough runs to see the noise
floor before any change is judged against it. Do not compare a future number to floor before any change is judged against it.
this one without first establishing that floor.
`workspace-repo-commit-protocol`'s pass is the one score that materially
improved: it now rests on twelve recorded `Write`/`Edit` paths, every one under
`/mission/repo`, instead of on the absence of a string in prose.
## What the measurement found ## What the measurement found
Four defects, none of which any test or log would have surfaced. ### 1–4: the 2026-08-19 findings
### 1. The prompt format made its own record unparseable Four defects, none of which any test or log would have surfaced: the prompt
format made its own record unparseable (`## <name>` against markdown bodies);
a prompt was recorded that was never sent; a pinned skill taught
`/workspace/repo`, a path the platform does not mount; and
`int-xx-marker-protocol` documented a `PLAN_COMPLETE` marker the parser had
never implemented. All four are fixed, with guards in
`skills_loader::contradiction_tests` and `topology_exec`. The detail is in this
file's git history.
Skills were introduced with `## <name>`, and skill bodies are markdown full of ### 5. The tool tap recorded the name and discarded the argument
`##` headings. Run 1 duly scored **"Sizing heuristic"** and **"The output
shape"** — both subheadings inside `decompose-int-items` — as skills with no
catalogue row.
Fixed with an unambiguous `--- SKILL: <name> ---` marker, and both writers now The first container-tier mission with telemetry recorded `Bash × 6` and not one
share one renderer so the reader cannot drift from the writer. of them said what it ran. `vm_tool_tap::parse` read `tool_input` to pull the
path out of it and dropped the rest.
### 2. A prompt was recorded that was never sent Every behavioural question was therefore unanswerable from a record that looked
complete — which is the recurring shape, not a new one. Fixed host-side: the
arguments were always in the tap file.
The phase prompt was recorded at the dispatch fork, before the tier was chosen. ### 6. Two more skills contradicted the platform
The container tier does not send that text — it sends the bare task and appends
skills per turn. So every container mission logged a `solo` prompt that reached
no agent.
A provenance record of something that did not happen is worse than no record: it Same class as finding 3, and both found by reading the source of truth before
is the wrong answer, delivered confidently. Recording now happens inside each writing a check against it.
tier, and a test asserts every launcher records the prompt it actually sends.
### 3. A pinned skill contradicted the platform in the same prompt - **`decompose-int-items` taught `PLAN_COMPLETE: INT-01..05`.** An id is
strictly `INT-` plus digits, so the range form is rejected outright: the plan
pass records nothing while every item stays open. A live planner emitted
exactly that line.
- **`workspace-repo-commit-protocol` claimed the task-card parser advances
mission state on the INT id in your commit subject.** Nothing in the platform
reads commit messages. `task_card_parser::apply_for_run` reads `run_events` —
the agent's turn output. An agent that believed this would commit with the id,
never emit `COMPLETED: INT-NN`, and leave the mission open on an item it had
already finished.
`workspace-repo-commit-protocol` told agents that **`/workspace/repo`** was "the `no_skill_shows_a_marker_the_parser_would_reject` now runs the real parser over
ONLY path where source-modifying edits belong". The platform mounts and every marker in every skill's fenced blocks, negative-controlled against the
advertises **`/mission/repo`** — 26 references in the code; `/workspace/repo` range form.
appears in none. The skill is bound on **29 role bindings** and was delivered
twice in run 2, so agents received the real path in the tool preamble and a
skill contradicting it a few hundred tokens later.
It also instructed `file_read` / `file_write` / `shell` — ZeroClaw's tool names, ### 7. A repo-less research mission is staffed with a Rust SDLC crew
the exact ones `phase_task_text` was fixed to stop advertising after five agents
on one mission spent 7.4k tokens describing the mismatch instead of working.
An agent that obeyed this skill wrote source into a directory nothing collects, This is the finding of run 3, and it explains most of the `not_applicable`
and reached for tools its subprocess does not expose. Rewritten against what the column above.
code actually does, with two guards in `skills_loader::contradiction_tests`: no
skill may name a repo path the platform does not mount, and none may instruct a
tool the agent does not have. Both negative-controlled.
This is the same shape as the finding below and it is worth stating as a class: `templates/workflows/research_only.toml` declares `requires_repo = false` and a
**the skills were never checked against the platform they describe.** Nothing single `research` phase — and `default_team_template = "rust_sdlc"`. So the
compared them, so a skill could contradict the prompt it ships inside and stay mission was staffed with **planner, coder, tester, reviewer, committer**, and
that way indefinitely. each received the skills its role is bound to:
### 4. The skill documents a marker the platform never implemented ```
coder :: write-rust-current-edition, cargo-test-driven-development,
workspace-repo-commit-protocol, small-focused-commits,
int-xx-marker-protocol
tester :: cargo-test-driven-development, criterion-benchmarking,
tdd-red-green-refactor
committer :: workspace-repo-commit-protocol, small-focused-commits
reviewer :: code-review-checklist, small-focused-commits
planner :: decompose-int-items, small-focused-commits
```
`int-xx-marker-protocol` lists `PLAN_COMPLETE: INT-NN` in its ladder. There is no repository, nothing to test, nothing to review and nothing to
`task_card_parser` has **no such kind** and never has. An agent following the commit. Four of the five roles have no work, and 50KB of prompt (~12.6k tokens)
skill exactly emits a marker that is silently ignored. is spent staffing them.
Observed live: run 2's planner emitted `PLAN_COMPLETE: INT-01..02`, which is **The skills are correctly bound to the roles. The roles are wrong for the
also the range form — on the kinds that *are* parsed, that yields the id workflow.** That distinction matters: a reader who saw only "7 of 9 skills
`INT-01..02`, a task card for an item that does not exist while the two real scored not_applicable" would conclude the skills are useless, when what the
items stay open. number actually measures is a staffing default.
**This is a skill/implementation mismatch, not an agent failure**, and it is `continuous_research` names its own team; the other four recipes all default to
precisely what this measurement exists to find: the agent did what it was told, `rust_sdlc`. `research_only` has no correct existing template to point at —
and what it was told was wrong. Both shapes are now scored as failures; the `papers_research` is arXiv-shaped, `insight_research` is vault-shaped, and
underlying reconciliation — implement `PLAN_COMPLETE` or remove it from the `codebase_research` needs a repo — so the fix is an operator decision, not a
skill — is deliberately left as a decision rather than guessed at here. one-line edit, and is deliberately left open.
### 8. The check that got it wrong first
Run 3's first scoring reported `cargo-test-driven-development` and
`tdd-red-green-refactor` as **compliance = fail**: files were written and no
test ever ran.
That verdict was wrong, and wrong in the way this whole document exists to
prevent. The phase wrote fifteen markdown notes and a helper script. There was
no code to test-drive. Reporting it as an agent failure would have been a system
defect wearing an agent's name — and it would have buried the real finding,
which is finding 7 above.
The check is now scoped to files with a source extension in the languages the
skill itself names. It is recorded here rather than quietly corrected, because
a measurement that hides its own false positives cannot be trusted about
anyone else's.
## Honest limits ## Honest limits
- **Two runs, one tier, one workflow.** Nothing here generalises to the microVM - **Three runs, one tier, one workflow.** Nothing here generalises to the
or session tiers yet, and this document should not be read as if it does. microVM or session tiers.
- **Most skills score `not_applicable`** on both observable axes. That is not a - **The new checks are not yet exercised live.** `research_only` writes no code
pass. It means we cannot currently tell whether those skills changed anything. and makes no commits, so the TDD and commit checks are proven by unit tests
`workspace-repo-commit-protocol` now has a Boundary check (writing outside and negative controls, not by a mission that could have violated them. A
`/mission/repo`); `small-focused-commits` and `tdd-red-green-refactor` remain coding run against a real repository is the next measurement.
candidates, and both need the repository diff rather than the turn text. - **Tool calls carry no agent attribution.** `record_vm_tools` writes
- **No agent-authored skill has been measured.** Self-authoring shipped in the `agent_id: None` — the container tier's tap is per-container, and all five
same pass; `source_kind` is carried through the scorer specifically so a roles share one container. Every score above is therefore per-**mission**,
rising score on agent-authored skills is visible rather than averaged in. not per-role, and the World's per-agent view gets nothing from it. Mapping
the hook payload's `session_id` back to a turn would fix it.
- **Most skills still score `not_applicable`** on both observable axes. That is
not a pass. See finding 7 for why the number is what it is.
- **No agent-authored skill has been measured.** `source_kind` is carried
through the scorer specifically so a rising score on agent-authored skills is
visible rather than averaged in.
- **Evidence expires.** Mission events are reaped after 7 days unless - **Evidence expires.** Mission events are reaped after 7 days unless
`retain_events_until` is set. Both runs here are held for 90 days. An empty `retain_events_until` is set; `scripts/skill-use-run.sh` holds every run for
score means "no evidence", never "no compliance", and the API says so in its 90 days so it stays re-scorable when the scorer changes again — which is
payload rather than leaving the caller to infer it. exactly what happened to run 3. An empty score means "no evidence", never "no
compliance", and the API says so in its payload.
## Reproducing
```
scripts/skill-use-run.sh "<title>" "<task>" # run and score
scripts/skill-use-run.sh --score <mission-id> # re-score, no new run
```
Local stack only. Production auth is Clerk and a mission cannot be launched
from a terminal there — which is also why, as of 2026-08-21, **production has
never run a mission at all** (`select count(*) from missions` → 0).
docs/TOOL-CALL-ARCHITECTURE.md
+19 -4
View File
@@ -26,17 +26,32 @@ a security posture, and it is the one we have.
## Observe versus gate — they are different, and both are partial ## Observe versus gate — they are different, and both are partial
As of 2026-08-21, with the container-tier hooks shipped:
| Path | Has tools | We observe | We gate | | Path | Has tools | We observe | We gate |
|---|---|---|---| |---|---|---|---|
| Solo microVM | yes | **yes** — `vm_tool_tap` | no | | Solo microVM | yes | **yes** — `vm_tool_tap` | **yes** — `vm_tool_gate` |
| Composed microVM | yes | **yes** — same tap | no | | Composed microVM | yes | **yes** — same tap | **yes** — same gate |
| Direct session | yes | **no mechanism at all** | no | | Direct session | yes | **no mechanism at all** | no |
| Container / ZeroClaw | yes (see below) | mechanism exists, receives nothing | no | | Container / ZeroClaw | yes (see below) | **yes** — `container_tool_hooks` | **yes** — same |
The direct-session row is the only remaining gap, and it is dormant:
`CLAWMATES_MISSION_EXECUTOR` is unset in production, so that tier never runs.
Checking that before building a tap for it is the reason there is no tap for it.
`vm_tool_tap` installs a **`PostToolUse`** hook, which fires *after* the tool has `vm_tool_tap` installs a **`PostToolUse`** hook, which fires *after* the tool has
already run, and `exit 0`s unconditionally because a non-zero `PostToolUse` already run, and `exit 0`s unconditionally because a non-zero `PostToolUse`
talks back to the model. It is telemetry and says so. It is structurally talks back to the model. It is telemetry and says so. It is structurally
incapable of gating. incapable of gating — which is why the gate is a separate `PreToolUse` hook
rather than a stricter version of this one.
**What the tap records.** Until 2026-08-21 it kept the tool's name and the path
its arguments named, and threw the arguments themselves away. A phase that
recorded `Bash × 6` could not answer whether it ran the tests, whether it
committed, or whether it called an API a skill forbids. It now keeps the
arguments, bounded: file bodies become a byte count, over-long strings are
truncated with a marker. That is what makes `skill_use` able to score behaviour
rather than the agent's own account of it.
The §15 `GatePolicy` has exactly **one** enforcement site — `Runtime::drive`, The §15 `GatePolicy` has exactly **one** enforcement site — `Runtime::drive`,
the chat loop — and its approvals are keyed to `(session_id, message_id)`, which the chat loop — and its approvals are keyed to `(session_id, message_id)`, which
scripts/skill-use-run.sh
+9 -3
View File
@@ -18,6 +18,9 @@
# PG local postgres ctr (default clawmates-postgres-1) # PG local postgres ctr (default clawmates-postgres-1)
# OWNER account to mint for (default [email protected]) # OWNER account to mint for (default [email protected])
# TEMPLATE workflow recipe (default research_only) # TEMPLATE workflow recipe (default research_only)
# REPO_ID repository to check out; required by the coding recipes, and
# the only way the TDD and commit checks can ever fire — a
# repo-less run writes markdown and commits nothing
# TIMEOUT seconds to wait (default 1800) # TIMEOUT seconds to wait (default 1800)
# RETAIN_DAYS hold events this long so the run stays re-scorable (default 90) # RETAIN_DAYS hold events this long so the run stays re-scorable (default 90)
@@ -87,13 +90,16 @@ fi
TITLE="${1:?title}" TITLE="${1:?title}"
TASK="${2:?task description}" TASK="${2:?task description}"
body=$(python3 - "$TITLE" "$TASK" "$TEMPLATE" <<'PY' body=$(python3 - "$TITLE" "$TASK" "$TEMPLATE" "${REPO_ID:-}" <<'PY'
import json, sys import json, sys
print(json.dumps({ req = {
"title": sys.argv[1], "title": sys.argv[1],
"description": sys.argv[2], "description": sys.argv[2],
"template_kind": sys.argv[3], "template_kind": sys.argv[3],
})) }
if sys.argv[4]:
req["repo_id"] = sys.argv[4]
print(json.dumps(req))
PY PY
) )