osobh/clawmates
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

test(skill-use): the coding run, and the parsing bug it found

Browse Source 
Run 4 (`research_and_code`, real repo) is the first mission that could
have violated the TDD and commit checks. It exercised both, and found a
bug in one.

Claude Code writes a multi-line commit message as a heredoc inside a
command substitution:

    git commit -m "$(cat <<'EOF'
    INT-01 Add slugify function to src/lib.rs
    …
    EOF
    )"

`commit_subjects` read the first line of the `-m` value, which is the
heredoc OPENER. Every commit check was scoring `$(cat <<'EOF'` — a string
the agent never wrote. It reported no violation only because that string
is not one of the never-merge messages, which is luck rather than a check.
Regression test built from the exact command in `mission_events`.

The TDD verdict came back `not_observable`, which is the honest answer and
also a real limit worth stating: the agents edited `src/lib.rs` once —
implementation and `#[cfg(test)] mod tests` in the same write — then ran
`cargo test` five times. In Rust the unit test lives in the file under
test, so that ordering is exactly what following the skill precisely looks
like from outside. The check detects "wrote source, never ran a test" and
cannot confirm red-first. Confirming it needs the diff, not the tool order.

Every one of run 4's 33 tool calls stayed inside /mission/repo.

Handoff and baseline updated: production has never run a mission (both
tables empty), a mission container has leaked since 2026-08-12 that no
reaper can see, and `research_only` staffs a five-role Rust SDLC crew on a
repo-less markdown mission — which is what "most skills score
not_applicable" has been measuring all along.

Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_018i9Ten1LU4jUr5d7TAWda9
This commit is contained in:
Omar Sobh
2026-08-21 08:43:37 -07:00
co-authored by Claude Opus 5
parent c209e654d9
commit 9560aaec41
3 changed files with 264 additions and 142 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/cm-api/src/skill_use.rs
+53 -2
View File
@@ -499,11 +499,39 @@ fn commit_subjects(ev: &Evidence<'_>) -> Vec<String> {
ev.commands() ev.commands()
.filter(|c| c.contains("git commit") || c.contains("git ci")) .filter(|c| c.contains("git commit") || c.contains("git ci"))
.filter_map(dash_m_value) .filter_map(dash_m_value)
.filter_map(|m| m.lines().next().map(|l| l.trim().to_string())) .filter_map(|m| subject_line(&m))
.filter(|s| !s.is_empty())
.collect() .collect()
} }
/// The subject out of a `-m` value.
///
/// Normally the first line. But Claude Code writes a multi-line message as
///
/// ```text
/// git commit -m "$(cat <<'EOF'
/// INT-01 Add slugify function
///
/// …body…
/// EOF
/// )"
/// ```
///
/// and the first line of that value is `$(cat <<'EOF'` — the heredoc *opener*,
/// not the subject. Observed on the first live coding run; it scored no
/// violation only because `$(cat <<'EOF'` happens not to be one of the
/// never-merge messages, which is luck, not a check.
///
/// So: if the first line opens a heredoc or a command substitution, the subject
/// is the next non-empty line.
fn subject_line(msg: &str) -> Option<String> {
let mut lines = msg.lines().map(str::trim).filter(|l| !l.is_empty());
let first = lines.next()?;
if first.starts_with("$(") || first.contains("<<") {
return lines.next().map(str::to_string);
}
Some(first.to_string())
}
/// The value of the `-m` flag in a shell command. /// The value of the `-m` flag in a shell command.
/// ///
/// Hand-scanned rather than split on whitespace: the value is the one argument /// Hand-scanned rather than split on whitespace: the value is the one argument
@@ -1080,6 +1108,29 @@ mod tests {
assert!(dash_m_value("git commit -F /tmp/msg").is_none()); assert!(dash_m_value("git commit -F /tmp/msg").is_none());
} }
/// The exact commit command the first live coding run ran.
///
/// Claude Code writes a multi-line message as a heredoc inside a command
/// substitution, so the first line of the `-m` value is the heredoc opener
/// and the subject is the line after it. Read from
/// `mission_events`, not invented.
#[test]
fn a_heredoc_commit_message_gives_up_its_real_subject() {
let live = "git add src/lib.rs && git commit -m \"$(cat <<'EOF'\n\
INT-01 Add slugify function to src/lib.rs\n\n\
The crate exposed only add() with no string-normalization utility.\n\n\
Refs: INT-01\n\
EOF\n\
)\"";
assert_eq!(
commit_subjects(&Evidence::new("", &acted(&[ran(live)]))),
vec!["INT-01 Add slugify function to src/lib.rs"],
"the heredoc OPENER is not the subject — reading it as one puts \
`$(cat <<'EOF'` in the record, and every commit check then scores \
a string the agent never wrote"
);
}
/// The skill tells the agent to `curl` the abstract page in the paragraph /// The skill tells the agent to `curl` the abstract page in the paragraph
/// under the one that forbids the query API. A check that cannot tell them /// under the one that forbids the query API. A check that cannot tell them
/// apart fails agents for obeying the skill. /// apart fails agents for obeying the skill.
docs/NEXT-SESSION.md
+139 -124
View File
@@ -1,177 +1,192 @@
# Where this left off — 2026-08-21 # Where this left off — 2026-08-21 (second pass)
Read `CAPABILITY-REVIEW.md` for the system picture, Read `CAPABILITY-REVIEW.md` for the system picture,
`TOOL-CALL-ARCHITECTURE.md` for how mission tools actually work (and the two `TOOL-CALL-ARCHITECTURE.md` for how mission tools actually work, and
wrong theories that preceded it), and `SKILL-USE-BASELINE.md` for the `SKILL-USE-BASELINE.md` for the measurement — which now scores behaviour rather
measurement — noting that its Trigger column is now out of date in a good way than the agent's own account of it.
(see "Next, in order" §2).
## State of the tree ## State of the tree
Everything is pushed. `main` is at `5a11fae`; the fork's Local suite green: **429 lib tests**, workspace builds with `--all-targets`.
`merge/upstream-v0.8.4` is at `be9c34b1c`. Local suite green: **107 test Four measurement missions ran on the local stack (`scripts/skill-use-run.sh`);
binaries, 412 lib tests**, frontend builds. all four are held 90 days and re-scorable with `--score <id>`.
Seven commits on `main` this pass, **not pushed** — a push to `main` auto-deploys
to gw-04, and the container-tier work already deployed is unexercised there
(see below).
**CI is green and production is current.** Run 507 passed `test` and `build`, ## The premise of the last handoff's item 1 was wrong
and gw-04 rolled to it. Production also runs `clawmates-runtime:hooks` for
per-mission containers (see below).
## Container-tier tool gate + telemetry — SHIPPED 2026-08-21 It said the container-tier gate and tap were "proven locally and unproven in
prod", and told you to watch for the first production mission.
The tier that actually runs missions now has both, verified end to end on a **Production has never run a mission.**
real mission (locally first, then deployed):
``` ```
tool.call 10 Bash 6, Read 3, Write 1 gw-04$ select count(*) from missions; -> 0
file.touch 4 research/tapproof.md gw-04$ select count(*) from mission_events; -> 0
``` ```
That is the first time the container tier has ever been observable. Prod is armed correctly — server restarted with
`CLAWMATES_RUNTIME_IMAGE=clawmates-runtime:hooks`, image present. There is
simply nothing to watch. Prod auth is Clerk, so a mission cannot be launched
from a terminal; someone has to click. Generalise the lesson: before debugging
why a deployed thing shows no evidence, check whether anything ran.
**How, after `stream-json` failed.** Claude Code runs its tools inside its own ### A leaked mission container nothing can reap
subprocess, so they never reach ZeroClaw's executor and never become a
`TurnEvent::ToolCall`. Hooks bypass that entirely: `claude -p --settings <doc>`
honours `PreToolUse` and `PostToolUse`, so the gate blocks and the tap records
without ZeroClaw being involved at all.
Pieces: `--settings` on `claude_cli` (fork `be9c34b1c`), `cm-runtime-mission-019ff5b157ce77028f308ebd3dc92748` has been `Up` since
`container_tool_hooks` writes both hook scripts and one settings document into **2026-08-12** on `clawmates-runtime:sync`, with no `missions` row behind it.
the mission container, `set_claude_cli_settings` points the provider at it, and
`phase_runner::drain_finished_container_phases` collects the tap into
`mission_events` (idempotent by truncation — no cursor column).
**Production state:** server on `f6e6037`; `mission_runtime`'s terminal sweeper selects `FROM missions WHERE status IN
`CLAWMATES_RUNTIME_IMAGE=clawmates-runtime:hooks` in `/opt/clawmates/.env` (…)`, and `teardown_container(mission_id)` is only ever called with an id from
(backup at `.env.bak.prehooks`; rollback = restore it and recreate). Note that that query. Nothing enumerates Docker for `cm-runtime-mission-*` containers with
host uses **legacy `docker-compose`**, not the v2 plugin. no matching row, so a container whose row is gone is invisible to every reaper.
Same shape as the earlier agent-container reap drift, different table.
**Not yet observed in production** — no prod mission has run since the flip. It holds disk on the host whose disk exhaustion broke CI runs 503–506. **Not
Prod auth is Clerk, so a mission cannot be launched from here by password. The fixed — your call**, and the fix is small: one sweep that lists containers by
check when one runs: name prefix and reaps those with no row.
``` ## What shipped this pass
ssh gw-04 'docker logs clawmates_server_1 2>&1 | grep -E "per-mission runtime image|drained .* tool call"'
```
### Three bugs the live test found, all the same shape ### The tool tap kept the name and discarded the argument
Each left every other link looking correct: The container tier's first measured mission recorded `Bash × 6` and not one of
them said what it ran. `vm_tool_tap::parse` read `tool_input` to pull the path
out of it and dropped the rest, so every behavioural question about a phase was
unanswerable from a record that looked complete.
1. The settings document pointed `PostToolUse` at a path the installer never `Observed.input` now keeps it, bounded: file bodies become a byte count, other
wrote. Claude Code does not complain about a missing hook command — it long strings truncate with a marker. **Host-side only, no image rebuild** — the
records nothing. A test now compares the document's commands against the arguments were always in the tap file. `tool.call` also gained `detail.path`
files the installer creates. (the World's SSE reads it and had been getting null on every container-tier
2. The mission container uses `CLAWMATES_RUNTIME_IMAGE`, not the shared call) and `file.touch` gained `detail.abs`.
`clawmates-runtime` container — it was on an older image whose daemon schema
had no `settings` field, so the prop write returned `404 path_not_found`.
3. The drain used `connect_with_local_defaults()`; the server reaches Docker
through a **socket proxy**, so it failed — and returned `Ok(())` silently.
### CI: build failures were disk, not code ### Skill-Use is scored from actions
Runs 503–506 failed at `build` with an unreadable log, and the first casualty `skill_use::Evidence` carries `tool.call` rows alongside the narrative, and
was a **docs-only** commit. Cause: building runtime images by hand on gw-04 every check prefers them. `workspace-repo-commit-protocol`'s boundary was a
competes with CI for the same 150G volume; the frontend image build lost. substring search for `/workspace/repo` in prose — an agent that wrote to the
`docker builder prune` reclaimed 34GB (22G → 57G free) and the next run went wrong root **without narrating it scored a clean pass**. Two verdicts changed
green. The build job now writes breadcrumbs, a `df -h` snapshot, and which for honesty: silence is `NotObservable` rather than `Pass`, and a test that ran
services actually pushed — the failing runs had pushed `server`, aborted on after the first write is undecidable rather than a failure.
`frontend`, and left `:latest` unmoved, which surfaced three steps later as
"the deploy did not happen".
**Operational note:** do not build images by hand on gw-04 while CI may run. **Trigger is still `NotObservable`, and half of its old reason is now wrong.**
"`claude_cli` cannot surface a tool call" is false. What still holds is that we
**inline** skill bodies, so there is no retrieval to observe. The blocker moved
from the transport to the delivery model, and the door (§3) closes it with no
scorer change at all.
### Two more skills contradicted the platform
Both found by reading the source of truth before writing a check against it —
which is the only reason they were found.
- `decompose-int-items` taught `PLAN_COMPLETE: INT-01..05`. Ids are strictly
`INT-<digits>`, so the range form is rejected and the plan pass records
nothing while every item stays open.
- `workspace-repo-commit-protocol` claimed the task-card parser advances mission
state on the INT id in your commit subject. **Nothing in the platform reads
commit messages** — `apply_for_run` reads `run_events`, the turn output.
`no_skill_shows_a_marker_the_parser_would_reject` guards the class, running the
real parser over every marker in every skill's fenced blocks.
## Next, in order ## Next, in order
1. **Watch the first production mission.** Nothing has run since the runtime 1. **`research_only` staffs a Rust SDLC crew, and it is your decision.**
flip, so the container-tier gate and tap are proven locally and unproven in The recipe declares `requires_repo = false`, one `research` phase, and
prod. Prod auth is Clerk, so a mission cannot be launched from here. `default_team_template = "rust_sdlc"` — so a repo-less markdown mission gets
``` planner, coder, tester, reviewer and committer, four of whom have nothing to
ssh gw-04 'docker logs clawmates_server_1 2>&1 | grep -E "per-mission runtime image|drained .* tool call"' do, at ~50KB of prompt. This is why most skills score `not_applicable`: the
ssh gw-04 'docker exec <cm-runtime-mission-…> cat /root/toolhooks/tap/tools.jsonl | head' skills are correctly bound to the roles, and the **roles are wrong for the
``` workflow**. No existing template fits — `papers_research` is arXiv-shaped,
If tools appear in `mission_events`, the loop is closed. If not, check the `insight_research` vault-shaped, `codebase_research` needs a repo — so this
three failure shapes listed above — each looked correct from every other is a new template or a recipe change, not a one-liner. `benchmark`,
angle. `refactor`, `research_and_code` and `security_hardening` all default to
`rust_sdlc` too; check whether each is right.
2. **Now that tool calls are observable, redo the Skill-Use measurement.** 2. **The TDD check cannot confirm red-first, and that is structural.** Run 4
`docs/SKILL-USE-BASELINE.md` reports Trigger as `not_observable` on the (`research_and_code`, real repo) edited `src/lib.rs` once — implementation
container tier because there was no tool evidence. There is now. *and* `#[cfg(test)] mod tests` in the same write — then ran `cargo test`
`mission_events` carries `tool.call` and `file.touch` per phase, so five times. In Rust the unit test lives in the file under test, so that
Trigger can be scored from behaviour instead of prose — which is what the ordering is what following the skill precisely looks like from outside. The
paper actually measures. This is the single highest-value follow-up: it check detects "wrote source, never ran a test" and nothing more. If you want
turns the baseline from "the first honest number" into a real one. red-first, it needs the diff (did the test exist before the impl?), not the
tool order.
3. **Give the microVM tier the same treatment, or retire the difference.** It 3. **Attribute tool calls to agents.** `record_vm_tools` writes
has the gate and a tap already, but by a different route (`vm_tool_tap` `agent_id: None`, because the container tap is per-container and all roles
installs into the guest, `microvm_executor` drains inside the turn). Two share one. Every Skill-Use score is therefore per-**mission**, not per-role,
mechanisms for one job is how they drift. Worth folding onto and the World's per-agent view gets nothing from the container tier. The
`container_tool_hooks` once the fleet is back — tank and morpheus have been hook payload carries `session_id`; mapping it back to a turn is the fix.
offline for over a week, so the microVM tier cannot be tested at all today.
4. **Deploy the door** (`docs/TOOL-CALL-ARCHITECTURE.md` §3). Config, not code: 4. **Deploy the door** (`TOOL-CALL-ARCHITECTURE.md` §3). Config, not code:
`/zeroclaw-data/clawmates-mcp.json` plus a door-shaped provider alias. Now `/zeroclaw-data/clawmates-mcp.json` plus a door-shaped provider alias. It is
less urgent than it looked — the gate no longer depends on it — but it is now the single change that makes **Trigger** a real measurement, and the
still the precondition for `clawmates_skills` being reachable, and therefore precondition for skills moving from inlined bodies to progressive
for skills moving from inlined bodies to progressive disclosure. disclosure — which would also cut the prompt cost in item 1.
5. **Pull upstream's egress policy** — `0db7d999a feat(plugins): add shared 5. **Fold the microVM tier onto `container_tool_hooks`.** It has a gate and a
tap by a different route (`vm_tool_tap` installs into the guest,
`microvm_executor` drains inside the turn). Two mechanisms for one job is how
they drift — and the argument-discarding bug above lived in the shared parser
precisely because nobody looked at it from the container side. The fleet has
been offline for over a week, so this cannot be tested today.
6. **Pull upstream's egress policy** — `0db7d999a feat(plugins): add shared
egress policy foundation (#9137)`. We are ~220 commits behind; this is the egress policy foundation (#9137)`. We are ~220 commits behind; this is the
one item identified as worth taking, and it is defence for a problem we have one item worth taking, and it is defence for a problem we have not solved.
not solved.
**Dropped from this list:** "give the direct-session tier a tap". That tier is
dormant — `CLAWMATES_MISSION_EXECUTOR` is unset in production, so it never
runs. Checking that before building for it saved the work.
## Open decisions that are yours ## Open decisions that are yours
- **Self-authoring scope.** Agents now apply their own `skill_candidate` items - **Push.** Seven commits are local. Pushing `main` triggers CI → auto-deploy to
with no human click (`CLAWMATES_SKILL_SELF_AUTHORING=0` restores the gate). gw-04.
- **The orphan container** above.
- **Self-authoring scope.** Agents apply their own `skill_candidate` items with
no human click (`CLAWMATES_SKILL_SELF_AUTHORING=0` restores the gate).
`identity_refinement` and `brain_consolidation` still wait for a human, `identity_refinement` and `brain_consolidation` still wait for a human,
because they change what an agent IS rather than adding a procedure it can because they change what an agent IS rather than adding a procedure it can
consult. Say if you want those autonomous too. consult.
- **Skill-Use Compliance coverage.** Most skills still score `not_applicable` —
we cannot tell whether they changed anything. `small-focused-commits` and
`tdd-red-green-refactor` are the next candidates and both need the repository
diff rather than the turn text.
## Deliberately not done ## Deliberately not done
- **The mission executor swap** (running turns through `ProviderExecutor` or the - **The mission executor swap.** Blockers are structural: `cm-runtime`'s `files`
chat `Runtime`). The blockers are structural, not wiring: `cm-runtime`'s tool rejects absolute paths by construction, `shell` runs in a per-agent
`files` tool rejects absolute paths *by construction*, `shell` runs in a sandbox with no mission mount, `ToolContext` carries no path or VM handle, and
per-agent sandbox with no mission mount, `ToolContext` carries no path or VM approvals key on `(session_id, message_id)`.
handle, and approvals key on `(session_id, message_id)`. The cheap fixes - **A tap for the direct-session tier.** Dormant —
deliver what it was wanted for. `CLAWMATES_MISSION_EXECUTOR` is unset in production, so it never runs.
- **`cm-brain` offline tests** — 6 of 9 need live `clawbrainhub.com`. Stubbing - **`cm-brain` offline tests** — 6 of 9 need live `clawbrainhub.com`.
means reproducing an external registry protocol we have no spec for.
- **Graph memory / `clawhdf5-agent`** — in the workspace manifest, used by no - **Graph memory / `clawhdf5-agent`** — in the workspace manifest, used by no
crate. Measure against a baseline before migrating. crate.
## Operational facts that cost time to learn ## Operational facts that cost time to learn
- The Gitea **actions-log API returns 403** for the token in - The Gitea **actions-log API returns 403** for the token in
`deploy/compose/.env`. Every CI failure this week was debugged blind because `deploy/compose/.env`. A token with the `actions` scope remains the
of it. Steps now write to `/tmp/ci-logs` on the runner host as a workaround; highest-value thing to obtain.
**a token with the `actions` scope** remains the highest-value thing to - **gw-04 uses legacy `docker-compose`**, not the v2 plugin.
obtain.
- **gw-04 uses legacy `docker-compose`**, not the v2 plugin. `docker compose`
fails there.
- **Do not build images by hand on gw-04 while CI may run** — same 150G volume, - **Do not build images by hand on gw-04 while CI may run** — same 150G volume,
and the frontend image build is what loses. and the frontend image build is what loses.
- The server reaches Docker through a **socket proxy** (`DOCKER_HOST`). Use - The server reaches Docker through a **socket proxy** (`DOCKER_HOST`). Use
`container_exec::connect()`, never `connect_with_local_defaults()`. `container_exec::connect()`, never `connect_with_local_defaults()`.
- Prod auth is **Clerk**; the bootstrap password in `deploy/compose/.env` works - Prod auth is **Clerk**; the bootstrap password in `deploy/compose/.env` works
only against the local stack. only against the local stack.
- Rebuilding the local server image is a **full Rust compile inside Docker**
(~8 min); the layer cache does not preserve `target/`. Budget for it before
any measurement that needs new server code.
- macOS has no `timeout(1)`.
## Two corrections made this session, worth remembering ## The recurring shape, now seven times over
- **"Missions can't call tools at all" was wrong.** They call `Bash` and `Write` **A claim in a comment or a doc, believed and never checked.** Every significant
with permissions pre-accepted. The gap was observing and gating, not having. finding this pass came from reading the source of truth — the parser, the
- **Raw test counts are a bad coverage metric.** They pointed at `cm-safety`, recipe, the production table — rather than the text describing it. The two new
whose seven tests already covered its critical paths, and missed a Slack skill contradictions were found *while writing checks against those skills*,
replay hole that let one captured request authenticate forever. which is the cheapest place to catch them and the reason to always read first.
The recurring shape, now seven times over: **a claim in a comment or a doc, The corollary the measurement itself demonstrated: **its own first verdict was
believed and never checked.** Every significant finding this session came from wrong**, and scoring a research phase as a TDD failure would have buried the
running the thing rather than reading about it. real finding (item 1). A check that reports a system defect as an agent defect
is worse than no check.
docs/SKILL-USE-BASELINE.md
+72 -16
View File
@@ -73,17 +73,21 @@ drift, and then the score would pass while the mission loop still stalled.
## The runs ## The runs
Three missions on the container/ZeroClaw tier, local stack, `research_only`. Four missions on the container/ZeroClaw tier, local stack. Runs 1–3 are
Run 3 uses the **same task text as run 2**, so the only variable is the scorer. `research_only`; run 3 uses the **same task text as run 2**, so the only
variable is the scorer. Run 4 is `research_and_code` against a real repository,
because a research mission writes no code and makes no commits — the TDD and
commit checks could never fire on one.
| | run 1 | run 2 | run 3 | | | run 1 | run 2 | run 3 | run 4 |
|---|---|---|---| |---|---|---|---|---|
| date | 08-19 | 08-19 | 08-21 | | date | 08-19 | 08-19 | 08-21 | 08-21 |
| distinct skills delivered | 3 | 9 | 9 | | workflow | research | research | research | **code** |
| total deliveries (per role prompt) | 3 | 14 | 14 | | distinct skills delivered | 3 | 9 | 9 | 9 |
| phantom "skills" scored | **2** | 0 | 0 | | total deliveries (per role prompt) | 3 | 14 | 14 | 28 |
| tool calls recorded | 0 | 0 | **49** | | phantom "skills" scored | **2** | 0 | 0 | 0 |
| axes scored from actions | 0 | 0 | **3 skills** | | tool calls recorded | 0 | 0 | **49** | **33** |
| writes outside `/mission/repo` | ? | ? | 0 | 0 |
Run 3, per skill (all `source_kind=builtin`; no agent-authored skill has been Run 3, per skill (all `source_kind=builtin`; no agent-authored skill has been
delivered yet): delivered yet):
@@ -100,7 +104,7 @@ delivered yet):
| `code-review-checklist` | 1 | n/a | n/a | | `code-review-checklist` | 1 | n/a | n/a |
| `criterion-benchmarking` | 1 | n/a | n/a | | `criterion-benchmarking` | 1 | n/a | n/a |
**n = 3 runs. No spread is reported because three cannot establish one.** This **n = 4 runs. No spread is reported because four cannot establish one.** This
is a baseline in the sense of "the first honest number", not in the sense of is a baseline in the sense of "the first honest number", not in the sense of
`metrics-baseline-comparison.md`, which requires enough runs to see the noise `metrics-baseline-comparison.md`, which requires enough runs to see the noise
floor before any change is judged against it. floor before any change is judged against it.
@@ -109,6 +113,57 @@ floor before any change is judged against it.
improved: it now rests on twelve recorded `Write`/`Edit` paths, every one under improved: it now rests on twelve recorded `Write`/`Edit` paths, every one under
`/mission/repo`, instead of on the absence of a string in prose. `/mission/repo`, instead of on the absence of a string in prose.
### Run 4 — the first run that could have violated the new checks
`research_and_code` against `clawmates-delivery-scratch`, task: add a `slugify`
utility and commit it. The task says nothing about testing; priming it would
have measured the prompt rather than the skill.
| skill | deliveries | compliance | boundary |
|---|---|---|---|
| `int-xx-marker-protocol` | 2 | **pass** | n/a |
| `workspace-repo-commit-protocol` | 4 | n/a | **pass** |
| `cargo-test-driven-development` | 4 | **not observable** | n/a |
| `tdd-red-green-refactor` | 2 | **not observable** | n/a |
| `small-focused-commits` | 8 | n/a | n/a |
| the other four | 2 each | n/a | n/a |
The agents edited `src/lib.rs` once, ran `cargo test` five times, and committed
with `INT-01` on the subject. Every one of 33 tool calls stayed inside
`/mission/repo`.
**The TDD verdict is `not_observable`, and that is the honest answer rather
than a gap in the run.** The single `Edit` to `src/lib.rs` added the
implementation *and* a `#[cfg(test)] mod tests` block, then the tests ran. In
Rust the unit test lives in the file under test, so "wrote the file, then ran
the test" is exactly what writing the failing test first looks like from the
outside. The check therefore detects one thing only — **a phase that wrote
source and never ran a test at all** — and cannot confirm red-first. That is a
real limit of scoring TDD from tool ordering, and it applies to the most common
Rust shape, not an edge case.
#### The parsing bug run 4 found
Claude Code writes a multi-line commit message as
```
git commit -m "$(cat <<'EOF'
INT-01 Add slugify function to src/lib.rs
…
EOF
)"
```
and `commit_subjects` read the first line of the `-m` value — which is the
heredoc *opener*, `$(cat <<'EOF'`. Every commit check was scoring a string the
agent never wrote. It happened to score no violation, because `$(cat <<'EOF'`
is not one of the never-merge messages; that is luck, not a check. Fixed, with
a regression test built from the exact command in `mission_events`.
The verdicts in the table above are unchanged by the fix — the real subject,
`INT-01 Add slugify function to src/lib.rs`, is not a never-merge message
either — so the table reproduces against the shipped scorer.
## What the measurement found ## What the measurement found
### 1–4: the 2026-08-19 findings ### 1–4: the 2026-08-19 findings
@@ -207,12 +262,13 @@ anyone else's.
## Honest limits ## Honest limits
- **Three runs, one tier, one workflow.** Nothing here generalises to the - **Four runs, one tier, two workflows.** Nothing here generalises to the
microVM or session tiers. microVM or session tiers.
- **The new checks are not yet exercised live.** `research_only` writes no code - **The TDD check is one-sided and the common Rust case is undecidable.** It
and makes no commits, so the TDD and commit checks are proven by unit tests catches "wrote source, never ran a test". It cannot confirm red-first,
and negative controls, not by a mission that could have violated them. A because a Rust unit test lives in the file under test — see run 4.
coding run against a real repository is the next measurement. - **Four runs, and run 4 is the only coding one.** The commit checks have been
*reached* live exactly once.
- **Tool calls carry no agent attribution.** `record_vm_tools` writes - **Tool calls carry no agent attribution.** `record_vm_tools` writes
`agent_id: None` — the container tier's tap is per-container, and all five `agent_id: None` — the container tier's tap is per-container, and all five
roles share one container. Every score above is therefore per-**mission**, roles share one container. Every score above is therefore per-**mission**,