fix(skills): a pinned skill contradicted the platform inside the same prompt
Extending the Skill-Use mechanical checks, per the baseline's own next step, found something bigger than a missing check. THE DEFECT `workspace-repo-commit-protocol` told agents that `/workspace/repo` was "the ONLY path where source-modifying edits belong". The platform mounts and advertises `/mission/repo` — 26 references in the code; `/workspace/repo` appears in none of them. The skill is bound on 29 role bindings and was delivered TWICE in the run already measured, so an agent received the real path in its tool preamble and a skill contradicting it a few hundred tokens later, in one prompt. An agent that obeyed the skill wrote source into a directory nothing collects — the phase then delivers nothing, and looks like an agent that did no work. The same skill instructed `file_read` / `file_write` / `shell`: ZeroClaw's names, the exact ones `phase_task_text` was fixed to stop advertising after five agents on a single mission spent 7.4k tokens describing the mismatch instead of working. The prompt was corrected and the skill kept saying it. Rewritten against what the code actually does, including the repo-less case (`/mission/repo` exists, is collected as artifacts, has nothing to push). THE CLASS, AND THE GUARD The skills were never checked against the platform they describe. Nothing compared them, so a skill could contradict the prompt it ships inside and stay that way indefinitely — the same shape as PLAN_COMPLETE being documented and never implemented. Two tests in `skills_loader::contradiction_tests` now hold it: no skill may name a repo path the platform does not mount, and none may instruct a tool the agent's subprocess does not expose. The second matches backticked instructions and skips corrective lines, so a skill may still WARN against the wrong names — as this one now does. Both negative-controlled by restoring the old wording. AND THE CHECK THAT STARTED IT `workspace-repo-commit-protocol` now has a Boundary check: writing outside `/mission/repo` fails, and the message names the consequence — a phase that delivers nothing — rather than just the wrong path. docs/SKILL-USE-BASELINE.md records this as the fourth defect the measurement found, and corrects the "next unit of work" note now that this one is done. Full workspace suite green: 106 binaries, zero build errors. Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 5
parent
113de610ec
commit
771092b165
@@ -92,7 +92,7 @@ this one without first establishing that floor.
|
||||
|
||||
## What the measurement found
|
||||
|
||||
Three defects, none of which any test or log would have surfaced.
|
||||
Four defects, none of which any test or log would have surfaced.
|
||||
|
||||
### 1. The prompt format made its own record unparseable
|
||||
|
||||
@@ -115,7 +115,31 @@ A provenance record of something that did not happen is worse than no record: it
|
||||
is the wrong answer, delivered confidently. Recording now happens inside each
|
||||
tier, and a test asserts every launcher records the prompt it actually sends.
|
||||
|
||||
### 3. The skill documents a marker the platform never implemented
|
||||
### 3. A pinned skill contradicted the platform in the same prompt
|
||||
|
||||
`workspace-repo-commit-protocol` told agents that **`/workspace/repo`** was "the
|
||||
ONLY path where source-modifying edits belong". The platform mounts and
|
||||
advertises **`/mission/repo`** — 26 references in the code; `/workspace/repo`
|
||||
appears in none. The skill is bound on **29 role bindings** and was delivered
|
||||
twice in run 2, so agents received the real path in the tool preamble and a
|
||||
skill contradicting it a few hundred tokens later.
|
||||
|
||||
It also instructed `file_read` / `file_write` / `shell` — ZeroClaw's tool names,
|
||||
the exact ones `phase_task_text` was fixed to stop advertising after five agents
|
||||
on one mission spent 7.4k tokens describing the mismatch instead of working.
|
||||
|
||||
An agent that obeyed this skill wrote source into a directory nothing collects,
|
||||
and reached for tools its subprocess does not expose. Rewritten against what the
|
||||
code actually does, with two guards in `skills_loader::contradiction_tests`: no
|
||||
skill may name a repo path the platform does not mount, and none may instruct a
|
||||
tool the agent does not have. Both negative-controlled.
|
||||
|
||||
This is the same shape as the finding below and it is worth stating as a class:
|
||||
**the skills were never checked against the platform they describe.** Nothing
|
||||
compared them, so a skill could contradict the prompt it ships inside and stay
|
||||
that way indefinitely.
|
||||
|
||||
### 4. The skill documents a marker the platform never implemented
|
||||
|
||||
`int-xx-marker-protocol` lists `PLAN_COMPLETE: INT-NN` in its ladder.
|
||||
`task_card_parser` has **no such kind** and never has. An agent following the
|
||||
@@ -136,11 +160,11 @@ skill — is deliberately left as a decision rather than guessed at here.
|
||||
|
||||
- **Two runs, one tier, one workflow.** Nothing here generalises to the microVM
|
||||
or session tiers yet, and this document should not be read as if it does.
|
||||
- **7 of 9 skills scored `not_applicable`** on both observable axes. That is not
|
||||
a pass. It means we cannot currently tell whether those skills changed
|
||||
anything, and the next unit of work is mechanical checks for more of them —
|
||||
`workspace-repo-commit-protocol` and `small-focused-commits` both have
|
||||
checkable consequences in git history.
|
||||
- **Most skills score `not_applicable`** on both observable axes. That is not a
|
||||
pass. It means we cannot currently tell whether those skills changed anything.
|
||||
`workspace-repo-commit-protocol` now has a Boundary check (writing outside
|
||||
`/mission/repo`); `small-focused-commits` and `tdd-red-green-refactor` remain
|
||||
candidates, and both need the repository diff rather than the turn text.
|
||||
- **No agent-authored skill has been measured.** Self-authoring shipped in the
|
||||
same pass; `source_kind` is carried through the scorer specifically so a
|
||||
rising score on agent-authored skills is visible rather than averaged in.
|
||||
|
||||
Reference in New Issue
Block a user