docs: the staffing fix, measured

Run 5 is run 3's task against the new staffing: 5 roles → 3, 14 skill
deliveries → 4, 50KB of prompt → 24KB, and 1 of 9 delivered skills
applicable → 4 of 4. The agents produced exactly the structure the new
team's task specifies — questions.md, evidence.md, REPORT.md — with zero
writes outside /mission/repo.

The baseline says plainly that the SCORES barely moved, because they did:
run 5 is one `pass` and three `not_applicable`. What changed is what
`not_applicable` means — "no machine-checkable consequence" rather than
"this skill had nothing to do with this phase". Halving the prompt is real
but incidental. The finding is that the denominator was wrong: seven of
run 3's nine skills were never applicable, so any ratio over them measured
staffing, not skill use.

Handoff item 1 is closed and the orphan-container section now records what
was actually in it.

Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_018i9Ten1LU4jUr5d7TAWda9
This commit is contained in:
Omar Sobh
2026-08-21 10:35:12 -07:00
co-authored by Claude Opus 5
parent 6af1149e45
commit 19c4de36e4
2 changed files with 86 additions and 33 deletions
+52 -25
View File
@@ -8,9 +8,9 @@ than the agent's own account of it.
## State of the tree ## State of the tree
Local suite green: **107 test binaries, 792 tests** (`cargo test --workspace`), and the workspace builds with `--all-targets`. Local suite green: **107 test binaries, 792 tests** (`cargo test --workspace`), and the workspace builds with `--all-targets`.
Four measurement missions ran on the local stack (`scripts/skill-use-run.sh`); Five measurement missions ran on the local stack (`scripts/skill-use-run.sh`);
all four are held 90 days and re-scorable with `--score <id>`. all five are held 90 days and re-scorable with `--score <id>`.
Seven commits on `main` this pass, **not pushed** — a push to `main` auto-deploys 10 commits on `main` this pass, **not pushed** — a push to `main` auto-deploys
to gw-04, and the container-tier work already deployed is unexercised there to gw-04, and the container-tier work already deployed is unexercised there
(see below). (see below).
@@ -43,9 +43,20 @@ that query. Nothing enumerates Docker for `cm-runtime-mission-*` containers with
no matching row, so a container whose row is gone is invisible to every reaper. no matching row, so a container whose row is gone is invisible to every reaper.
Same shape as the earlier agent-container reap drift, different table. Same shape as the earlier agent-container reap drift, different table.
It holds disk on the host whose disk exhaustion broke CI runs 503–506. **Not **It was not idle.** Its checkout held **ten commits on a branch that had never
fixed — your call**, and the fix is small: one sweep that lists containers by been pushed** — +3451/-30 across 30 files, eighteen INT items on `clawhdf5`
name prefix and reaps those with no row. including AES-256-GCM, Ed25519 signing and HNSW batch insert. The remote had
eight other `clawmates/*` branches and not this one.
Handled: bundled and verified, branch pushed to git.redclaw.dev, confirmed on
the remote at the tip (`87039e9`), container removed. 57G → 59G free. The
bundle is kept at `/opt/clawmates/rescued/rescue-019ff5b1.bundle`.
`mission_runtime::sweep_orphans` now closes the gap, and **that container is
why it refuses to reap a checkout holding commits no remote has.** A reaper
that deleted on sight would have destroyed all of it silently, as its designed
behaviour. Every unanswerable case — docker will not date it, git will not
answer, the clock skewed — resolves to *do not reap*.
## What shipped this pass ## What shipped this pass
@@ -77,6 +88,35 @@ after the first write is undecidable rather than a failure.
from the transport to the delivery model, and the door (§3) closes it with no from the transport to the delivery model, and the door (§3) closes it with no
scorer change at all. scorer change at all.
### Research phases are staffed by a research team
`research_only` — repo-less, one research phase — defaulted to `rust_sdlc`, so
it was staffed with a planner, coder, tester, reviewer and committer, four of
whom had nothing to do. New `topic_research` team, plus `default_phase_teams`
so a recipe can staff each phase *purpose* separately. Measured: 5 roles → 3,
14 skill deliveries → 4, 50KB of prompt → 24KB, and **1 of 9 delivered skills
applicable → 4 of 4**.
The scores barely moved, and that is the honest reading: what changed is that
`not_applicable` now means "no machine-checkable consequence" rather than "this
skill had nothing to do with this phase".
Three existing research templates were also wrong in ways nothing checked.
`papers_research` bound **`arxiv-daily`** — a skill whose content is "do not
search arXiv yourself" — to the DOMAIN SCOUT, the role whose job is searching.
Its PAPER READER was told to "fetch the PDF, extract text"; the runtime image
has no pdftotext, no mutool and no pypdf, so every paper would have hit the
`[read: abstract only]` fallback, which reads exactly like the fallback working.
`insight_research` cross-referenced "our repos'" history when a mission binds
one. `codebase_research` wrote to a vault that is not mounted.
### The wrong repo path was in the team templates too
The `/workspace/repo` guard was written against `skills/` only. The same path
was in four team templates — including `rust_sdlc`, default for five of six
recipes, whose coder was told "your working directory is /workspace/repo". The
guards now walk one corpus: skills, team templates and recipes together.
### Two more skills contradicted the platform ### Two more skills contradicted the platform
Both found by reading the source of truth before writing a check against it — Both found by reading the source of truth before writing a check against it —
@@ -94,19 +134,7 @@ real parser over every marker in every skill's fenced blocks.
## Next, in order ## Next, in order
1. **`research_only` staffs a Rust SDLC crew, and it is your decision.** 1. **The TDD check cannot confirm red-first, and that is structural.** Run 4
The recipe declares `requires_repo = false`, one `research` phase, and
`default_team_template = "rust_sdlc"` — so a repo-less markdown mission gets
planner, coder, tester, reviewer and committer, four of whom have nothing to
do, at ~50KB of prompt. This is why most skills score `not_applicable`: the
skills are correctly bound to the roles, and the **roles are wrong for the
workflow**. No existing template fits — `papers_research` is arXiv-shaped,
`insight_research` vault-shaped, `codebase_research` needs a repo — so this
is a new template or a recipe change, not a one-liner. `benchmark`,
`refactor`, `research_and_code` and `security_hardening` all default to
`rust_sdlc` too; check whether each is right.
2. **The TDD check cannot confirm red-first, and that is structural.** Run 4
(`research_and_code`, real repo) edited `src/lib.rs` once — implementation (`research_and_code`, real repo) edited `src/lib.rs` once — implementation
*and* `#[cfg(test)] mod tests` in the same write — then ran `cargo test` *and* `#[cfg(test)] mod tests` in the same write — then ran `cargo test`
five times. In Rust the unit test lives in the file under test, so that five times. In Rust the unit test lives in the file under test, so that
@@ -115,34 +143,33 @@ real parser over every marker in every skill's fenced blocks.
red-first, it needs the diff (did the test exist before the impl?), not the red-first, it needs the diff (did the test exist before the impl?), not the
tool order. tool order.
3. **Attribute tool calls to agents.** `record_vm_tools` writes 2. **Attribute tool calls to agents.** `record_vm_tools` writes
`agent_id: None`, because the container tap is per-container and all roles `agent_id: None`, because the container tap is per-container and all roles
share one. Every Skill-Use score is therefore per-**mission**, not per-role, share one. Every Skill-Use score is therefore per-**mission**, not per-role,
and the World's per-agent view gets nothing from the container tier. The and the World's per-agent view gets nothing from the container tier. The
hook payload carries `session_id`; mapping it back to a turn is the fix. hook payload carries `session_id`; mapping it back to a turn is the fix.
4. **Deploy the door** (`TOOL-CALL-ARCHITECTURE.md` §3). Config, not code: 3. **Deploy the door** (`TOOL-CALL-ARCHITECTURE.md` §3). Config, not code:
`/zeroclaw-data/clawmates-mcp.json` plus a door-shaped provider alias. It is `/zeroclaw-data/clawmates-mcp.json` plus a door-shaped provider alias. It is
now the single change that makes **Trigger** a real measurement, and the now the single change that makes **Trigger** a real measurement, and the
precondition for skills moving from inlined bodies to progressive precondition for skills moving from inlined bodies to progressive
disclosure — which would also cut the prompt cost in item 1. disclosure — which would also cut the prompt cost in item 1.
5. **Fold the microVM tier onto `container_tool_hooks`.** It has a gate and a 4. **Fold the microVM tier onto `container_tool_hooks`.** It has a gate and a
tap by a different route (`vm_tool_tap` installs into the guest, tap by a different route (`vm_tool_tap` installs into the guest,
`microvm_executor` drains inside the turn). Two mechanisms for one job is how `microvm_executor` drains inside the turn). Two mechanisms for one job is how
they drift — and the argument-discarding bug above lived in the shared parser they drift — and the argument-discarding bug above lived in the shared parser
precisely because nobody looked at it from the container side. The fleet has precisely because nobody looked at it from the container side. The fleet has
been offline for over a week, so this cannot be tested today. been offline for over a week, so this cannot be tested today.
6. **Pull upstream's egress policy** — `0db7d999a feat(plugins): add shared 5. **Pull upstream's egress policy** — `0db7d999a feat(plugins): add shared
egress policy foundation (#9137)`. We are ~220 commits behind; this is the egress policy foundation (#9137)`. We are ~220 commits behind; this is the
one item worth taking, and it is defence for a problem we have not solved. one item worth taking, and it is defence for a problem we have not solved.
## Open decisions that are yours ## Open decisions that are yours
- **Push.** Seven commits are local. Pushing `main` triggers CI → auto-deploy to - **Push.** 10 commits are local. Pushing `main` triggers CI → auto-deploy to
gw-04. gw-04.
- **The orphan container** above.
- **Self-authoring scope.** Agents apply their own `skill_candidate` items with - **Self-authoring scope.** Agents apply their own `skill_candidate` items with
no human click (`CLAWMATES_SKILL_SELF_AUTHORING=0` restores the gate). no human click (`CLAWMATES_SKILL_SELF_AUTHORING=0` restores the gate).
`identity_refinement` and `brain_consolidation` still wait for a human, `identity_refinement` and `brain_consolidation` still wait for a human,
+34 -8
View File
@@ -73,7 +73,7 @@ drift, and then the score would pass while the mission loop still stalled.
## The runs ## The runs
Four missions on the container/ZeroClaw tier, local stack. Runs 1–3 are Five missions on the container/ZeroClaw tier, local stack. Runs 1–3 are
`research_only`; run 3 uses the **same task text as run 2**, so the only `research_only`; run 3 uses the **same task text as run 2**, so the only
variable is the scorer. Run 4 is `research_and_code` against a real repository, variable is the scorer. Run 4 is `research_and_code` against a real repository,
because a research mission writes no code and makes no commits — the TDD and because a research mission writes no code and makes no commits — the TDD and
@@ -104,7 +104,7 @@ delivered yet):
| `code-review-checklist` | 1 | n/a | n/a | | `code-review-checklist` | 1 | n/a | n/a |
| `criterion-benchmarking` | 1 | n/a | n/a | | `criterion-benchmarking` | 1 | n/a | n/a |
**n = 4 runs. No spread is reported because four cannot establish one.** This **n = 5 runs. No spread is reported because five cannot establish one.** This
is a baseline in the sense of "the first honest number", not in the sense of is a baseline in the sense of "the first honest number", not in the sense of
`metrics-baseline-comparison.md`, which requires enough runs to see the noise `metrics-baseline-comparison.md`, which requires enough runs to see the noise
floor before any change is judged against it. floor before any change is judged against it.
@@ -237,11 +237,37 @@ workflow.** That distinction matters: a reader who saw only "7 of 9 skills
scored not_applicable" would conclude the skills are useless, when what the scored not_applicable" would conclude the skills are useless, when what the
number actually measures is a staffing default. number actually measures is a staffing default.
`continuous_research` names its own team; the other four recipes all default to **Fixed the same day, and measured again as run 5.** `research_only` now
`rust_sdlc`. `research_only` has no correct existing template to point at — defaults to a new `topic_research` team, and `default_phase_teams` lets a
`papers_research` is arXiv-shaped, `insight_research` is vault-shaped, and recipe staff each phase *purpose* separately — `research_and_code` and
`codebase_research` needs a repo — so the fix is an operator decision, not a `security_hardening` give their research phases the research team and keep
one-line edit, and is deliberately left open. `rust_sdlc` for coding. `benchmark` and `refactor` were checked and left alone:
one coding-purpose phase each, correctly staffed already.
Run 5 is run 3's task, re-run against the new staffing:
| | run 3 | run 5 |
|---|---|---|
| roles staffed | 5 | **3** |
| distinct skills delivered | 9 | **4** |
| total deliveries | 14 | **4** |
| prompt bytes across roles | 50,449 | **24,065** |
| skills applicable to the phase | 1 of 9 | **4 of 4** |
**Read the last row carefully, and not the ones above it.** The *scores* barely
moved: run 5 has one `pass` and three `not_applicable`. What changed is what
`not_applicable` now means. In run 3 it mostly meant "this skill had nothing to
do with what this phase was doing"; in run 5 it means "this skill's procedure
has no machine-checkable consequence" — which is the honest, permanent reason,
and the one this measurement was designed to report.
Cutting the prompt in half is real but incidental. The finding is that the
denominator was wrong: seven of the nine skills in run 3 were never applicable,
so any ratio computed over them measured staffing, not skill use.
The agents also produced exactly the structure the new team's task specifies —
`research/questions.md`, `research/evidence.md`, `research/REPORT.md` — with
zero writes outside `/mission/repo`.
### 8. The check that got it wrong first ### 8. The check that got it wrong first
@@ -262,7 +288,7 @@ anyone else's.
## Honest limits ## Honest limits
- **Four runs, one tier, two workflows.** Nothing here generalises to the - **Five runs, one tier, two workflows.** Nothing here generalises to the
microVM or session tiers. microVM or session tiers.
- **The TDD check is one-sided and the common Rust case is undecidable.** It - **The TDD check is one-sided and the common Rust case is undecidable.** It
catches "wrote source, never ran a test". It cannot confirm red-first, catches "wrote source, never ran a test". It cannot confirm red-first,