docs: the staffing fix, measured
Run 5 is run 3's task against the new staffing: 5 roles → 3, 14 skill deliveries → 4, 50KB of prompt → 24KB, and 1 of 9 delivered skills applicable → 4 of 4. The agents produced exactly the structure the new team's task specifies — questions.md, evidence.md, REPORT.md — with zero writes outside /mission/repo. The baseline says plainly that the SCORES barely moved, because they did: run 5 is one `pass` and three `not_applicable`. What changed is what `not_applicable` means — "no machine-checkable consequence" rather than "this skill had nothing to do with this phase". Halving the prompt is real but incidental. The finding is that the denominator was wrong: seven of run 3's nine skills were never applicable, so any ratio over them measured staffing, not skill use. Handoff item 1 is closed and the orphan-container section now records what was actually in it. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_018i9Ten1LU4jUr5d7TAWda9
This commit is contained in:
co-authored by
Claude Opus 5
parent
6af1149e45
commit
19c4de36e4
+52
-25
@@ -8,9 +8,9 @@ than the agent's own account of it.
|
|||||||
## State of the tree
|
## State of the tree
|
||||||
|
|
||||||
Local suite green: **107 test binaries, 792 tests** (`cargo test --workspace`), and the workspace builds with `--all-targets`.
|
Local suite green: **107 test binaries, 792 tests** (`cargo test --workspace`), and the workspace builds with `--all-targets`.
|
||||||
Four measurement missions ran on the local stack (`scripts/skill-use-run.sh`);
|
Five measurement missions ran on the local stack (`scripts/skill-use-run.sh`);
|
||||||
all four are held 90 days and re-scorable with `--score <id>`.
|
all five are held 90 days and re-scorable with `--score <id>`.
|
||||||
Seven commits on `main` this pass, **not pushed** — a push to `main` auto-deploys
|
10 commits on `main` this pass, **not pushed** — a push to `main` auto-deploys
|
||||||
to gw-04, and the container-tier work already deployed is unexercised there
|
to gw-04, and the container-tier work already deployed is unexercised there
|
||||||
(see below).
|
(see below).
|
||||||
|
|
||||||
@@ -43,9 +43,20 @@ that query. Nothing enumerates Docker for `cm-runtime-mission-*` containers with
|
|||||||
no matching row, so a container whose row is gone is invisible to every reaper.
|
no matching row, so a container whose row is gone is invisible to every reaper.
|
||||||
Same shape as the earlier agent-container reap drift, different table.
|
Same shape as the earlier agent-container reap drift, different table.
|
||||||
|
|
||||||
It holds disk on the host whose disk exhaustion broke CI runs 503–506. **Not
|
**It was not idle.** Its checkout held **ten commits on a branch that had never
|
||||||
fixed — your call**, and the fix is small: one sweep that lists containers by
|
been pushed** — +3451/-30 across 30 files, eighteen INT items on `clawhdf5`
|
||||||
name prefix and reaps those with no row.
|
including AES-256-GCM, Ed25519 signing and HNSW batch insert. The remote had
|
||||||
|
eight other `clawmates/*` branches and not this one.
|
||||||
|
|
||||||
|
Handled: bundled and verified, branch pushed to git.redclaw.dev, confirmed on
|
||||||
|
the remote at the tip (`87039e9`), container removed. 57G → 59G free. The
|
||||||
|
bundle is kept at `/opt/clawmates/rescued/rescue-019ff5b1.bundle`.
|
||||||
|
|
||||||
|
`mission_runtime::sweep_orphans` now closes the gap, and **that container is
|
||||||
|
why it refuses to reap a checkout holding commits no remote has.** A reaper
|
||||||
|
that deleted on sight would have destroyed all of it silently, as its designed
|
||||||
|
behaviour. Every unanswerable case — docker will not date it, git will not
|
||||||
|
answer, the clock skewed — resolves to *do not reap*.
|
||||||
|
|
||||||
## What shipped this pass
|
## What shipped this pass
|
||||||
|
|
||||||
@@ -77,6 +88,35 @@ after the first write is undecidable rather than a failure.
|
|||||||
from the transport to the delivery model, and the door (§3) closes it with no
|
from the transport to the delivery model, and the door (§3) closes it with no
|
||||||
scorer change at all.
|
scorer change at all.
|
||||||
|
|
||||||
|
### Research phases are staffed by a research team
|
||||||
|
|
||||||
|
`research_only` — repo-less, one research phase — defaulted to `rust_sdlc`, so
|
||||||
|
it was staffed with a planner, coder, tester, reviewer and committer, four of
|
||||||
|
whom had nothing to do. New `topic_research` team, plus `default_phase_teams`
|
||||||
|
so a recipe can staff each phase *purpose* separately. Measured: 5 roles → 3,
|
||||||
|
14 skill deliveries → 4, 50KB of prompt → 24KB, and **1 of 9 delivered skills
|
||||||
|
applicable → 4 of 4**.
|
||||||
|
|
||||||
|
The scores barely moved, and that is the honest reading: what changed is that
|
||||||
|
`not_applicable` now means "no machine-checkable consequence" rather than "this
|
||||||
|
skill had nothing to do with this phase".
|
||||||
|
|
||||||
|
Three existing research templates were also wrong in ways nothing checked.
|
||||||
|
`papers_research` bound **`arxiv-daily`** — a skill whose content is "do not
|
||||||
|
search arXiv yourself" — to the DOMAIN SCOUT, the role whose job is searching.
|
||||||
|
Its PAPER READER was told to "fetch the PDF, extract text"; the runtime image
|
||||||
|
has no pdftotext, no mutool and no pypdf, so every paper would have hit the
|
||||||
|
`[read: abstract only]` fallback, which reads exactly like the fallback working.
|
||||||
|
`insight_research` cross-referenced "our repos'" history when a mission binds
|
||||||
|
one. `codebase_research` wrote to a vault that is not mounted.
|
||||||
|
|
||||||
|
### The wrong repo path was in the team templates too
|
||||||
|
|
||||||
|
The `/workspace/repo` guard was written against `skills/` only. The same path
|
||||||
|
was in four team templates — including `rust_sdlc`, default for five of six
|
||||||
|
recipes, whose coder was told "your working directory is /workspace/repo". The
|
||||||
|
guards now walk one corpus: skills, team templates and recipes together.
|
||||||
|
|
||||||
### Two more skills contradicted the platform
|
### Two more skills contradicted the platform
|
||||||
|
|
||||||
Both found by reading the source of truth before writing a check against it —
|
Both found by reading the source of truth before writing a check against it —
|
||||||
@@ -94,19 +134,7 @@ real parser over every marker in every skill's fenced blocks.
|
|||||||
|
|
||||||
## Next, in order
|
## Next, in order
|
||||||
|
|
||||||
1. **`research_only` staffs a Rust SDLC crew, and it is your decision.**
|
1. **The TDD check cannot confirm red-first, and that is structural.** Run 4
|
||||||
The recipe declares `requires_repo = false`, one `research` phase, and
|
|
||||||
`default_team_template = "rust_sdlc"` — so a repo-less markdown mission gets
|
|
||||||
planner, coder, tester, reviewer and committer, four of whom have nothing to
|
|
||||||
do, at ~50KB of prompt. This is why most skills score `not_applicable`: the
|
|
||||||
skills are correctly bound to the roles, and the **roles are wrong for the
|
|
||||||
workflow**. No existing template fits — `papers_research` is arXiv-shaped,
|
|
||||||
`insight_research` vault-shaped, `codebase_research` needs a repo — so this
|
|
||||||
is a new template or a recipe change, not a one-liner. `benchmark`,
|
|
||||||
`refactor`, `research_and_code` and `security_hardening` all default to
|
|
||||||
`rust_sdlc` too; check whether each is right.
|
|
||||||
|
|
||||||
2. **The TDD check cannot confirm red-first, and that is structural.** Run 4
|
|
||||||
(`research_and_code`, real repo) edited `src/lib.rs` once — implementation
|
(`research_and_code`, real repo) edited `src/lib.rs` once — implementation
|
||||||
*and* `#[cfg(test)] mod tests` in the same write — then ran `cargo test`
|
*and* `#[cfg(test)] mod tests` in the same write — then ran `cargo test`
|
||||||
five times. In Rust the unit test lives in the file under test, so that
|
five times. In Rust the unit test lives in the file under test, so that
|
||||||
@@ -115,34 +143,33 @@ real parser over every marker in every skill's fenced blocks.
|
|||||||
red-first, it needs the diff (did the test exist before the impl?), not the
|
red-first, it needs the diff (did the test exist before the impl?), not the
|
||||||
tool order.
|
tool order.
|
||||||
|
|
||||||
3. **Attribute tool calls to agents.** `record_vm_tools` writes
|
2. **Attribute tool calls to agents.** `record_vm_tools` writes
|
||||||
`agent_id: None`, because the container tap is per-container and all roles
|
`agent_id: None`, because the container tap is per-container and all roles
|
||||||
share one. Every Skill-Use score is therefore per-**mission**, not per-role,
|
share one. Every Skill-Use score is therefore per-**mission**, not per-role,
|
||||||
and the World's per-agent view gets nothing from the container tier. The
|
and the World's per-agent view gets nothing from the container tier. The
|
||||||
hook payload carries `session_id`; mapping it back to a turn is the fix.
|
hook payload carries `session_id`; mapping it back to a turn is the fix.
|
||||||
|
|
||||||
4. **Deploy the door** (`TOOL-CALL-ARCHITECTURE.md` §3). Config, not code:
|
3. **Deploy the door** (`TOOL-CALL-ARCHITECTURE.md` §3). Config, not code:
|
||||||
`/zeroclaw-data/clawmates-mcp.json` plus a door-shaped provider alias. It is
|
`/zeroclaw-data/clawmates-mcp.json` plus a door-shaped provider alias. It is
|
||||||
now the single change that makes **Trigger** a real measurement, and the
|
now the single change that makes **Trigger** a real measurement, and the
|
||||||
precondition for skills moving from inlined bodies to progressive
|
precondition for skills moving from inlined bodies to progressive
|
||||||
disclosure — which would also cut the prompt cost in item 1.
|
disclosure — which would also cut the prompt cost in item 1.
|
||||||
|
|
||||||
5. **Fold the microVM tier onto `container_tool_hooks`.** It has a gate and a
|
4. **Fold the microVM tier onto `container_tool_hooks`.** It has a gate and a
|
||||||
tap by a different route (`vm_tool_tap` installs into the guest,
|
tap by a different route (`vm_tool_tap` installs into the guest,
|
||||||
`microvm_executor` drains inside the turn). Two mechanisms for one job is how
|
`microvm_executor` drains inside the turn). Two mechanisms for one job is how
|
||||||
they drift — and the argument-discarding bug above lived in the shared parser
|
they drift — and the argument-discarding bug above lived in the shared parser
|
||||||
precisely because nobody looked at it from the container side. The fleet has
|
precisely because nobody looked at it from the container side. The fleet has
|
||||||
been offline for over a week, so this cannot be tested today.
|
been offline for over a week, so this cannot be tested today.
|
||||||
|
|
||||||
6. **Pull upstream's egress policy** — `0db7d999a feat(plugins): add shared
|
5. **Pull upstream's egress policy** — `0db7d999a feat(plugins): add shared
|
||||||
egress policy foundation (#9137)`. We are ~220 commits behind; this is the
|
egress policy foundation (#9137)`. We are ~220 commits behind; this is the
|
||||||
one item worth taking, and it is defence for a problem we have not solved.
|
one item worth taking, and it is defence for a problem we have not solved.
|
||||||
|
|
||||||
## Open decisions that are yours
|
## Open decisions that are yours
|
||||||
|
|
||||||
- **Push.** Seven commits are local. Pushing `main` triggers CI → auto-deploy to
|
- **Push.** 10 commits are local. Pushing `main` triggers CI → auto-deploy to
|
||||||
gw-04.
|
gw-04.
|
||||||
- **The orphan container** above.
|
|
||||||
- **Self-authoring scope.** Agents apply their own `skill_candidate` items with
|
- **Self-authoring scope.** Agents apply their own `skill_candidate` items with
|
||||||
no human click (`CLAWMATES_SKILL_SELF_AUTHORING=0` restores the gate).
|
no human click (`CLAWMATES_SKILL_SELF_AUTHORING=0` restores the gate).
|
||||||
`identity_refinement` and `brain_consolidation` still wait for a human,
|
`identity_refinement` and `brain_consolidation` still wait for a human,
|
||||||
|
|||||||
@@ -73,7 +73,7 @@ drift, and then the score would pass while the mission loop still stalled.
|
|||||||
|
|
||||||
## The runs
|
## The runs
|
||||||
|
|
||||||
Four missions on the container/ZeroClaw tier, local stack. Runs 1–3 are
|
Five missions on the container/ZeroClaw tier, local stack. Runs 1–3 are
|
||||||
`research_only`; run 3 uses the **same task text as run 2**, so the only
|
`research_only`; run 3 uses the **same task text as run 2**, so the only
|
||||||
variable is the scorer. Run 4 is `research_and_code` against a real repository,
|
variable is the scorer. Run 4 is `research_and_code` against a real repository,
|
||||||
because a research mission writes no code and makes no commits — the TDD and
|
because a research mission writes no code and makes no commits — the TDD and
|
||||||
@@ -104,7 +104,7 @@ delivered yet):
|
|||||||
| `code-review-checklist` | 1 | n/a | n/a |
|
| `code-review-checklist` | 1 | n/a | n/a |
|
||||||
| `criterion-benchmarking` | 1 | n/a | n/a |
|
| `criterion-benchmarking` | 1 | n/a | n/a |
|
||||||
|
|
||||||
**n = 4 runs. No spread is reported because four cannot establish one.** This
|
**n = 5 runs. No spread is reported because five cannot establish one.** This
|
||||||
is a baseline in the sense of "the first honest number", not in the sense of
|
is a baseline in the sense of "the first honest number", not in the sense of
|
||||||
`metrics-baseline-comparison.md`, which requires enough runs to see the noise
|
`metrics-baseline-comparison.md`, which requires enough runs to see the noise
|
||||||
floor before any change is judged against it.
|
floor before any change is judged against it.
|
||||||
@@ -237,11 +237,37 @@ workflow.** That distinction matters: a reader who saw only "7 of 9 skills
|
|||||||
scored not_applicable" would conclude the skills are useless, when what the
|
scored not_applicable" would conclude the skills are useless, when what the
|
||||||
number actually measures is a staffing default.
|
number actually measures is a staffing default.
|
||||||
|
|
||||||
`continuous_research` names its own team; the other four recipes all default to
|
**Fixed the same day, and measured again as run 5.** `research_only` now
|
||||||
`rust_sdlc`. `research_only` has no correct existing template to point at —
|
defaults to a new `topic_research` team, and `default_phase_teams` lets a
|
||||||
`papers_research` is arXiv-shaped, `insight_research` is vault-shaped, and
|
recipe staff each phase *purpose* separately — `research_and_code` and
|
||||||
`codebase_research` needs a repo — so the fix is an operator decision, not a
|
`security_hardening` give their research phases the research team and keep
|
||||||
one-line edit, and is deliberately left open.
|
`rust_sdlc` for coding. `benchmark` and `refactor` were checked and left alone:
|
||||||
|
one coding-purpose phase each, correctly staffed already.
|
||||||
|
|
||||||
|
Run 5 is run 3's task, re-run against the new staffing:
|
||||||
|
|
||||||
|
| | run 3 | run 5 |
|
||||||
|
|---|---|---|
|
||||||
|
| roles staffed | 5 | **3** |
|
||||||
|
| distinct skills delivered | 9 | **4** |
|
||||||
|
| total deliveries | 14 | **4** |
|
||||||
|
| prompt bytes across roles | 50,449 | **24,065** |
|
||||||
|
| skills applicable to the phase | 1 of 9 | **4 of 4** |
|
||||||
|
|
||||||
|
**Read the last row carefully, and not the ones above it.** The *scores* barely
|
||||||
|
moved: run 5 has one `pass` and three `not_applicable`. What changed is what
|
||||||
|
`not_applicable` now means. In run 3 it mostly meant "this skill had nothing to
|
||||||
|
do with what this phase was doing"; in run 5 it means "this skill's procedure
|
||||||
|
has no machine-checkable consequence" — which is the honest, permanent reason,
|
||||||
|
and the one this measurement was designed to report.
|
||||||
|
|
||||||
|
Cutting the prompt in half is real but incidental. The finding is that the
|
||||||
|
denominator was wrong: seven of the nine skills in run 3 were never applicable,
|
||||||
|
so any ratio computed over them measured staffing, not skill use.
|
||||||
|
|
||||||
|
The agents also produced exactly the structure the new team's task specifies —
|
||||||
|
`research/questions.md`, `research/evidence.md`, `research/REPORT.md` — with
|
||||||
|
zero writes outside `/mission/repo`.
|
||||||
|
|
||||||
### 8. The check that got it wrong first
|
### 8. The check that got it wrong first
|
||||||
|
|
||||||
@@ -262,7 +288,7 @@ anyone else's.
|
|||||||
|
|
||||||
## Honest limits
|
## Honest limits
|
||||||
|
|
||||||
- **Four runs, one tier, two workflows.** Nothing here generalises to the
|
- **Five runs, one tier, two workflows.** Nothing here generalises to the
|
||||||
microVM or session tiers.
|
microVM or session tiers.
|
||||||
- **The TDD check is one-sided and the common Rust case is undecidable.** It
|
- **The TDD check is one-sided and the common Rust case is undecidable.** It
|
||||||
catches "wrote source, never ran a test". It cannot confirm red-first,
|
catches "wrote source, never ran a test". It cannot confirm red-first,
|
||||||
|
|||||||
Reference in New Issue
Block a user