docs: the staffing fix, measured

Run 5 is run 3's task against the new staffing: 5 roles → 3, 14 skill
deliveries → 4, 50KB of prompt → 24KB, and 1 of 9 delivered skills
applicable → 4 of 4. The agents produced exactly the structure the new
team's task specifies — questions.md, evidence.md, REPORT.md — with zero
writes outside /mission/repo.

The baseline says plainly that the SCORES barely moved, because they did:
run 5 is one `pass` and three `not_applicable`. What changed is what
`not_applicable` means — "no machine-checkable consequence" rather than
"this skill had nothing to do with this phase". Halving the prompt is real
but incidental. The finding is that the denominator was wrong: seven of
run 3's nine skills were never applicable, so any ratio over them measured
staffing, not skill use.

Handoff item 1 is closed and the orphan-container section now records what
was actually in it.

Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_018i9Ten1LU4jUr5d7TAWda9
This commit is contained in:
Omar Sobh
2026-08-21 10:35:12 -07:00
co-authored by Claude Opus 5
parent 6af1149e45
commit 19c4de36e4
2 changed files with 86 additions and 33 deletions
+34 -8
View File
@@ -73,7 +73,7 @@ drift, and then the score would pass while the mission loop still stalled.
## The runs
Four missions on the container/ZeroClaw tier, local stack. Runs 13 are
Five missions on the container/ZeroClaw tier, local stack. Runs 13 are
`research_only`; run 3 uses the **same task text as run 2**, so the only
variable is the scorer. Run 4 is `research_and_code` against a real repository,
because a research mission writes no code and makes no commits — the TDD and
@@ -104,7 +104,7 @@ delivered yet):
| `code-review-checklist` | 1 | n/a | n/a |
| `criterion-benchmarking` | 1 | n/a | n/a |
**n = 4 runs. No spread is reported because four cannot establish one.** This
**n = 5 runs. No spread is reported because five cannot establish one.** This
is a baseline in the sense of "the first honest number", not in the sense of
`metrics-baseline-comparison.md`, which requires enough runs to see the noise
floor before any change is judged against it.
@@ -237,11 +237,37 @@ workflow.** That distinction matters: a reader who saw only "7 of 9 skills
scored not_applicable" would conclude the skills are useless, when what the
number actually measures is a staffing default.
`continuous_research` names its own team; the other four recipes all default to
`rust_sdlc`. `research_only` has no correct existing template to point at —
`papers_research` is arXiv-shaped, `insight_research` is vault-shaped, and
`codebase_research` needs a repo — so the fix is an operator decision, not a
one-line edit, and is deliberately left open.
**Fixed the same day, and measured again as run 5.** `research_only` now
defaults to a new `topic_research` team, and `default_phase_teams` lets a
recipe staff each phase *purpose* separately — `research_and_code` and
`security_hardening` give their research phases the research team and keep
`rust_sdlc` for coding. `benchmark` and `refactor` were checked and left alone:
one coding-purpose phase each, correctly staffed already.
Run 5 is run 3's task, re-run against the new staffing:
| | run 3 | run 5 |
|---|---|---|
| roles staffed | 5 | **3** |
| distinct skills delivered | 9 | **4** |
| total deliveries | 14 | **4** |
| prompt bytes across roles | 50,449 | **24,065** |
| skills applicable to the phase | 1 of 9 | **4 of 4** |
**Read the last row carefully, and not the ones above it.** The *scores* barely
moved: run 5 has one `pass` and three `not_applicable`. What changed is what
`not_applicable` now means. In run 3 it mostly meant "this skill had nothing to
do with what this phase was doing"; in run 5 it means "this skill's procedure
has no machine-checkable consequence" — which is the honest, permanent reason,
and the one this measurement was designed to report.
Cutting the prompt in half is real but incidental. The finding is that the
denominator was wrong: seven of the nine skills in run 3 were never applicable,
so any ratio computed over them measured staffing, not skill use.
The agents also produced exactly the structure the new team's task specifies —
`research/questions.md`, `research/evidence.md`, `research/REPORT.md` — with
zero writes outside `/mission/repo`.
### 8. The check that got it wrong first
@@ -262,7 +288,7 @@ anyone else's.
## Honest limits
- **Four runs, one tier, two workflows.** Nothing here generalises to the
- **Five runs, one tier, two workflows.** Nothing here generalises to the
microVM or session tiers.
- **The TDD check is one-sided and the common Rust case is undecidable.** It
catches "wrote source, never ran a test". It cannot confirm red-first,