docs: the staffing fix, measured
Run 5 is run 3's task against the new staffing: 5 roles → 3, 14 skill deliveries → 4, 50KB of prompt → 24KB, and 1 of 9 delivered skills applicable → 4 of 4. The agents produced exactly the structure the new team's task specifies — questions.md, evidence.md, REPORT.md — with zero writes outside /mission/repo. The baseline says plainly that the SCORES barely moved, because they did: run 5 is one `pass` and three `not_applicable`. What changed is what `not_applicable` means — "no machine-checkable consequence" rather than "this skill had nothing to do with this phase". Halving the prompt is real but incidental. The finding is that the denominator was wrong: seven of run 3's nine skills were never applicable, so any ratio over them measured staffing, not skill use. Handoff item 1 is closed and the orphan-container section now records what was actually in it. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_018i9Ten1LU4jUr5d7TAWda9
This commit is contained in:
co-authored by
Claude Opus 5
parent
6af1149e45
commit
19c4de36e4
@@ -73,7 +73,7 @@ drift, and then the score would pass while the mission loop still stalled.
|
||||
|
||||
## The runs
|
||||
|
||||
Four missions on the container/ZeroClaw tier, local stack. Runs 1–3 are
|
||||
Five missions on the container/ZeroClaw tier, local stack. Runs 1–3 are
|
||||
`research_only`; run 3 uses the **same task text as run 2**, so the only
|
||||
variable is the scorer. Run 4 is `research_and_code` against a real repository,
|
||||
because a research mission writes no code and makes no commits — the TDD and
|
||||
@@ -104,7 +104,7 @@ delivered yet):
|
||||
| `code-review-checklist` | 1 | n/a | n/a |
|
||||
| `criterion-benchmarking` | 1 | n/a | n/a |
|
||||
|
||||
**n = 4 runs. No spread is reported because four cannot establish one.** This
|
||||
**n = 5 runs. No spread is reported because five cannot establish one.** This
|
||||
is a baseline in the sense of "the first honest number", not in the sense of
|
||||
`metrics-baseline-comparison.md`, which requires enough runs to see the noise
|
||||
floor before any change is judged against it.
|
||||
@@ -237,11 +237,37 @@ workflow.** That distinction matters: a reader who saw only "7 of 9 skills
|
||||
scored not_applicable" would conclude the skills are useless, when what the
|
||||
number actually measures is a staffing default.
|
||||
|
||||
`continuous_research` names its own team; the other four recipes all default to
|
||||
`rust_sdlc`. `research_only` has no correct existing template to point at —
|
||||
`papers_research` is arXiv-shaped, `insight_research` is vault-shaped, and
|
||||
`codebase_research` needs a repo — so the fix is an operator decision, not a
|
||||
one-line edit, and is deliberately left open.
|
||||
**Fixed the same day, and measured again as run 5.** `research_only` now
|
||||
defaults to a new `topic_research` team, and `default_phase_teams` lets a
|
||||
recipe staff each phase *purpose* separately — `research_and_code` and
|
||||
`security_hardening` give their research phases the research team and keep
|
||||
`rust_sdlc` for coding. `benchmark` and `refactor` were checked and left alone:
|
||||
one coding-purpose phase each, correctly staffed already.
|
||||
|
||||
Run 5 is run 3's task, re-run against the new staffing:
|
||||
|
||||
| | run 3 | run 5 |
|
||||
|---|---|---|
|
||||
| roles staffed | 5 | **3** |
|
||||
| distinct skills delivered | 9 | **4** |
|
||||
| total deliveries | 14 | **4** |
|
||||
| prompt bytes across roles | 50,449 | **24,065** |
|
||||
| skills applicable to the phase | 1 of 9 | **4 of 4** |
|
||||
|
||||
**Read the last row carefully, and not the ones above it.** The *scores* barely
|
||||
moved: run 5 has one `pass` and three `not_applicable`. What changed is what
|
||||
`not_applicable` now means. In run 3 it mostly meant "this skill had nothing to
|
||||
do with what this phase was doing"; in run 5 it means "this skill's procedure
|
||||
has no machine-checkable consequence" — which is the honest, permanent reason,
|
||||
and the one this measurement was designed to report.
|
||||
|
||||
Cutting the prompt in half is real but incidental. The finding is that the
|
||||
denominator was wrong: seven of the nine skills in run 3 were never applicable,
|
||||
so any ratio computed over them measured staffing, not skill use.
|
||||
|
||||
The agents also produced exactly the structure the new team's task specifies —
|
||||
`research/questions.md`, `research/evidence.md`, `research/REPORT.md` — with
|
||||
zero writes outside `/mission/repo`.
|
||||
|
||||
### 8. The check that got it wrong first
|
||||
|
||||
@@ -262,7 +288,7 @@ anyone else's.
|
||||
|
||||
## Honest limits
|
||||
|
||||
- **Four runs, one tier, two workflows.** Nothing here generalises to the
|
||||
- **Five runs, one tier, two workflows.** Nothing here generalises to the
|
||||
microVM or session tiers.
|
||||
- **The TDD check is one-sided and the common Rust case is undecidable.** It
|
||||
catches "wrote source, never ran a test". It cannot confirm red-first,
|
||||
|
||||
Reference in New Issue
Block a user