Omar SobhandClaude Opus 5
594cd99e54
A review of what our agents can actually be asked to do, graded by evidence rather than by what the TOML declares. Recipes: research_and_code (13 harness scenarios) and research_only (staffing measured and corrected — 1 of 9 applicable skills under the old rust_sdlc default, 4 of 4 under topic_research) are proven. security_hardening, benchmark and refactor are exercised once each. continuous_research is the outlier: the most moving parts of any recipe, ran twice today, and NOTHING in the harness would notice if it broke. Dead keys the recipes lean on, from phase_config::DECLARED_BUT_UNREAD: loop, produces, input_from_phase, and mcp_bundles at phase level. Not hidden — security_hardening.toml annotates its own decoration inline, and the other five should copy that. The risk is a reader taking loop = "until_no_more_int_items" for a loop. Teams: all 12 resolve every declared skill (0 unbound, the skill-binding-repair work holding), but only rust_sdlc, topic_research and continuous_research have a run behind them. The other nine are well-formed scaffolding. Five are blocked on a target stack we do not have a repo for; four could be exercised against repositories we already have, and nobody currently knows whether they run at all. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01WZb5A2kfVfjpdwSochkuHz
112 lines
6.4 KiB
Markdown
112 lines
6.4 KiB
Markdown
# Template maturity — what our agents can actually be asked to do
|
|
|
|
A review of the 6 workflow recipes and 12 team templates, on 2026-09-22.
|
|
Graded by evidence, not by what the TOML declares: a recipe is mature when a
|
|
mission built from it has completed and something checked the result.
|
|
|
|
## Workflow recipes
|
|
|
|
| recipe | phases | standing check | staffing | grade |
|
|
|---|---|---|---|---|
|
|
| `research_and_code` | 2 (research → coding) | **13 harness scenarios** | research→`topic_research`, coding→`rust_sdlc` | **Proven** |
|
|
| `research_only` | 1 (research, repo-less) | 2 scenarios | `topic_research` | **Proven** |
|
|
| `security_hardening` | 3 (scan → research → coding) | 1 scenario | +`rust_sdlc` | Exercised |
|
|
| `benchmark` | 1 | 1 scenario | `rust_sdlc` | Exercised |
|
|
| `refactor` | 1 | 1 scenario | `rust_sdlc` | Exercised |
|
|
| `continuous_research` | 2 (analysis → script) | **none** | `continuous_research` | Runs live, unguarded |
|
|
|
|
`research_and_code` is the workhorse: every delivery, gate, judge, memory and
|
|
triage scenario is built on it, so it is re-proven on every harness run.
|
|
`research_only` earned its grade differently — its staffing was **measured and
|
|
corrected**: under the old `rust_sdlc` default it delivered 5 roles, 14 skill
|
|
deliveries and ~50 KB of prompt with **1 of 9** skills applicable; on
|
|
`topic_research` it is 3 roles, 4 deliveries, 24 KB, **4 of 4**.
|
|
|
|
`continuous_research` is the outlier and the one to fix first. It has the most
|
|
moving parts of any recipe — harvest → triage → manifest → analysis → ranking
|
|
→ script → `episode.json` → vault commit — it ran twice today and produced
|
|
real output both times, and **nothing in the harness would notice if it broke
|
|
tomorrow.** Every other recipe has a standing check; this one has operator
|
|
attention, which is not the same thing.
|
|
|
|
## What the recipes declare that does not fire
|
|
|
|
`phase_config::DECLARED_BUT_UNREAD` is the honest list, and recipes lean on it:
|
|
|
|
| key | used by | status |
|
|
|---|---|---|
|
|
| `loop` | `research_and_code` (`until_no_more_int_items`), `refactor` (`single_pass`) | **inert** — iteration is `max_iterations` + `done_when` |
|
|
| `produces` | almost every recipe (`["md"]`) | **inert** — artifact rendering is not driven by it |
|
|
| `input_from_phase` | `security_hardening` | **inert** — phases share a checkout |
|
|
| `mcp_bundles` (phase level) | `security_hardening` | **inert** — bundles come from the TEAM template |
|
|
|
|
None of this is hidden: `security_hardening.toml` carries a "what is real here
|
|
and what is decoration" section and annotates its own dead keys inline. That is
|
|
the right pattern and the other five should copy it. The risk is not the dead
|
|
keys themselves — it is that a reader takes `loop = "until_no_more_int_items"`
|
|
for a loop.
|
|
|
|
## Team templates
|
|
|
|
All 12 are structurally sound and **every declared skill resolves to a real
|
|
file — 0 unbound across all of them**, which is the `skill-binding-repair` work
|
|
holding (it was 55 of 85 dangling once).
|
|
|
|
| team | slots | risk profile | skills | evidenced? |
|
|
|---|---|---|---|---|
|
|
| `rust_sdlc` | planner coder tester reviewer committer | coding_readwrite | 19 | **yes** — harness default |
|
|
| `topic_research` | lead_researcher evidence_checker report_writer | research_web_readonly | 4 | **yes** — measured |
|
|
| `continuous_research` | paper_reader signal_ranker script_writer | research_readonly | 11 | **yes** — live runs |
|
|
| `backend` | api_designer db_engineer coder tester committer | coding_readwrite | 20 | no |
|
|
| `frontend` | designer coder tester committer | coding_readwrite | 12 | no |
|
|
| `mobile` | designer coder tester committer | coding_readwrite | 11 | no |
|
|
| `gpu` | arch_analyst kernel_author bench_engineer coder committer | coding_readwrite | 13 | no |
|
|
| `threejs` | scene_designer coder shader_author perf_engineer committer | coding_readwrite | 13 | no |
|
|
| `codebase_research` | code_archeologist architecture_mapper flow_tracer vault_scribe | research_readonly | 11 | no |
|
|
| `papers_research` | domain_scout paper_reader library_curator | research_web_readonly | 8 | no |
|
|
| `insight_research` | implementation_tracker novelty_hunter publication_drafter | research_readonly | 8 | no |
|
|
| `continuous_improvement` | brain_inspector improvement_proposer improvement_evaluator | research_readonly | 7 | no |
|
|
|
|
**Three of twelve are evidenced.** The other nine are well-formed scaffolding:
|
|
roles, prompts, brain seeds and resolving skills, and no run behind any of
|
|
them. They will probably work — they are structurally identical to the three
|
|
that do — but "probably" is the word, and this codebase has a name for the gap
|
|
between a thing being wired and a thing being proven.
|
|
|
|
Five of the nine need something we do not have: `frontend`, `mobile`,
|
|
`threejs` and `gpu` target stacks with no repository in the harness to point
|
|
them at, and `insight_research` needs a vault plus commit history. Those are
|
|
blocked on a target, not on the template. The remaining four —
|
|
`codebase_research`, `papers_research`, `continuous_improvement`, `backend` —
|
|
could be exercised against repositories we already have.
|
|
|
|
## What this means we can ask for today
|
|
|
|
**With confidence:** a repo-backed research→code loop on a Rust project, with
|
|
a judge, a commit gate, a verifier subagent and delivery to a branch. A
|
|
repo-less research report with sourced claims. Both are re-proven on every
|
|
harness run.
|
|
|
|
**With supervision:** a security scan and plan, a benchmark, a refactor. Each
|
|
has completed once under a standing check; none has the depth of evidence the
|
|
first two have.
|
|
|
|
**With attention:** continuous research. It works, it produced two good
|
|
digests today, and it has no guard.
|
|
|
|
**Not yet:** anything staffed from the other nine teams. Nothing is known to
|
|
be wrong with them; nothing is known to be right either.
|
|
|
|
## The two things worth doing next
|
|
|
|
1. **A `continuous-research` harness scenario.** The recipe with the most
|
|
moving parts is the only one with no standing check, and it now also
|
|
carries the paper triage (`kind` + `evidence`). A scenario that launches
|
|
it, waits, and asserts the manifest has tags and a spread, `analysis.md`
|
|
covers every paper, and `episode.json` parses would turn operator
|
|
attention into a guard.
|
|
2. **Exercise the four unblocked teams once each**, against repositories we
|
|
already have, and record what came out. Not to prove them mature — one run
|
|
is not maturity — but to find out whether they run at all, which nobody
|
|
currently knows.
|