From 594cd99e544844cf1c198834be3917c3aebeae0f Mon Sep 17 00:00:00 2001 From: Omar Sobh Date: Tue, 22 Sep 2026 10:26:19 -0500 Subject: [PATCH] =?UTF-8?q?docs:=20template=20maturity=20=E2=80=94=203=20o?= =?UTF-8?q?f=2012=20teams=20evidenced,=202=20of=206=20recipes=20proven?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A review of what our agents can actually be asked to do, graded by evidence rather than by what the TOML declares. Recipes: research_and_code (13 harness scenarios) and research_only (staffing measured and corrected — 1 of 9 applicable skills under the old rust_sdlc default, 4 of 4 under topic_research) are proven. security_hardening, benchmark and refactor are exercised once each. continuous_research is the outlier: the most moving parts of any recipe, ran twice today, and NOTHING in the harness would notice if it broke. Dead keys the recipes lean on, from phase_config::DECLARED_BUT_UNREAD: loop, produces, input_from_phase, and mcp_bundles at phase level. Not hidden — security_hardening.toml annotates its own decoration inline, and the other five should copy that. The risk is a reader taking loop = "until_no_more_int_items" for a loop. Teams: all 12 resolve every declared skill (0 unbound, the skill-binding-repair work holding), but only rust_sdlc, topic_research and continuous_research have a run behind them. The other nine are well-formed scaffolding. Five are blocked on a target stack we do not have a repo for; four could be exercised against repositories we already have, and nobody currently knows whether they run at all. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01WZb5A2kfVfjpdwSochkuHz --- docs/TEMPLATE-MATURITY.md | 111 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 111 insertions(+) create mode 100644 docs/TEMPLATE-MATURITY.md diff --git a/docs/TEMPLATE-MATURITY.md b/docs/TEMPLATE-MATURITY.md new file mode 100644 index 0000000..9b2c6bb --- /dev/null +++ b/docs/TEMPLATE-MATURITY.md @@ -0,0 +1,111 @@ +# Template maturity — what our agents can actually be asked to do + +A review of the 6 workflow recipes and 12 team templates, on 2026-09-22. +Graded by evidence, not by what the TOML declares: a recipe is mature when a +mission built from it has completed and something checked the result. + +## Workflow recipes + +| recipe | phases | standing check | staffing | grade | +|---|---|---|---|---| +| `research_and_code` | 2 (research → coding) | **13 harness scenarios** | research→`topic_research`, coding→`rust_sdlc` | **Proven** | +| `research_only` | 1 (research, repo-less) | 2 scenarios | `topic_research` | **Proven** | +| `security_hardening` | 3 (scan → research → coding) | 1 scenario | +`rust_sdlc` | Exercised | +| `benchmark` | 1 | 1 scenario | `rust_sdlc` | Exercised | +| `refactor` | 1 | 1 scenario | `rust_sdlc` | Exercised | +| `continuous_research` | 2 (analysis → script) | **none** | `continuous_research` | Runs live, unguarded | + +`research_and_code` is the workhorse: every delivery, gate, judge, memory and +triage scenario is built on it, so it is re-proven on every harness run. +`research_only` earned its grade differently — its staffing was **measured and +corrected**: under the old `rust_sdlc` default it delivered 5 roles, 14 skill +deliveries and ~50 KB of prompt with **1 of 9** skills applicable; on +`topic_research` it is 3 roles, 4 deliveries, 24 KB, **4 of 4**. + +`continuous_research` is the outlier and the one to fix first. It has the most +moving parts of any recipe — harvest → triage → manifest → analysis → ranking +→ script → `episode.json` → vault commit — it ran twice today and produced +real output both times, and **nothing in the harness would notice if it broke +tomorrow.** Every other recipe has a standing check; this one has operator +attention, which is not the same thing. + +## What the recipes declare that does not fire + +`phase_config::DECLARED_BUT_UNREAD` is the honest list, and recipes lean on it: + +| key | used by | status | +|---|---|---| +| `loop` | `research_and_code` (`until_no_more_int_items`), `refactor` (`single_pass`) | **inert** — iteration is `max_iterations` + `done_when` | +| `produces` | almost every recipe (`["md"]`) | **inert** — artifact rendering is not driven by it | +| `input_from_phase` | `security_hardening` | **inert** — phases share a checkout | +| `mcp_bundles` (phase level) | `security_hardening` | **inert** — bundles come from the TEAM template | + +None of this is hidden: `security_hardening.toml` carries a "what is real here +and what is decoration" section and annotates its own dead keys inline. That is +the right pattern and the other five should copy it. The risk is not the dead +keys themselves — it is that a reader takes `loop = "until_no_more_int_items"` +for a loop. + +## Team templates + +All 12 are structurally sound and **every declared skill resolves to a real +file — 0 unbound across all of them**, which is the `skill-binding-repair` work +holding (it was 55 of 85 dangling once). + +| team | slots | risk profile | skills | evidenced? | +|---|---|---|---|---| +| `rust_sdlc` | planner coder tester reviewer committer | coding_readwrite | 19 | **yes** — harness default | +| `topic_research` | lead_researcher evidence_checker report_writer | research_web_readonly | 4 | **yes** — measured | +| `continuous_research` | paper_reader signal_ranker script_writer | research_readonly | 11 | **yes** — live runs | +| `backend` | api_designer db_engineer coder tester committer | coding_readwrite | 20 | no | +| `frontend` | designer coder tester committer | coding_readwrite | 12 | no | +| `mobile` | designer coder tester committer | coding_readwrite | 11 | no | +| `gpu` | arch_analyst kernel_author bench_engineer coder committer | coding_readwrite | 13 | no | +| `threejs` | scene_designer coder shader_author perf_engineer committer | coding_readwrite | 13 | no | +| `codebase_research` | code_archeologist architecture_mapper flow_tracer vault_scribe | research_readonly | 11 | no | +| `papers_research` | domain_scout paper_reader library_curator | research_web_readonly | 8 | no | +| `insight_research` | implementation_tracker novelty_hunter publication_drafter | research_readonly | 8 | no | +| `continuous_improvement` | brain_inspector improvement_proposer improvement_evaluator | research_readonly | 7 | no | + +**Three of twelve are evidenced.** The other nine are well-formed scaffolding: +roles, prompts, brain seeds and resolving skills, and no run behind any of +them. They will probably work — they are structurally identical to the three +that do — but "probably" is the word, and this codebase has a name for the gap +between a thing being wired and a thing being proven. + +Five of the nine need something we do not have: `frontend`, `mobile`, +`threejs` and `gpu` target stacks with no repository in the harness to point +them at, and `insight_research` needs a vault plus commit history. Those are +blocked on a target, not on the template. The remaining four — +`codebase_research`, `papers_research`, `continuous_improvement`, `backend` — +could be exercised against repositories we already have. + +## What this means we can ask for today + +**With confidence:** a repo-backed research→code loop on a Rust project, with +a judge, a commit gate, a verifier subagent and delivery to a branch. A +repo-less research report with sourced claims. Both are re-proven on every +harness run. + +**With supervision:** a security scan and plan, a benchmark, a refactor. Each +has completed once under a standing check; none has the depth of evidence the +first two have. + +**With attention:** continuous research. It works, it produced two good +digests today, and it has no guard. + +**Not yet:** anything staffed from the other nine teams. Nothing is known to +be wrong with them; nothing is known to be right either. + +## The two things worth doing next + +1. **A `continuous-research` harness scenario.** The recipe with the most + moving parts is the only one with no standing check, and it now also + carries the paper triage (`kind` + `evidence`). A scenario that launches + it, waits, and asserts the manifest has tags and a spread, `analysis.md` + covers every paper, and `episode.json` parses would turn operator + attention into a guard. +2. **Exercise the four unblocked teams once each**, against repositories we + already have, and record what came out. Not to prove them mature — one run + is not maturity — but to find out whether they run at all, which nobody + currently knows.