clawmates/docs/TEMPLATE-MATURITY.md

# Template maturity — what our agents can actually be asked to do

A review of the 6 workflow recipes and 12 team templates, on 2026-09-22.
Graded by evidence, not by what the TOML declares: a recipe is mature when a
mission built from it has completed and something checked the result.

## Workflow recipes

| recipe | phases | standing check | staffing | grade |
|---|---|---|---|---|
| `research_and_code` | 2 (research → coding) | **13 harness scenarios** | research→`topic_research`, coding→`rust_sdlc` | **Proven** |
| `research_only` | 1 (research, repo-less) | 2 scenarios | `topic_research` | **Proven** |
| `security_hardening` | 3 (scan → research → coding) | 1 scenario | +`rust_sdlc` | Exercised |
| `benchmark` | 1 | 1 scenario | `rust_sdlc` | Exercised |
| `refactor` | 1 | 1 scenario | `rust_sdlc` | Exercised |
| `continuous_research` | 2 (analysis → script) | 1 scenario (`continuous-research`: digest on main, episode rendered, triage fields, saturation spread) | `continuous_research` | **Proven** — end to end incl. audio |
| `self_audit` | 1 (research) | none (3 live runs, 2026-09-22) | `continuous_improvement` | Exercised — diagnosed a planted cause |

`research_and_code` is the workhorse: every delivery, gate, judge, memory and
triage scenario is built on it, so it is re-proven on every harness run.
`research_only` earned its grade differently — its staffing was **measured and
corrected**: under the old `rust_sdlc` default it delivered 5 roles, 14 skill
deliveries and ~50 KB of prompt with **1 of 9** skills applicable; on
`topic_research` it is 3 roles, 4 deliveries, 24 KB, **4 of 4**.

`continuous_research` is the outlier and the one to fix first. It has the most
moving parts of any recipe — harvest → triage → manifest → analysis → ranking
→ script → `episode.json` → vault commit — it ran twice today and produced
real output both times, and **nothing in the harness would notice if it broke
tomorrow.** Every other recipe has a standing check; this one has operator
attention, which is not the same thing.

## What the recipes declare that does not fire

`phase_config::DECLARED_BUT_UNREAD` is the honest list, and recipes lean on it:

| key | used by | status |
|---|---|---|
| `loop` | `research_and_code` (`until_no_more_int_items`), `refactor` (`single_pass`) | **inert** — iteration is `max_iterations` + `done_when` |
| `produces` | almost every recipe (`["md"]`) | **inert** — artifact rendering is not driven by it |
| `input_from_phase` | `security_hardening` | **inert** — phases share a checkout |
| `mcp_bundles` (phase level) | `security_hardening` | **inert** — bundles come from the TEAM template |

None of this is hidden: `security_hardening.toml` carries a "what is real here
and what is decoration" section and annotates its own dead keys inline. That is
the right pattern and the other five should copy it. The risk is not the dead
keys themselves — it is that a reader takes `loop = "until_no_more_int_items"`
for a loop.

## Team templates

All 12 are structurally sound and **every declared skill resolves to a real
file — 0 unbound across all of them**, which is the `skill-binding-repair` work
holding (it was 55 of 85 dangling once).

| team | slots | risk profile | skills | evidenced? |
|---|---|---|---|---|
| `rust_sdlc` | planner coder tester reviewer committer | coding_readwrite | 19 | **yes** — harness default |
| `topic_research` | lead_researcher evidence_checker report_writer | research_web_readonly | 4 | **yes** — measured |
| `continuous_research` | paper_reader signal_ranker script_writer | research_readonly | 11 | **yes** — live runs |
| `backend` | api_designer db_engineer coder tester committer | coding_readwrite | 20 | **yes** — first run 2026-09-22 |
| `frontend` | designer coder tester committer | coding_readwrite | 12 | **work verified by hand; judge cannot run npm** |
| `mobile` | designer coder tester committer | coding_readwrite | 11 | no |
| `gpu` | arch_analyst kernel_author bench_engineer coder committer | coding_readwrite | 13 | no |
| `threejs` | scene_designer coder shader_author perf_engineer committer | coding_readwrite | 13 | no |
| `codebase_research` | code_archeologist architecture_mapper flow_tracer vault_scribe | research_readonly | 11 | **yes** — first run 2026-09-22 |
| `papers_research` | domain_scout paper_reader library_curator | research_web_readonly | 8 | **yes** — first run 2026-09-22 |
| `insight_research` | implementation_tracker novelty_hunter publication_drafter | research_readonly | 8 | no |
| `continuous_improvement` | brain_inspector improvement_proposer improvement_evaluator | research_readonly | 7 | **evidenced** — `self_audit` recipe; diagnosed a planted cause and found a real one |

**Six of twelve are evidenced.** `backend` was exercised on 2026-09-22,
the first time anything had been staffed from it: all five roles
provisioned with real agents (api_designer, db_engineer, coder, tester,
committer), and the mission delivered cursor pagination — `Paged<T>`,
`paginate<T: Clone>`, module declared, `cargo test` 3 passed — judged
**met on the first pass**, 2 files pushed. It also gave the task-permission
shadow its first confirmed `Edit` call.

`codebase_research` was exercised the same day, against the real
`clawmates` repository rather than the toy scratch crate, with a task that
cannot be faked from filenames: trace every hop of a mission agent's tool
call from the guest hook to the recorded event, naming file, function and
whether each hop runs in the guest or on the host. It produced a 189-line
`research/GATE-MAP.md` describing code committed **the same day** — the
four-field `NODE_EXTRACT` including `agent_type`, the shadow-mode
`would-deny.jsonl` semantics, `install_with` — and **all six of its line
citations verify exactly** (`hook_script_with` 562, `NODE_EXTRACT` 789,
`TaskPolicy` 297, `ROLE_POLICIES` 261, `settings_hook` 809,
`install_command_with` 819). Nothing hallucinated. **Five of twelve.**

`papers_research` followed: asked for a bounded library on microVM and
sandbox isolation for agents, it delivered a README index and four notes
with complete frontmatter, judged met. The judge could only check the
frontmatter was *present*; a paper team whose container has no
pdf-to-text tool is exactly where invented citations live, so every
arXiv id was checked against arXiv itself — **all four exist, with exact
title matches**. One of them, `2603.02277` (SandboxEscapeBench), is a paper
the operator's own research pass had already cited, which is independent
evidence it found on-topic work rather than plausible filler. **Six of
twelve.**

`continuous_improvement` ran too, and gets a different grade, because
running and working came apart. All three roles handed off with
attributed commits, and it was scrupulously honest: it filed **zero**
level-up proposals, its proposer committed *"no proposals warranted,
evidence base too thin"*, and its audit opens by stating the problem
exactly — *"No `.brain` files are accessible within the repo or mission
filesystem — brains are held in the platform, not checked in."*

That is the defect, and it is structural rather than a matter of data.
The template's subject is "every project agent's `.brain`"; those live in
the server's `/data/brains` volume, and **nothing delivers them into a
mission** — the same shape as `research-has-no-delivery-channel` and
`skills-had-no-delivery-channel`. So it audited the only agent-shaped
thing in reach, a `ROSTER.md` in the scratch repo, and read that file's
`6.1.128` (a guest kernel version) as an agent version, having no way to
know better. It is not counted as evidenced.

A channel alone would not rescue it. Mission crews are minted per mission
(reuse is off by decision), and missions write memory to the *repo* brain,
so every agent brain is a ~2 KB seed with no history to audit. The
template assumes long-lived agents that accumulate a record; the platform
makes disposable ones. That is a design decision to resolve, not a bug to
patch.

**Retargeted (2026-09-22).** The decision taken: audit the *repo* brain,
which is where missions actually write what they learn. The server now
exports it as `/mission/memory/PROJECT-MEMORY.md`, one line per judged
phase, and installs it into every mission that has a repository.

Two runs followed, and they separate the channel from the steering:

- **v2 (01a0cad8) — channel works, steering does not.** The 2.4 KB record
  was installed. The agent made 50 tool calls, 0 of them touched it, 13
  went to the roster, and the judge passed it. The brief and `done_when`
  were reused from v1 ("read each agent's brain"), and the template's
  retargeted role prompts are **inert on mission turns** (see
  `prompt-injection-two-paths`). Rewriting `system_prompt` cannot aim a
  mission; only the brief can, and nothing supplies one for this team.
- **v3 (01a0cade) — right answer, failed by the judge.** With a brief and
  `done_when` pointing at the record, 3 of 7 tool calls read it. The audit
  states the record exactly (4 lines, MET 4 / UNMET 0), finds no pattern,
  and files **zero** proposals ("a single UNMET line would be an incident;
  this record has zero"), the correct result on an all-MET record. The
  judge failed it, correctly by the letter: the condition required each
  finding to *quote* the record lines, and the audit summarised them in a
  table instead. `max_iterations` was 1, so there was no retry.

So the subject is now reachable and the reasoning holds up. The unproven
part is the case this team exists for, a record *with* failures. That
needs a repository whose history contains UNMET verdicts, and a default
brief so no caller has to write one. Still not counted as evidenced.

**The `self_audit` recipe, and a record with real failures (2026-09-22).**
The brief now lives in `templates/workflows/self_audit.toml`, so a mission
naming only `template_kind = "self_audit"` and a repo is staffed and aimed
without the caller writing anything. To give it something to find, two
missions were planted on the scratch repo whose `done_when` required a
*Limitations* section their brief never mentioned. The judge marked both
UNMET for exactly that. The record then held 7 lines: 4 MET, 3 UNMET, the
third being v3's quoting failure, a natural single-incident control.

Mission 01a0cb3f, launched with no brief of its own, read the record (8 of
25 tool calls), stated 7 / 4 / 3, found the one pattern, quoted both lines
verbatim, and named the quoting failure as an incident with no proposal. The
judge passed it on iteration 1 of 2.

Its **diagnosis was wrong**, and the record is why. It concluded the agents
"produced the main content but omitted the final required section" and
proposed a write-then-check checklist. The recorded prompts show the agents
were never told: none of the three mentions *Limitations*, because working
agents are not shown their completion condition. The checklist would not have
helped. The record carried the condition and not the brief, so a
brief/condition mismatch was invisible to any reader of it. Three fixes
followed:

- verdict lines now carry the phase's **brief** beside its condition;
- the mission id in a line is the uuid **tail**: the two planted missions,
  launched 34 s apart, both rendered as `01a0cb38` (a UUIDv7 leads with its
  timestamp), and the audit read them as one mission;
- the recipe has the auditor **copy the record** into the checkout, since the
  judge cannot read `/mission/memory`. Iteration 0 failed as "could not verify
  the quotes" until the agent did this unprompted.

Counts, pattern-vs-incident discipline and verbatim evidence: shown. A
correct diagnosis of a cause: not yet shown, and until now not possible.

**Re-run with the fixes (mission 01a0cb54) — evidenced.** A second planted
pair, whose lines now carry their briefs, joined the record (11 lines: 5 MET,
6 UNMET). Launched again with nothing but `template_kind` and a repo, the
audit copied the record into the checkout, and the judge confirmed its counts
by grep against that copy. It passed on iteration 0. Its findings:

- **The planted pattern, diagnosed correctly.** *"The requirement appeared in
  the condition but not in the brief, so the agent was never asked to write
  it … a setup defect."* The proposal is the right one: the brief, old wording
  quoted, new wording asking for the section.
- **A pattern nobody planted.** Its own two earlier UNMET audits (v3, and
  01a0cb3f's first attempt) were judged for the same missing thing. It noted
  that their briefs predate the record carrying briefs, so it could not tell
  what those agents were told, and proposed "copy the record, quote verbatim".
  That is the fix made by hand in `9ca71e5`, found independently from the
  record alone.

**Seven of twelve.** Limits: the planted pattern is one shape (brief omits a
condition's requirement); the pre-fix lines still render two missions as
`01a0cb38`, which the audit counted correctly as two lines; and it proposes
into a markdown file. Nothing reads `research/IMPROVEMENT-AUDIT.md` or files
`level_up_proposals` from it.

**Two of the remaining six are still unevidenced**, and the other four
are blocked on a target stack. The other nine are well-formed scaffolding:
roles, prompts, brain seeds and resolving skills, and no run behind any of
them. They will probably work — they are structurally identical to the three
that do — but "probably" is the word, and this codebase has a name for the gap
between a thing being wired and a thing being proven.

Five of the nine need something we do not have: `frontend`, `mobile`,
`threejs` and `gpu` target stacks with no repository in the harness to point
them at, and `insight_research` needs a vault plus commit history. Those are
blocked on a target, not on the template. The remaining four —
`codebase_research`, `papers_research`, `continuous_improvement`, `backend` —
could be exercised against repositories we already have.

## `frontend` — the work is right; the judge cannot check it (2026-09-22)

Target: `osobh/clawmates-frontend-scratch` (private, disposable), a Vite +
React 19 + TypeScript + Tailwind v4 app with Vitest and Testing Library,
checked locally before any mission touched it. Task: an accessible Tabs
component (ARIA tablist/tab/tabpanel wiring, roving tabindex, Arrow/Home/End
with wrap-around, Tailwind focus style), with the brief and `done_when`
stating the same requirements.

Mission 01a0cbc7 delivered one commit, +191 lines, three files. Re-run
locally from the delivered branch: **10/10 tests pass (9 new), `tsc -b`
clean**. The component is correct on every stated requirement; one nit, the
tab buttons lack `type="button"`. The agents ran `npm test` and the
typecheck repeatedly in the mission container, all green.

The mission **failed**, both iterations, on the judge's side. The judge
verifies in `clawmates-runtime` against a copy that, by design, excludes
`node_modules` (the transport packer's list). It cannot reinstall:
`clawmates_core` has no gateway, so `npm ci` gets `EAI_AGAIN`. Every npm
project, which covers `frontend`, `mobile` and `threejs`, therefore fails any
`done_when` that asks for a passing suite, however good the work. Rust
escapes it only because the scratch crates have no dependencies. The judge
read the source correctly both times; it failed only on the suite it could
not run.

Not counted as evidenced until the judge can run the suite itself. How it
should get the dependencies (offline from the lockfile, registry egress, or
not at all) is an operator decision about the verifier's isolation.

## One thing both runs showed: agents reach for Bash

Cumulative tool calls across every mission since the policy shipped:
**Bash 74, Read 34, Write 15, Glob 3, Edit 1, Agent 1**. A read-only
mapping mission that could have used `Grep` and `Glob` used `Bash` 33
times and `Read` 5. That matters for task permission: the allowlist's
dedicated-tool entries (`Grep`, `ToolSearch`, `WebFetch`, `TodoWrite`)
may simply never be exercised, so "unconfirmed by a real run" will not
converge for them — and it means the surface that actually needs
governing is `Bash`, which the floor rules already cover.

## What this means we can ask for today

**With confidence:** a repo-backed research→code loop on a Rust project, with
a judge, a commit gate, a verifier subagent and delivery to a branch. A
repo-less research report with sourced claims. Both are re-proven on every
harness run.

**With supervision:** a security scan and plan, a benchmark, a refactor. Each
has completed once under a standing check; none has the depth of evidence the
first two have.

**With attention:** continuous research. It works, it produced two good
digests today, and it has no guard.

**Not yet:** anything staffed from the other nine teams. Nothing is known to
be wrong with them; nothing is known to be right either.

## The two things worth doing next

1. **A `continuous-research` harness scenario.** The recipe with the most
   moving parts is the only one with no standing check, and it now also
   carries the paper triage (`kind` + `evidence`). A scenario that launches
   it, waits, and asserts the manifest has tags and a spread, `analysis.md`
   covers every paper, and `episode.json` parses would turn operator
   attention into a guard.
2. **Exercise the four unblocked teams once each**, against repositories we
   already have, and record what came out. Not to prove them mature — one run
   is not maturity — but to find out whether they run at all, which nobody
   currently knows.