fix(memory): verdict lines carry the brief and a distinguishing mission id
The first self_audit run on a planted brief/condition mismatch found the pattern and quoted it, then diagnosed "the agent skipped the section": the record held the condition but not the brief, and working agents never see the condition. It also read two missions as one — a UUIDv7's first 8 chars are a timestamp, and missions 34 s apart both rendered as 01a0cb38. - verdict_line records the phase brief (config.task) beside the condition - the short mission id is the uuid tail - self_audit copies the record into the checkout (the judge cannot read /mission/memory) and compares brief with condition Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
fca828b5a1
commit
9ca71e5fa4
@@ -65,7 +65,7 @@ holding (it was 55 of 85 dangling once).
|
||||
| `codebase_research` | code_archeologist architecture_mapper flow_tracer vault_scribe | research_readonly | 11 | **yes** — first run 2026-09-22 |
|
||||
| `papers_research` | domain_scout paper_reader library_curator | research_web_readonly | 8 | **yes** — first run 2026-09-22 |
|
||||
| `insight_research` | implementation_tracker novelty_hunter publication_drafter | research_readonly | 8 | no |
|
||||
| `continuous_improvement` | brain_inspector improvement_proposer improvement_evaluator | research_readonly | 7 | **reaches its subject; right answer, judge-failed on quoting** |
|
||||
| `continuous_improvement` | brain_inspector improvement_proposer improvement_evaluator | research_readonly | 7 | **finds real patterns; diagnosis blocked by the record (fixed, unre-run)** |
|
||||
|
||||
**Six of twelve are evidenced.** `backend` was exercised on 2026-09-22,
|
||||
the first time anything had been staffed from it: all five roles
|
||||
@@ -150,6 +150,40 @@ part is the case this team exists for, a record *with* failures. That
|
||||
needs a repository whose history contains UNMET verdicts, and a default
|
||||
brief so no caller has to write one. Still not counted as evidenced.
|
||||
|
||||
**The `self_audit` recipe, and a record with real failures (2026-09-22).**
|
||||
The brief now lives in `templates/workflows/self_audit.toml`, so a mission
|
||||
naming only `template_kind = "self_audit"` and a repo is staffed and aimed
|
||||
without the caller writing anything. To give it something to find, two
|
||||
missions were planted on the scratch repo whose `done_when` required a
|
||||
*Limitations* section their brief never mentioned. The judge marked both
|
||||
UNMET for exactly that. The record then held 7 lines: 4 MET, 3 UNMET, the
|
||||
third being v3's quoting failure, a natural single-incident control.
|
||||
|
||||
Mission 01a0cb3f, launched with no brief of its own, read the record (8 of
|
||||
25 tool calls), stated 7 / 4 / 3, found the one pattern, quoted both lines
|
||||
verbatim, and named the quoting failure as an incident with no proposal. The
|
||||
judge passed it on iteration 1 of 2.
|
||||
|
||||
Its **diagnosis was wrong**, and the record is why. It concluded the agents
|
||||
"produced the main content but omitted the final required section" and
|
||||
proposed a write-then-check checklist. The recorded prompts show the agents
|
||||
were never told: none of the three mentions *Limitations*, because working
|
||||
agents are not shown their completion condition. The checklist would not have
|
||||
helped. The record carried the condition and not the brief, so a
|
||||
brief/condition mismatch was invisible to any reader of it. Three fixes
|
||||
followed:
|
||||
|
||||
- verdict lines now carry the phase's **brief** beside its condition;
|
||||
- the mission id in a line is the uuid **tail**: the two planted missions,
|
||||
launched 34 s apart, both rendered as `01a0cb38` (a UUIDv7 leads with its
|
||||
timestamp), and the audit read them as one mission;
|
||||
- the recipe has the auditor **copy the record** into the checkout, since the
|
||||
judge cannot read `/mission/memory`. Iteration 0 failed as "could not verify
|
||||
the quotes" until the agent did this unprompted.
|
||||
|
||||
Counts, pattern-vs-incident discipline and verbatim evidence: shown. A
|
||||
correct diagnosis of a cause: not yet shown, and until now not possible.
|
||||
|
||||
**Two of the remaining six are still unevidenced**, and the other four
|
||||
are blocked on a target stack. The other nine are well-formed scaffolding:
|
||||
roles, prompts, brain seeds and resolving skills, and no run behind any of
|
||||
|
||||
Reference in New Issue
Block a user