fix(memory): verdict lines carry the brief and a distinguishing mission id
deploy / test (push) Successful in 4m50s
deploy / build (push) Successful in 5m45s

The first self_audit run on a planted brief/condition mismatch found the
pattern and quoted it, then diagnosed "the agent skipped the section": the
record held the condition but not the brief, and working agents never see the
condition. It also read two missions as one — a UUIDv7's first 8 chars are a
timestamp, and missions 34 s apart both rendered as 01a0cb38.

- verdict_line records the phase brief (config.task) beside the condition
- the short mission id is the uuid tail
- self_audit copies the record into the checkout (the judge cannot read
  /mission/memory) and compares brief with condition

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
Omar Sobh
2026-09-22 17:42:37 -05:00
co-authored by Claude Opus 5.5
parent fca828b5a1
commit 9ca71e5fa4
4 changed files with 96 additions and 15 deletions
+49 -9
View File
@@ -49,6 +49,7 @@ fn brain_path(dir: &Path, repo_id: Uuid) -> PathBuf {
pub fn verdict_line(
mission_id: Uuid,
phase_kind: &str,
brief: &str,
condition: &str,
verdict: &crate::evaluator::Verdict,
) -> Option<String> {
@@ -65,10 +66,26 @@ pub fn verdict_line(
if finding.is_empty() {
return None;
}
// The TAIL. A UUIDv7 leads with its timestamp, so two missions launched
// seconds apart share their first eight characters — measured: two planted
// missions 34 s apart both rendered as `01a0cb38`, and the self-audit read
// them as one mission failing twice.
let short = mission_id.simple().to_string();
let short = &short[short.len() - 8..];
// The brief is what the agent was TOLD; the condition is what it was
// judged against, and working agents are not shown it. Without the brief a
// reader of the record cannot tell "the agent skipped a requirement" from
// "nobody asked for it" — the first self-audit on a planted brief/condition
// mismatch diagnosed the former and proposed a fix that would not have
// helped.
let brief = brief.trim();
let told = if brief.is_empty() {
String::new()
} else {
format!(" — brief: {}", head(&brief.split_whitespace().collect::<Vec<_>>().join(" "), 200))
};
Some(format!(
"{outcome}{phase_kind} phase of mission {} — condition: {} — judge: {}",
&short[..8],
"{outcome}{phase_kind} phase of mission {short}{told} — condition: {} — judge: {}",
head(condition, 200),
head(finding, 400),
))
@@ -81,6 +98,7 @@ pub fn remember_verdict(
repo_id: Uuid,
mission_id: Uuid,
phase_kind: &str,
brief: &str,
condition: &str,
verdict: &crate::evaluator::Verdict,
) {
@@ -89,6 +107,7 @@ pub fn remember_verdict(
repo_id,
mission_id,
phase_kind,
brief,
condition,
verdict,
)
@@ -99,10 +118,11 @@ fn remember_in(
repo_id: Uuid,
mission_id: Uuid,
phase_kind: &str,
brief: &str,
condition: &str,
verdict: &crate::evaluator::Verdict,
) {
let Some(line) = verdict_line(mission_id, phase_kind, condition, verdict) else {
let Some(line) = verdict_line(mission_id, phase_kind, brief, condition, verdict) else {
return;
};
let path = brain_path(dir, repo_id);
@@ -329,7 +349,7 @@ mod tests {
#[test]
fn unmet_remembers_guidance_not_reason() {
let v = verdict(false, "token ZZQX-9 is absent", "the required marker is absent", None);
let line = verdict_line(Uuid::nil(), "coding", "cond", &v).unwrap();
let line = verdict_line(Uuid::nil(), "coding", "", "cond", &v).unwrap();
assert!(line.starts_with("UNMET — coding phase"));
assert!(line.contains("the required marker is absent"));
assert!(!line.contains("ZZQX-9"));
@@ -338,16 +358,36 @@ mod tests {
#[test]
fn met_remembers_what_the_judge_found() {
let v = verdict(true, "MICROVM.md holds both lines", "", None);
let line = verdict_line(Uuid::nil(), "coding", "cond", &v).unwrap();
let line = verdict_line(Uuid::nil(), "coding", "", "cond", &v).unwrap();
assert!(line.starts_with("MET — "));
assert!(line.contains("MICROVM.md holds both lines"));
}
/// What the agent was told sits beside what it was judged against, and two
/// missions launched back to back stay two missions. Both were missing when
/// the first self-audit read a planted brief/condition mismatch as "the
/// agent skipped the section" and two missions as one.
#[test]
fn a_line_carries_the_brief_and_a_distinguishing_mission_id() {
let v = verdict(false, "r", "add a Limitations section", None);
let a = Uuid::now_v7();
let b = Uuid::now_v7();
let brief = "Write NOTES.md:\n five bullet points";
let la = verdict_line(a, "research", brief, "ends with Limitations", &v).unwrap();
let lb = verdict_line(b, "research", brief, "ends with Limitations", &v).unwrap();
assert!(la.contains(" — brief: Write NOTES.md: five bullet points — condition: "), "{la}");
assert_ne!(la, lb, "same-second UUIDv7s must not render as one mission");
let tail = a.simple().to_string();
assert!(la.contains(&format!("mission {}", &tail[tail.len() - 8..])), "{la}");
let none = verdict_line(a, "research", " ", "c", &v).unwrap();
assert!(!none.contains("brief:"), "an empty brief adds no segment: {none}");
}
/// No judgement, no lesson.
#[test]
fn an_unreachable_judge_leaves_no_memory() {
let v = verdict(false, "could not evaluate", "could not evaluate", Some("429"));
assert!(verdict_line(Uuid::nil(), "coding", "cond", &v).is_none());
assert!(verdict_line(Uuid::nil(), "coding", "", "cond", &v).is_none());
}
#[test]
@@ -366,10 +406,10 @@ mod tests {
let repo = Uuid::now_v7();
assert!(export_in(&dir, repo).is_none(), "no brain, no export");
remember_in(&dir, repo, Uuid::now_v7(), "coding", "first",
remember_in(&dir, repo, Uuid::now_v7(), "coding", "", "first",
&verdict(false, "r", "the tests do not cover the empty case", None));
std::thread::sleep(std::time::Duration::from_millis(5));
remember_in(&dir, repo, Uuid::now_v7(), "coding", "second",
remember_in(&dir, repo, Uuid::now_v7(), "coding", "", "second",
&verdict(true, "all three tests pass", "", None));
let md = export_in(&dir, repo).expect("two verdicts, so an export");
@@ -398,7 +438,7 @@ mod tests {
"",
None,
);
remember_in(&dir, repo, Uuid::now_v7(), "benchmark", "a baseline is recorded", &v);
remember_in(&dir, repo, Uuid::now_v7(), "benchmark", "", "a baseline is recorded", &v);
let got = recall_in(&dir, repo, "record a performance baseline for the hot path", RECALL_K);
assert_eq!(got.len(), 1, "{got:?}");
assert!(got[0].starts_with("MET — benchmark phase"), "{}", got[0]);
+5 -2
View File
@@ -2622,7 +2622,7 @@ async fn evaluate_finished_phases(
) -> Result<(), String> {
let rows = sqlx::query(
"SELECT mp.id, mp.mission_id, mp.kind, mp.done_when, mp.max_iterations, mp.iteration,
m.runtime_kind, m.repo_id
mp.config->>'task' AS task, m.runtime_kind, m.repo_id
FROM mission_phases mp
JOIN missions m ON m.id = mp.mission_id
WHERE mp.status = 'evaluating' AND m.status = 'running'
@@ -2692,7 +2692,10 @@ async fn evaluate_finished_phases(
// The project remembers the verdict. Only a repo-backed mission has a
// project to remember into; a repo-less one leaves no trace here.
if let Some(repo) = repo_id {
crate::mission_memory::remember_verdict(repo, mission_id, &kind, &condition, &verdict);
let brief = row.get::<Option<String>, _>("task").unwrap_or_default();
crate::mission_memory::remember_verdict(
repo, mission_id, &kind, &brief, &condition, &verdict,
);
}
// A judge that could not be REACHED has not judged. `Verdict.error` is
+35 -1
View File
@@ -65,7 +65,7 @@ holding (it was 55 of 85 dangling once).
| `codebase_research` | code_archeologist architecture_mapper flow_tracer vault_scribe | research_readonly | 11 | **yes** — first run 2026-09-22 |
| `papers_research` | domain_scout paper_reader library_curator | research_web_readonly | 8 | **yes** — first run 2026-09-22 |
| `insight_research` | implementation_tracker novelty_hunter publication_drafter | research_readonly | 8 | no |
| `continuous_improvement` | brain_inspector improvement_proposer improvement_evaluator | research_readonly | 7 | **reaches its subject; right answer, judge-failed on quoting** |
| `continuous_improvement` | brain_inspector improvement_proposer improvement_evaluator | research_readonly | 7 | **finds real patterns; diagnosis blocked by the record (fixed, unre-run)** |
**Six of twelve are evidenced.** `backend` was exercised on 2026-09-22,
the first time anything had been staffed from it: all five roles
@@ -150,6 +150,40 @@ part is the case this team exists for, a record *with* failures. That
needs a repository whose history contains UNMET verdicts, and a default
brief so no caller has to write one. Still not counted as evidenced.
**The `self_audit` recipe, and a record with real failures (2026-09-22).**
The brief now lives in `templates/workflows/self_audit.toml`, so a mission
naming only `template_kind = "self_audit"` and a repo is staffed and aimed
without the caller writing anything. To give it something to find, two
missions were planted on the scratch repo whose `done_when` required a
*Limitations* section their brief never mentioned. The judge marked both
UNMET for exactly that. The record then held 7 lines: 4 MET, 3 UNMET, the
third being v3's quoting failure, a natural single-incident control.
Mission 01a0cb3f, launched with no brief of its own, read the record (8 of
25 tool calls), stated 7 / 4 / 3, found the one pattern, quoted both lines
verbatim, and named the quoting failure as an incident with no proposal. The
judge passed it on iteration 1 of 2.
Its **diagnosis was wrong**, and the record is why. It concluded the agents
"produced the main content but omitted the final required section" and
proposed a write-then-check checklist. The recorded prompts show the agents
were never told: none of the three mentions *Limitations*, because working
agents are not shown their completion condition. The checklist would not have
helped. The record carried the condition and not the brief, so a
brief/condition mismatch was invisible to any reader of it. Three fixes
followed:
- verdict lines now carry the phase's **brief** beside its condition;
- the mission id in a line is the uuid **tail**: the two planted missions,
launched 34 s apart, both rendered as `01a0cb38` (a UUIDv7 leads with its
timestamp), and the audit read them as one mission;
- the recipe has the auditor **copy the record** into the checkout, since the
judge cannot read `/mission/memory`. Iteration 0 failed as "could not verify
the quotes" until the agent did this unprompted.
Counts, pattern-vs-incident discipline and verbatim evidence: shown. A
correct diagnosis of a cause: not yet shown, and until now not possible.
**Two of the remaining six are still unevidenced**, and the other four
are blocked on a target stack. The other nine are well-formed scaffolding:
roles, prompts, brain seeds and resolving skills, and no run behind any of
+7 -3
View File
@@ -22,16 +22,17 @@ default_topology = "pipeline"
task = """
Audit what this repository's missions have learned.
The record is /mission/memory/PROJECT-MEMORY.md: one line per judged phase of every mission on this repository — MET or UNMET, the phase kind, the completion condition, and the judge's reason. That record is the whole subject of this audit. Do not look for agent brain files, rosters or anything else to audit; there is nothing else.
The record is /mission/memory/PROJECT-MEMORY.md: one line per judged phase of every mission on this repository — MET or UNMET, the phase kind, the brief the agent was given (newer lines), the completion condition it was judged against, and the judge's reason. Working agents are NOT shown their completion condition; they see only the brief. That record is the whole subject of this audit. Do not look for agent brain files, rosters or anything else to audit; there is nothing else.
If the file is absent, the repository has no judged history yet. Write that in research/IMPROVEMENT-AUDIT.md and stop — that is a complete audit.
Otherwise:
Otherwise, first copy the record verbatim to research/PROJECT-RECORD.md. The judge of this audit cannot read /mission/memory; that copy is how your quotes get checked.
1. Count the lines, and how many are MET and how many UNMET. State the numbers.
2. Look for patterns, not incidents: the same kind of work failing more than once, the judge asking for the same missing thing more than once, a condition that passed only after several iterations. One UNMET line is an incident, not a pattern — name it as an incident and propose nothing for it.
3. For each pattern, quote the record lines it rests on, copied verbatim as they appear in the file. A paraphrase or a summary table is not evidence; the quoted lines are.
4. For each pattern, propose ONE change to how the work is set up — a reworded completion condition (old and new wording), a missing skill, a recipe setting, a task brief — and say why it would have turned those UNMET lines into MET. A proposal that fewer than two lines support is dropped, not softened.
When a line carries a brief, compare it with the condition. A requirement that appears in the condition and not in the brief was never asked of the agent — that is a setup defect, not the agent skipping work, and the fix belongs in the brief, not in the agent's discipline.
5. If the record shows no failure pattern, say so, and say that no change is warranted. That is a correct result, and on a thin or clean record it is the expected one. Never invent a pattern to have something to report.
You cannot apply anything and there is nothing to submit to. Write everything to research/IMPROVEMENT-AUDIT.md; the operator decides.
@@ -39,8 +40,11 @@ You cannot apply anything and there is nothing to submit to. Write everything to
# Wording follows the measured rule: say what the file CONTAINS. The v3 run's
# condition said "quotes the record lines each finding rests on" and the judge
# failed an audit with no failure patterns for paraphrasing its observations.
# The judge cannot read /mission/memory, so iteration 0 of the first recipe run
# failed as "could not verify the quotes"; the agent then copied the record into
# the checkout unprompted and passed. The copy is now part of the brief.
# The quoting requirement now attaches to what needs evidence — a reported
# pattern — and the clean-record outcome is named so it reads as a pass.
done_when = "research/IMPROVEMENT-AUDIT.md exists and either states that no project record was present, or states how many verdict lines the record contains with the MET and UNMET counts and then, for each failure pattern it reports, quotes at least two record lines verbatim and proposes one change, or states that the record shows no failure pattern and no change is warranted"
done_when = "research/IMPROVEMENT-AUDIT.md exists and either states that no project record was present, or research/PROJECT-RECORD.md holds the record's lines and the audit states how many verdict lines the record contains with the MET and UNMET counts and then, for each failure pattern it reports, quotes at least two record lines verbatim and proposes one change, or states that the record shows no failure pattern and no change is warranted"
max_iterations = 2
commit_policy = "always"