Omar SobhandClaude Opus 5
18dc0b964b
Four defects, found by checking the audit's claims instead of trusting them. Two of the audit's own findings turned out to be wrong, and the registry that exists to record which config keys are read was itself inaccurate — so the corrections are part of the change. upsert_task raised 42P10 on every call, for every caller `mission_tasks_external_uniq` is a PARTIAL unique index (WHERE external_id IS NOT NULL). Postgres will not match a partial index to an ON CONFLICT target unless the statement repeats the predicate, so the upsert failed on its first row. Both callers — the task-card parser that turns INT markers into tasks, and the security scanner — map the error to a string their caller logs. Two features were broken and nothing was red. Regression test in cm-db with a negative control: reverting the WHERE reproduces 42P10 exactly. the security scan never ran `security_scan::run` was reachable only from an operator button, so security_hardening.toml — a workflow whose entire first phase is a scan — ran an agent that was never told to scan and never fired the scanner either. phase_runner now sweeps finished security_scan phases, mirroring the benchmark baseline sweep that was added for the identical defect. Guarded on a new completion marker rather than on findings: a clean scan writes no findings, so a findings-guard would rescan forever. The marker also answers the question an operator actually asks, which is not "how many findings" but "was this looked at, by what, and when". two recipes could not fail security_hardening.toml and benchmark.toml carried no `task` and no `done_when` on any phase. A phase without done_when never enters evaluating, is never judged, and reports completed whatever it did — so a security mission could scan nothing and go green, and a benchmark mission could record no baseline that the next refactor would then compare against. Both now state the work and the condition, with inert keys annotated inline rather than deleted, so the gap between what a recipe asks for and what a phase receives stays visible. the config registry was wrong in both directions `harness` was listed NOT IMPLEMENTED while benchmark_runner reads it and phase_runner runs a baseline through it. `tools` was listed NOT IMPLEMENTED while security_scan::run reads it. A registry that exists so an operator can trust what a recipe does is worse than useless when it is inaccurate. Both corrected, `bench_name` and `cmd` added, and `test_command` deleted — it had neither a reader nor a writer, so it described a situation that could not arise. Also: CLAWMATES_JUDGE_MODEL had two different defaults (opus-4-8 in routes/topology.rs vs opus-5 in cm_runtime::judge_model) and a doc comment naming a third; topology now calls the one function. GITEA_TOKEN's absence in mission_plan is stated rather than degrading to the same "could not be read" string a private repo produces. BRAINHUB_API_KEY needed no change — hub::push already rejects an unset key with a named error. That half of the finding was overstated. Co-Authored-By: Claude Opus 5 <[email protected]>
7.8 KiB
Provenance: what we can answer today, and what we cannot
Assessment only. No schema, no migration, nothing built. Written 2026-08-19.
The question this exists to answer is narrow and practical:
An agent said something. Where did it come from?
That is the question a provenance layer has to make answerable. Everything below is measured against it.
The short answer
We can reconstruct what an agent did, on both execution paths, for seven days. We cannot reconstruct why it said what it said on any path, at any retention, because no store links a statement to the evidence that produced it. There is no claim as a first-class object anywhere in the system.
That is a design gap, not a bug. Nothing is broken; the edge was never built.
What each store actually holds
mission_events — the action record
The main one. Structured rows for the mission path: tool calls, phase transitions, agent lifecycle.
- Retention: 7 days (
EVENT_RETENTION_DAYS,mission_gc.rs). This is the single most consequential fact in this document. The richest signal we have expires before most retrospectives happen, and a level-up proposal citing an event id older than a week points at nothing. reasoningrows are written (topology_worker.rs) and are read by nothing.routes/world.rsexplicitly excludes them, with a comment saying so. The model's stated rationale is recorded and then discarded unread.- No foreign key to
run_id—0075dropped it deliberately. Events reference runs by convention, so nothing enforces that the reference resolves.
steps — the rich record, on the wrong path
Structurally the best provenance we have: kind, tool_name, input,
output, and taint[] — per tool call, with the taint sources carried
through.
It is chat-path only, and not by omission. steps.message_id references a
chat message, and runtime::record_step is the sole writer. A mission phase
has no message, so it cannot write a step even in principle.
The consequence is worth stating plainly: the execution path that does the substantial work — missions — produces the poorer provenance record, and the path that produces the good one is the conversational one. Any real provenance work starts by resolving that asymmetry, and it is a schema change, not a call site.
audit_log — genuinely immutable, narrowly used
A BEFORE UPDATE OR DELETE trigger (0001_init.sql) makes rows append-only for
real, not by convention. It is the only tamper-evident store in the system.
It is used mostly as a rate-limit counter. The mechanism we would want for provenance already exists here and holds almost none of the content we would want in it.
Approvals — the best-shaped record we have
Approval rows carry taint_sources[] and the exact preview a human was shown.
For the narrow slice of actions that pass a gate, we can answer "what was the
human told, what did they decide, and what data influenced it" — completely.
This is the shape to copy. It is also, currently, an island.
The .brain — memory, versioned, chat-only
Full .onion revision history with commit/revisions/rollback, so a
definition's evolution is fully recoverable. Memory is BM25 keyword recall and
is written on the chat path; mission work writes none, so an agent that ran
missions for a week has an empty memory section.
cm-brain/src/lib.rs:234-239 exposes set_provenance / provenance — a wired
slot with zero callers in the entire workspace. It is a place to put this,
already plumbed, already versioned, already per-agent.
mission_phase_summaries — prose, and overwritten
A model's post-hoc narrative of a phase, overwritten on retry. A summary is a model's later account of its own behaviour, and the retry that changed the outcome erases the account of the attempt that failed — which is precisely the one worth reading.
The questions we cannot answer
Concretely, with what breaks each:
| Question | Why not |
|---|---|
| "Why did the agent claim X?" | No claim object; no edge from a statement to a tool result |
| "Which tool output led to this file change?" | Mission path writes no steps, so no input/output pair exists |
| "What did the agent believe when it decided?" | reasoning rows are written and read by nothing |
| "Was this conclusion derived from tainted input?" | taint[] exists only on the chat path |
| "What happened on the attempt that failed?" | Phase summaries are overwritten on retry |
| "Why did this mission go wrong last month?" | 7-day retention |
The recurring shape: the pieces exist, individually, on the wrong path or unread. This is not a system that lacks provenance primitives. It is one where they were never connected into something that answers a question.
Two candidate paths
A. Postgres claim/edge tables
Add a claim as a first-class row, and edges from claim → evidence (tool call,
file, event). Extend steps off the mission path so mission tool calls record
input/output/taint the way chat ones do.
- For: stays in the store we already query, back up and migrate; the approvals record already proves the shape works here; no new dependency.
- Against: a real schema addition on a 79-migration database; retention policy must be decided deliberately (7 days makes the whole thing pointless); graph queries in SQL get awkward exactly when they get interesting.
B. Adopt clawhdf5-agent::knowledge + ::provenance
Already in the dependency graph via the clawsync patch — declared in the
workspace Cargo.toml, and no crate depends on it. Roughly 21k lines:
typed entities, relations including RelationType::Causal, BFS and spreading
activation; provenance.rs with MemorySource / content_hash /
session_id.
- For: the causal structure is the thing we lack, and it is written; it is
per-agent and per-file, matching the
.brainmodel;set_provenanceis already the slot it would fill. - Against: 21k lines of unexercised code entering a critical path; it is
file-local, so cross-agent queries need a second mechanism; the
.brainis reaped with the agent, which is the wrong lifetime for an audit record.
What the research says about choosing
MemoryLake on MemoryArena (2026-08-14) compared memory backends with the
backend as the only changed component. Structured beat vector RAG and
long-context — on success rates of 9/40, 12/20 and 4/20, with every system
scoring zero somewhere. Harness the Memory (08-15) found no substrate
dominates, and that excessive retrieval actively harms agent decision-making
even while helping factual QA.
Read together: a substrate swap is not where the win is, and adopting a graph memory because it is more sophisticated is not supported by the evidence. That argues for A first — make the record complete and correctly retained on the path that matters — and to treat B as a question to measure later, against a baseline captured beforehand.
D²ACCI (08-18) is the sharper finding for us: "end-to-end evaluation reveals
that an error occurred, but not which stage caused it", and its DCR metric
grades whether failures stay localizable. That is this project's recurring
defect class stated as a research problem, and localizability — not
completeness — is the property a provenance layer here should be judged on.
The cheapest thing that would help, if we do nothing else
- Read the
reasoningrows we already write. They exist. Nothing consumes them. This is a query, not a schema. - Raise retention for a subset. Seven days is right for volume, wrong for audit. The distinction is which rows, not how long.
- Stop overwriting phase summaries on retry. The erased attempt is the informative one.
None of these is the provenance layer. All three are cheap, and each closes a question we currently cannot answer at all.