Omar SobhandClaude Opus 5
18dc0b964b
Four defects, found by checking the audit's claims instead of trusting them. Two of the audit's own findings turned out to be wrong, and the registry that exists to record which config keys are read was itself inaccurate — so the corrections are part of the change. upsert_task raised 42P10 on every call, for every caller `mission_tasks_external_uniq` is a PARTIAL unique index (WHERE external_id IS NOT NULL). Postgres will not match a partial index to an ON CONFLICT target unless the statement repeats the predicate, so the upsert failed on its first row. Both callers — the task-card parser that turns INT markers into tasks, and the security scanner — map the error to a string their caller logs. Two features were broken and nothing was red. Regression test in cm-db with a negative control: reverting the WHERE reproduces 42P10 exactly. the security scan never ran `security_scan::run` was reachable only from an operator button, so security_hardening.toml — a workflow whose entire first phase is a scan — ran an agent that was never told to scan and never fired the scanner either. phase_runner now sweeps finished security_scan phases, mirroring the benchmark baseline sweep that was added for the identical defect. Guarded on a new completion marker rather than on findings: a clean scan writes no findings, so a findings-guard would rescan forever. The marker also answers the question an operator actually asks, which is not "how many findings" but "was this looked at, by what, and when". two recipes could not fail security_hardening.toml and benchmark.toml carried no `task` and no `done_when` on any phase. A phase without done_when never enters evaluating, is never judged, and reports completed whatever it did — so a security mission could scan nothing and go green, and a benchmark mission could record no baseline that the next refactor would then compare against. Both now state the work and the condition, with inert keys annotated inline rather than deleted, so the gap between what a recipe asks for and what a phase receives stays visible. the config registry was wrong in both directions `harness` was listed NOT IMPLEMENTED while benchmark_runner reads it and phase_runner runs a baseline through it. `tools` was listed NOT IMPLEMENTED while security_scan::run reads it. A registry that exists so an operator can trust what a recipe does is worse than useless when it is inaccurate. Both corrected, `bench_name` and `cmd` added, and `test_command` deleted — it had neither a reader nor a writer, so it described a situation that could not arise. Also: CLAWMATES_JUDGE_MODEL had two different defaults (opus-4-8 in routes/topology.rs vs opus-5 in cm_runtime::judge_model) and a doc comment naming a third; topology now calls the one function. GITEA_TOKEN's absence in mission_plan is stated rather than degrading to the same "could not be read" string a private repo produces. BRAINHUB_API_KEY needed no change — hub::push already rejects an unset key with a named error. That half of the finding was overstated. Co-Authored-By: Claude Opus 5 <[email protected]>
167 lines
7.8 KiB
Markdown
167 lines
7.8 KiB
Markdown
# Provenance: what we can answer today, and what we cannot
|
|
|
|
*Assessment only. No schema, no migration, nothing built. Written 2026-08-19.*
|
|
|
|
The question this exists to answer is narrow and practical:
|
|
|
|
> An agent said something. Where did it come from?
|
|
|
|
That is the question a provenance layer has to make answerable. Everything
|
|
below is measured against it.
|
|
|
|
## The short answer
|
|
|
|
We can reconstruct **what an agent did**, on both execution paths, for seven
|
|
days. We cannot reconstruct **why it said what it said** on any path, at any
|
|
retention, because no store links a statement to the evidence that produced it.
|
|
There is no claim as a first-class object anywhere in the system.
|
|
|
|
That is a design gap, not a bug. Nothing is broken; the edge was never built.
|
|
|
|
## What each store actually holds
|
|
|
|
### `mission_events` — the action record
|
|
|
|
The main one. Structured rows for the mission path: tool calls, phase
|
|
transitions, agent lifecycle.
|
|
|
|
- **Retention: 7 days** (`EVENT_RETENTION_DAYS`, `mission_gc.rs`). This is the
|
|
single most consequential fact in this document. The richest signal we have
|
|
expires before most retrospectives happen, and a level-up proposal citing an
|
|
event id older than a week points at nothing.
|
|
- `reasoning` rows **are written** (`topology_worker.rs`) and **are read by
|
|
nothing**. `routes/world.rs` explicitly excludes them, with a comment saying
|
|
so. The model's stated rationale is recorded and then discarded unread.
|
|
- No foreign key to `run_id` — `0075` dropped it deliberately. Events reference
|
|
runs by convention, so nothing enforces that the reference resolves.
|
|
|
|
### `steps` — the rich record, on the wrong path
|
|
|
|
Structurally the best provenance we have: `kind`, `tool_name`, `input`,
|
|
`output`, **and `taint[]`** — per tool call, with the taint sources carried
|
|
through.
|
|
|
|
It is **chat-path only**, and not by omission. `steps.message_id` references a
|
|
chat message, and `runtime::record_step` is the sole writer. A mission phase
|
|
has no message, so it cannot write a step even in principle.
|
|
|
|
The consequence is worth stating plainly: the execution path that does the
|
|
substantial work — missions — produces the *poorer* provenance record, and the
|
|
path that produces the good one is the conversational one. Any real provenance
|
|
work starts by resolving that asymmetry, and it is a schema change, not a call
|
|
site.
|
|
|
|
### `audit_log` — genuinely immutable, narrowly used
|
|
|
|
A `BEFORE UPDATE OR DELETE` trigger (`0001_init.sql`) makes rows append-only for
|
|
real, not by convention. It is the only tamper-evident store in the system.
|
|
|
|
It is used mostly as a rate-limit counter. The mechanism we would want for
|
|
provenance already exists here and holds almost none of the content we would
|
|
want in it.
|
|
|
|
### Approvals — the best-shaped record we have
|
|
|
|
Approval rows carry `taint_sources[]` and the exact preview a human was shown.
|
|
For the narrow slice of actions that pass a gate, we can answer "what was the
|
|
human told, what did they decide, and what data influenced it" — completely.
|
|
|
|
This is the shape to copy. It is also, currently, an island.
|
|
|
|
### The `.brain` — memory, versioned, chat-only
|
|
|
|
Full `.onion` revision history with `commit`/`revisions`/`rollback`, so a
|
|
definition's evolution is fully recoverable. Memory is BM25 keyword recall and
|
|
is written on the **chat** path; mission work writes none, so an agent that ran
|
|
missions for a week has an empty memory section.
|
|
|
|
`cm-brain/src/lib.rs:234-239` exposes `set_provenance` / `provenance` — a wired
|
|
slot with **zero callers** in the entire workspace. It is a place to put this,
|
|
already plumbed, already versioned, already per-agent.
|
|
|
|
### `mission_phase_summaries` — prose, and overwritten
|
|
|
|
A model's post-hoc narrative of a phase, **overwritten on retry**. A summary is
|
|
a model's later account of its own behaviour, and the retry that changed the
|
|
outcome erases the account of the attempt that failed — which is precisely the
|
|
one worth reading.
|
|
|
|
## The questions we cannot answer
|
|
|
|
Concretely, with what breaks each:
|
|
|
|
| Question | Why not |
|
|
|---|---|
|
|
| "Why did the agent claim X?" | No claim object; no edge from a statement to a tool result |
|
|
| "Which tool output led to this file change?" | Mission path writes no `steps`, so no input/output pair exists |
|
|
| "What did the agent believe when it decided?" | `reasoning` rows are written and read by nothing |
|
|
| "Was this conclusion derived from tainted input?" | `taint[]` exists only on the chat path |
|
|
| "What happened on the attempt that failed?" | Phase summaries are overwritten on retry |
|
|
| "Why did this mission go wrong last month?" | 7-day retention |
|
|
|
|
The recurring shape: **the pieces exist, individually, on the wrong path or
|
|
unread.** This is not a system that lacks provenance primitives. It is one where
|
|
they were never connected into something that answers a question.
|
|
|
|
## Two candidate paths
|
|
|
|
### A. Postgres claim/edge tables
|
|
|
|
Add a claim as a first-class row, and edges from claim → evidence (tool call,
|
|
file, event). Extend `steps` off the mission path so mission tool calls record
|
|
input/output/taint the way chat ones do.
|
|
|
|
- **For:** stays in the store we already query, back up and migrate; the
|
|
approvals record already proves the shape works here; no new dependency.
|
|
- **Against:** a real schema addition on a 79-migration database; retention
|
|
policy must be decided deliberately (7 days makes the whole thing pointless);
|
|
graph queries in SQL get awkward exactly when they get interesting.
|
|
|
|
### B. Adopt `clawhdf5-agent::knowledge` + `::provenance`
|
|
|
|
Already in the dependency graph via the clawsync patch — declared in the
|
|
workspace `Cargo.toml`, and **no crate depends on it**. Roughly 21k lines:
|
|
typed entities, relations including `RelationType::Causal`, BFS and spreading
|
|
activation; `provenance.rs` with `MemorySource` / `content_hash` /
|
|
`session_id`.
|
|
|
|
- **For:** the causal structure is the thing we lack, and it is written; it is
|
|
per-agent and per-file, matching the `.brain` model; `set_provenance` is
|
|
already the slot it would fill.
|
|
- **Against:** 21k lines of unexercised code entering a critical path; it is
|
|
file-local, so cross-agent queries need a second mechanism; the `.brain` is
|
|
reaped with the agent, which is the wrong lifetime for an audit record.
|
|
|
|
### What the research says about choosing
|
|
|
|
`MemoryLake on MemoryArena` (2026-08-14) compared memory backends with the
|
|
backend as the only changed component. Structured beat vector RAG and
|
|
long-context — on success rates of 9/40, 12/20 and 4/20, with every system
|
|
scoring zero somewhere. `Harness the Memory` (08-15) found no substrate
|
|
dominates, and that *excessive* retrieval actively harms agent decision-making
|
|
even while helping factual QA.
|
|
|
|
Read together: a substrate swap is not where the win is, and adopting a graph
|
|
memory because it is more sophisticated is not supported by the evidence. That
|
|
argues for **A first** — make the record complete and correctly retained on the
|
|
path that matters — and to treat B as a question to measure later, against a
|
|
baseline captured beforehand.
|
|
|
|
`D²ACCI` (08-18) is the sharper finding for us: "end-to-end evaluation reveals
|
|
that an error occurred, but not which stage caused it", and its **DCR** metric
|
|
grades whether failures stay *localizable*. That is this project's recurring
|
|
defect class stated as a research problem, and localizability — not
|
|
completeness — is the property a provenance layer here should be judged on.
|
|
|
|
## The cheapest thing that would help, if we do nothing else
|
|
|
|
1. **Read the `reasoning` rows we already write.** They exist. Nothing consumes
|
|
them. This is a query, not a schema.
|
|
2. **Raise retention for a subset.** Seven days is right for volume, wrong for
|
|
audit. The distinction is which rows, not how long.
|
|
3. **Stop overwriting phase summaries on retry.** The erased attempt is the
|
|
informative one.
|
|
|
|
None of these is the provenance layer. All three are cheap, and each closes a
|
|
question we currently cannot answer at all.
|