Omar SobhandClaude Opus 5
6e5ccc25a6
Slice 2 of the adopt-or-build plan. A recurring mission's hard problem is
not running the agent — that is 23 seconds — it is knowing what it did
last time. This repository already tried continuous research once:
migrations 0030-0044 built research_topics/loops, 0053 dropped them all,
and the reason they could not survive is that research_topics carried a
status lifecycle but no seen-set. It could run forever and never know
what it had covered.
Two kinds of row, because the real vault forced it. The plan assumed
notes carry arxiv:/doi:/url: frontmatter. Measured against the actual
valhalla-vault: 416 notes, 145 with frontmatter, and ZERO with any of
those keys — the dominant keys are repo-sync metadata (node, org, gitea)
and course fields (presenter, session). An ingester keyed only on
external identity would have indexed nothing, which is the same shape of
failure as everything else found this week. So `note` rows record
coverage (keyed by path) and `source` rows record consumption (keyed by
natural id); a continuous mission needs both.
Two decisions the data forced:
- `source:` is deliberately NOT an identity key. The vault uses it for
local paths of course material (/Users/quantum/Downloads/...), which is
provenance, not citable identity. Accepting it would fill the seen-set
with 25 rows keyed on a laptop path.
- The hash covers the body, not the whole file. Repo-sync notes rewrite
updated:/size_kb: on every sync without the prose changing; hashing the
file would report 103 phantom edits per run and make "unchanged"
meaningless.
Authoritative in Postgres rather than ZeroClaw memory, per the Slice 1
spike: memory is agent-scoped and mission agents are ephemeral
claw_<uuid> aliases (~100 already present). A seen-set that disappears
with the agent that wrote it is not a seen-set. The spike did find that
POST /api/memory upserts by key, so mirroring content there later would
inherit idempotence for free if keyed by source_id.
Verified against the live 416-note vault, not a fixture:
PASS1 { scanned: 416, inserted: 416, updated: 0, unchanged: 0 }
PASS2 { scanned: 416, inserted: 0, updated: 0, unchanged: 416 }
382 tests, clippy clean.
Co-Authored-By: Claude Opus 5 <[email protected]>
70 lines
3.4 KiB
SQL
70 lines
3.4 KiB
SQL
-- The seen-set for continuous missions.
|
|
--
|
|
-- Every "continuous X" mission has the same failure mode: it runs again and
|
|
-- redoes work it already did. Research resurfaces papers it already read; a
|
|
-- security scan re-reports findings already triaged. Orchestration does not
|
|
-- fix that — a record of what has already been covered does.
|
|
--
|
|
-- This repository already tried continuous research once. Migrations 0030-0044
|
|
-- built `research_topics`, `research_outcomes` and `loops`; 0053 dropped them
|
|
-- all. `research_topics` carried a status lifecycle but no seen-set, so it
|
|
-- could run forever and never know what it had covered. That is the gap this
|
|
-- table exists to close, and it is the reason it lands before any scheduling.
|
|
--
|
|
-- Authoritative here rather than in the runtime's memory: ZeroClaw memory is
|
|
-- scoped per agent, and mission agents are ephemeral `claw_<uuid>` aliases
|
|
-- minted per mission (measured: ~100 of them already). A seen-set that
|
|
-- disappears with the agent that wrote it is not a seen-set.
|
|
CREATE TABLE corpus_items (
|
|
id UUID PRIMARY KEY,
|
|
workspace_id UUID NOT NULL REFERENCES workspaces (id) ON DELETE CASCADE,
|
|
-- Which corpus this belongs to, e.g. 'valhalla-vault'. A workspace can
|
|
-- track several (a vault, a findings ledger, a paper collection).
|
|
corpus_id TEXT NOT NULL,
|
|
|
|
-- 'note' = something already in the corpus (a vault file). Establishes
|
|
-- coverage: what has this vault already got?
|
|
-- 'source' = an external thing a mission consumed (a paper, an advisory).
|
|
-- This is the dedupe key that stops re-reading.
|
|
--
|
|
-- Both are needed and they answer different questions. Measured against
|
|
-- the real vault: 416 notes, and ZERO carry an arxiv/doi/url key — so an
|
|
-- ingester keyed only on external identity would index nothing at all.
|
|
kind TEXT NOT NULL CHECK (kind IN ('note', 'source')),
|
|
|
|
-- Stable identity within the corpus. For notes, 'note:<vault-relative
|
|
-- path>'; for sources, a natural id like 'arxiv:2401.12345', 'doi:10...'
|
|
-- or 'url:<sha256>'. Uniqueness is on this, which is what makes
|
|
-- re-ingestion idempotent.
|
|
source_id TEXT NOT NULL,
|
|
|
|
title TEXT,
|
|
-- Vault-relative path for notes; NULL for external sources.
|
|
path TEXT,
|
|
url TEXT,
|
|
|
|
-- SHA-256 of the content at last sight. Lets a re-index distinguish
|
|
-- "unchanged" from "edited" without diffing, so an unchanged vault is a
|
|
-- genuine no-op rather than 416 pointless updates.
|
|
content_hash TEXT NOT NULL,
|
|
|
|
first_seen_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
|
last_seen_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
|
-- Which mission first recorded this. NULL for the initial vault index,
|
|
-- which is derived from files nobody's mission wrote.
|
|
mission_id UUID REFERENCES missions (id) ON DELETE SET NULL,
|
|
|
|
UNIQUE (workspace_id, corpus_id, source_id)
|
|
);
|
|
|
|
-- The hot query is "have I seen this?", which the UNIQUE index already covers.
|
|
-- This one serves "what does this corpus contain?" for briefing assembly.
|
|
CREATE INDEX corpus_items_corpus_idx
|
|
ON corpus_items (workspace_id, corpus_id, kind, last_seen_at DESC);
|
|
|
|
-- "What did this mission add?" — the verification predicate for a continuous
|
|
-- run is that it contributed at least one NEW source.
|
|
CREATE INDEX corpus_items_mission_idx
|
|
ON corpus_items (mission_id)
|
|
WHERE mission_id IS NOT NULL;
|