clawhdf5/research/03_provenance.md

Research: Provenance — clawhdf5

Scope: data lineage, source attribution, and tamper-evidence for both the low-level HDF5 format layer and the higher-level agent-memory / migration tools built on top of it.

Finding PR1 — SHINES provenance (SHA-256 + creator/timestamp/source) is fully built and tested, but zero production write paths use it

Location: crates/clawhdf5-format/src/provenance.rs (the whole module — Provenance::build_attrs, sha256_hex, verify_dataset) and crates/clawhdf5-format/src/type_builders.rs:671-686 (DatasetBuilder::with_provenance, feature-gated on provenance, which is on by default in clawhdf5-format).

Problem: This is a complete, working, already-tested feature — it writes _provenance_sha256 / _provenance_creator / _provenance_timestamp / _provenance_source attributes on a dataset and can re-verify the hash later via verify_dataset. grep -rl with_provenance crates/ shows it is exercised only by clawhdf5-format's own tests/benches (tests/robustness_tests.rs, tests/writer_h5py_tests.rs, benches/bench.rs). Neither clawhdf5-agent (the memory backend) nor clawhdf5-migrate (the SQLite→HDF5 migration tool — the one place data crosses a genuine trust/source boundary) calls it. Concretely, crates/clawhdf5-migrate/src/hdf5_writer.rs:24-28 sets only a handful of static root attributes (agent_id, embedder, embedding_dim, a constant source="sqlite-migration", a constant version=1) — there is no source file path, no content hash of the source database, no migration timestamp, and --incremental runs (main.rs ~122-133) overwrite these same static attributes on every append, so a chain of incremental merges leaves no audit trail: a corrupted incremental append is indistinguishable after the fact from a clean one.

Fix implemented (INT-03): Wired the existing SHINES provenance mechanism into the migration write path instead of inventing a new one:

• clawhdf5-migrate/src/hdf5_writer.rs: the embeddings and text chunk datasets are now built with .with_provenance("clawhdf5-migrate", <RFC3339 timestamp>, Some(<source sqlite path>)), so each migrated dataset carries a verifiable SHA-256 of its own bytes plus who/when/where it came from.
• clawhdf5-migrate/src/sqlite_reader.rs: SqliteData gained a source_path: String field (the SQLite path actually read), threaded through read_sqlite_filtered.
• clawhdf5-migrate/src/main.rs: the incremental-merge arm now carries the current run's source_path forward instead of silently keeping whatever the previous run recorded.
• clawhdf5-migrate/src/validate.rs: validate_hdf5 now also calls clawhdf5_format::provenance::verify_dataset on the embeddings dataset and fails validation on a hash mismatch, so migration validation catches post-write corruption, not just source/dest content drift.

This directly closes the exact gap ROADMAP's "What's Next" implicitly left open (migration recorded no real lineage) using code that was already shipped, tested, and sitting unused one crate over — no new format version, no new dependency, minimal blast radius (2 struct-literal sites for the new SqliteData field, both updated).

Finding PR2 — agent-level MemoryProvenance/AnomalyDetector are dead code on the real save path (ROADMAP claims Track 5 "complete")

Location: crates/clawhdf5-agent/src/lib.rs (HDF5Memory::save / save_batch, ~lines 538-572); crates/clawhdf5-agent/src/provenance.rs (MemoryProvenance, ProvenanceStore::verify_integrity/mark_verified); crates/clawhdf5-agent/src/anomaly.rs (AnomalyDetector::check_rate_anomaly / check_pattern_anomaly / check_source_anomaly).

Problem: ROADMAP.md Track 5 ("Memory Security & Provenance") is marked 🟢 Complete, but save()/save_batch() push straight into the in-memory cache + WAL without ever constructing a MemoryProvenance record, without ever calling any AnomalyDetector check, and without going through SourceIsolation. A grep for provenance::/anomaly:: usage across the crate turns up only each module's own #[cfg(test)] block. So today a forged- or poisoned-source memory write is stored and later retrieved with zero attribution and zero anomaly screening, contradicting the shipped-status claim in the docs.

Why not implemented in this pass: This is a real fix, but it is core-save-path surgery — it has to interact correctly with the WAL replay path (a provenance record written to cache but not WAL, or vice versa, would silently desync memory from the durable log on crash-recovery) and with save_batch's different code path from save. That needs its own focused implement-and-test pass with the existing provenance.rs/anomaly.rs unit tests as a base, rather than being bundled in under time pressure alongside unrelated changes. Tracked as INT-05 below.

Finding PR3 — nothing on the retrieval path ever calls verify_integrity

Location: crates/clawhdf5-agent/src/provenance.rs:128 (ProvenanceStore::verify_integrity), vs. search.rs/hybrid.rs (no callers).

Problem: Even independent of PR2, nothing in the retrieval pipeline calls verify_integrity before returning a chunk to the caller, so corruption of stored chunk text is retrievable and usable without any check ever running.

Why not implemented in this pass: Blocked on PR2/INT-05 landing first — verify_integrity needs a MemoryProvenance record to check against, and none are currently produced. Tracked as INT-06, sequenced after INT-05.