clawhdf5/research/03_provenance.md

# Research: Provenance — clawhdf5

Scope: data lineage, source attribution, and tamper-evidence for both the
low-level HDF5 format layer and the higher-level agent-memory / migration
tools built on top of it.

## Finding PR1 — SHINES provenance (SHA-256 + creator/timestamp/source) is fully built and tested, but zero production write paths use it

**Location:** `crates/clawhdf5-format/src/provenance.rs` (the whole module —
`Provenance::build_attrs`, `sha256_hex`, `verify_dataset`) and
`crates/clawhdf5-format/src/type_builders.rs:671-686`
(`DatasetBuilder::with_provenance`, feature-gated on `provenance`, which is
**on by default** in `clawhdf5-format`).

**Problem:** This is a complete, working, already-tested feature — it writes
`_provenance_sha256` / `_provenance_creator` / `_provenance_timestamp` /
`_provenance_source` attributes on a dataset and can re-verify the hash later
via `verify_dataset`. `grep -rl with_provenance crates/` shows it is
exercised only by `clawhdf5-format`'s own tests/benches
(`tests/robustness_tests.rs`, `tests/writer_h5py_tests.rs`,
`benches/bench.rs`). Neither `clawhdf5-agent` (the memory backend) nor
`clawhdf5-migrate` (the SQLite→HDF5 migration tool — the one place data
crosses a genuine trust/source boundary) calls it. Concretely,
`crates/clawhdf5-migrate/src/hdf5_writer.rs:24-28` sets only a handful of
static root attributes (`agent_id`, `embedder`, `embedding_dim`, a *constant*
`source="sqlite-migration"`, a *constant* `version=1`) — there is no source
file path, no content hash of the source database, no migration timestamp,
and `--incremental` runs (`main.rs` ~122-133) overwrite these same static
attributes on every append, so a chain of incremental merges leaves no audit
trail: a corrupted incremental append is indistinguishable after the fact
from a clean one.

**Fix implemented (INT-03):** Wired the *existing* SHINES provenance
mechanism into the migration write path instead of inventing a new one:

- `clawhdf5-migrate/src/hdf5_writer.rs`: the `embeddings` and `text` chunk
  datasets are now built with `.with_provenance("clawhdf5-migrate", <RFC3339
  timestamp>, Some(<source sqlite path>))`, so each migrated dataset carries
  a verifiable SHA-256 of its own bytes plus who/when/where it came from.
- `clawhdf5-migrate/src/sqlite_reader.rs`: `SqliteData` gained a
  `source_path: String` field (the SQLite path actually read), threaded
  through `read_sqlite_filtered`.
- `clawhdf5-migrate/src/main.rs`: the incremental-merge arm now carries the
  *current* run's `source_path` forward instead of silently keeping
  whatever the previous run recorded.
- `clawhdf5-migrate/src/validate.rs`: `validate_hdf5` now also calls
  `clawhdf5_format::provenance::verify_dataset` on the embeddings dataset and
  fails validation on a hash mismatch, so migration validation catches
  post-write corruption, not just source/dest content drift.

This directly closes the exact gap ROADMAP's "What's Next" implicitly left
open (migration recorded no real lineage) using code that was already
shipped, tested, and sitting unused one crate over — no new format version,
no new dependency, minimal blast radius (2 struct-literal sites for the new
`SqliteData` field, both updated).

## Finding PR2 — agent-level `MemoryProvenance`/`AnomalyDetector` are dead code on the real save path (ROADMAP claims Track 5 "complete")

**Location:** `crates/clawhdf5-agent/src/lib.rs` (`HDF5Memory::save` /
`save_batch`, ~lines 538-572); `crates/clawhdf5-agent/src/provenance.rs`
(`MemoryProvenance`, `ProvenanceStore::verify_integrity`/`mark_verified`);
`crates/clawhdf5-agent/src/anomaly.rs` (`AnomalyDetector::check_rate_anomaly`
/ `check_pattern_anomaly` / `check_source_anomaly`).

**Problem:** `ROADMAP.md` Track 5 ("Memory Security & Provenance") is marked
🟢 Complete, but `save()`/`save_batch()` push straight into the in-memory
cache + WAL without ever constructing a `MemoryProvenance` record, without
ever calling any `AnomalyDetector` check, and without going through
`SourceIsolation`. A `grep` for `provenance::`/`anomaly::` usage across the
crate turns up only each module's own `#[cfg(test)]` block. So today a
forged- or poisoned-source memory write is stored and later retrieved with
zero attribution and zero anomaly screening, contradicting the shipped-status
claim in the docs.

**Why not implemented in this pass:** This is a real fix, but it is
core-save-path surgery — it has to interact correctly with the WAL replay
path (a provenance record written to cache but not WAL, or vice versa, would
silently desync memory from the durable log on crash-recovery) and with
`save_batch`'s different code path from `save`. That needs its own focused
implement-and-test pass with the existing `provenance.rs`/`anomaly.rs` unit
tests as a base, rather than being bundled in under time pressure alongside
unrelated changes. Tracked as **INT-05** below.

## Finding PR3 — nothing on the retrieval path ever calls `verify_integrity`

**Location:** `crates/clawhdf5-agent/src/provenance.rs:128`
(`ProvenanceStore::verify_integrity`), vs. `search.rs`/`hybrid.rs` (no
callers).

**Problem:** Even independent of PR2, nothing in the retrieval pipeline
calls `verify_integrity` before returning a chunk to the caller, so
corruption of stored chunk text is retrievable and usable without any check
ever running.

**Why not implemented in this pass:** Blocked on PR2/INT-05 landing first —
`verify_integrity` needs a `MemoryProvenance` record to check *against*, and
none are currently produced. Tracked as **INT-06**, sequenced after INT-05.