docs: changelog and CLAUDE.md for the durability & integrity work

Co-Authored-By: Claude Fable 5.1 <[email protected]>
This commit is contained in:
osobh
2026-09-19 06:14:56 -07:00
co-authored by Claude Fable 5.1
parent bf8bbec87e
commit 005f37e846
2 changed files with 67 additions and 0 deletions
+19
View File
@@ -40,6 +40,25 @@ Cargo workspace with 16 crates under `crates/` (plus `libaec-sys`, an internal F
format (v2) is still fully readable; the oldest no-CRC format (v1) is only
reachable through the one-time migration path in `HDF5Memory::open`, not
through the public `WalFile::read_entries`.
**What the WAL guarantees:** integrity, ordering, and recovery from a
*process* crash at any point — including between a checkpoint and the WAL
truncate (each checkpoint records a `WalMark` in `/meta`, and `open()` skips
the WAL prefix the `.h5` already contains, so entries are never applied
twice). Checkpoints and snapshots are made durable as a unit (temp file
synced, renamed, directory synced). **What it does not guarantee:**
individual WAL appends are *not* fsynced (a deliberate latency trade-off), so
saves made since the last checkpoint can be lost on power failure or kernel
panic. Current header version is 4 (adds the `Update` record used by
`save_or_update`); v3 files are read and upgraded in place.
- A store has a **single writer**: `HDF5Memory::create`/`open` hold an exclusive
advisory lock on `<store>.h5.lock` and a second opener gets
`MemoryError::Locked`. Use `HDF5Memory::open_read_only` for a lock-free,
never-writing point-in-time view (the CLI's `recall`/`stats`/`agents-md`/
`export` do). An unreadable WAL (torn header, bad magic) is quarantined to
`<store>.h5.wal.corrupt-<ts>` rather than blocking `open()`; a WAL with an
unknown *newer* version still fails and is left untouched.
- `MemoryConfig::compression` uses deflate by default; enable the agent's
`zstd` feature to compress embeddings with Zstd instead (links libzstd).
- `Dataset::verify_provenance()` (clawhdf5 facade, `provenance` feature, on by
default) recomputes a dataset's SHA-256 and compares it against the
`_provenance_sha256` attribute written automatically on save when