docs: changelog and CLAUDE.md for the durability & integrity work

Co-Authored-By: Claude Fable 5.1 <[email protected]>

CHANGELOG.md

@@ -19,6 +19,54 @@
 - `clawhdf5-agent`: `benches/bench.rs` and `benches/memory_bench.rs` no longer
   compiled against the current `strategy`/`consolidation` APIs.
 
+### Durability & Integrity
+- `clawhdf5-agent`: a crash between writing a checkpoint and truncating the WAL
+  no longer **duplicates every pending entry** on the next open. Each
+  checkpoint records a `WalMark` (byte length + chained CRC of the WAL prefix it
+  folded in) in `/meta`; `open()` skips exactly that prefix when it is still
+  present. No WAL format change for this; older files behave as before.
+- `clawhdf5-agent`: checkpoints and snapshots are durable as a unit — the temp
+  file is synced before the rename and the directory after it. Individual WAL
+  appends remain unsynced by design (documented in `CLAUDE.md`).
+- `clawhdf5-agent`: `save_or_update` hits are logged as a new `Update` WAL
+  record, so replay updates in place instead of appending a duplicate. WAL
+  header version 3 → 4 (so older builds refuse the file rather than truncating
+  a record they can't parse); v3 files are read and upgraded in place.
+- `clawhdf5-agent`: loading validates every per-record dataset length (a
+  truncated store is now `MemoryError::Schema`, not a later panic), fixes the
+  `n.len() == n.len()` tautology that trusted a norms dataset of any length,
+  and rejects `embedding_dim == 0` with records present.
+- `clawhdf5-agent`: eight behavioural `MemoryConfig` fields are now persisted in
+  `/meta`. Previously they reset to defaults on every open — a compressed store
+  was rewritten uncompressed, `wal_enabled = false` flipped back to `true`.
+- `clawhdf5-agent`: `compression = true` never worked in a default build (it
+  requested Zstd without enabling the feature, so every checkpoint failed with
+  `unsupported filter: 32015`). Default builds now use deflate; Zstd is the new
+  opt-in `zstd` feature.
+- `clawhdf5-agent`: **single-writer lock** (`<store>.h5.lock`,
+  `MemoryError::Locked`) — two handles on one store used to silently destroy
+  each other's data. New `HDF5Memory::open_read_only` gives a lock-free,
+  never-writing view; the CLI's read-only subcommands use it.
+- `clawhdf5-agent`: an unreadable WAL (torn header / bad magic) is quarantined
+  (`HDF5Memory::quarantined_wal()`) instead of blocking `open()` of a healthy
+  store. A WAL from an unknown newer version still fails and is left intact.
+- `clawhdf5-agent`: provenance records are renumbered on compaction (they
+  weren't, so every later `save_or_update` raised a false High integrity
+  alert); pending anomaly alerts and tracked sessions are bounded;
+  `snapshot()` includes entries still in the WAL.
+- `clawhdf5-agent`: hybrid ranking is deterministic (index tie-breaks instead
+  of `HashMap` order); a set of identical positive scores — including a single
+  candidate — normalises to 1.0 rather than 0.0; the Hebbian boost no longer
+  reinforces zero-score filler results.
+- `clawhdf5-format`: chunked/VDS/hyperslab reads size their buffers with
+  overflow-checked arithmetic and fallible allocation, so crafted dimensions
+  are `FormatError::Overflow` instead of a wrapped size or a process abort;
+  `parallel_read` bounds checks use `checked_add`.
+- `clawhdf5`: a malformed filter-pipeline message is an error instead of being
+  treated as "no filters" (which returned compressed bytes as data);
+  `FileBuilder::write` is atomic and synced instead of truncating the
+  destination first.
+
 ### CI / Testing
 - CI now lints every target (`cargo clippy --all-targets`) plus
   `clawhdf5-format`'s optional features, compiles all benches, and tests the

CLAUDE.md

@@ -40,6 +40,25 @@ Cargo workspace with 16 crates under `crates/` (plus `libaec-sys`, an internal F
   format (v2) is still fully readable; the oldest no-CRC format (v1) is only
   reachable through the one-time migration path in `HDF5Memory::open`, not
   through the public `WalFile::read_entries`.
+  **What the WAL guarantees:** integrity, ordering, and recovery from a
+  *process* crash at any point — including between a checkpoint and the WAL
+  truncate (each checkpoint records a `WalMark` in `/meta`, and `open()` skips
+  the WAL prefix the `.h5` already contains, so entries are never applied
+  twice). Checkpoints and snapshots are made durable as a unit (temp file
+  synced, renamed, directory synced). **What it does not guarantee:**
+  individual WAL appends are *not* fsynced (a deliberate latency trade-off), so
+  saves made since the last checkpoint can be lost on power failure or kernel
+  panic. Current header version is 4 (adds the `Update` record used by
+  `save_or_update`); v3 files are read and upgraded in place.
+- A store has a **single writer**: `HDF5Memory::create`/`open` hold an exclusive
+  advisory lock on `<store>.h5.lock` and a second opener gets
+  `MemoryError::Locked`. Use `HDF5Memory::open_read_only` for a lock-free,
+  never-writing point-in-time view (the CLI's `recall`/`stats`/`agents-md`/
+  `export` do). An unreadable WAL (torn header, bad magic) is quarantined to
+  `<store>.h5.wal.corrupt-<ts>` rather than blocking `open()`; a WAL with an
+  unknown *newer* version still fails and is left untouched.
+- `MemoryConfig::compression` uses deflate by default; enable the agent's
+  `zstd` feature to compress embeddings with Zstd instead (links libzstd).
 - `Dataset::verify_provenance()` (clawhdf5 facade, `provenance` feature, on by
   default) recomputes a dataset's SHA-256 and compares it against the
   `_provenance_sha256` attribute written automatically on save when