docs: changelog and CLAUDE.md for the durability & integrity work

Co-Authored-By: Claude Fable 5.1 <[email protected]>
This commit is contained in:
osobh
2026-09-19 06:14:56 -07:00
co-authored by Claude Fable 5.1
parent bf8bbec87e
commit 005f37e846
2 changed files with 67 additions and 0 deletions
+48
View File
@@ -19,6 +19,54 @@
- `clawhdf5-agent`: `benches/bench.rs` and `benches/memory_bench.rs` no longer
compiled against the current `strategy`/`consolidation` APIs.
### Durability & Integrity
- `clawhdf5-agent`: a crash between writing a checkpoint and truncating the WAL
no longer **duplicates every pending entry** on the next open. Each
checkpoint records a `WalMark` (byte length + chained CRC of the WAL prefix it
folded in) in `/meta`; `open()` skips exactly that prefix when it is still
present. No WAL format change for this; older files behave as before.
- `clawhdf5-agent`: checkpoints and snapshots are durable as a unit — the temp
file is synced before the rename and the directory after it. Individual WAL
appends remain unsynced by design (documented in `CLAUDE.md`).
- `clawhdf5-agent`: `save_or_update` hits are logged as a new `Update` WAL
record, so replay updates in place instead of appending a duplicate. WAL
header version 3 → 4 (so older builds refuse the file rather than truncating
a record they can't parse); v3 files are read and upgraded in place.
- `clawhdf5-agent`: loading validates every per-record dataset length (a
truncated store is now `MemoryError::Schema`, not a later panic), fixes the
`n.len() == n.len()` tautology that trusted a norms dataset of any length,
and rejects `embedding_dim == 0` with records present.
- `clawhdf5-agent`: eight behavioural `MemoryConfig` fields are now persisted in
`/meta`. Previously they reset to defaults on every open — a compressed store
was rewritten uncompressed, `wal_enabled = false` flipped back to `true`.
- `clawhdf5-agent`: `compression = true` never worked in a default build (it
requested Zstd without enabling the feature, so every checkpoint failed with
`unsupported filter: 32015`). Default builds now use deflate; Zstd is the new
opt-in `zstd` feature.
- `clawhdf5-agent`: **single-writer lock** (`<store>.h5.lock`,
`MemoryError::Locked`) — two handles on one store used to silently destroy
each other's data. New `HDF5Memory::open_read_only` gives a lock-free,
never-writing view; the CLI's read-only subcommands use it.
- `clawhdf5-agent`: an unreadable WAL (torn header / bad magic) is quarantined
(`HDF5Memory::quarantined_wal()`) instead of blocking `open()` of a healthy
store. A WAL from an unknown newer version still fails and is left intact.
- `clawhdf5-agent`: provenance records are renumbered on compaction (they
weren't, so every later `save_or_update` raised a false High integrity
alert); pending anomaly alerts and tracked sessions are bounded;
`snapshot()` includes entries still in the WAL.
- `clawhdf5-agent`: hybrid ranking is deterministic (index tie-breaks instead
of `HashMap` order); a set of identical positive scores — including a single
candidate — normalises to 1.0 rather than 0.0; the Hebbian boost no longer
reinforces zero-score filler results.
- `clawhdf5-format`: chunked/VDS/hyperslab reads size their buffers with
overflow-checked arithmetic and fallible allocation, so crafted dimensions
are `FormatError::Overflow` instead of a wrapped size or a process abort;
`parallel_read` bounds checks use `checked_add`.
- `clawhdf5`: a malformed filter-pipeline message is an error instead of being
treated as "no filters" (which returned compressed bytes as data);
`FileBuilder::write` is atomic and synced instead of truncating the
destination first.
### CI / Testing
- CI now lints every target (`cargo clippy --all-targets`) plus
`clawhdf5-format`'s optional features, compiles all benches, and tests the