diff --git a/CHANGELOG.md b/CHANGELOG.md index 7cc517b..6a4eeb6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,6 +19,54 @@ - `clawhdf5-agent`: `benches/bench.rs` and `benches/memory_bench.rs` no longer compiled against the current `strategy`/`consolidation` APIs. +### Durability & Integrity +- `clawhdf5-agent`: a crash between writing a checkpoint and truncating the WAL + no longer **duplicates every pending entry** on the next open. Each + checkpoint records a `WalMark` (byte length + chained CRC of the WAL prefix it + folded in) in `/meta`; `open()` skips exactly that prefix when it is still + present. No WAL format change for this; older files behave as before. +- `clawhdf5-agent`: checkpoints and snapshots are durable as a unit — the temp + file is synced before the rename and the directory after it. Individual WAL + appends remain unsynced by design (documented in `CLAUDE.md`). +- `clawhdf5-agent`: `save_or_update` hits are logged as a new `Update` WAL + record, so replay updates in place instead of appending a duplicate. WAL + header version 3 → 4 (so older builds refuse the file rather than truncating + a record they can't parse); v3 files are read and upgraded in place. +- `clawhdf5-agent`: loading validates every per-record dataset length (a + truncated store is now `MemoryError::Schema`, not a later panic), fixes the + `n.len() == n.len()` tautology that trusted a norms dataset of any length, + and rejects `embedding_dim == 0` with records present. +- `clawhdf5-agent`: eight behavioural `MemoryConfig` fields are now persisted in + `/meta`. Previously they reset to defaults on every open — a compressed store + was rewritten uncompressed, `wal_enabled = false` flipped back to `true`. +- `clawhdf5-agent`: `compression = true` never worked in a default build (it + requested Zstd without enabling the feature, so every checkpoint failed with + `unsupported filter: 32015`). Default builds now use deflate; Zstd is the new + opt-in `zstd` feature. +- `clawhdf5-agent`: **single-writer lock** (`.h5.lock`, + `MemoryError::Locked`) — two handles on one store used to silently destroy + each other's data. New `HDF5Memory::open_read_only` gives a lock-free, + never-writing view; the CLI's read-only subcommands use it. +- `clawhdf5-agent`: an unreadable WAL (torn header / bad magic) is quarantined + (`HDF5Memory::quarantined_wal()`) instead of blocking `open()` of a healthy + store. A WAL from an unknown newer version still fails and is left intact. +- `clawhdf5-agent`: provenance records are renumbered on compaction (they + weren't, so every later `save_or_update` raised a false High integrity + alert); pending anomaly alerts and tracked sessions are bounded; + `snapshot()` includes entries still in the WAL. +- `clawhdf5-agent`: hybrid ranking is deterministic (index tie-breaks instead + of `HashMap` order); a set of identical positive scores — including a single + candidate — normalises to 1.0 rather than 0.0; the Hebbian boost no longer + reinforces zero-score filler results. +- `clawhdf5-format`: chunked/VDS/hyperslab reads size their buffers with + overflow-checked arithmetic and fallible allocation, so crafted dimensions + are `FormatError::Overflow` instead of a wrapped size or a process abort; + `parallel_read` bounds checks use `checked_add`. +- `clawhdf5`: a malformed filter-pipeline message is an error instead of being + treated as "no filters" (which returned compressed bytes as data); + `FileBuilder::write` is atomic and synced instead of truncating the + destination first. + ### CI / Testing - CI now lints every target (`cargo clippy --all-targets`) plus `clawhdf5-format`'s optional features, compiles all benches, and tests the diff --git a/CLAUDE.md b/CLAUDE.md index a787b4d..041cfa3 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -40,6 +40,25 @@ Cargo workspace with 16 crates under `crates/` (plus `libaec-sys`, an internal F format (v2) is still fully readable; the oldest no-CRC format (v1) is only reachable through the one-time migration path in `HDF5Memory::open`, not through the public `WalFile::read_entries`. + **What the WAL guarantees:** integrity, ordering, and recovery from a + *process* crash at any point — including between a checkpoint and the WAL + truncate (each checkpoint records a `WalMark` in `/meta`, and `open()` skips + the WAL prefix the `.h5` already contains, so entries are never applied + twice). Checkpoints and snapshots are made durable as a unit (temp file + synced, renamed, directory synced). **What it does not guarantee:** + individual WAL appends are *not* fsynced (a deliberate latency trade-off), so + saves made since the last checkpoint can be lost on power failure or kernel + panic. Current header version is 4 (adds the `Update` record used by + `save_or_update`); v3 files are read and upgraded in place. +- A store has a **single writer**: `HDF5Memory::create`/`open` hold an exclusive + advisory lock on `.h5.lock` and a second opener gets + `MemoryError::Locked`. Use `HDF5Memory::open_read_only` for a lock-free, + never-writing point-in-time view (the CLI's `recall`/`stats`/`agents-md`/ + `export` do). An unreadable WAL (torn header, bad magic) is quarantined to + `.h5.wal.corrupt-` rather than blocking `open()`; a WAL with an + unknown *newer* version still fails and is left untouched. +- `MemoryConfig::compression` uses deflate by default; enable the agent's + `zstd` feature to compress embeddings with Zstd instead (links libzstd). - `Dataset::verify_provenance()` (clawhdf5 facade, `provenance` feature, on by default) recomputes a dataset's SHA-256 and compares it against the `_provenance_sha256` attribute written automatically on save when