docs: changelog and CLAUDE.md for the durability & integrity work
Co-Authored-By: Claude Fable 5.1 <[email protected]>
This commit is contained in:
co-authored by
Claude Fable 5.1
parent
bf8bbec87e
commit
005f37e846
@@ -40,6 +40,25 @@ Cargo workspace with 16 crates under `crates/` (plus `libaec-sys`, an internal F
|
||||
format (v2) is still fully readable; the oldest no-CRC format (v1) is only
|
||||
reachable through the one-time migration path in `HDF5Memory::open`, not
|
||||
through the public `WalFile::read_entries`.
|
||||
**What the WAL guarantees:** integrity, ordering, and recovery from a
|
||||
*process* crash at any point — including between a checkpoint and the WAL
|
||||
truncate (each checkpoint records a `WalMark` in `/meta`, and `open()` skips
|
||||
the WAL prefix the `.h5` already contains, so entries are never applied
|
||||
twice). Checkpoints and snapshots are made durable as a unit (temp file
|
||||
synced, renamed, directory synced). **What it does not guarantee:**
|
||||
individual WAL appends are *not* fsynced (a deliberate latency trade-off), so
|
||||
saves made since the last checkpoint can be lost on power failure or kernel
|
||||
panic. Current header version is 4 (adds the `Update` record used by
|
||||
`save_or_update`); v3 files are read and upgraded in place.
|
||||
- A store has a **single writer**: `HDF5Memory::create`/`open` hold an exclusive
|
||||
advisory lock on `<store>.h5.lock` and a second opener gets
|
||||
`MemoryError::Locked`. Use `HDF5Memory::open_read_only` for a lock-free,
|
||||
never-writing point-in-time view (the CLI's `recall`/`stats`/`agents-md`/
|
||||
`export` do). An unreadable WAL (torn header, bad magic) is quarantined to
|
||||
`<store>.h5.wal.corrupt-<ts>` rather than blocking `open()`; a WAL with an
|
||||
unknown *newer* version still fails and is left untouched.
|
||||
- `MemoryConfig::compression` uses deflate by default; enable the agent's
|
||||
`zstd` feature to compress embeddings with Zstd instead (links libzstd).
|
||||
- `Dataset::verify_provenance()` (clawhdf5 facade, `provenance` feature, on by
|
||||
default) recomputes a dataset's SHA-256 and compares it against the
|
||||
`_provenance_sha256` attribute written automatically on save when
|
||||
|
||||
Reference in New Issue
Block a user