fix(agent): crash between checkpoint and WAL truncate no longer duplicates entries

flush() writes the new .h5 and only then truncates the WAL. A crash in that
window left a .h5 that already contained the pending entries AND a WAL that
still listed them, and open() replayed the WAL unconditionally — every pending
entry came back twice.

A checkpoint now records a WalMark in /meta (wal_applied_len/wal_applied_crc):
the byte length and chained CRC of the WAL prefix it folded in. On open, if
the WAL's v3 CRC chain passes through exactly that position, the entries up to
it are skipped; otherwise (the normal case: the WAL was truncated) everything
is replayed. No WAL format change; files without the attributes behave as
before. WalFile tracks its chain length alongside running_crc and resumes both
on reopen.

Also make the checkpoint and snapshot durable as a unit: sync the temp file
before the rename and the parent directory after it, so a power loss can't
leave an empty or partial .h5 under the final name. This is per-checkpoint
cost only; individual WAL appends remain unsynced by design.

Co-Authored-By: Claude Fable 5.1 <[email protected]>

crates/clawhdf5-agent/src/lib.rs

@@ -287,7 +287,8 @@ impl HDF5Memory {
 
     /// Open an existing HDF5 memory file.
     pub fn open(path: &Path) -> Result<Self> {
-        let (config, mut cache, sessions, knowledge) = storage::read_from_disk(path)?;
+        let ((config, mut cache, sessions, knowledge), wal_applied) =
+            storage::read_from_disk_with_mark(path)?;
 
         // Replay WAL if present
         let wal_path = path.with_extension("h5.wal");
@@ -295,7 +296,10 @@ impl HDF5Memory {
             // Uses the migration-only reader since this is the one legitimate
             // path that may need to read a legacy (pre-CRC) WAL file — see
             // WalFile::read_entries_for_migration.
-            let entries = wal::WalFile::read_entries_for_migration(&wal_path)?;
+            // `wal_applied` drops the prefix a checkpoint already folded in,
+            // in case the process died between writing the .h5 and
+            // truncating the WAL.
+            let entries = wal::WalFile::read_entries_for_migration(&wal_path, wal_applied)?;
             wal::replay_into_cache(&entries, &mut cache);
             Some(wal::WalFile::open(&wal_path)?)
         } else if config.wal_enabled {
@@ -336,12 +340,16 @@ impl HDF5Memory {
     /// also clear the WAL, otherwise `open()` will replay stale entries
     /// on top of the already-persisted data, duplicating them.
     fn flush(&mut self) -> Result<()> {
-        storage::write_to_disk(
+        // Record which WAL prefix this checkpoint contains, so a crash before
+        // the truncate below can't replay those entries a second time.
+        let wal_applied = self.wal.as_ref().map(|w| w.mark());
+        storage::write_to_disk_with_mark(
             &self.config.path,
             &self.config,
             &self.cache,
             &self.sessions,
             &self.knowledge,
+            wal_applied,
         )?;
         if let Some(ref mut w) = self.wal {
             w.truncate()?;