Formatting only. cargo fmt --check was already failing on main (accel SIMD
kernels, agent, format, migrate, bench); CI now enforces it.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
- clippy --all-targets plus a clawhdf5-format feature matrix (parallel, lz4,
zstd, pcodec, fast-checksum); fix the accumulated lint backlog in test,
bench and feature-gated code (no behaviour changes).
- Install python3 + h5py/numpy/netCDF4/xarray in the CI container and set
CLAWHDF5_REQUIRE_INTEROP=1, which makes a missing interop dependency a test
failure. Every h5py/netCDF4 interop test used to skip silently in CI. Run
the #[ignore]d writer_h5py_tests suite explicitly.
- cargo bench --no-run so benches can't rot; fix bench.rs and memory_bench.rs,
which no longer compiled against the current strategy/consolidation APIs.
- Optional fuzz smoke run via CLAWHDF5_FUZZ_SECONDS.
- CHANGELOG and docs/known-issues.md updated.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
`WalFile::open` scanned the chained entries to resume the CRC chain, then
seeked to END OF FILE to append. After a crash mid-append — the ordinary way a
WAL ends up damaged — that puts the next entry BEHIND the torn bytes:
[1..N verified][torn tail][N+1, chained to N]
`read_chained_entries` stops at the torn tail, so N+1 is unreachable forever
even though its `append` returned Ok and synced. Silent loss of an acknowledged
write, in the one situation a WAL exists for.
`read_chained_entries` now also returns the byte length of the verified prefix,
and `open` truncates to it and appends there. The torn tail was never
acknowledged to any caller, so discarding it loses nothing, and the file offset
then matches the `running_crc` the chain continues from.
Verified by negative control: with the previous `seek(End(0))` the new test
fails with "got 1 entr(y/ies) — the post-crash write was silently lost".
Introduced by neither this branch nor the chaining work — v2 seeked to EOF too.
What changed is that `open` now scans and therefore KNOWS where the verified
prefix ends, which is what makes the fix a two-line consequence of information
already in hand.
Co-Authored-By: Claude Opus 5 <[email protected]>
Two related gaps in the WAL format, both closed:
1. Each entry's CRC32 covered only its own bytes, with no sequence number
or chaining — entries could be reordered, duplicated, or spliced (e.g.
a Tombstone moved before/after its target Save) while every individual
entry still passed its own CRC check, silently changing replayed cache
state. Bump to WAL_VERSION 3: each entry's CRC32 trailer is now computed
over its own bytes chained with the previous entry's stored CRC
(crc32(entry_bytes ++ prev_crc)), seeded at 0 after a truncation. Moving,
duplicating, or reordering an entry breaks the chain at that point, and
replay stops there — same handling as a bit-flip or truncation. The
previous per-entry-CRC-only format becomes WAL_VERSION_CRC_UNCHAINED (2)
and remains fully readable (not restricted, since it still verifies each
entry); WalFile::open migrates it to v3 by recreating the file fresh,
same as the existing v1 migration.
WalFile::open() on an existing v3 file scans it once to resume the CRC
chain correctly for further appends — required because a process
restart without an intervening flush reopens the same (non-truncated)
WAL and keeps appending to it, so new entries must chain against the
real last entry already on disk, not restart from 0.
2. WAL_VERSION_LEGACY_NO_CRC (v1, no integrity verification at all) was
reachable through the public WalFile::read_entries — a version byte
flipped from 2/3 down to 1 silently downgraded every entry to the
fully-unverified pre-hardening parser for any caller, not just the
one-time migration path. Split into WalFile::read_entries (rejects v1
with a typed error; still reads v2/v3) and the pub(crate)
read_entries_for_migration (accepts v1 too), used exclusively by
HDF5Memory::open's migration flow.
INT-09
Bump WAL_VERSION to 2: every entry (Save and Tombstone) now ends with a
4-byte CRC32 trailer computed over its type+timestamp+payload bytes, using
the existing clawhdf5_format::checksum::crc32 (already available since
clawhdf5-agent depends on clawhdf5-format with fast-checksum enabled).
A bit-flip inside an entry is now detected and replay stops there, instead
of silently accepting corrupted data as before.
Write side needed no restructuring — append_save/append_tombstone already
buffer an entry's bytes before a single write_all, so the CRC is just
appended to that buffer first.
Read side: read_len_prefixed_str/read_embedding are generalized from
&mut File to R: Read, and a new TeeReader<R> wraps the file handle for one
entry at a time, accumulating every byte actually consumed (via read_exact)
into a buffer. This lets read_entries compute the CRC over exactly the
bytes read for a Save entry without needing to know its length up front
(its sub-fields are length-prefixed and interleaved with the length itself
only becoming known as parsing proceeds). A new read_one_entry<R: Read>
factors the per-entry-type field parsing shared by both the legacy and
current read paths.
Backward compatibility: WAL_VERSION_LEGACY_NO_CRC (1) files are still
readable via WalFile::read_entries (old field-by-file-handle path,
unchanged, no CRC expected). WalFile::open migrates a legacy file by
recreating it fresh in the current format — safe because the only two
real call sites (HDF5Memory::open/create) always call read_entries before
open, so entries are already replayed by the time migration happens.
New tests: a corrupted-payload-byte test confirming replay stops cleanly
at the corrupted entry (no prior coverage existed for mid-entry bit-flip
detection), a legacy-v1-format read test, and an open()-migration test.
- clawhdf5-android: validate embedding_len/query_embedding_len against
the handle's configured embedding_dim (and reject null pointers)
before constructing a slice via from_raw_parts in edgehdf5_save and
edgehdf5_hybrid_search. Strengthen the # Safety docs to state the
now-enforced invariant and its limits. Add unit tests covering
mismatched length and null-pointer rejection.
- clawhdf5-py: bump pyo3/numpy 0.28 -> 0.29, clearing RUSTSEC-2026-0176
(OOB read in PyList/PyTuple iterator) and RUSTSEC-2026-0177 (missing
Sync bound on PyCFunction::new_closure). No source changes needed;
confirmed via cargo audit that both advisories no longer appear.
- clawhdf5-agent/wal.rs: cap read_len_prefixed_str/read_embedding's
length claims at a new MAX_WAL_FIELD_LEN (64 MiB) before allocating,
so a corrupted/truncated WAL length field fails cleanly instead of
attempting a huge allocation. Add regression tests for both.
- BENCHMARKS.md: add a top-of-file traceability note distinguishing the
dated/hardware-cited/reproducible h5bench and tank-validation sections
from the older sections that don't yet meet that bar.
Implements WAL group commit optimizations (arXiv:2507.13062):
1. Serialize each WAL entry to a local Vec<u8> before writing, reducing
write() syscalls per entry from ~8 to 1.
2. Defer header entry_count updates to every GROUP_COMMIT_SIZE (8) entries
instead of per-entry, eliminating 3 lseek() + 1 write() per entry.
3. Fix read_entries() to read until EOF instead of looping entry_count
times — the header count is now a pre-allocation hint only. This is
strictly more robust: tolerates stale counts from deferred updates AND
truncated files from crashes mid-write.
Benchmark results:
- wal_flush_100_entries: -7.8% latency improvement (469 µs)
- save_with_wal_single: -1.7% (18.2 µs)
- save_without_wal_single: -2.3% (67 µs, full HDF5 write)
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
