- tests/wal_properties.rs — deterministic generator, reproducible by seed:
everything appended is read back intact (300 cases), and after ANY damage
to the file (bit flips, truncation, inserted/deleted bytes, duplicated or
rotated regions, overwritten ranges; 1500 cases) reading never panics and
yields an exact prefix of what was written — the guarantee the chained CRC
exists to give. Opening for append then repairs the tail and a new entry
lands right behind the surviving prefix.
- tests/crash_recovery.rs — builds the on-disk images a process crash can
leave and reopens each against a model of what was acknowledged: an image
after every operation (random saves, in-place updates, checkpoints, small
wal_max_entries), the checkpoint window (new .h5 + not-yet-truncated WAL)
over several rounds, and the WAL torn at every byte length, which must
recover the checkpoint plus a prefix of the operations logged since.
- fuzz/fuzz_wal_replay — arbitrary bytes as a WAL: read and open-for-append
must not panic, and open() must not change what is replayable. Based on the
target from the clawmates mission branch (4aee2fa), with the repair
property added. The CI fuzz step now covers both fuzz crates.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
- HDF5Memory::create/open take an exclusive advisory lock on <store>.h5.lock
(std File::try_lock, no new dependency). The store lives in memory and is
rewritten wholesale at each checkpoint, so two handles on one store used to
silently destroy each other's data; a second writer now gets
MemoryError::Locked. The OS drops the lock with the descriptor, so a crash
never leaves a stale lock. Acquisition retries for ~250 ms to absorb a
previous owner that is mid-teardown; AsyncHDF5Memory::shutdown releases the
lock once its writer task has stopped.
- HDF5Memory::open_read_only: a lock-free, point-in-time view (checkpoint +
current WAL contents, replayed in memory) that never writes — it does not
repair, upgrade or move the WAL, and anything that would persist returns an
error. The CLI's recall/stats/agents-md/export use it, so a store can be
inspected while an agent has it open. Tests that reopened a store purely to
verify on-disk state now use it.
- open() no longer fails on a WAL that cannot possibly be replayed (torn
header, bad magic): it is moved to <store>.h5.wal.corrupt-<ts>, reported via
HDF5Memory::quarantined_wal(), and the healthy .h5 opens from its last
checkpoint. A well-formed header with an unknown version still fails and is
left untouched — most likely a newer build's WAL, which must not be
discarded.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
- clippy --all-targets plus a clawhdf5-format feature matrix (parallel, lz4,
zstd, pcodec, fast-checksum); fix the accumulated lint backlog in test,
bench and feature-gated code (no behaviour changes).
- Install python3 + h5py/numpy/netCDF4/xarray in the CI container and set
CLAWHDF5_REQUIRE_INTEROP=1, which makes a missing interop dependency a test
failure. Every h5py/netCDF4 interop test used to skip silently in CI. Run
the #[ignore]d writer_h5py_tests suite explicitly.
- cargo bench --no-run so benches can't rot; fix bench.rs and memory_bench.rs,
which no longer compiled against the current strategy/consolidation APIs.
- Optional fuzz smoke run via CLAWHDF5_FUZZ_SECONDS.
- CHANGELOG and docs/known-issues.md updated.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Missed in the INT-16 flat-embedding-buffer commit — this integration
test call site lives under tests/, outside the src/ tree that was
grepped for callers.
- Add .gitea/workflows/ci.yml running scripts/ci-test.sh (fmt, clippy,
test, no_std check) on push/PR to main.
- Fix stale rustyhdf5-py/rustyhdf5-format package names in
ci-test.sh/check-nostd.sh, which had been silently no-op'ing those
checks (cargo warns but doesn't fail on an unknown --exclude/-p
target).
- With those checks actually running, fix the real issues they surface:
- clippy: useless_conversion in chunked_write.rs, byte_char_slices in
global_heap.rs/object_header.rs.
- cargo fmt: apply formatting across the workspace (whitespace only).
- no_std (thumbv7em-none-eabihf) build errors in clawhdf5-format:
core::sync::atomic::AtomicU64 doesn't exist on that target (no
native 64-bit atomics) — switch profiling.rs's counters to
portable-atomic, which falls back to a CAS-based emulation there
and is a no-op wrapper elsewhere. Add missing alloc imports for
Box (filters.rs), Vec (filters_szip.rs), and format! (dict_encoding.rs)
on no_std paths. Replace f64::powi (std/libm-only) with a small
local exponentiation-by-squaring helper in the scale-offset filter.
Resolves two gaps found in a project-state review:
1. Python build was broken: PyO3/numpy 0.23 caps at Python 3.13 but the
environment has 3.14. Bumped to 0.28 and updated the two breaking APIs
(PyObject -> Py<PyAny>, allow_threads -> detach). The extension module now
imports and round-trips under Python 3.14, unblocking cargo build --workspace.
2. The "HNSW vector search over agent memories" headline was unwired:
clawhdf5-ann had zero dependents and the agent used a linear cosine+BM25 scan.
- clawhdf5-ann is now a live index: insert, mark_deleted (soft delete with a
deleted bitset, traversed but never returned), compact, and a format
version tag (v2) with backward-compatible load of v1 files.
- clawhdf5-agent wires HNSW behind the `hnsw` feature (ON by default). The
index mirrors the cache (node id == cache index) and self-heals: it rebuilds
whenever hnsw_synced_len drifts from cache.len(), so unhooked pushes can't
desync it. Non-indexable stores (no/zero-dim/mixed embeddings) and queries
whose dim doesn't match fall back to the exact linear scan.
- hybrid.rs gains merge_vector_keyword, shared by the linear and HNSW paths.
- tests/hnsw_integration.rs validates recall vs a brute-force oracle plus
insert/delete/batch behaviour.
Disable HNSW for exact search with `--no-default-features --features float16`.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
