Omar Sobh
2013fa94a0
CI / test (push) Failing after 3s
Bump WAL_VERSION to 2: every entry (Save and Tombstone) now ends with a 4-byte CRC32 trailer computed over its type+timestamp+payload bytes, using the existing clawhdf5_format::checksum::crc32 (already available since clawhdf5-agent depends on clawhdf5-format with fast-checksum enabled). A bit-flip inside an entry is now detected and replay stops there, instead of silently accepting corrupted data as before. Write side needed no restructuring — append_save/append_tombstone already buffer an entry's bytes before a single write_all, so the CRC is just appended to that buffer first. Read side: read_len_prefixed_str/read_embedding are generalized from &mut File to R: Read, and a new TeeReader<R> wraps the file handle for one entry at a time, accumulating every byte actually consumed (via read_exact) into a buffer. This lets read_entries compute the CRC over exactly the bytes read for a Save entry without needing to know its length up front (its sub-fields are length-prefixed and interleaved with the length itself only becoming known as parsing proceeds). A new read_one_entry<R: Read> factors the per-entry-type field parsing shared by both the legacy and current read paths. Backward compatibility: WAL_VERSION_LEGACY_NO_CRC (1) files are still readable via WalFile::read_entries (old field-by-file-handle path, unchanged, no CRC expected). WalFile::open migrates a legacy file by recreating it fresh in the current format — safe because the only two real call sites (HDF5Memory::open/create) always call read_entries before open, so entries are already replayed by the time migration happens. New tests: a corrupted-payload-byte test confirming replay stops cleanly at the corrupted entry (no prior coverage existed for mid-entry bit-flip detection), a legacy-v1-format read test, and an open()-migration test.
clawhdf5-agent
HDF5-backed persistent memory store for on-device AI agents.
Built on clawhdf5, clawhdf5-agent provides a vector-searchable memory backend optimized for edge AI workloads. Store embeddings, text chunks, and metadata in a single HDF5 file with SIMD-accelerated similarity search.
Features
- Persistent vector store in HDF5 format
- Cosine similarity and L2 distance search
- SIMD-accelerated via clawhdf5-accel (AVX2, NEON)
- Optional GPU acceleration via clawhdf5-gpu
- Memory-mapped access for large stores
- f16 storage support for compact embeddings
Usage
[dependencies]
clawhdf5-agent = "2.1.0"
License
MIT