Omar Sobh
dfae9e2cc1
CI / test (push) Failing after 2s
Found via a real-world integration audit against omni-cortex (a JEPA-based cognitive architecture built on clawhdf5 as its tiered Working/Episodic/ Semantic memory store). - Add DatasetBuilder::with_u64_data (crates/clawhdf5-format/type_builders.rs). The read side already has read_u64/read_as_u64, but there was no symmetric write-side builder — only signed with_i32_data/with_i64_data existed. Every consumer needing full-range u64 (timestamps, IDs) had to bit-cast through i64 via `i64::from_ne_bytes(v.to_ne_bytes())` on write and reverse it on read. omni-cortex does this in at least 6 places across its writer/reader/mmap-reader/consolidate crates. Confirmed the new builder round-trips full-range u64 (including values with the high bit set) end-to-end in a standalone sanity check mirroring their usage. - Fix Dataset::read_selection(&Selection::All) to route through the same per-file chunk cache read_raw()/read_f64() etc. already use, instead of the uncached read_chunked_data path. Selection::All is semantically a full read; there's no reason two ways of asking for "everything" should have different caching behavior. Also gains read_raw()'s virtual-dataset resolver support for free. omni-cortex's Reader/mmap-reader/consolidate crates all call read_selection(&Selection::All) for their chunked/ compressed dataset reads, so this was a real, if currently low-traffic (single-pass read pattern), inconsistency in the public API's behavior. - README: fix a stale crate-map claim that clawhdf5-filters supports "blosc" compression — it never did (the crate only ever held fast_deflate.rs; lz4/zstd/pcodec/szip filters live in clawhdf5-format). New tests: u64_data_roundtrip, read_selection_all_matches_read_raw_on_chunked_dataset.
clawhdf5
Pure-Rust HDF5 reader/writer — no C dependencies.
Features
- Read and write HDF5 files entirely in Rust
- Memory-mapped I/O for large files (
mmapfeature, enabled by default) - Parallel chunk reads via Rayon (
parallelfeature) - Lazy dataset access for minimal memory usage
- h5py-compatible file output
Usage
use clawhdf5::File;
let file = File::open("data.h5").unwrap();
let dataset = file.dataset("/group/data").unwrap();
let values: Vec<f64> = dataset.read_1d().unwrap();
License
MIT