Omar Sobh
dfae9e2cc1
CI / test (push) Failing after 2s
Found via a real-world integration audit against omni-cortex (a JEPA-based cognitive architecture built on clawhdf5 as its tiered Working/Episodic/ Semantic memory store). - Add DatasetBuilder::with_u64_data (crates/clawhdf5-format/type_builders.rs). The read side already has read_u64/read_as_u64, but there was no symmetric write-side builder — only signed with_i32_data/with_i64_data existed. Every consumer needing full-range u64 (timestamps, IDs) had to bit-cast through i64 via `i64::from_ne_bytes(v.to_ne_bytes())` on write and reverse it on read. omni-cortex does this in at least 6 places across its writer/reader/mmap-reader/consolidate crates. Confirmed the new builder round-trips full-range u64 (including values with the high bit set) end-to-end in a standalone sanity check mirroring their usage. - Fix Dataset::read_selection(&Selection::All) to route through the same per-file chunk cache read_raw()/read_f64() etc. already use, instead of the uncached read_chunked_data path. Selection::All is semantically a full read; there's no reason two ways of asking for "everything" should have different caching behavior. Also gains read_raw()'s virtual-dataset resolver support for free. omni-cortex's Reader/mmap-reader/consolidate crates all call read_selection(&Selection::All) for their chunked/ compressed dataset reads, so this was a real, if currently low-traffic (single-pass read pattern), inconsistency in the public API's behavior. - README: fix a stale crate-map claim that clawhdf5-filters supports "blosc" compression — it never did (the crate only ever held fast_deflate.rs; lz4/zstd/pcodec/szip filters live in clawhdf5-format). New tests: u64_data_roundtrip, read_selection_all_matches_read_raw_on_chunked_dataset.
clawhdf5-format
Pure-Rust HDF5 binary format parsing and writing — no C dependencies.
Features
- Zero-copy superblock, object header, and B-tree parsing
- Chunked dataset read/write with filter pipelines
no_stdsupport (disablestdfeature)- Optional parallel reads via Rayon
- SHA-256 provenance tracking
Usage
use clawhdf5_format::Superblock;
let data = std::fs::read("data.h5").unwrap();
let sb = Superblock::from_bytes(&data).unwrap();
println!("HDF5 version {}.{}", sb.version_major(), sb.version_minor());
License
MIT