osobhandClaude Opus 5.5
9e9b849dd7
read_f32/read_f64/read_i32/read_i64/read_u64 of a chunked dataset that stores exactly that type in native byte order now decode every chunk straight into the Vec<T> they return (data_read::read_chunked_native), on File (through its chunk cache), MmapFile and LazyFile. Before, the chunks went into a byte buffer that read_as_* then copied into a second, typed one: two dataset-sized allocations and a full extra copy per read. The output is zeroed pages from the allocator, backed by transparent huge pages when large, like the byte reader's. Other types and byte orders, and datasets with no storage or external data, keep converting through the byte readers; unallocated chunks read as the fill value as before. tests/chunked_read_paths_interop.rs checks every chunked read path (File twice, so cached; from_bytes; MmapFile; LazyFile; small, strided and point selections; with and without the parallel feature) against h5py for 1-8 byte integers and 2-8 byte floats in both byte orders, through deflate, shuffle, Fletcher32, LZF, SZIP and Blosc, with partial edge chunks, sparse datasets with default and non-default fill values, and datasets larger than the chunk cache. A filter this build lacks must be an error (or, when an optional filter declined every chunk, the right data). Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
clawhdf5
Pure-Rust HDF5 reader/writer — no C dependencies.
Features
- Read and write HDF5 files entirely in Rust
- Memory-mapped I/O for large files (
mmapfeature, enabled by default) - Parallel chunk reads via Rayon (
parallelfeature) - Lazy dataset access for minimal memory usage
- h5py-compatible file output
Usage
use clawhdf5::File;
let file = File::open("data.h5").unwrap();
let dataset = file.dataset("/group/data").unwrap();
let values: Vec<f64> = dataset.read_1d().unwrap();
License
MIT