osobhandClaude Fable 5.1
c6a7bbfc67
read_raw_data_selection computed which chunks a selection intersects, threw the answer away, decoded the entire dataset and picked elements out of it — for contiguous layouts too. A 64x64 window of a 64 MB deflate dataset cost 105 ms, about half a full read; every selection cost the same whatever its size. New partial_read module: materialise only the selection's bounding box — the overlapping rows of a contiguous dataset (straight from the file bytes) or the overlapping chunks (only those are decompressed) — then run the existing extractor over that buffer with the selection translated to the box origin, so extraction semantics are exactly the full-read ones. It declines (falling back to the old path) for All/None, compact/virtual/storage-less layouts, and boxes covering more than half the dataset. That window now takes 0.39 ms, one row 2.7 ms, one column 5.2 ms. Selections are validated against the dataset shape first. They were not: a hyperslab past an edge came back padded with zeros and a point with an out-of-range column wrapped into the next row, returning the wrong element with no error. Now FormatError::SelectionOutOfBounds (also rank mismatch and overlapping blocks); the facade's fill-aware path validates too. Tests: equivalence against a reference extraction from a full read over 60 random hyperslabs/point lists per layout (contiguous, chunked, deflate) for ranks 1-3. New read_harness bench binary with before/after in BENCHMARKS.md. Co-Authored-By: Claude Fable 5.1 <[email protected]>
105 lines
2.9 KiB
Rust
105 lines
2.9 KiB
Rust
//! Pure-Rust HDF5 binary format parsing and writing.
|
|
//!
|
|
//! `clawhdf5-format` is a zero-dependency (no C libhdf5) crate for reading and
|
|
//! writing HDF5 files. It supports `no_std` environments with the `alloc` crate.
|
|
//!
|
|
//! # Writing files
|
|
//!
|
|
//! Use [`file_writer::FileWriter`] to create HDF5 files:
|
|
//!
|
|
//! ```rust
|
|
//! use clawhdf5_format::file_writer::{FileWriter, AttrValue};
|
|
//!
|
|
//! let mut fw = FileWriter::new();
|
|
//! fw.create_dataset("data")
|
|
//! .with_f64_data(&[1.0, 2.0, 3.0])
|
|
//! .with_shape(&[3])
|
|
//! .set_attr("unit", AttrValue::String("m/s".into()));
|
|
//! let bytes = fw.finish().unwrap();
|
|
//! ```
|
|
//!
|
|
//! # Reading files
|
|
//!
|
|
//! Parsing follows the HDF5 object model: superblock → object header → messages.
|
|
//!
|
|
//! ```rust,no_run
|
|
//! use clawhdf5_format::{signature, superblock, object_header, group_v2,
|
|
//! datatype, dataspace, data_layout, data_read, message_type::MessageType};
|
|
//!
|
|
//! let file_data = std::fs::read("output.h5").unwrap();
|
|
//! let sig = signature::find_signature(&file_data).unwrap();
|
|
//! let sb = superblock::Superblock::parse(&file_data, sig).unwrap();
|
|
//! let addr = group_v2::resolve_path_any(&file_data, &sb, "data").unwrap();
|
|
//! let hdr = object_header::ObjectHeader::parse(
|
|
//! &file_data, addr as usize, sb.offset_size, sb.length_size).unwrap();
|
|
//! ```
|
|
//!
|
|
//! # Features
|
|
//!
|
|
//! | Feature | Default | Description |
|
|
//! |---------|---------|-------------|
|
|
//! | `std` | yes | Standard library support |
|
|
//! | `checksum` | yes | Jenkins lookup3 checksum validation |
|
|
//! | `deflate` | yes | Deflate (gzip) compression via `flate2` |
|
|
//! | `provenance` | yes | SHINES provenance — SHA-256 hashing & verification |
|
|
|
|
#![cfg_attr(not(feature = "std"), no_std)]
|
|
|
|
#[cfg(not(feature = "std"))]
|
|
extern crate alloc;
|
|
|
|
pub mod attribute;
|
|
pub mod attribute_info;
|
|
pub mod btree_v1;
|
|
pub mod btree_v2;
|
|
pub mod checksum;
|
|
pub mod chunk_cache;
|
|
pub mod chunk_index;
|
|
pub mod chunked_read;
|
|
pub mod chunked_write;
|
|
pub mod data_layout;
|
|
pub mod data_layout_write;
|
|
pub mod data_read;
|
|
pub mod dataspace;
|
|
pub mod datatype;
|
|
pub mod dict_encoding;
|
|
pub mod ea_writer;
|
|
pub mod error;
|
|
pub mod extensible_array;
|
|
pub mod file_writer;
|
|
pub mod fill_value;
|
|
pub mod filter_pipeline;
|
|
pub mod filters;
|
|
mod filters_szip;
|
|
pub mod fixed_array;
|
|
pub mod fractal_heap;
|
|
pub mod global_heap;
|
|
pub mod group_info;
|
|
pub mod group_v1;
|
|
pub mod group_v2;
|
|
#[cfg(feature = "parallel")]
|
|
pub mod lane_partition;
|
|
pub mod link_info;
|
|
pub mod link_message;
|
|
pub mod local_heap;
|
|
pub mod message_type;
|
|
pub mod metadata_cache;
|
|
pub mod metadata_index;
|
|
pub mod object_header;
|
|
pub mod object_header_writer;
|
|
#[cfg(feature = "parallel")]
|
|
pub mod parallel_read;
|
|
pub mod partial_read;
|
|
pub mod profiling;
|
|
pub mod property_list;
|
|
pub mod selection;
|
|
pub mod shared_message;
|
|
pub mod signature;
|
|
pub mod superblock;
|
|
pub mod symbol_table;
|
|
pub mod type_builders;
|
|
pub mod vl_data;
|
|
|
|
#[cfg(feature = "provenance")]
|
|
pub mod provenance;
|