osobhandClaude Opus 5.5
7f52a6f3ba
Audited LZF, bitshuffle, bzip2 and Blosc/BloscLZ for arithmetic on header fields and unchecked slicing. The only live bug was the Blosc frame-size underflow fixed in the previous commit; bzip2's output-growth step now uses a saturating subtraction as well (the allocator may hand back more capacity than asked for). src/test_fuzz.rs (tests only) feeds each decoder random bytes, truncated seeds and one-to-four-edit mutations of valid frames, biased towards edge-case u32 values in size and offset fields, and asserts no panic and no output over the limit (tests build with overflow checks and debug assertions). Per decoder: LZF, bzip2, bitshuffle in all six mode/block settings plus hostile cd_values, Blosc across four codecs, three shuffles, stored frames and a hand-built BloscLZ frame, and BloscLZ streams alone. With the previous commit's check removed, fuzzed_frames_never_panic panics at the same subtraction. A 100x-iteration soak (different seed) found no other panic. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
136 lines
3.8 KiB
Rust
136 lines
3.8 KiB
Rust
//! Pure-Rust HDF5 binary format parsing and writing.
|
|
//!
|
|
//! `clawhdf5-format` is a zero-dependency (no C libhdf5) crate for reading and
|
|
//! writing HDF5 files. It supports `no_std` environments with the `alloc` crate.
|
|
//!
|
|
//! # Writing files
|
|
//!
|
|
//! Use [`file_writer::FileWriter`] to create HDF5 files:
|
|
//!
|
|
//! ```rust
|
|
//! use clawhdf5_format::file_writer::{FileWriter, AttrValue};
|
|
//!
|
|
//! let mut fw = FileWriter::new();
|
|
//! fw.create_dataset("data")
|
|
//! .with_f64_data(&[1.0, 2.0, 3.0])
|
|
//! .with_shape(&[3])
|
|
//! .set_attr("unit", AttrValue::String("m/s".into()));
|
|
//! let bytes = fw.finish().unwrap();
|
|
//! ```
|
|
//!
|
|
//! # Reading files
|
|
//!
|
|
//! Parsing follows the HDF5 object model: superblock → object header → messages.
|
|
//!
|
|
//! ```rust,no_run
|
|
//! use clawhdf5_format::{signature, superblock, object_header, group_v2,
|
|
//! datatype, dataspace, data_layout, data_read, message_type::MessageType};
|
|
//!
|
|
//! let bytes = std::fs::read("output.h5").unwrap();
|
|
//! // Addresses are relative to the superblock: skip any user block.
|
|
//! let (_user_block, file_data) = signature::split_user_block(&bytes).unwrap();
|
|
//! let sb = superblock::Superblock::parse(file_data, 0).unwrap();
|
|
//! let addr = group_v2::resolve_path_any(file_data, &sb, "data").unwrap();
|
|
//! let hdr = object_header::ObjectHeader::parse(
|
|
//! file_data, addr as usize, sb.offset_size, sb.length_size).unwrap();
|
|
//! ```
|
|
//!
|
|
//! # Features
|
|
//!
|
|
//! | Feature | Default | Description |
|
|
//! |---------|---------|-------------|
|
|
//! | `std` | yes | Standard library support |
|
|
//! | `checksum` | yes | Jenkins lookup3 checksum validation |
|
|
//! | `deflate` | yes | Deflate (gzip) compression via `flate2` |
|
|
//! | `provenance` | yes | SHINES provenance — SHA-256 hashing & verification |
|
|
//! | `lzf` | yes | LZF filter (32000), h5py's `compression="lzf"` |
|
|
//! | `bitshuffle` | no | Bitshuffle filter (32008), none/LZ4/Zstandard |
|
|
//! | `bzip2` | no | bzip2 filter (307) |
|
|
//! | `blosc` | no | Blosc 1 filter (32001) |
|
|
//! | `plugin-filters` | no | The four above |
|
|
//!
|
|
//! Filters are looked up by ID in [`filter_registry`], which also takes
|
|
//! codecs registered at run time for other IDs.
|
|
|
|
#![cfg_attr(not(feature = "std"), no_std)]
|
|
|
|
#[cfg(not(feature = "std"))]
|
|
extern crate alloc;
|
|
|
|
pub mod attribute;
|
|
pub mod attribute_info;
|
|
pub mod btree_v1;
|
|
pub mod btree_v2;
|
|
pub mod checksum;
|
|
pub mod chunk_cache;
|
|
mod chunk_grid;
|
|
pub mod chunk_index;
|
|
pub mod chunked_read;
|
|
pub mod chunked_write;
|
|
pub mod data_layout;
|
|
pub mod data_layout_write;
|
|
pub mod data_read;
|
|
pub mod dataspace;
|
|
pub mod datatype;
|
|
pub mod dict_encoding;
|
|
pub mod ea_writer;
|
|
pub mod error;
|
|
pub mod extensible_array;
|
|
pub mod file_writer;
|
|
pub mod fill_value;
|
|
pub mod filter_pipeline;
|
|
pub mod filter_registry;
|
|
pub mod filters;
|
|
#[cfg(any(feature = "bitshuffle", feature = "blosc"))]
|
|
mod filters_bitshuffle;
|
|
#[cfg(feature = "blosc")]
|
|
pub mod filters_blosc;
|
|
#[cfg(feature = "bzip2")]
|
|
mod filters_bzip2;
|
|
#[cfg(feature = "lzf")]
|
|
pub mod filters_lzf;
|
|
mod filters_szip;
|
|
pub mod fixed_array;
|
|
pub mod float16;
|
|
pub mod fractal_heap;
|
|
pub mod global_heap;
|
|
pub mod group_info;
|
|
pub mod group_v1;
|
|
pub mod group_v2;
|
|
#[cfg(feature = "parallel")]
|
|
pub mod lane_partition;
|
|
pub mod link_info;
|
|
pub mod link_message;
|
|
pub mod local_heap;
|
|
pub mod message_type;
|
|
pub mod metadata_cache;
|
|
pub mod metadata_index;
|
|
pub mod object_header;
|
|
pub mod object_header_writer;
|
|
#[cfg(feature = "parallel")]
|
|
pub mod parallel_read;
|
|
pub mod partial_read;
|
|
pub mod profiling;
|
|
pub mod property_list;
|
|
pub mod selection;
|
|
pub mod shared_message;
|
|
pub mod signature;
|
|
pub mod superblock;
|
|
pub mod symbol_table;
|
|
#[cfg(all(
|
|
test,
|
|
any(
|
|
feature = "lzf",
|
|
feature = "bitshuffle",
|
|
feature = "bzip2",
|
|
feature = "blosc"
|
|
)
|
|
))]
|
|
mod test_fuzz;
|
|
pub mod type_builders;
|
|
pub mod vds;
|
|
pub mod vl_data;
|
|
|
|
#[cfg(feature = "provenance")]
|
|
pub mod provenance;
|