osobhandClaude Opus 5.5
b5a5041655
New `LibVer` (V18, V110, V112, V114, V200, Latest) and
`libver_bounds(low, high)` on the format crate's `FileWriter` and the
facade's `FileBuilder`, as libhdf5's H5Pset_libver_bounds / h5py's
libver=(low, high). The default stays (V110, Latest), byte for byte what
was written before.
With a low bound of 1.8: superblock version 2, layout message version 3
(contiguous, compact, chunked) and a version-1 B-tree chunk index for
every chunked dataset, resizable ones included -- what libhdf5 2.x writes
under libver=('v108', 'latest'). The new chunk B-tree writer
(btree_v1_write.rs) replays H5B_insert with the H5Dbtree.c callbacks for
row-major insertion (split ratios 0.1/0.5/0.9, right keys moved as
H5D__btree_cmp3 moves them, root kept in place): its trees equal
libhdf5's node for node for 1-D/2-D/3-D, 2- and 3-level, filtered and
unfiltered datasets (libhdf5 writing without a chunk cache).
The high bound refuses, with FormatError::LibverBound before anything is
written, what needs a newer format: virtual datasets and the paged
file-space strategy (1.10), the 1.12 reference types (datatype v4),
native complex (datatype v5, HDF5 2.0), and a low bound above the high.
Tests: tools/tests/libver_v18.rs writes every writer feature under
(V18, V18), and HDF5 1.8.23's h5dump (scripts/build-hdf5-1.8.sh; skipped
when absent) dumps it exactly as h5dump 1.14 does and returns our bytes
for every numeric dataset; h5py, clawhdf5 and h5rs check --data agree;
then FileEditor grows/appends/annotates it and h5py appends, and every
reader checks again. read_harness gains --v18 and --chunk N.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
150 lines
4.0 KiB
Rust
150 lines
4.0 KiB
Rust
//! Pure-Rust HDF5 binary format parsing and writing.
|
|
//!
|
|
//! `clawhdf5-format` is a zero-dependency (no C libhdf5) crate for reading and
|
|
//! writing HDF5 files. It supports `no_std` environments with the `alloc` crate.
|
|
//!
|
|
//! # Writing files
|
|
//!
|
|
//! Use [`file_writer::FileWriter`] to create HDF5 files:
|
|
//!
|
|
//! ```rust
|
|
//! use clawhdf5_format::file_writer::{FileWriter, AttrValue};
|
|
//!
|
|
//! let mut fw = FileWriter::new();
|
|
//! fw.create_dataset("data")
|
|
//! .with_f64_data(&[1.0, 2.0, 3.0])
|
|
//! .with_shape(&[3])
|
|
//! .set_attr("unit", AttrValue::String("m/s".into()));
|
|
//! let bytes = fw.finish().unwrap();
|
|
//! ```
|
|
//!
|
|
//! # Reading files
|
|
//!
|
|
//! Parsing follows the HDF5 object model: superblock → object header → messages.
|
|
//!
|
|
//! ```rust,no_run
|
|
//! use clawhdf5_format::{signature, superblock, object_header, group_v2,
|
|
//! datatype, dataspace, data_layout, data_read, message_type::MessageType};
|
|
//!
|
|
//! let bytes = std::fs::read("output.h5").unwrap();
|
|
//! // Addresses are relative to the superblock: skip any user block.
|
|
//! let (_user_block, file_data) = signature::split_user_block(&bytes).unwrap();
|
|
//! let sb = superblock::Superblock::parse(file_data, 0).unwrap();
|
|
//! let addr = group_v2::resolve_path_any(file_data, &sb, "data").unwrap();
|
|
//! let hdr = object_header::ObjectHeader::parse(
|
|
//! file_data, addr as usize, sb.offset_size, sb.length_size).unwrap();
|
|
//! ```
|
|
//!
|
|
//! # Features
|
|
//!
|
|
//! | Feature | Default | Description |
|
|
//! |---------|---------|-------------|
|
|
//! | `std` | yes | Standard library support |
|
|
//! | `checksum` | yes | Jenkins lookup3 checksum validation |
|
|
//! | `deflate` | yes | Deflate (gzip) compression via `flate2` |
|
|
//! | `provenance` | yes | SHINES provenance — SHA-256 hashing & verification |
|
|
//! | `lzf` | yes | LZF filter (32000), h5py's `compression="lzf"` |
|
|
//! | `bitshuffle` | no | Bitshuffle filter (32008), none/LZ4/Zstandard |
|
|
//! | `bzip2` | no | bzip2 filter (307) |
|
|
//! | `blosc` | no | Blosc 1 filter (32001) |
|
|
//! | `plugin-filters` | no | The four above |
|
|
//!
|
|
//! Filters are looked up by ID in [`filter_registry`], which also takes
|
|
//! codecs registered at run time for other IDs.
|
|
|
|
#![cfg_attr(not(feature = "std"), no_std)]
|
|
|
|
#[cfg(not(feature = "std"))]
|
|
extern crate alloc;
|
|
|
|
pub mod addr;
|
|
pub mod attribute;
|
|
pub mod attribute_info;
|
|
pub mod btree_v1;
|
|
mod btree_v1_write;
|
|
pub mod btree_v2;
|
|
mod btree_v2_write;
|
|
mod bulk_alloc;
|
|
pub mod checksum;
|
|
pub mod chunk_cache;
|
|
mod chunk_grid;
|
|
pub mod chunk_index;
|
|
pub mod chunked_read;
|
|
pub mod chunked_write;
|
|
pub mod data_layout;
|
|
pub mod data_layout_write;
|
|
pub mod data_read;
|
|
pub mod dataspace;
|
|
pub mod datatype;
|
|
pub mod dict_encoding;
|
|
pub mod ea_writer;
|
|
pub mod error;
|
|
pub mod extensible_array;
|
|
pub mod file_writer;
|
|
pub mod fill_value;
|
|
pub mod filter_pipeline;
|
|
pub mod filter_registry;
|
|
pub mod filters;
|
|
#[cfg(any(feature = "bitshuffle", feature = "blosc"))]
|
|
mod filters_bitshuffle;
|
|
#[cfg(feature = "blosc")]
|
|
pub mod filters_blosc;
|
|
#[cfg(feature = "blosc2")]
|
|
pub mod filters_blosc2;
|
|
#[cfg(feature = "bzip2")]
|
|
mod filters_bzip2;
|
|
#[cfg(feature = "lzf")]
|
|
pub mod filters_lzf;
|
|
mod filters_szip;
|
|
#[cfg(feature = "zfp")]
|
|
pub mod filters_zfp;
|
|
pub mod fixed_array;
|
|
pub mod float16;
|
|
pub mod fractal_heap;
|
|
mod gather;
|
|
pub mod global_heap;
|
|
pub mod group_info;
|
|
pub mod group_v1;
|
|
pub mod group_v2;
|
|
#[cfg(feature = "parallel")]
|
|
pub mod lane_partition;
|
|
pub mod libver;
|
|
pub mod link_info;
|
|
pub mod link_message;
|
|
pub mod local_heap;
|
|
pub mod lookup_stats;
|
|
pub mod message_type;
|
|
pub mod metadata_cache;
|
|
pub mod metadata_index;
|
|
pub mod object_header;
|
|
pub mod object_header_writer;
|
|
#[cfg(feature = "parallel")]
|
|
pub mod parallel_read;
|
|
pub mod partial_read;
|
|
pub mod profiling;
|
|
pub mod property_list;
|
|
pub mod selection;
|
|
pub mod shared_message;
|
|
pub mod signature;
|
|
pub mod storage;
|
|
pub mod superblock;
|
|
pub mod superblock_ext;
|
|
pub mod symbol_table;
|
|
#[cfg(all(
|
|
test,
|
|
any(
|
|
feature = "lzf",
|
|
feature = "bitshuffle",
|
|
feature = "bzip2",
|
|
feature = "blosc"
|
|
)
|
|
))]
|
|
mod test_fuzz;
|
|
pub mod type_builders;
|
|
pub mod vds;
|
|
pub mod vl_data;
|
|
mod writer_tree;
|
|
|
|
#[cfg(feature = "provenance")]
|
|
pub mod provenance;
|