clawhdf5/crates/clawhdf5

clawhdf5

The main crate: a pure-Rust HDF5 reader, writer and in-place editor, with no libhdf5 and, by default, no C code. It wraps clawhdf5-format (the binary format) and clawhdf5-io (memory-mapped reads) in an h5py-like API.

Not on crates.io yet; depend on it from git:

[dependencies]
clawhdf5 = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" }

Main types

Type	What it does
File	Opens a file (open, open_buffered, from_bytes, open_storage for any Storage), walks groups (root, group, dataset), lists datasets/groups/attrs. File is Send + Sync: several threads can read one open file.
Dataset	shape, dtype, max_dimensions, attrs; reads read_f64/read_f32/read_i32/read_i64/read_u64, strings (read_string, read_string_bytes), variable-length data (read_vlen), hyperslabs and point selections (read_selection, read_f64_selection, ...), zero-copy views of contiguous data (read_f64_zerocopy, ...), verify_provenance.
FileBuilder	Writes a new file: datasets of every numeric type, strings, compounds (CompoundTypeBuilder), enums, chunked and compressed layouts (deflate, shuffle, Fletcher-32, LZF, and with features LZ4, Zstd, bitshuffle, bzip2, Blosc, pcodec), nested groups, soft/hard/external links, virtual datasets, attribute creation order. Files open in h5py and h5dump.
FileEditor	Changes an existing file in place without rewriting it: write_values/write_selection/write_all, resize of chunked datasets (every chunk index), set_attr (compact and dense storage). Anything it cannot do safely is Error::Unsupported before any write.
MmapFile, LazyFile	Alternative readers: memory-mapped, and one that reads lazily and caches.
File::open_swmr	Reads a file a libhdf5 SWMR writer is still appending to (Dataset::refresh, bounded retries), as h5py's swmr=True reader does.

Examples

use clawhdf5::{AttrValue, File, FileBuilder, FileEditor, Selection};

// Write
let mut b = FileBuilder::new();
b.create_dataset("sensors/temperature")
    .with_f64_data(&[20.5, 21.0, 21.5, 22.0])
    .with_shape(&[4])
    .with_maxshape(&[u64::MAX]) // unlimited, so it can grow
    .with_chunks(&[2])
    .with_deflate(4);
b.set_attr("version", AttrValue::I64(1));
b.write("data.h5")?;

// Read
let file = File::open("data.h5")?;
let ds = file.dataset("sensors/temperature")?;
assert_eq!(ds.shape()?, vec![4]);
let values = ds.read_f64()?;

// Edit in place: grow the dataset and fill the new tail
let mut ed = FileEditor::open("data.h5")?;
ed.resize("sensors/temperature", &[6])?;
let tail = Selection::Hyperslab {
    start: vec![4],
    stride: vec![1],
    count: vec![2],
    block: vec![1],
};
ed.write_values("sensors/temperature", &tail, &[22.5f64, 23.0])?;
# Ok::<(), clawhdf5::Error>(())

Remote files (HTTP range requests, S3/GCS/Azure) are read through File::open_storage; clawhdf5-remote provides the storage and its block cache.

Features

Feature	Default	What	Builds C
mmap	yes	memory-mapped reads (File::open maps the file; MmapFile)	no
provenance	yes	SHA-256 _provenance_sha256 attributes (DatasetBuilder::with_provenance, Dataset::verify_provenance)	no
lzf	yes	LZF filter (32000), h5py's compression="lzf"	no
parallel	no	chunk decoding on a rayon pool	no
lz4	no	LZ4 filter (32004)	no
pcodec	no	pcodec filter	no
bitshuffle, bzip2, blosc	no	plugin filters 32008, 307, 32001 (read and write)	no (bzip2 uses the pure-Rust libbz2-rs-sys)
blosc2, zfp	no	plugin filters 32026 and 32013, read only	no
plugin-filters	no	lzf, bitshuffle, bzip2, blosc, blosc2, zfp	no
zstd	no	Zstandard filter (32015)	yes (libzstd)
fast-deflate	no	zlib-ng instead of the pure-Rust zlib-rs	yes (cmake)
blake3_hash	no	provenance::blake3_hash helpers	yes (cc, for blake3's SIMD code)
apple-compression	no	currently has no effect in this crate (it is not forwarded)	—

SZIP decoding is a clawhdf5-format feature (szip, links the system libaec); the facade does not forward it.

Limits and further reading

• What is known not to work, and what was wrong in earlier releases: docs/known-issues.md (editor limits, range reads, external links and external raw data, which are explicit errors).
• Read coverage against libhdf5/h5py on eight public corpora: CONFORMANCE.md.
• Read and write speed against libhdf5 and h5py: BENCHMARKS.md.
• Range reads and SWMR design: docs/design/range-reads.md, docs/design/swmr.md.
• Changes: CHANGELOG.md.

License

MIT