docs: crate READMEs describe each crate as it is today
Every crate under crates/ now has a README (android, bench, cli, napi and wasm had none), each saying what the crate is, its main types and functions (names checked against the code), its cargo features with defaults and which ones build C (checked with `cargo tree`), and links to the top-level docs. Corrections to the old stubs: - clawhdf5-derive: the derive is `H5Type`, not `HDF5Type`, and it needs clawhdf5-format as a dependency. - clawhdf5-filters: deflate backends only, and no library crate depends on it; the filter pipeline and every other codec are in -format. - clawhdf5-gpu: vector distance compute, not I/O; not used by HDF5Memory::search. - clawhdf5-io: MpiVol is root-read + broadcast, not collective MPI-IO. - clawhdf5-ann: from_hdf5/search(q, k) did not exist; load_from_hdf5 and search(q, k, ef). - clawhdf5-accel: checksum::crc32_simd did not exist; the SSE4 and wasm backends are reported but run the scalar kernels. - clawhdf5-gpu: the old example called l2_distances, which does not exist (l2_search). - clawhdf5-agent: it described a "vector store" with "GPU acceleration"; it now covers HDF5Memory, search options, WAL, signing, the graph. - crates.io/docs.rs badges removed and `cargo install <crate>` replaced: nothing is published; depend on git. - fuzz: the opt-in CLAWHDF5_FUZZ_SECONDS smoke run in ci-test.sh. - tools: the FileEditor interop tests that live in this crate. - remote, py: license, other front ends, limits, File.mode/flush/chunks. The Rust examples of the facade, format, filters, accel, ann, derive and agent READMEs were compiled and run as tests (netcdf4, gpu and remote compiled only) in a scratch crate; the CLI example was run. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
@@ -1,27 +1,106 @@
|
||||
# clawhdf5-format
|
||||
|
||||
[](https://crates.io/crates/clawhdf5-format)
|
||||
[](https://docs.rs/clawhdf5-format)
|
||||
The HDF5 file format in pure Rust: parsers and writers for every on-disk
|
||||
structure, the filter pipeline and its codecs, and the shared type
|
||||
definitions the other crates use. Most users want the
|
||||
[`clawhdf5`](../clawhdf5/README.md) facade, which wraps this crate in an
|
||||
h5py-like API; use this one directly for low-level access or in `no_std`
|
||||
code.
|
||||
|
||||
Pure-Rust HDF5 binary format parsing and writing — no C dependencies.
|
||||
Not on crates.io yet; depend on it from git:
|
||||
|
||||
```toml
|
||||
[dependencies]
|
||||
clawhdf5-format = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" }
|
||||
```
|
||||
|
||||
## What is in it
|
||||
|
||||
- **Parsing:** superblock v0–v3 (`superblock`, with the superblock
|
||||
extension and metadata cache images, `superblock_ext`), object headers v1
|
||||
and v2 (`object_header`), every header message the readers use
|
||||
(`datatype`, `dataspace`, `data_layout` v1–v4 including virtual datasets,
|
||||
`fill_value`, `attribute`, `link_message`, `shared_message`, ...), groups
|
||||
old and new (`group_v1` symbol tables with local heaps, `group_v2` with
|
||||
fractal heaps and v2 B-trees), and every chunk index (v1 B-tree, single
|
||||
chunk, implicit, fixed array, extensible array, v2 B-tree).
|
||||
- **Reading data:** `data_read` (contiguous, compact, chunked),
|
||||
`partial_read` and `selection` (hyperslabs and points), `vl_data`
|
||||
(variable-length strings and sequences through the global heap),
|
||||
`chunk_cache`.
|
||||
- **Storage:** the `storage::Storage` trait (`read_at`, `read_ranges`,
|
||||
`len`, `hint`) that every read path goes through, so a file can be read
|
||||
from memory, a file handle or a remote backend
|
||||
([`clawhdf5-remote`](../clawhdf5-remote/README.md)).
|
||||
- **Writing:** `file_writer::FileWriter` and the builders in
|
||||
`type_builders` (datasets, groups, attributes, compound and enum types,
|
||||
links, virtual datasets, creation-order tracking); chunk indexes and
|
||||
dense-storage B-trees of any size (`chunked_write`, `btree_v2_write`,
|
||||
`ea_writer`). Output is read by h5py and h5dump.
|
||||
- **Filters:** `filter_pipeline` and `filter_registry` (look up by ID; other
|
||||
IDs can be registered at run time with `register_filter`). Built in:
|
||||
deflate, shuffle, Fletcher-32, N-Bit, scale-offset; behind features LZ4,
|
||||
Zstd, SZIP (decode), pcodec, and the plugin filters LZF, bitshuffle,
|
||||
bzip2, Blosc 1 (read and write), Blosc2 and ZFP (read only).
|
||||
- **Shared pieces:** `float16` (the one IEEE half-precision conversion the
|
||||
workspace uses), `provenance` (SHA-256 dataset hashes), `checksum`
|
||||
(Jenkins lookup3 for v2+ structures).
|
||||
|
||||
## Example
|
||||
|
||||
```rust
|
||||
use clawhdf5_format::file_writer::{AttrValue, FileWriter};
|
||||
use clawhdf5_format::{group_v2, object_header, signature, superblock};
|
||||
|
||||
// Write a file to memory
|
||||
let mut fw = FileWriter::new();
|
||||
fw.create_dataset("data")
|
||||
.with_f64_data(&[1.0, 2.0, 3.0])
|
||||
.with_shape(&[3])
|
||||
.set_attr("unit", AttrValue::String("m/s".into()));
|
||||
let bytes = fw.finish().unwrap();
|
||||
|
||||
// Parse it back: superblock -> path -> object header
|
||||
let (_user_block, file) = signature::split_user_block(&bytes).unwrap();
|
||||
let sb = superblock::Superblock::parse(file, 0).unwrap();
|
||||
let addr = group_v2::resolve_path_any(file, &sb, "data").unwrap();
|
||||
let hdr = object_header::ObjectHeader::parse(file, addr as usize, sb.offset_size, sb.length_size)
|
||||
.unwrap();
|
||||
assert!(!hdr.messages.is_empty());
|
||||
```
|
||||
|
||||
## Features
|
||||
|
||||
- Zero-copy superblock, object header, and B-tree parsing
|
||||
- Chunked dataset read/write with filter pipelines
|
||||
- `no_std` support (disable `std` feature)
|
||||
- Optional parallel reads via Rayon
|
||||
- SHA-256 provenance tracking
|
||||
| Feature | Default | What | Builds C |
|
||||
|---|---|---|---|
|
||||
| `std` | yes | standard library; without it the crate is `no_std` + `alloc` (CI builds it for `thumbv7em-none-eabihf`) | no |
|
||||
| `checksum` | yes | verify Jenkins lookup3 checksums | no |
|
||||
| `deflate` | yes | deflate through flate2 | no |
|
||||
| `zlib-rs` | yes | flate2's pure-Rust zlib-rs backend, with `runtime_detection` (without it zlib-rs loses SIMD and inflates 3.5x slower) | no |
|
||||
| `system-zlib-decompress` | yes | no effect (nothing reads it; kept so existing feature lists still build) | no |
|
||||
| `provenance` | yes | SHA-256 provenance hashes | no |
|
||||
| `lzf` | yes | LZF (32000) | no |
|
||||
| `parallel` | no | rayon-parallel chunk decoding | no |
|
||||
| `fast-checksum` | no | hardware CRC32 through `crc32fast` | no |
|
||||
| `lz4` | no | LZ4 (32004) | no |
|
||||
| `pcodec` | no | pcodec | no |
|
||||
| `bitshuffle`, `bzip2`, `blosc` | no | 32008, 307, 32001, read and write | no |
|
||||
| `blosc2`, `zfp` | no | 32026, 32013, read only | no |
|
||||
| `plugin-filters` | no | all six plugin filters above | no |
|
||||
| `lookup-stats` | no | counters for name-lookup benchmarks | no |
|
||||
| `zstd` | no | Zstandard (32015) | yes (libzstd) |
|
||||
| `szip` | no | SZIP (4) decoding | links the system libaec (`libaec-dev`) |
|
||||
| `fast-deflate` | no | zlib-ng | yes (cmake) |
|
||||
| `system-zlib` | no | the system zlib | yes (`libz-sys`) |
|
||||
| `blake3_hash` | no | `provenance::blake3_hash` | yes (`cc`) |
|
||||
|
||||
## Usage
|
||||
## Robustness
|
||||
|
||||
```rust
|
||||
use clawhdf5_format::Superblock;
|
||||
|
||||
let data = std::fs::read("data.h5").unwrap();
|
||||
let sb = Superblock::from_bytes(&data).unwrap();
|
||||
println!("HDF5 version {}.{}", sb.version_major(), sb.version_minor());
|
||||
```
|
||||
Every parser is meant to return an error, never panic, on hostile input:
|
||||
nine cargo-fuzz targets live in [`fuzz/`](fuzz/README.md), the conformance
|
||||
sweep includes the HDF Group's CVE corpus
|
||||
([`CONFORMANCE.md`](../../CONFORMANCE.md)), and header checks follow
|
||||
libhdf5's. Open gaps are in [`docs/known-issues.md`](../../docs/known-issues.md).
|
||||
|
||||
## License
|
||||
|
||||
|
||||
@@ -51,10 +51,18 @@ done
|
||||
|
||||
## CI
|
||||
|
||||
These targets are **not** run in CI (`.gitea/workflows/ci.yml`) — cargo-fuzz
|
||||
requires nightly and each meaningful run takes minutes, which doesn't fit a
|
||||
per-PR gate. Run them manually on a schedule (e.g. before a release, or after
|
||||
touching parser code) instead.
|
||||
These targets are **not** run by the CI workflows (`.gitea/workflows/ci.yml`)
|
||||
— cargo-fuzz requires nightly and each meaningful run takes minutes, which
|
||||
doesn't fit a per-PR gate. Run them by hand before a release or after
|
||||
touching parser code. `scripts/ci-test.sh` has an opt-in smoke run: with
|
||||
`CLAWHDF5_FUZZ_SECONDS=N` it runs every target of this crate and of
|
||||
`crates/clawhdf5-agent/fuzz` (the WAL parser) for N seconds each.
|
||||
|
||||
Other robustness checks that do run: the nightly conformance sweep reads
|
||||
the HDF Group's CVE reproducers and fails on any panic, hang, crash or
|
||||
out-of-memory ([`conformance/README.md`](../../../conformance/README.md)),
|
||||
and `scripts/h5rs-fuzz.sh` runs every `h5rs` subcommand over them, optionally
|
||||
on byte-flipped copies.
|
||||
|
||||
## Reproducing Crashes
|
||||
|
||||
|
||||
Reference in New Issue
Block a user