From e01160299add4a488a4dcd97d842af234bf9290e Mon Sep 17 00:00:00 2001 From: osobh Date: Sat, 26 Sep 2026 00:14:47 -0500 Subject: [PATCH] docs: plugin filters and the filter registry CHANGELOG (Unreleased): LZF, bitshuffle, bzip2 and Blosc read and write in pure Rust, their features, the ChunkOptions::plugin field (breaking for struct-literal construction), the filter registry, the named UnsupportedFilter message, and Blosc2/ZFP still unimplemented. README: the clawhdf5-format feature table gains lzf (default), bitshuffle, bzip2, blosc and plugin-filters, with how to write them and what is not implemented; no speed claims. docs/known-issues.md: the audit's filter gap is marked fixed 2026-09-26 for LZF/bitshuffle/bzip2/Blosc, Blosc2 and ZFP still open. CLAUDE.md: the clawhdf5-filters row no longer says "No Blosc". clawhdf5-format's crate docs list the new features. Co-Authored-By: Claude Opus 5.5 (1M context) --- CHANGELOG.md | 30 ++++++++++++++++++++++++++++++ CLAUDE.md | 2 +- README.md | 18 +++++++++++++++++- crates/clawhdf5-format/src/lib.rs | 8 ++++++++ docs/known-issues.md | 9 ++++++++- 5 files changed, 64 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1fc429d..9318dbc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,36 @@ ## Unreleased +### Plugin filters (2026-09-26) +- **LZF, bitshuffle, bzip2 and Blosc read and write, in pure Rust.** Files + written by h5py with `compression="lzf"`, or with hdf5plugin's + `Bitshuffle`, `BZip2` and `Blosc`, failed with `UnsupportedFilter`. New + `clawhdf5-format`/`clawhdf5` features: `lzf` (32000, **on by default**, no + dependencies), `bitshuffle` (32008: transpose only, LZ4 and Zstandard + modes), `bzip2` (307), `blosc` (32001: Blosc 1 frames with BloscLZ, + LZ4/LZ4HC, Snappy, Zlib and Zstandard codecs and byte/bit shuffle; + BloscLZ is decoded by a port of c-blosc 1.21's decoder, and cannot be + written), and `plugin-filters` for all four. None compiles C: Zstandard is + ruzstd, bzip2 is libbz2-rs-sys. Write with `DatasetBuilder::with_lzf()`, + `with_bitshuffle(..)`, `with_bzip2(..)`, `with_blosc(..)` or + `with_plugin_filter(PluginFilter::..)`; `ChunkOptions` gains a `plugin` + field (**breaking** for code that builds `ChunkOptions` with a struct + literal and no `..Default::default()`). Tested both ways against h5py 3.16 + + hdf5plugin 7.1 over 1-3-D shapes with partial edge chunks, 1-8-byte + types in both byte orders and incompressible data + (`crates/clawhdf5/tests/plugin_filters_interop.rs`). Conformance: 573 of + 697 files ok (was 569) — h5ex_d_lzf/bshuf/bzip2/blosc. +- **Filter registry.** Filters are looked up by ID in + `clawhdf5_format::filter_registry` instead of a `match`: the built-in + table (per build), then codecs registered at run time with + `register_filter(id, codec)` — a decoding closure or a `FilterCodec` that + can also encode. Built-in IDs cannot be overridden; a registered decoder's + output is held to the chunk-size bound. Unknown IDs still fail with + `UnsupportedFilter(id)`, whose message now names known filters and the + missing feature ("unsupported filter: 32026 (Blosc2, not implemented by + clawhdf5)"). +- **Not implemented:** Blosc2 (32026) and ZFP (32013) remain a clear error. + ### Upgrade Notes - **HDF5 correctness audit (2026-09-25).** A sweep of 686 public files (the libhdf5 test files, the HDF Group's CVE reproducers, pyfive, netcdf-c, diff --git a/CLAUDE.md b/CLAUDE.md index 3f8d9c1..12a4b93 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -11,7 +11,7 @@ Cargo workspace with 16 crates under `crates/` (plus `libaec-sys`, an internal F |-------|------| | `clawhdf5-format` | HDF5 binary spec parser (superblock, B-tree, heap) — also holds shared type definitions and physical constants | | `clawhdf5-io` | Read/write implementation | -| `clawhdf5-filters` | Deflate backends (zlib-rs, zlib-ng, Apple Compression); the HDF5 filter pipeline and the other codecs (LZ4, Zstd, SZIP, N-Bit, scale-offset, pcodec) live in `clawhdf5-format`. No Blosc. | +| `clawhdf5-filters` | Deflate backends (zlib-rs, zlib-ng, Apple Compression); the HDF5 filter pipeline, the filter registry (`clawhdf5_format::filter_registry`) and the other codecs (LZ4, Zstd, SZIP, N-Bit, scale-offset, pcodec, and the pure-Rust plugin filters LZF, bitshuffle, bzip2, Blosc 1) live in `clawhdf5-format`. No Blosc2 or ZFP. | | `clawhdf5-derive` | Proc-macro derive for HDF5-serializable structs | | `clawhdf5` | Main facade crate | | `clawhdf5-netcdf4` | NetCDF-4 compatibility layer | diff --git a/README.md b/README.md index 5d4c707..1088194 100644 --- a/README.md +++ b/README.md @@ -594,7 +594,7 @@ clawhdf5 workspace (16 crates, ~86K lines of Rust in src/, ~104K with tests ├── Core HDF5 │ ├── clawhdf5-format — Binary parser/writer (no_std-capable), shared type definitions │ ├── clawhdf5-io — I/O abstraction (file/memory readers; optional mmap, async, HSDS, MPI) -│ ├── clawhdf5-filters — Fast deflate path (zlib-ng); lz4/zstd/pcodec/szip filters live in clawhdf5-format +│ ├── clawhdf5-filters — Fast deflate path (zlib-ng); the filter registry and the lz4/zstd/pcodec/szip/LZF/bitshuffle/bzip2/Blosc filters live in clawhdf5-format │ ├── clawhdf5-derive — Proc macros │ ├── clawhdf5 — High-level API │ ├── clawhdf5-netcdf4 — NetCDF-4 support @@ -700,6 +700,22 @@ stores keep their setting. Opt out with `float16 = false` or | `system-zlib` | no | System zlib backend for deflate (C) | | `blake3_hash` | no | BLAKE3 content hashing for provenance | | `szip` | no | SZIP filter (id 4) via libaec (C, through the internal `libaec-sys` crate) | +| `lzf` | **yes** | LZF filter (id 32000), h5py's built-in `compression="lzf"`: read and write. No dependencies | +| `bitshuffle` | no | Bitshuffle filter (id 32008) with its LZ4 and Zstandard modes: read and write. Pure Rust (lz4_flex, ruzstd) | +| `bzip2` | no | bzip2 filter (id 307): read and write. Pure Rust (the `bzip2` crate's libbz2-rs-sys backend compiles no C) | +| `blosc` | no | Blosc 1 filter (id 32001): reads BloscLZ, LZ4/LZ4HC, Snappy, Zlib and Zstandard frames with byte or bit shuffle; writes LZ4, Snappy, Zlib or Zstandard (not BloscLZ). Pure Rust | +| `plugin-filters` | no | All four above | + +Blosc2 (32026) and ZFP (32013) are not implemented: reading them fails with +`UnsupportedFilter`, whose message names the filter. Any other filter can be +supplied at run time with `filter_registry::register_filter` (a decoder +closure, or a `FilterCodec` that also encodes). The facade (`clawhdf5`) +forwards `lzf`, `bitshuffle`, `bzip2`, `blosc` and `plugin-filters`. Write +with `DatasetBuilder::with_lzf()`, `with_bitshuffle(..)`, `with_bzip2(..)` +and `with_blosc(..)`; h5py + hdf5plugin read the result (tested both ways in +`crates/clawhdf5/tests/plugin_filters_interop.rs`). The pure-Rust Zstandard +encoder has one level (about zstd's level 1); no speed or ratio claims are +made for these codecs. ### `clawhdf5-ann` diff --git a/crates/clawhdf5-format/src/lib.rs b/crates/clawhdf5-format/src/lib.rs index d9bba65..3e31686 100644 --- a/crates/clawhdf5-format/src/lib.rs +++ b/crates/clawhdf5-format/src/lib.rs @@ -43,6 +43,14 @@ //! | `checksum` | yes | Jenkins lookup3 checksum validation | //! | `deflate` | yes | Deflate (gzip) compression via `flate2` | //! | `provenance` | yes | SHINES provenance — SHA-256 hashing & verification | +//! | `lzf` | yes | LZF filter (32000), h5py's `compression="lzf"` | +//! | `bitshuffle` | no | Bitshuffle filter (32008), none/LZ4/Zstandard | +//! | `bzip2` | no | bzip2 filter (307) | +//! | `blosc` | no | Blosc 1 filter (32001) | +//! | `plugin-filters` | no | The four above | +//! +//! Filters are looked up by ID in [`filter_registry`], which also takes +//! codecs registered at run time for other IDs. #![cfg_attr(not(feature = "std"), no_std)] diff --git a/docs/known-issues.md b/docs/known-issues.md index 4ea7518..53c4996 100644 --- a/docs/known-issues.md +++ b/docs/known-issues.md @@ -133,7 +133,14 @@ fill-value item that did is fixed). - x87 long double and binary128 are refused. - N-Bit on 64-bit scale-offset data and some N-Bit parameter layouts fail. - **Filters:** blosc, blosc2, bitshuffle, bzip2, LZF and zfp are not - implemented. + implemented. **Fixed 2026-09-26** for LZF (default-on `lzf` feature), + bitshuffle, bzip2 and Blosc 1 (`bitshuffle`, `bzip2`, `blosc`, or + `plugin-filters` for all), read and write, pure Rust; h5ex_d_lzf, + h5ex_d_bshuf, h5ex_d_bzip2 and h5ex_d_blosc now read (conformance 573 of + 697 ok). **Still open:** Blosc2 (32026 — hdf5plugin stores each chunk as a + Blosc2 super-chunk frame, and n-D chunks as B2ND arrays) and ZFP (32013); + both fail with an `UnsupportedFilter` error that names the filter, and + either can be plugged in with `filter_registry::register_filter`. - **Header checks:** on 12 CVE datasets libhdf5 rejects a corrupt header and we read data anyway. We need stricter header checks. - **Writer:**