docs: plugin filters and the filter registry

CHANGELOG (Unreleased): LZF, bitshuffle, bzip2 and Blosc read and write
in pure Rust, their features, the ChunkOptions::plugin field (breaking for
struct-literal construction), the filter registry, the named
UnsupportedFilter message, and Blosc2/ZFP still unimplemented.
README: the clawhdf5-format feature table gains lzf (default),
bitshuffle, bzip2, blosc and plugin-filters, with how to write them and
what is not implemented; no speed claims. docs/known-issues.md: the audit's
filter gap is marked fixed 2026-09-26 for LZF/bitshuffle/bzip2/Blosc,
Blosc2 and ZFP still open. CLAUDE.md: the clawhdf5-filters row no longer
says "No Blosc". clawhdf5-format's crate docs list the new features.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-26 00:14:47 -05:00
co-authored by Claude Opus 5.5
parent 056092b082
commit e01160299a
5 changed files with 64 additions and 3 deletions
+30
View File
@@ -2,6 +2,36 @@
## Unreleased
### Plugin filters (2026-09-26)
- **LZF, bitshuffle, bzip2 and Blosc read and write, in pure Rust.** Files
written by h5py with `compression="lzf"`, or with hdf5plugin's
`Bitshuffle`, `BZip2` and `Blosc`, failed with `UnsupportedFilter`. New
`clawhdf5-format`/`clawhdf5` features: `lzf` (32000, **on by default**, no
dependencies), `bitshuffle` (32008: transpose only, LZ4 and Zstandard
modes), `bzip2` (307), `blosc` (32001: Blosc 1 frames with BloscLZ,
LZ4/LZ4HC, Snappy, Zlib and Zstandard codecs and byte/bit shuffle;
BloscLZ is decoded by a port of c-blosc 1.21's decoder, and cannot be
written), and `plugin-filters` for all four. None compiles C: Zstandard is
ruzstd, bzip2 is libbz2-rs-sys. Write with `DatasetBuilder::with_lzf()`,
`with_bitshuffle(..)`, `with_bzip2(..)`, `with_blosc(..)` or
`with_plugin_filter(PluginFilter::..)`; `ChunkOptions` gains a `plugin`
field (**breaking** for code that builds `ChunkOptions` with a struct
literal and no `..Default::default()`). Tested both ways against h5py 3.16
+ hdf5plugin 7.1 over 1-3-D shapes with partial edge chunks, 1-8-byte
types in both byte orders and incompressible data
(`crates/clawhdf5/tests/plugin_filters_interop.rs`). Conformance: 573 of
697 files ok (was 569) — h5ex_d_lzf/bshuf/bzip2/blosc.
- **Filter registry.** Filters are looked up by ID in
`clawhdf5_format::filter_registry` instead of a `match`: the built-in
table (per build), then codecs registered at run time with
`register_filter(id, codec)` — a decoding closure or a `FilterCodec` that
can also encode. Built-in IDs cannot be overridden; a registered decoder's
output is held to the chunk-size bound. Unknown IDs still fail with
`UnsupportedFilter(id)`, whose message now names known filters and the
missing feature ("unsupported filter: 32026 (Blosc2, not implemented by
clawhdf5)").
- **Not implemented:** Blosc2 (32026) and ZFP (32013) remain a clear error.
### Upgrade Notes
- **HDF5 correctness audit (2026-09-25).** A sweep of 686 public files (the
libhdf5 test files, the HDF Group's CVE reproducers, pyfive, netcdf-c,