docs: plugin filters and the filter registry
CHANGELOG (Unreleased): LZF, bitshuffle, bzip2 and Blosc read and write in pure Rust, their features, the ChunkOptions::plugin field (breaking for struct-literal construction), the filter registry, the named UnsupportedFilter message, and Blosc2/ZFP still unimplemented. README: the clawhdf5-format feature table gains lzf (default), bitshuffle, bzip2, blosc and plugin-filters, with how to write them and what is not implemented; no speed claims. docs/known-issues.md: the audit's filter gap is marked fixed 2026-09-26 for LZF/bitshuffle/bzip2/Blosc, Blosc2 and ZFP still open. CLAUDE.md: the clawhdf5-filters row no longer says "No Blosc". clawhdf5-format's crate docs list the new features. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
@@ -2,6 +2,36 @@
|
||||
|
||||
## Unreleased
|
||||
|
||||
### Plugin filters (2026-09-26)
|
||||
- **LZF, bitshuffle, bzip2 and Blosc read and write, in pure Rust.** Files
|
||||
written by h5py with `compression="lzf"`, or with hdf5plugin's
|
||||
`Bitshuffle`, `BZip2` and `Blosc`, failed with `UnsupportedFilter`. New
|
||||
`clawhdf5-format`/`clawhdf5` features: `lzf` (32000, **on by default**, no
|
||||
dependencies), `bitshuffle` (32008: transpose only, LZ4 and Zstandard
|
||||
modes), `bzip2` (307), `blosc` (32001: Blosc 1 frames with BloscLZ,
|
||||
LZ4/LZ4HC, Snappy, Zlib and Zstandard codecs and byte/bit shuffle;
|
||||
BloscLZ is decoded by a port of c-blosc 1.21's decoder, and cannot be
|
||||
written), and `plugin-filters` for all four. None compiles C: Zstandard is
|
||||
ruzstd, bzip2 is libbz2-rs-sys. Write with `DatasetBuilder::with_lzf()`,
|
||||
`with_bitshuffle(..)`, `with_bzip2(..)`, `with_blosc(..)` or
|
||||
`with_plugin_filter(PluginFilter::..)`; `ChunkOptions` gains a `plugin`
|
||||
field (**breaking** for code that builds `ChunkOptions` with a struct
|
||||
literal and no `..Default::default()`). Tested both ways against h5py 3.16
|
||||
+ hdf5plugin 7.1 over 1-3-D shapes with partial edge chunks, 1-8-byte
|
||||
types in both byte orders and incompressible data
|
||||
(`crates/clawhdf5/tests/plugin_filters_interop.rs`). Conformance: 573 of
|
||||
697 files ok (was 569) — h5ex_d_lzf/bshuf/bzip2/blosc.
|
||||
- **Filter registry.** Filters are looked up by ID in
|
||||
`clawhdf5_format::filter_registry` instead of a `match`: the built-in
|
||||
table (per build), then codecs registered at run time with
|
||||
`register_filter(id, codec)` — a decoding closure or a `FilterCodec` that
|
||||
can also encode. Built-in IDs cannot be overridden; a registered decoder's
|
||||
output is held to the chunk-size bound. Unknown IDs still fail with
|
||||
`UnsupportedFilter(id)`, whose message now names known filters and the
|
||||
missing feature ("unsupported filter: 32026 (Blosc2, not implemented by
|
||||
clawhdf5)").
|
||||
- **Not implemented:** Blosc2 (32026) and ZFP (32013) remain a clear error.
|
||||
|
||||
### Upgrade Notes
|
||||
- **HDF5 correctness audit (2026-09-25).** A sweep of 686 public files (the
|
||||
libhdf5 test files, the HDF Group's CVE reproducers, pyfive, netcdf-c,
|
||||
|
||||
Reference in New Issue
Block a user