Merge branch 'feat/p1-plugin-filters' into feat/p1-proof

# Conflicts:
#	crates/clawhdf5/tests/h5py_chunked_read_tests.rs
#	docs/known-issues.md
This commit is contained in:
osobh
2026-09-26 01:37:58 -05:00
27 changed files with 3492 additions and 114 deletions
+51
View File
@@ -2,6 +2,57 @@
## Unreleased
### Plugin filters (2026-09-26)
- **LZF, bitshuffle, bzip2 and Blosc read and write, in pure Rust.** Files
written by h5py with `compression="lzf"`, or with hdf5plugin's
`Bitshuffle`, `BZip2` and `Blosc`, failed with `UnsupportedFilter`. New
`clawhdf5-format`/`clawhdf5` features: `lzf` (32000, **on by default**, no
dependencies), `bitshuffle` (32008: transpose only, LZ4 and Zstandard
modes), `bzip2` (307), `blosc` (32001: Blosc 1 frames with BloscLZ,
LZ4/LZ4HC, Snappy, Zlib and Zstandard codecs and byte/bit shuffle;
BloscLZ is decoded by a port of c-blosc 1.21's decoder, and cannot be
written), and `plugin-filters` for all four. None compiles C: Zstandard is
ruzstd, bzip2 is libbz2-rs-sys. Write with `DatasetBuilder::with_lzf()`,
`with_bitshuffle(..)`, `with_bzip2(..)`, `with_blosc(..)` or
`with_plugin_filter(PluginFilter::..)`; `ChunkOptions` gains a `plugin`
field (**breaking** for code that builds `ChunkOptions` with a struct
literal and no `..Default::default()`). Tested both ways against h5py 3.16
+ hdf5plugin 7.1 over 1-3-D shapes with partial edge chunks, 1-8-byte
types in both byte orders and incompressible data
(`crates/clawhdf5/tests/plugin_filters_interop.rs`). Conformance: 573 of
697 files ok (was 569) — h5ex_d_lzf/bshuf/bzip2/blosc.
- **Filter registry.** Filters are looked up by ID in
`clawhdf5_format::filter_registry` instead of a `match`: the built-in
table (per build), then codecs registered at run time with
`register_filter(id, codec)` — a decoding closure or a `FilterCodec` that
can also encode. Built-in IDs cannot be overridden; a registered decoder's
output is held to the chunk-size bound. Unknown IDs still fail with
`UnsupportedFilter(id)`, whose message now names known filters and the
missing feature ("unsupported filter: 32026 (Blosc2, not implemented by
clawhdf5)").
- **Not implemented:** Blosc2 (32026) and ZFP (32013) remain a clear error.
- **Wrong data: a chunk that decodes short read as zeros** (pre-existing, every
filter). HDF5 stores every chunk at the full chunk size, so a filter
pipeline that decodes to fewer bytes means a corrupt chunk; every chunk
reader (full, cached, selection, parallel, partial) padded it with zeros.
It is now an error naming the chunk ("chunk at [16] decoded to 16 bytes,
expected 32"), via the new `filters::decompress_chunk_exact`. libhdf5
returns the rest of such a chunk uninitialised, or fails when the filter
checks. A Blosc frame declaring no data for a non-empty chunk is an error
too. Legitimate edge chunks are unaffected (they are stored full-size,
filtered or not); conformance is unchanged at 573 of 697, with no file
changing class.
- **Crash: a hostile Blosc chunk panicked** in builds with overflow checks
(debug builds, `cargo test`, `maturin develop`): a frame size below the
16-byte header underflowed. It is now an error. Every new decoder (LZF,
bitshuffle, bzip2, Blosc/BloscLZ) is fuzzed with random and mutated frames
in the unit tests.
- **`register_filter(32023, ..)` works with the `pcodec` feature.** 32023 is
Granular BitRound's ID; the built-in entry there only reads clawhdf5
<= 2.7.0's pcodec chunks (filter name `"pcodec"`), so a registered codec now
handles every other chunk with that ID, and writes. It was refused as
"built in".
### Upgrade Notes
- **HDF5 correctness audit (2026-09-25).** A sweep of 686 public files (the
libhdf5 test files, the HDF Group's CVE reproducers, pyfive, netcdf-c,