Merge branch 'feat/p1-plugin-filters' into feat/p1-proof
# Conflicts: # crates/clawhdf5/tests/h5py_chunked_read_tests.rs # docs/known-issues.md
This commit is contained in:
@@ -2,6 +2,57 @@
|
||||
|
||||
## Unreleased
|
||||
|
||||
### Plugin filters (2026-09-26)
|
||||
- **LZF, bitshuffle, bzip2 and Blosc read and write, in pure Rust.** Files
|
||||
written by h5py with `compression="lzf"`, or with hdf5plugin's
|
||||
`Bitshuffle`, `BZip2` and `Blosc`, failed with `UnsupportedFilter`. New
|
||||
`clawhdf5-format`/`clawhdf5` features: `lzf` (32000, **on by default**, no
|
||||
dependencies), `bitshuffle` (32008: transpose only, LZ4 and Zstandard
|
||||
modes), `bzip2` (307), `blosc` (32001: Blosc 1 frames with BloscLZ,
|
||||
LZ4/LZ4HC, Snappy, Zlib and Zstandard codecs and byte/bit shuffle;
|
||||
BloscLZ is decoded by a port of c-blosc 1.21's decoder, and cannot be
|
||||
written), and `plugin-filters` for all four. None compiles C: Zstandard is
|
||||
ruzstd, bzip2 is libbz2-rs-sys. Write with `DatasetBuilder::with_lzf()`,
|
||||
`with_bitshuffle(..)`, `with_bzip2(..)`, `with_blosc(..)` or
|
||||
`with_plugin_filter(PluginFilter::..)`; `ChunkOptions` gains a `plugin`
|
||||
field (**breaking** for code that builds `ChunkOptions` with a struct
|
||||
literal and no `..Default::default()`). Tested both ways against h5py 3.16
|
||||
+ hdf5plugin 7.1 over 1-3-D shapes with partial edge chunks, 1-8-byte
|
||||
types in both byte orders and incompressible data
|
||||
(`crates/clawhdf5/tests/plugin_filters_interop.rs`). Conformance: 573 of
|
||||
697 files ok (was 569) — h5ex_d_lzf/bshuf/bzip2/blosc.
|
||||
- **Filter registry.** Filters are looked up by ID in
|
||||
`clawhdf5_format::filter_registry` instead of a `match`: the built-in
|
||||
table (per build), then codecs registered at run time with
|
||||
`register_filter(id, codec)` — a decoding closure or a `FilterCodec` that
|
||||
can also encode. Built-in IDs cannot be overridden; a registered decoder's
|
||||
output is held to the chunk-size bound. Unknown IDs still fail with
|
||||
`UnsupportedFilter(id)`, whose message now names known filters and the
|
||||
missing feature ("unsupported filter: 32026 (Blosc2, not implemented by
|
||||
clawhdf5)").
|
||||
- **Not implemented:** Blosc2 (32026) and ZFP (32013) remain a clear error.
|
||||
- **Wrong data: a chunk that decodes short read as zeros** (pre-existing, every
|
||||
filter). HDF5 stores every chunk at the full chunk size, so a filter
|
||||
pipeline that decodes to fewer bytes means a corrupt chunk; every chunk
|
||||
reader (full, cached, selection, parallel, partial) padded it with zeros.
|
||||
It is now an error naming the chunk ("chunk at [16] decoded to 16 bytes,
|
||||
expected 32"), via the new `filters::decompress_chunk_exact`. libhdf5
|
||||
returns the rest of such a chunk uninitialised, or fails when the filter
|
||||
checks. A Blosc frame declaring no data for a non-empty chunk is an error
|
||||
too. Legitimate edge chunks are unaffected (they are stored full-size,
|
||||
filtered or not); conformance is unchanged at 573 of 697, with no file
|
||||
changing class.
|
||||
- **Crash: a hostile Blosc chunk panicked** in builds with overflow checks
|
||||
(debug builds, `cargo test`, `maturin develop`): a frame size below the
|
||||
16-byte header underflowed. It is now an error. Every new decoder (LZF,
|
||||
bitshuffle, bzip2, Blosc/BloscLZ) is fuzzed with random and mutated frames
|
||||
in the unit tests.
|
||||
- **`register_filter(32023, ..)` works with the `pcodec` feature.** 32023 is
|
||||
Granular BitRound's ID; the built-in entry there only reads clawhdf5
|
||||
<= 2.7.0's pcodec chunks (filter name `"pcodec"`), so a registered codec now
|
||||
handles every other chunk with that ID, and writes. It was refused as
|
||||
"built in".
|
||||
|
||||
### Upgrade Notes
|
||||
- **HDF5 correctness audit (2026-09-25).** A sweep of 686 public files (the
|
||||
libhdf5 test files, the HDF Group's CVE reproducers, pyfive, netcdf-c,
|
||||
|
||||
Reference in New Issue
Block a user