Merge branch 'feat/p2b-blosc2' into feat/p2b-scale

# Conflicts:
#	CHANGELOG.md
This commit is contained in:
osobh
2026-09-26 11:57:11 -05:00
60 changed files with 2276 additions and 33 deletions
+36
View File
@@ -89,6 +89,41 @@
`names_whose_hashes_collide_are_found_by_name` in
`crates/clawhdf5/tests/writer_groups_interop.rs`.
### Blosc2 (2026-09-26)
- **Blosc2 (filter 32026) reads, in pure Rust.** Files written with
hdf5plugin's `Blosc2` failed with `UnsupportedFilter`. New feature
`blosc2` (`clawhdf5-format` and `clawhdf5`, included in `plugin-filters`)
decodes the Blosc2 contiguous frame hdf5-blosc2 stores per chunk, the
B2ND arrays it uses for chunks of 2 or more dimensions (blocks gathered
back into C order), Blosc2 chunks with their special values (zeros, NaN,
uninitialised, one repeated value), and the shuffle, bit-shuffle, delta
and truncate-precision filters, over the BloscLZ, LZ4/LZ4HC, Zlib and
Zstandard codecs shared with Blosc 1. Read only: there is no Blosc2
encoder. Dictionaries, lazy chunks, variable-length blocks, user-defined
codecs and registered Blosc2 filters (e.g. bytedelta) are errors;
uninitialised chunks read as zeros. Tested against h5py 3.16 +
hdf5plugin 7.1 (every codec, filter and level 0-9; 1- to 5-D chunks with
partial edge chunks; every integer width and f4/f8; datasets of zeros,
one value and NaN; Fletcher32 before Blosc2) and against frames from
python-blosc2 4.13.1 for what hdf5plugin never writes
(`crates/clawhdf5-format/tests/fixtures/blosc2/`); the decoder is
fuzzed. Conformance: 576 of 697 files ok (was 575) — h5ex_d_blosc2.
- **A crafted Blosc2 chunk cannot allocate more than a few times its HDF5
chunk size.** Found in review before release: the sizes a frame declares
sized the decoder's buffers. A 173-byte frame whose offsets chunk claimed
2 GiB was decoded in full for a 1 MiB chunk; an empty chunk allocated its
declared block size twice (about 1 GiB); a B2ND chunk was decoded whole
with its padding (up to 16x the chunk); and a Zstandard stream's declared
window (up to 100 MiB) was reserved as the decoder was reused, which also
affected Blosc 1 and bitshuffle with Zstandard. Now the offsets chunk is
capped at the chunk size, block sizes are clamped to the chunk, B2ND
blocks are placed as they are decoded (padding is never held), and the
Zstandard window is capped at twice the stream's output (at least
128 KiB). A B2ND chunk may no longer be larger than its array, which
hdf5-blosc2 never writes. `tests/blosc2_alloc_bounds.rs` measures peak
allocation for these frames and for 45,000 fuzzed ones: at most 6x the
chunk size, twice the input and 2 MiB of Zstandard state.
### Concurrent reads (2026-09-26)
- **Full reads of chunked datasets scale with threads again when rayon's
pool has one thread.** Each full read handed its chunks to rayon to
@@ -459,6 +494,7 @@
missing feature ("unsupported filter: 32026 (Blosc2, not implemented by
clawhdf5)").
- **Not implemented:** Blosc2 (32026) and ZFP (32013) remain a clear error.
(Blosc2 reads since the `blosc2` feature, see above.)
- **Wrong data: a chunk that decodes short read as zeros** (pre-existing, every
filter). HDF5 stores every chunk at the full chunk size, so a filter
pipeline that decodes to fewer bytes means a corrupt chunk; every chunk