Merge branch 'feat/p2b-blosc2' into feat/p2b-scale
# Conflicts: # CHANGELOG.md
This commit is contained in:
@@ -89,6 +89,41 @@
|
||||
`names_whose_hashes_collide_are_found_by_name` in
|
||||
`crates/clawhdf5/tests/writer_groups_interop.rs`.
|
||||
|
||||
### Blosc2 (2026-09-26)
|
||||
- **Blosc2 (filter 32026) reads, in pure Rust.** Files written with
|
||||
hdf5plugin's `Blosc2` failed with `UnsupportedFilter`. New feature
|
||||
`blosc2` (`clawhdf5-format` and `clawhdf5`, included in `plugin-filters`)
|
||||
decodes the Blosc2 contiguous frame hdf5-blosc2 stores per chunk, the
|
||||
B2ND arrays it uses for chunks of 2 or more dimensions (blocks gathered
|
||||
back into C order), Blosc2 chunks with their special values (zeros, NaN,
|
||||
uninitialised, one repeated value), and the shuffle, bit-shuffle, delta
|
||||
and truncate-precision filters, over the BloscLZ, LZ4/LZ4HC, Zlib and
|
||||
Zstandard codecs shared with Blosc 1. Read only: there is no Blosc2
|
||||
encoder. Dictionaries, lazy chunks, variable-length blocks, user-defined
|
||||
codecs and registered Blosc2 filters (e.g. bytedelta) are errors;
|
||||
uninitialised chunks read as zeros. Tested against h5py 3.16 +
|
||||
hdf5plugin 7.1 (every codec, filter and level 0-9; 1- to 5-D chunks with
|
||||
partial edge chunks; every integer width and f4/f8; datasets of zeros,
|
||||
one value and NaN; Fletcher32 before Blosc2) and against frames from
|
||||
python-blosc2 4.13.1 for what hdf5plugin never writes
|
||||
(`crates/clawhdf5-format/tests/fixtures/blosc2/`); the decoder is
|
||||
fuzzed. Conformance: 576 of 697 files ok (was 575) — h5ex_d_blosc2.
|
||||
- **A crafted Blosc2 chunk cannot allocate more than a few times its HDF5
|
||||
chunk size.** Found in review before release: the sizes a frame declares
|
||||
sized the decoder's buffers. A 173-byte frame whose offsets chunk claimed
|
||||
2 GiB was decoded in full for a 1 MiB chunk; an empty chunk allocated its
|
||||
declared block size twice (about 1 GiB); a B2ND chunk was decoded whole
|
||||
with its padding (up to 16x the chunk); and a Zstandard stream's declared
|
||||
window (up to 100 MiB) was reserved as the decoder was reused, which also
|
||||
affected Blosc 1 and bitshuffle with Zstandard. Now the offsets chunk is
|
||||
capped at the chunk size, block sizes are clamped to the chunk, B2ND
|
||||
blocks are placed as they are decoded (padding is never held), and the
|
||||
Zstandard window is capped at twice the stream's output (at least
|
||||
128 KiB). A B2ND chunk may no longer be larger than its array, which
|
||||
hdf5-blosc2 never writes. `tests/blosc2_alloc_bounds.rs` measures peak
|
||||
allocation for these frames and for 45,000 fuzzed ones: at most 6x the
|
||||
chunk size, twice the input and 2 MiB of Zstandard state.
|
||||
|
||||
### Concurrent reads (2026-09-26)
|
||||
- **Full reads of chunked datasets scale with threads again when rayon's
|
||||
pool has one thread.** Each full read handed its chunks to rayon to
|
||||
@@ -459,6 +494,7 @@
|
||||
missing feature ("unsupported filter: 32026 (Blosc2, not implemented by
|
||||
clawhdf5)").
|
||||
- **Not implemented:** Blosc2 (32026) and ZFP (32013) remain a clear error.
|
||||
(Blosc2 reads since the `blosc2` feature, see above.)
|
||||
- **Wrong data: a chunk that decodes short read as zeros** (pre-existing, every
|
||||
filter). HDF5 stores every chunk at the full chunk size, so a filter
|
||||
pipeline that decodes to fewer bytes means a corrupt chunk; every chunk
|
||||
|
||||
Reference in New Issue
Block a user