Chunked reads beat an h5py process pool; unlimited writer B-trees; Blosc2; 599/697 conformance #16
@@ -21,6 +21,21 @@
|
||||
python-blosc2 4.13.1 for what hdf5plugin never writes
|
||||
(`crates/clawhdf5-format/tests/fixtures/blosc2/`); the decoder is
|
||||
fuzzed. Conformance: 576 of 697 files ok (was 575) — h5ex_d_blosc2.
|
||||
- **A crafted Blosc2 chunk cannot allocate more than a few times its HDF5
|
||||
chunk size.** Found in review before release: the sizes a frame declares
|
||||
sized the decoder's buffers. A 173-byte frame whose offsets chunk claimed
|
||||
2 GiB was decoded in full for a 1 MiB chunk; an empty chunk allocated its
|
||||
declared block size twice (about 1 GiB); a B2ND chunk was decoded whole
|
||||
with its padding (up to 16x the chunk); and a Zstandard stream's declared
|
||||
window (up to 100 MiB) was reserved as the decoder was reused, which also
|
||||
affected Blosc 1 and bitshuffle with Zstandard. Now the offsets chunk is
|
||||
capped at the chunk size, block sizes are clamped to the chunk, B2ND
|
||||
blocks are placed as they are decoded (padding is never held), and the
|
||||
Zstandard window is capped at twice the stream's output (at least
|
||||
128 KiB). A B2ND chunk may no longer be larger than its array, which
|
||||
hdf5-blosc2 never writes. `tests/blosc2_alloc_bounds.rs` measures peak
|
||||
allocation for these frames and for 45,000 fuzzed ones: at most 6x the
|
||||
chunk size, twice the input and 2 MiB of Zstandard state.
|
||||
|
||||
### Concurrent reads (2026-09-26)
|
||||
- **Full reads of chunked datasets scale with threads again when rayon's
|
||||
|
||||
Reference in New Issue
Block a user