docs: chunked full reads decode in place; small pools no longer block

CHANGELOG entry for the chunked read changes, and the known-issues entry
on concurrent chunked reads updated: both causes it names (per-read page
faults, readers waiting on a small pool) are fixed; the 16-thread
comparison with h5py stays open until re-measured on an idle machine.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-26 10:36:44 -05:00
co-authored by Claude Opus 5.5
parent 9e608b975c
commit 94b6df986c
2 changed files with 62 additions and 5 deletions
+46
View File
@@ -2,6 +2,52 @@
## Unreleased
### Chunked full reads (2026-09-26)
- **Chunks are decoded straight into the output, into reused buffers.** A
full read of a chunked dataset faulted in about three times its size in
fresh pages: every chunk was decoded into a new buffer per filter stage
(the cached reader behind `read_*` decoded 128 chunks at a time before
placing any; the uncached one behind `MmapFile`, `LazyFile` and
`verify_provenance` decoded the whole dataset first), then assembled into
a byte buffer, which the typed readers copied once more. Now each chunk
is decoded into buffers the thread keeps between chunks and reads
(`clawhdf5_format::filters::DecodeScratch`,
`decompress_chunk_exact_with`: deflate inflates into a kept buffer with a
reset inflater, shuffle into the other one, Fletcher32 is checked in
place; other filters go through the registry as before) and copied
directly to its place in the output. Chunks still go into the file's
chunk cache when the whole dataset fits. Selection reads decode the
chunks they touch the same way.
- **Typed full reads of chunked data skip the byte buffer.** `read_f32`,
`read_f64`, `read_i32`, `read_i64` and `read_u64` (on `File`, `MmapFile`
and `LazyFile`) of a chunked dataset stored as that type in native byte
order decode every chunk into the returned `Vec` (huge-page backed when
large, like the byte readers' output); other types and byte orders
convert as before. New public
`clawhdf5_format::data_read::read_chunked_native`.
- **Reading threads no longer wait for a busy rayon pool.** A full read
handed its chunks to rayon and the calling thread slept until the pool had
decoded them, so with a small pool (2-4 threads) readers outside it queued
behind its workers. The calling thread now decodes too, and pool workers
join in only when free; a helper the pool starts after the read has
finished returns at once. A single read still spreads over the default
pool. This replaces the one-thread-pool special case below for full
reads. Chunks are placed from several threads only when the chunk index
puts them on the chunk grid at distinct places; a corrupt index is read
one chunk at a time, and the error reported is still the first failing
chunk's. New test `crates/clawhdf5/tests/busy_decode_pool.rs`.
- **Fixed:** in a filtered dataset, a chunk stored with every filter skipped
(filter mask) and shorter than a chunk read with zeros in place of its
missing part through `File`'s `read_*`; it is now an error naming the
chunk, as `MmapFile`/`LazyFile` already made it.
- New h5py comparison `crates/clawhdf5/tests/chunked_read_paths_interop.rs`:
every chunked read path (cached and uncached full reads, `MmapFile`,
`LazyFile`, small, strided and point selections, with and without the
`parallel` feature) for 1-8-byte integers and 2-8-byte floats in both
byte orders, through deflate, shuffle, Fletcher32, LZF, SZIP and Blosc,
with partial edge chunks, sparse datasets and fill values, and datasets
larger than the chunk cache.
### Concurrent reads (2026-09-26)
- **Full reads of chunked datasets scale with threads again when rayon's
pool has one thread.** Each full read handed its chunks to rayon to