docs: chunked full reads decode in place; small pools no longer block
CHANGELOG entry for the chunked read changes, and the known-issues entry on concurrent chunked reads updated: both causes it names (per-read page faults, readers waiting on a small pool) are fixed; the 16-thread comparison with h5py stays open until re-measured on an idle machine. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
@@ -2,6 +2,52 @@
|
||||
|
||||
## Unreleased
|
||||
|
||||
### Chunked full reads (2026-09-26)
|
||||
- **Chunks are decoded straight into the output, into reused buffers.** A
|
||||
full read of a chunked dataset faulted in about three times its size in
|
||||
fresh pages: every chunk was decoded into a new buffer per filter stage
|
||||
(the cached reader behind `read_*` decoded 128 chunks at a time before
|
||||
placing any; the uncached one behind `MmapFile`, `LazyFile` and
|
||||
`verify_provenance` decoded the whole dataset first), then assembled into
|
||||
a byte buffer, which the typed readers copied once more. Now each chunk
|
||||
is decoded into buffers the thread keeps between chunks and reads
|
||||
(`clawhdf5_format::filters::DecodeScratch`,
|
||||
`decompress_chunk_exact_with`: deflate inflates into a kept buffer with a
|
||||
reset inflater, shuffle into the other one, Fletcher32 is checked in
|
||||
place; other filters go through the registry as before) and copied
|
||||
directly to its place in the output. Chunks still go into the file's
|
||||
chunk cache when the whole dataset fits. Selection reads decode the
|
||||
chunks they touch the same way.
|
||||
- **Typed full reads of chunked data skip the byte buffer.** `read_f32`,
|
||||
`read_f64`, `read_i32`, `read_i64` and `read_u64` (on `File`, `MmapFile`
|
||||
and `LazyFile`) of a chunked dataset stored as that type in native byte
|
||||
order decode every chunk into the returned `Vec` (huge-page backed when
|
||||
large, like the byte readers' output); other types and byte orders
|
||||
convert as before. New public
|
||||
`clawhdf5_format::data_read::read_chunked_native`.
|
||||
- **Reading threads no longer wait for a busy rayon pool.** A full read
|
||||
handed its chunks to rayon and the calling thread slept until the pool had
|
||||
decoded them, so with a small pool (2-4 threads) readers outside it queued
|
||||
behind its workers. The calling thread now decodes too, and pool workers
|
||||
join in only when free; a helper the pool starts after the read has
|
||||
finished returns at once. A single read still spreads over the default
|
||||
pool. This replaces the one-thread-pool special case below for full
|
||||
reads. Chunks are placed from several threads only when the chunk index
|
||||
puts them on the chunk grid at distinct places; a corrupt index is read
|
||||
one chunk at a time, and the error reported is still the first failing
|
||||
chunk's. New test `crates/clawhdf5/tests/busy_decode_pool.rs`.
|
||||
- **Fixed:** in a filtered dataset, a chunk stored with every filter skipped
|
||||
(filter mask) and shorter than a chunk read with zeros in place of its
|
||||
missing part through `File`'s `read_*`; it is now an error naming the
|
||||
chunk, as `MmapFile`/`LazyFile` already made it.
|
||||
- New h5py comparison `crates/clawhdf5/tests/chunked_read_paths_interop.rs`:
|
||||
every chunked read path (cached and uncached full reads, `MmapFile`,
|
||||
`LazyFile`, small, strided and point selections, with and without the
|
||||
`parallel` feature) for 1-8-byte integers and 2-8-byte floats in both
|
||||
byte orders, through deflate, shuffle, Fletcher32, LZF, SZIP and Blosc,
|
||||
with partial edge chunks, sparse datasets and fill values, and datasets
|
||||
larger than the chunk cache.
|
||||
|
||||
### Concurrent reads (2026-09-26)
|
||||
- **Full reads of chunked datasets scale with threads again when rayon's
|
||||
pool has one thread.** Each full read handed its chunks to rayon to
|
||||
|
||||
Reference in New Issue
Block a user