fix(read): decode on the calling thread when rayon's pool has one thread
Full reads of chunked datasets handed their chunks to rayon. With a one-thread pool (concurrent_read --decode-threads 1, RAYON_NUM_THREADS=1) every thread reading through a File queued behind that single worker, so 16 readers decoded on one core: per-thread CPU time showed one thread doing all the decoding and the readers almost none, and full reads stopped at about 2x one thread. The cached full-read path and the uncached reader behind verify_provenance now decode inline when the pool cannot parallelise (parallel_read::pool_can_parallelise). The File's chunk cache was the suspect but not the cause: datasets over its budget were already read without inserting, and skipping its lookups gained only a few percent at 16 threads. The regression test keeps a one-thread global pool's worker busy and requires a full read and verify_provenance to finish anyway; before the fix both waited for the worker (timed out). Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
@@ -2,6 +2,19 @@
|
||||
|
||||
## Unreleased
|
||||
|
||||
### Concurrent reads (2026-09-26)
|
||||
- **Full reads of chunked datasets scale with threads again when rayon's
|
||||
pool has one thread.** Each full read handed its chunks to rayon to
|
||||
decode; with a one-thread pool (`RAYON_NUM_THREADS=1`, or
|
||||
`concurrent_read --decode-threads 1`) every thread reading through a
|
||||
`File` queued behind that single worker, so N readers decoded on one core
|
||||
and throughput stopped at about 2x one thread. Such reads, and
|
||||
`verify_provenance`'s uncached reader, now decode on the calling thread
|
||||
(`clawhdf5_format::parallel_read::pool_can_parallelise`). The `File`'s
|
||||
chunk cache, the suspect in `docs/known-issues.md`, was not the cause:
|
||||
reads of datasets larger than its budget already skipped inserting, and
|
||||
its lookups cost a few percent at 16 threads.
|
||||
|
||||
### Plugin filters (2026-09-26)
|
||||
- **LZF, bitshuffle, bzip2 and Blosc read and write, in pure Rust.** Files
|
||||
written by h5py with `compression="lzf"`, or with hdf5plugin's
|
||||
|
||||
Reference in New Issue
Block a user