fix(read): decode on the calling thread when rayon's pool has one thread
Full reads of chunked datasets handed their chunks to rayon. With a one-thread pool (concurrent_read --decode-threads 1, RAYON_NUM_THREADS=1) every thread reading through a File queued behind that single worker, so 16 readers decoded on one core: per-thread CPU time showed one thread doing all the decoding and the readers almost none, and full reads stopped at about 2x one thread. The cached full-read path and the uncached reader behind verify_provenance now decode inline when the pool cannot parallelise (parallel_read::pool_can_parallelise). The File's chunk cache was the suspect but not the cause: datasets over its budget were already read without inserting, and skipping its lookups gained only a few percent at 16 threads. The regression test keeps a one-thread global pool's worker busy and requires a full read and verify_provenance to finish anyway; before the fix both waited for the worker (timed out). Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
+19
-6
@@ -9,12 +9,25 @@ deleting it.
|
||||
|
||||
## Concurrent and contiguous read performance (measured 2026-09-26)
|
||||
|
||||
**Status:** open. Measured on tank with `concurrent_read` against h5py
|
||||
3.16 / HDF5 2.0 (`BENCHMARKS.md`, "Concurrent reads"):
|
||||
- Full reads of chunked datasets from several threads through one `File`
|
||||
stop scaling at about 4 threads (880 MB/s on deflate data vs 4424 MB/s
|
||||
for 16 h5py processes). Hyperslab reads, which skip the chunk cache,
|
||||
scale to 1244 MB/s, so the `File`'s shared chunk cache is the suspect.
|
||||
**Status:** first bullet fixed (2026-09-26), second open. Measured on
|
||||
tank with `concurrent_read` against h5py 3.16 / HDF5 2.0 (`BENCHMARKS.md`,
|
||||
"Concurrent reads"):
|
||||
- **Fixed 2026-09-26.** Full reads of chunked datasets from several threads
|
||||
through one `File` stop scaling at about 4 threads (880 MB/s on deflate
|
||||
data vs 4424 MB/s for 16 h5py processes). Hyperslab reads, which skip the
|
||||
chunk cache, scale to 1244 MB/s, so the `File`'s shared chunk cache is the
|
||||
suspect. *Cause:* not the cache. Those numbers were taken with
|
||||
`--decode-threads 1`, a one-thread rayon pool, and every full read handed
|
||||
its chunks to that pool, so all reader threads queued behind its single
|
||||
worker (per-thread CPU time: one thread did all the decoding, the 16
|
||||
readers almost none). Hyperslab reads touch one chunk each and never used
|
||||
the pool. Reads now decode on the calling thread when the pool has one
|
||||
thread (`tests/single_thread_decode_pool.rs`). Datasets larger than the
|
||||
cache's budget were already read without inserting into it, and skipping
|
||||
its lookups entirely gained only a few percent at 16 threads. Remaining
|
||||
per-read overhead, not yet addressed: each full `read_f32` of a chunked
|
||||
dataset faults in about three times its size in fresh pages (the output,
|
||||
the `f32` copy of it, and a new buffer per decoded chunk).
|
||||
- Contiguous datasets read 4x slower than h5py on one thread (2.5 vs
|
||||
9.8 GB/s full, 0.12x for 256 x 256 hyperslabs).
|
||||
Values are correct; this is speed only.
|
||||
|
||||
Reference in New Issue
Block a user