fix(read): decode on the calling thread when rayon's pool has one thread

Full reads of chunked datasets handed their chunks to rayon. With a
one-thread pool (concurrent_read --decode-threads 1, RAYON_NUM_THREADS=1)
every thread reading through a File queued behind that single worker, so
16 readers decoded on one core: per-thread CPU time showed one thread
doing all the decoding and the readers almost none, and full reads
stopped at about 2x one thread. The cached full-read path and the
uncached reader behind verify_provenance now decode inline when the pool
cannot parallelise (parallel_read::pool_can_parallelise).

The File's chunk cache was the suspect but not the cause: datasets over
its budget were already read without inserting, and skipping its lookups
gained only a few percent at 16 threads.

The regression test keeps a one-thread global pool's worker busy and
requires a full read and verify_provenance to finish anyway; before the
fix both waited for the worker (timed out).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-26 08:40:10 -05:00
co-authored by Claude Opus 5.5
parent 63648c7000
commit a5e41c1a53
6 changed files with 152 additions and 14 deletions
+19 -6
View File
@@ -9,12 +9,25 @@ deleting it.
## Concurrent and contiguous read performance (measured 2026-09-26)
**Status:** open. Measured on tank with `concurrent_read` against h5py
3.16 / HDF5 2.0 (`BENCHMARKS.md`, "Concurrent reads"):
- Full reads of chunked datasets from several threads through one `File`
stop scaling at about 4 threads (880 MB/s on deflate data vs 4424 MB/s
for 16 h5py processes). Hyperslab reads, which skip the chunk cache,
scale to 1244 MB/s, so the `File`'s shared chunk cache is the suspect.
**Status:** first bullet fixed (2026-09-26), second open. Measured on
tank with `concurrent_read` against h5py 3.16 / HDF5 2.0 (`BENCHMARKS.md`,
"Concurrent reads"):
- **Fixed 2026-09-26.** Full reads of chunked datasets from several threads
through one `File` stop scaling at about 4 threads (880 MB/s on deflate
data vs 4424 MB/s for 16 h5py processes). Hyperslab reads, which skip the
chunk cache, scale to 1244 MB/s, so the `File`'s shared chunk cache is the
suspect. *Cause:* not the cache. Those numbers were taken with
`--decode-threads 1`, a one-thread rayon pool, and every full read handed
its chunks to that pool, so all reader threads queued behind its single
worker (per-thread CPU time: one thread did all the decoding, the 16
readers almost none). Hyperslab reads touch one chunk each and never used
the pool. Reads now decode on the calling thread when the pool has one
thread (`tests/single_thread_decode_pool.rs`). Datasets larger than the
cache's budget were already read without inserting into it, and skipping
its lookups entirely gained only a few percent at 16 threads. Remaining
per-read overhead, not yet addressed: each full `read_f32` of a chunked
dataset faults in about three times its size in fresh pages (the output,
the `f32` copy of it, and a new buffer per decoded chunk).
- Contiguous datasets read 4x slower than h5py on one thread (2.5 vs
9.8 GB/s full, 0.12x for 256 x 256 hyperslabs).
Values are correct; this is speed only.