quantumclaw/clawhdf5
3
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

docs: the rayon fix covers a one-thread pool, not the h5py-process gap

Browse Source 
The review measured the default pool unchanged (about 2900 MB/s at 16
threads before and after) and still short of 16 h5py processes; small
pools still make outside readers wait. Say so instead of marking the
scaling issue fixed.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-26 08:48:05 -05:00
co-authored by Claude Opus 5.5
parent a5e41c1a53
commit 37770f594a
3 changed files with 14 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/known-issues.md
+7 -3
View File
@@ -9,10 +9,10 @@ deleting it.
## Concurrent and contiguous read performance (measured 2026-09-26)
**Status:** first bullet fixed (2026-09-26), second open. Measured on
**Status:** open; one cause of the first bullet fixed (2026-09-26). Measured on
tank with `concurrent_read` against h5py 3.16 / HDF5 2.0 (`BENCHMARKS.md`,
"Concurrent reads"):
- **Fixed 2026-09-26.** Full reads of chunked datasets from several threads
- **Partly fixed 2026-09-26.** Full reads of chunked datasets from several threads
through one `File` stop scaling at about 4 threads (880 MB/s on deflate
data vs 4424 MB/s for 16 h5py processes). Hyperslab reads, which skip the
chunk cache, scale to 1244 MB/s, so the `File`'s shared chunk cache is the
@@ -22,7 +22,11 @@ tank with `concurrent_read` against h5py 3.16 / HDF5 2.0 (`BENCHMARKS.md`,
worker (per-thread CPU time: one thread did all the decoding, the 16
readers almost none). Hyperslab reads touch one chunk each and never used
the pool. Reads now decode on the calling thread when the pool has one
thread (`tests/single_thread_decode_pool.rs`). Datasets larger than the
thread (`tests/single_thread_decode_pool.rs`). **Still open:** this
fixes only a one-thread pool. With the default pool, 16 reader threads
ran at about 2900 MB/s before and after the change, still short of 16
h5py processes (4424 MB/s); with a small pool (2-4 threads) readers
outside it still wait on its workers. Datasets larger than the
cache's budget were already read without inserting into it, and skipping
its lookups entirely gained only a few percent at 16 threads. Remaining
per-read overhead, not yet addressed: each full `read_f32` of a chunked