bench: concurrent reads re-measured after the read fixes
CI / test-arm64 (pull_request) Successful in 1m34s
CI / test (pull_request) Successful in 7m30s

Idle tank at 408f69e, h5py re-run in the same session. Contiguous reads
went from 0.25x to 1.44x h5py (full) and 0.12x to 6.3x (256x256
hyperslabs) on one thread; deflate full reads at 8 threads 887 -> 2943
MB/s (h5py processes 3042). Full chunked reads at 16 threads are still
0.69x-0.76x h5py processes; the issue stays open.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-26 09:28:51 -05:00
co-authored by Claude Opus 5.5
parent 408f69ec1d
commit dda28d6c72
2 changed files with 55 additions and 7 deletions
+9 -6
View File
@@ -40,11 +40,12 @@ tank with `concurrent_read` against h5py 3.16 / HDF5 2.0 (`BENCHMARKS.md`,
worker (per-thread CPU time: one thread did all the decoding, the 16
readers almost none). Hyperslab reads touch one chunk each and never used
the pool. Reads now decode on the calling thread when the pool has one
thread (`tests/single_thread_decode_pool.rs`). **Still open:** this
fixes only a one-thread pool. With the default pool, 16 reader threads
ran at about 2900 MB/s before and after the change, still short of 16
h5py processes (4424 MB/s); with a small pool (2-4 threads) readers
outside it still wait on its workers. Datasets larger than the
thread (`tests/single_thread_decode_pool.rs`); re-measured at
`408f69e`, 8 threads went from 887 to 2943 MB/s (h5py processes: 3042).
**Still open:** at 16 threads full chunked reads reach 2142-2341 MB/s,
0.69x-0.76x 16 h5py processes (3083 MB/s in the same run), with the
default pool as with a one-thread one; with a small pool (2-4 threads)
readers outside it still wait on its workers. Datasets larger than the
cache's budget were already read without inserting into it, and skipping
its lookups entirely gained only a few percent at 16 threads. Remaining
per-read overhead, not yet addressed: each full `read_f32` of a chunked
@@ -52,7 +53,9 @@ tank with `concurrent_read` against h5py 3.16 / HDF5 2.0 (`BENCHMARKS.md`,
the `f32` copy of it, and a new buffer per decoded chunk).
- Contiguous datasets read 4x slower than h5py on one thread (2.5 vs
9.8 GB/s full, 0.12x for 256 x 256 hyperslabs).
**Fixed 2026-09-26** (not yet re-measured for `BENCHMARKS.md`): full
**Fixed 2026-09-26** (re-measured on tank at `408f69e`: 13665 MB/s
full and 31991 MB/s for 256 x 256 hyperslabs on one thread, 1.44x and
6.3x h5py; `BENCHMARKS.md`): full
reads were dominated by 4 KiB page faults on the fresh output buffer,
which is now backed by transparent huge pages as numpy's is; hyperslab
reads copied the selection three times, element by element, and now copy