bench: concurrent reads re-measured after the read fixes
Idle tank at 408f69e, h5py re-run in the same session. Contiguous reads
went from 0.25x to 1.44x h5py (full) and 0.12x to 6.3x (256x256
hyperslabs) on one thread; deflate full reads at 8 threads 887 -> 2943
MB/s (h5py processes 3042). Full chunked reads at 16 threads are still
0.69x-0.76x h5py processes; the issue stays open.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
@@ -40,11 +40,12 @@ tank with `concurrent_read` against h5py 3.16 / HDF5 2.0 (`BENCHMARKS.md`,
|
||||
worker (per-thread CPU time: one thread did all the decoding, the 16
|
||||
readers almost none). Hyperslab reads touch one chunk each and never used
|
||||
the pool. Reads now decode on the calling thread when the pool has one
|
||||
thread (`tests/single_thread_decode_pool.rs`). **Still open:** this
|
||||
fixes only a one-thread pool. With the default pool, 16 reader threads
|
||||
ran at about 2900 MB/s before and after the change, still short of 16
|
||||
h5py processes (4424 MB/s); with a small pool (2-4 threads) readers
|
||||
outside it still wait on its workers. Datasets larger than the
|
||||
thread (`tests/single_thread_decode_pool.rs`); re-measured at
|
||||
`408f69e`, 8 threads went from 887 to 2943 MB/s (h5py processes: 3042).
|
||||
**Still open:** at 16 threads full chunked reads reach 2142-2341 MB/s,
|
||||
0.69x-0.76x 16 h5py processes (3083 MB/s in the same run), with the
|
||||
default pool as with a one-thread one; with a small pool (2-4 threads)
|
||||
readers outside it still wait on its workers. Datasets larger than the
|
||||
cache's budget were already read without inserting into it, and skipping
|
||||
its lookups entirely gained only a few percent at 16 threads. Remaining
|
||||
per-read overhead, not yet addressed: each full `read_f32` of a chunked
|
||||
@@ -52,7 +53,9 @@ tank with `concurrent_read` against h5py 3.16 / HDF5 2.0 (`BENCHMARKS.md`,
|
||||
the `f32` copy of it, and a new buffer per decoded chunk).
|
||||
- Contiguous datasets read 4x slower than h5py on one thread (2.5 vs
|
||||
9.8 GB/s full, 0.12x for 256 x 256 hyperslabs).
|
||||
**Fixed 2026-09-26** (not yet re-measured for `BENCHMARKS.md`): full
|
||||
**Fixed 2026-09-26** (re-measured on tank at `408f69e`: 13665 MB/s
|
||||
full and 31991 MB/s for 256 x 256 hyperslabs on one thread, 1.44x and
|
||||
6.3x h5py; `BENCHMARKS.md`): full
|
||||
reads were dominated by 4 KiB page faults on the fresh output buffer,
|
||||
which is now backed by transparent huge pages as numpy's is; hyperslab
|
||||
reads copied the selection three times, element by element, and now copy
|
||||
|
||||
Reference in New Issue
Block a user