quantumclaw/clawhdf5
3
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

bench: chunked full reads now beat an h5py process pool
CI / test-arm64 (pull_request) Successful in 1m25s
Details
CI / test (pull_request) Successful in 13m0s
Details

Browse Source 
Idle-start run on tank at c5334b1 (noisier than the last: compare ratios
within the run). Full reads of deflate data at 16 threads: 4944 MB/s vs
3135 for 16 h5py processes (1.58x; 0.69x-0.76x before). One thread with
the default pool: 6143 MB/s, 15x one h5py call. The concurrent-read
known issue is closed.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-26 12:20:47 -05:00
co-authored by Claude Opus 5.5
parent c5334b1c97
commit f2ff2c424f
2 changed files with 44 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
BENCHMARKS.md
+39
View File
@@ -484,6 +484,45 @@ explain the slower windows.
## Concurrent reads
### Results after in-place chunk decoding (2026-09-26, tank, `c5334b1`)
Same machine, files and commands, re-run after chunked reads started
decoding into reusable per-thread buffers straight into the (typed) output,
with the calling thread decoding alongside the pool. Load average 1.78 at
the start; it rose to 6-9 during the runs (the clawhdf5 runs' own threads,
and it stayed around 5-6 through the h5py runs, so something else was
active). **This run was noisier than the previous one: h5py's own contiguous
figures are about 40% lower than in the run below, and ours dropped
similarly, so compare ratios within a run rather than MB/s across runs.**
h5py was re-run in the same session.
Each read decoding on its calling thread (`--decode-threads 1`, like h5py):
| layout | mode | threads | clawhdf5 MB/s (eff) | h5py threads MB/s (eff) | h5py processes MB/s (eff) | vs h5py processes |
|---|---|---:|---:|---:|---:|---:|
| deflate | distinct | 1 | 670 (1.00) | 410 (1.00) | 397 (1.00) | 1.69x |
| deflate | distinct | 4 | 2434 (0.91) | 406 (0.25) | 1470 (0.93) | 1.66x |
| deflate | distinct | 8 | 3749 (0.70) | 406 (0.12) | 2398 (0.76) | 1.56x |
| deflate | distinct | 16 | 4944 (0.46) | 390 (0.06) | 3135 (0.49) | 1.58x |
| deflate | same | 1 | 211 (1.00) | 125 (1.00) | 124 (1.00) | 1.70x |
| deflate | same | 16 | 1835 (0.54) | 122 (0.06) | 961 (0.48) | 1.91x |
| contiguous | distinct | 1 | 6718 (1.00) | 5545 (1.00) | 5200 (1.00) | 1.29x |
| contiguous | distinct | 16 | 11035 (0.10) | 4950 (0.06) | 10558 (0.13) | 1.05x |
| contiguous | same | 1 | 14483 (1.00) | 2593 (1.00) | 2737 (1.00) | 5.29x |
| contiguous | same | 16 | 132175 (0.57) | 2224 (0.05) | 14809 (0.34) | 8.93x |
With the default rayon pool, deflate `distinct` reads 6143 MB/s from a single
thread (15x h5py's 410 on one call) and 4556 MB/s at 16 threads (1.45x h5py
processes); the other rows are within the noise of the table above.
What changed: full reads of chunked datasets were 0.69x-0.76x of h5py
processes at 16 threads in the run below, and are 1.58x here; with one
thread they were 1.44x and are 1.69x. Minor page faults for the 16-thread
run fell from about 4.6M to 0.2M (`/usr/bin/time -v`, provisional, loaded
machine). clawhdf5 now reads faster than 16 h5py processes in every row of
this benchmark except contiguous full reads at 16 threads, where both
saturate memory bandwidth (1.05x).
### Results after the read fixes (2026-09-26, tank, `408f69e`)
Same machine, files and commands as the first run below, re-run on an idle