quantumclaw/clawhdf5
3
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

bench: concurrent reads re-measured after the read fixes
CI / test-arm64 (pull_request) Successful in 1m34s
Details
CI / test (pull_request) Successful in 7m30s
Details

Browse Source 
Idle tank at 408f69e, h5py re-run in the same session. Contiguous reads
went from 0.25x to 1.44x h5py (full) and 0.12x to 6.3x (256x256
hyperslabs) on one thread; deflate full reads at 8 threads 887 -> 2943
MB/s (h5py processes 3042). Full chunked reads at 16 threads are still
0.69x-0.76x h5py processes; the issue stays open.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-26 09:28:51 -05:00
co-authored by Claude Opus 5.5
parent 408f69ec1d
commit dda28d6c72
2 changed files with 55 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files
BENCHMARKS.md
+46 -1
View File
@@ -484,7 +484,52 @@ explain the slower windows.
## Concurrent reads
### Results (2026-09-26, tank)
### Results after the read fixes (2026-09-26, tank, `408f69e`)
Same machine, files and commands as the first run below, re-run on an idle
tank (load average 1.60 at the start; the 1-minute figure rose to about 5
during the clawhdf5 runs, mostly their own threads) after two fixes:
contiguous reads back their output with transparent huge pages and copy
hyperslabs run by run, and full chunked reads no longer queue behind a
one-thread rayon pool. h5py was re-run in the same session.
Each read decoding on its calling thread (`--decode-threads 1`, like h5py):
| layout | mode | threads | clawhdf5 MB/s (eff) | h5py threads MB/s (eff) | h5py processes MB/s (eff) |
|---|---|---:|---:|---:|---:|
| deflate | distinct | 1 | 606 (1.00) | 432 (1.00) | 421 (1.00) |
| deflate | distinct | 4 | 1816 (0.75) | 428 (0.25) | 1654 (0.98) |
| deflate | distinct | 8 | 2943 (0.61) | 428 (0.12) | 3042 (0.90) |
| deflate | distinct | 16 | 2142 (0.22) | 375 (0.05) | 3083 (0.46) |
| deflate | same | 1 | 154 (1.00) | 130 (1.00) | 129 (1.00) |
| deflate | same | 4 | 599 (0.98) | 129 (0.25) | 499 (0.97) |
| deflate | same | 16 | 1592 (0.65) | 128 (0.06) | 1399 (0.68) |
| contiguous | distinct | 1 | 13665 (1.00) | 9490 (1.00) | 8781 (1.00) |
| contiguous | distinct | 16 | 12674 (0.06) | 2285 (0.02) | 6942 (0.05) |
| contiguous | same | 1 | 31991 (1.00) | 5087 (1.00) | 5078 (1.00) |
| contiguous | same | 16 | 237151 (0.46) | 4304 (0.05) | 35772 (0.44) |
With the default rayon pool: deflate `distinct` 2117 MB/s at 1 thread (4.9x
h5py), 3163 at 4, 2341 at 16 (0.76x h5py processes); deflate `same` 1439 MB/s
at 16; contiguous as above within a few percent.
Before -> after for clawhdf5 (`--decode-threads 1` unless noted):
contiguous full read at 1 thread 2495 -> 13665 MB/s (0.25x -> 1.44x h5py);
contiguous 256 x 256 hyperslabs at 1 thread 624 -> 31991 MB/s (0.12x ->
6.3x); deflate full reads at 8 threads 887 -> 2943 MB/s; deflate
hyperslabs at 16 threads 1244 -> 1592 MB/s.
Read with care:
- `contiguous same` reads 1024 slabs of one 64 MiB dataset over and over, so
it mostly measures copies out of the CPU's caches (the 7800X3D has 96 MiB
of L3); the per-call overhead is what differs (h5py's is about 50 us).
- At 16 threads every tool dropped in this run (h5py threads on contiguous
data from 8002 to 2285 MB/s, processes from 12846 to 6942), so the
16-thread rows are noisier than the others.
- Still behind: full reads of chunked data at 16 threads (0.69x-0.76x h5py
processes). See `docs/known-issues.md`.
### First run, before the read fixes (2026-09-26, tank, `91644d8`)
Measured on tank (AMD Ryzen 7 7800X3D, 8 cores / 16 threads, 61 GiB, Linux
7.0) at commit `91644d8`, load average 1.84 when the run started (the