quantumclaw/clawhdf5
3
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

bench: chunked full reads now beat an h5py process pool
CI / test-arm64 (pull_request) Successful in 1m25s
Details
CI / test (pull_request) Successful in 13m0s
Details

Browse Source 
Idle-start run on tank at c5334b1 (noisier than the last: compare ratios
within the run). Full reads of deflate data at 16 threads: 4944 MB/s vs
3135 for 16 h5py processes (1.58x; 0.69x-0.76x before). One thread with
the default pool: 6143 MB/s, 15x one h5py call. The concurrent-read
known issue is closed.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-26 12:20:47 -05:00
co-authored by Claude Opus 5.5
parent c5334b1c97
commit f2ff2c424f
2 changed files with 44 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/known-issues.md
+5 -3
View File
@@ -26,9 +26,11 @@ performance" below.
## Concurrent and contiguous read performance (measured 2026-09-26)
**Status:** open for chunked full reads at 16 threads until re-measured
(the causes identified below are fixed as of 2026-09-26); the contiguous
item is fixed (2026-09-26). Measured on
**Status:** fixed (2026-09-26). Re-measured on tank at `c5334b1`: full
reads of chunked deflate data at 16 threads run at 4944 MB/s against 3135
for 16 h5py processes (1.58x; 0.69x-0.76x before), and contiguous reads
are 1.29x h5py on one thread (`BENCHMARKS.md`, "Results after in-place
chunk decoding"). The history below is kept for reference. Measured on
tank with `concurrent_read` against h5py 3.16 / HDF5 2.0 (`BENCHMARKS.md`,
"Concurrent reads"):
- **Partly fixed 2026-09-26.** Full reads of chunked datasets from several threads