quantumclaw/clawhdf5
3
1
Fork 0
Code Issues Pull Requests 1 Actions 1 Packages Projects Releases 1 Wiki Activity

bench: concurrent-read harness against h5py threads and processes

Browse Source 
concurrent_read reads one shared File from 1-16 threads: every dataset
in full (distinct datasets per thread) and random hyperslabs of one
dataset, over a deflate and a contiguous file it generates (or reuses
while manifest.json matches). It reports decoded MB/s and scaling
efficiency, warm or --cold (posix_fadvise) page cache, sizes the decode
pool with --decode-threads, and writes JSON.

scripts/concurrent_read_h5py.py runs the same workload on the same files
with h5py threads or spawned processes (same splitmix64 data and slab
stream, checked at spot elements), and compare_concurrent_read.py prints
one table and refuses runs with different workloads. A smoke test runs
all three end to end on tiny files (h5py half honours CLAWHDF5_PYTHON /
CLAWHDF5_REQUIRE_INTEROP).

BENCHMARKS.md gets a "Concurrent reads" section with the commands, marked
not yet measured.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-25 23:56:36 -05:00
co-authored by Claude Opus 5.5
parent bb78d70b99
commit 3b24e6753b
9 changed files with 1100 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files