bench: concurrent-read harness against h5py threads and processes

concurrent_read reads one shared File from 1-16 threads: every dataset
in full (distinct datasets per thread) and random hyperslabs of one
dataset, over a deflate and a contiguous file it generates (or reuses
while manifest.json matches). It reports decoded MB/s and scaling
efficiency, warm or --cold (posix_fadvise) page cache, sizes the decode
pool with --decode-threads, and writes JSON.

scripts/concurrent_read_h5py.py runs the same workload on the same files
with h5py threads or spawned processes (same splitmix64 data and slab
stream, checked at spot elements), and compare_concurrent_read.py prints
one table and refuses runs with different workloads. A smoke test runs
all three end to end on tiny files (h5py half honours CLAWHDF5_PYTHON /
CLAWHDF5_REQUIRE_INTEROP).

BENCHMARKS.md gets a "Concurrent reads" section with the commands, marked
not yet measured.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-25 23:56:36 -05:00
co-authored by Claude Opus 5.5
parent bb78d70b99
commit 3b24e6753b
9 changed files with 1100 additions and 0 deletions
+7
View File
@@ -195,6 +195,13 @@
README claimed but nothing measured.
- `footprint_bench` reports whether it built `float16` or `f32` stores and
takes `--f32`; it had kept printing "f32" after the default changed.
- New `concurrent_read` harness, with an h5py counterpart
(`crates/clawhdf5-bench/scripts/concurrent_read_h5py.py`, threads or
processes) and `compare_concurrent_read.py`: decoded read throughput and
scaling efficiency at 1-16 threads on one open file, full reads of distinct
datasets and random hyperslabs of one dataset, deflate and contiguous, warm
or `--cold` page cache, JSON output. Not yet measured — `BENCHMARKS.md`
("Concurrent reads") has the commands and no numbers.
### Interop
- **Conformance sweep in the repo** (`conformance/`, report in