bench: concurrent-read results on tank, including where we lose

h5py threads stay flat (global lock); clawhdf5 hyperslab reads of deflate data scale to 1244 MB/s at 16 threads (9.7x h5py threads, 0.89x h5py processes). Two deficits recorded as open issues: full chunked reads stop scaling at ~4 threads (chunk cache suspected), and contiguous reads are 4x (full) to 8x (hyperslab) slower than h5py single-threaded. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
2026-09-26 02:02:17 -05:00
parent 91644d8aaf
commit 63648c7000
2 changed files with 61 additions and 3 deletions
@@ -7,6 +7,18 @@ deleting it.

 ---

+## Concurrent and contiguous read performance (measured 2026-09-26)
+
+**Status:** open. Measured on tank with `concurrent_read` against h5py
+3.16 / HDF5 2.0 (`BENCHMARKS.md`, "Concurrent reads"):
+- Full reads of chunked datasets from several threads through one `File`
+  stop scaling at about 4 threads (880 MB/s on deflate data vs 4424 MB/s
+  for 16 h5py processes). Hyperslab reads, which skip the chunk cache,
+  scale to 1244 MB/s, so the `File`'s shared chunk cache is the suspect.
+- Contiguous datasets read 4x slower than h5py on one thread (2.5 vs
+  9.8 GB/s full, 0.12x for 256 x 256 hyperslabs).
+Values are correct; this is speed only.
+
 ## Silent wrong data found by the 2026-09-25 HDF5 audit

 **Status:** fixed after v2.7.0 (2026-09-25). **Every release up