bench: concurrent-read results on tank, including where we lose

h5py threads stay flat (global lock); clawhdf5 hyperslab reads of deflate
data scale to 1244 MB/s at 16 threads (9.7x h5py threads, 0.89x h5py
processes). Two deficits recorded as open issues: full chunked reads stop
scaling at ~4 threads (chunk cache suspected), and contiguous reads are
4x (full) to 8x (hyperslab) slower than h5py single-threaded.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

BENCHMARKS.md

@@ -484,9 +484,55 @@ explain the slower windows.
 
 ## Concurrent reads
 
-**Not yet measured.** The harness exists; no numbers are published until it
+### Results (2026-09-26, tank)
-has been run on an idle machine. The smoke runs used while building it (tiny
+
-files, other jobs compiling on the box) are not results.
+Measured on tank (AMD Ryzen 7 7800X3D, 8 cores / 16 threads, 61 GiB, Linux
+7.0) at commit `91644d8`, load average 1.84 when the run started (the
+1-minute figure rose to 3.7 during the runs; that is mostly the benchmark's
+own threads). Warm page cache. clawhdf5 2.7.0 (workspace), h5py 3.16.0 on
+HDF5 2.0.0. Commands exactly as in the **Run** box below; files at their
+defaults (64 datasets of 16384 x 1024 `f32`, 64 MiB each; deflate chunks
+256 x 256, level 4). MB/s is decoded data, the median of the repetitions;
+eff is scaling efficiency against the same tool's 1-thread row.
+
+Each read decoding on its calling thread (`--decode-threads 1`, like h5py):
+
+| layout | mode | threads | clawhdf5 MB/s (eff) | h5py threads MB/s (eff) | h5py processes MB/s (eff) |
+|---|---|---:|---:|---:|---:|
+| deflate | distinct | 1 | 421 (1.00) | 433 (1.00) | 421 (1.00) |
+| deflate | distinct | 4 | 890 (0.53) | 428 (0.25) | 1651 (0.98) |
+| deflate | distinct | 16 | 880 (0.13) | 427 (0.06) | 4424 (0.66) |
+| deflate | same | 1 | 151 (1.00) | 130 (1.00) | 129 (1.00) |
+| deflate | same | 4 | 490 (0.81) | 129 (0.25) | 497 (0.96) |
+| deflate | same | 16 | 1244 (0.52) | 128 (0.06) | 1402 (0.68) |
+| contiguous | distinct | 1 | 2495 (1.00) | 9789 (1.00) | 9169 (1.00) |
+| contiguous | distinct | 16 | 8083 (0.20) | 8096 (0.05) | 12272 (0.08) |
+| contiguous | same | 1 | 624 (1.00) | 5022 (1.00) | 5172 (1.00) |
+| contiguous | same | 16 | 4778 (0.48) | 4411 (0.05) | 37138 (0.45) |
+
+With the default rayon pool decoding inside each read, deflate `distinct`
+is 912 MB/s at 1 thread (2.1x h5py) and 2824 MB/s at 16 (6.6x h5py threads,
+0.64x h5py processes); the other rows are within a few percent of the table
+above. Full tables (2, 4, 8 threads, both decode modes) come from
+`compare_concurrent_read.py` on the JSON files.
+
+What this shows:
+- **h5py threads do not scale** (flat at about 430 MB/s on deflate, every
+  thread count): libhdf5's global lock.
+- **clawhdf5 threads on one `File` do, for hyperslab reads of compressed
+  data:** 1244 MB/s at 16 threads, 9.7x h5py threads and 0.89x h5py
+  processes, without a process pool.
+- **Where clawhdf5 is behind** (open performance bugs, see
+  `docs/known-issues.md`):
+  - *Full reads of chunked datasets stop scaling at about 4 threads*
+    (about 880 MB/s) while h5py processes reach 4424 MB/s. Hyperslab
+    reads, which bypass the `File`'s chunk cache, keep scaling, so the
+    cache (one mutex and one 16 MiB budget per `File`, thrashed by 64 MiB
+    datasets) is the suspect.
+  - *Contiguous reads are slow*: 2.5 GB/s for a single-threaded full read
+    against h5py's 9.8 GB/s (0.25x), and 0.12x for 256 x 256 hyperslabs.
+    Threads close the gap (about 1.0x h5py at 16), but single-thread
+    contiguous I/O is a real deficit.
 
 The question: libhdf5's threadsafe build serialises every API call under one
 global mutex, and h5py holds a global lock around every call too, so threads

docs/known-issues.md

@@ -7,6 +7,18 @@ deleting it.
 
 ---
 
+## Concurrent and contiguous read performance (measured 2026-09-26)
+
+**Status:** open. Measured on tank with `concurrent_read` against h5py
+3.16 / HDF5 2.0 (`BENCHMARKS.md`, "Concurrent reads"):
+- Full reads of chunked datasets from several threads through one `File`
+  stop scaling at about 4 threads (880 MB/s on deflate data vs 4424 MB/s
+  for 16 h5py processes). Hyperslab reads, which skip the chunk cache,
+  scale to 1244 MB/s, so the `File`'s shared chunk cache is the suspect.
+- Contiguous datasets read 4x slower than h5py on one thread (2.5 vs
+  9.8 GB/s full, 0.12x for 256 x 256 hyperslabs).
+Values are correct; this is speed only.
+
 ## Silent wrong data found by the 2026-09-25 HDF5 audit
 
 **Status:** fixed after v2.7.0 (2026-09-25). **Every release up