diff --git a/BENCHMARKS.md b/BENCHMARKS.md index 645c6d6..c105698 100644 --- a/BENCHMARKS.md +++ b/BENCHMARKS.md @@ -484,9 +484,55 @@ explain the slower windows. ## Concurrent reads -**Not yet measured.** The harness exists; no numbers are published until it -has been run on an idle machine. The smoke runs used while building it (tiny -files, other jobs compiling on the box) are not results. +### Results (2026-09-26, tank) + +Measured on tank (AMD Ryzen 7 7800X3D, 8 cores / 16 threads, 61 GiB, Linux +7.0) at commit `91644d8`, load average 1.84 when the run started (the +1-minute figure rose to 3.7 during the runs; that is mostly the benchmark's +own threads). Warm page cache. clawhdf5 2.7.0 (workspace), h5py 3.16.0 on +HDF5 2.0.0. Commands exactly as in the **Run** box below; files at their +defaults (64 datasets of 16384 x 1024 `f32`, 64 MiB each; deflate chunks +256 x 256, level 4). MB/s is decoded data, the median of the repetitions; +eff is scaling efficiency against the same tool's 1-thread row. + +Each read decoding on its calling thread (`--decode-threads 1`, like h5py): + +| layout | mode | threads | clawhdf5 MB/s (eff) | h5py threads MB/s (eff) | h5py processes MB/s (eff) | +|---|---|---:|---:|---:|---:| +| deflate | distinct | 1 | 421 (1.00) | 433 (1.00) | 421 (1.00) | +| deflate | distinct | 4 | 890 (0.53) | 428 (0.25) | 1651 (0.98) | +| deflate | distinct | 16 | 880 (0.13) | 427 (0.06) | 4424 (0.66) | +| deflate | same | 1 | 151 (1.00) | 130 (1.00) | 129 (1.00) | +| deflate | same | 4 | 490 (0.81) | 129 (0.25) | 497 (0.96) | +| deflate | same | 16 | 1244 (0.52) | 128 (0.06) | 1402 (0.68) | +| contiguous | distinct | 1 | 2495 (1.00) | 9789 (1.00) | 9169 (1.00) | +| contiguous | distinct | 16 | 8083 (0.20) | 8096 (0.05) | 12272 (0.08) | +| contiguous | same | 1 | 624 (1.00) | 5022 (1.00) | 5172 (1.00) | +| contiguous | same | 16 | 4778 (0.48) | 4411 (0.05) | 37138 (0.45) | + +With the default rayon pool decoding inside each read, deflate `distinct` +is 912 MB/s at 1 thread (2.1x h5py) and 2824 MB/s at 16 (6.6x h5py threads, +0.64x h5py processes); the other rows are within a few percent of the table +above. Full tables (2, 4, 8 threads, both decode modes) come from +`compare_concurrent_read.py` on the JSON files. + +What this shows: +- **h5py threads do not scale** (flat at about 430 MB/s on deflate, every + thread count): libhdf5's global lock. +- **clawhdf5 threads on one `File` do, for hyperslab reads of compressed + data:** 1244 MB/s at 16 threads, 9.7x h5py threads and 0.89x h5py + processes, without a process pool. +- **Where clawhdf5 is behind** (open performance bugs, see + `docs/known-issues.md`): + - *Full reads of chunked datasets stop scaling at about 4 threads* + (about 880 MB/s) while h5py processes reach 4424 MB/s. Hyperslab + reads, which bypass the `File`'s chunk cache, keep scaling, so the + cache (one mutex and one 16 MiB budget per `File`, thrashed by 64 MiB + datasets) is the suspect. + - *Contiguous reads are slow*: 2.5 GB/s for a single-threaded full read + against h5py's 9.8 GB/s (0.25x), and 0.12x for 256 x 256 hyperslabs. + Threads close the gap (about 1.0x h5py at 16), but single-thread + contiguous I/O is a real deficit. The question: libhdf5's threadsafe build serialises every API call under one global mutex, and h5py holds a global lock around every call too, so threads diff --git a/docs/known-issues.md b/docs/known-issues.md index f2351a0..bf8f77a 100644 --- a/docs/known-issues.md +++ b/docs/known-issues.md @@ -7,6 +7,18 @@ deleting it. --- +## Concurrent and contiguous read performance (measured 2026-09-26) + +**Status:** open. Measured on tank with `concurrent_read` against h5py +3.16 / HDF5 2.0 (`BENCHMARKS.md`, "Concurrent reads"): +- Full reads of chunked datasets from several threads through one `File` + stop scaling at about 4 threads (880 MB/s on deflate data vs 4424 MB/s + for 16 h5py processes). Hyperslab reads, which skip the chunk cache, + scale to 1244 MB/s, so the `File`'s shared chunk cache is the suspect. +- Contiguous datasets read 4x slower than h5py on one thread (2.5 vs + 9.8 GB/s full, 0.12x for 256 x 256 hyperslabs). +Values are correct; this is speed only. + ## Silent wrong data found by the 2026-09-25 HDF5 audit **Status:** fixed after v2.7.0 (2026-09-25). **Every release up