osobh
b2dce41532
CI / test (push) Failing after 3s
than h5py (5e) stable-worldmodel (arXiv 2605.21800, LeCun/Balestriero) supports HDF5 as one of three native formats and measures generic HDF5 at 1,416-1,474 samples/s for per-frame sample loading. This measures clawhdf5 against that shape, hardware-controlled: clawhdf5 and h5py reading the SAME file on the SAME machine. worldmodel_sampling example: mmap an (N,H,W,C) uint8 observation dataset, read each frame once per pass in shuffled (dataloader) order. The file is written by h5py (benchmarks/gen_worldmodel_frames.py) — clawhdf5 parsing an externally-produced HDF5 file is itself the interop result — and read by both clawhdf5 and the h5py counterpart (benchmarks/bench_worldmodel_h5py.py, opening exactly stable-worldmodel's HDF5Dataset: swmr + 256 MB cache). Results (tank, Ryzen 7 7800X3D, 20000x64x64x3 = 246 MB, in page cache, median of 3): clawhdf5 zero-copy view 593k samples/sec 8.1x clawhdf5 materialised copy 518k samples/sec 7.1x h5py (swmr, 256 MB cache) 73k samples/sec 1.0x The materialised-copy row is the fair equal-work comparison (to_vec per frame, matching h5py's numpy materialisation) and is still 7.1x faster; that the copy costs almost nothing shows the gap is h5py's per-frame call overhead, not data movement. Honest caveats in BENCHMARKS.md: absolute numbers are NOT comparable to the paper's (different hardware, smaller frames, no torch/transform), only the same-machine ratio is; this is an in-page-cache measurement isolating read-path overhead, not disk bandwidth. Adds only an example, two benchmark scripts, and a BENCHMARKS.md section — no library code. (Workspace clippy has pre-existing toolchain drift unrelated to this change; tracked separately.)