than h5py (5e)
stable-worldmodel (arXiv 2605.21800, LeCun/Balestriero) supports HDF5 as
one of three native formats and measures generic HDF5 at 1,416-1,474
samples/s for per-frame sample loading. This measures clawhdf5 against
that shape, hardware-controlled: clawhdf5 and h5py reading the SAME file
on the SAME machine.
worldmodel_sampling example: mmap an (N,H,W,C) uint8 observation dataset,
read each frame once per pass in shuffled (dataloader) order. The file is
written by h5py (benchmarks/gen_worldmodel_frames.py) — clawhdf5 parsing
an externally-produced HDF5 file is itself the interop result — and read
by both clawhdf5 and the h5py counterpart (benchmarks/bench_worldmodel_h5py.py,
opening exactly stable-worldmodel's HDF5Dataset: swmr + 256 MB cache).
Results (tank, Ryzen 7 7800X3D, 20000x64x64x3 = 246 MB, in page cache,
median of 3):
clawhdf5 zero-copy view 593k samples/sec 8.1x
clawhdf5 materialised copy 518k samples/sec 7.1x
h5py (swmr, 256 MB cache) 73k samples/sec 1.0x
The materialised-copy row is the fair equal-work comparison (to_vec per
frame, matching h5py's numpy materialisation) and is still 7.1x faster;
that the copy costs almost nothing shows the gap is h5py's per-frame call
overhead, not data movement. Honest caveats in BENCHMARKS.md: absolute
numbers are NOT comparable to the paper's (different hardware, smaller
frames, no torch/transform), only the same-machine ratio is; this is an
in-page-cache measurement isolating read-path overhead, not disk
bandwidth.
Adds only an example, two benchmark scripts, and a BENCHMARKS.md section —
no library code. (Workspace clippy has pre-existing toolchain drift
unrelated to this change; tracked separately.)
