quantumclaw/clawhdf5
3
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

perf(format): back large read buffers with transparent huge pages

Browse Source 
A full read of a contiguous dataset is one memcpy from the mapped file,
yet ran at a quarter of h5py's speed on one thread: the fresh output Vec
took a page fault and a kernel page clear for every 4 KiB page written,
16384 per 64 MiB, costing several times the copy (the benchmark spent
6.2 s of 8 s in the kernel, 4.3M minor faults). numpy, so h5py, madvises
MADV_HUGEPAGE on allocations of 4 MiB or more; the typed readers' output,
the raw contiguous read and the chunk assembly buffer now do the same
(Linux only, libc as a Linux-only dependency; no-op otherwise).

New h5py comparison tests cover full and selection reads of contiguous
data for every 1-8-byte integer and float type, both byte orders, ranks
1-4, empty selections, and datasets past the 4 MiB threshold.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-26 08:12:55 -05:00
co-authored by Claude Opus 5.5
parent 63648c7000
commit 78c769f179
7 changed files with 521 additions and 11 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+17
View File
@@ -2,6 +2,23 @@
## Unreleased
### Contiguous read speed (2026-09-26)
- **Large read buffers are backed by transparent huge pages.** A full read
of a contiguous dataset was one `memcpy` from the mapped file, yet ran at
a quarter of h5py's speed on one thread: the fresh output `Vec` took a
page fault (and a kernel page clear) for every 4 KiB page it was written
to, 16384 of them for 64 MiB, and those cost several times the copy.
numpy, and so h5py, asks for transparent huge pages on every allocation of
4 MiB or more; clawhdf5-format's read buffers now do too
(`madvise(MADV_HUGEPAGE)` on Linux, `libc` added as a Linux-only
dependency; a no-op elsewhere or when THP is disabled). It applies to the
typed readers' output (`read_f32`, `read_f64`, `read_i32`, `read_i64`,
`read_u64`, both byte orders), the raw contiguous read and the chunk
assembly buffer. Values are unchanged; new h5py comparison
`crates/clawhdf5/tests/contiguous_read_interop.rs` covers every 1-8-byte
integer and float type in both byte orders, ranks 1-4, and datasets past
the 4 MiB threshold.
### Plugin filters (2026-09-26)
- **LZF, bitshuffle, bzip2 and Blosc read and write, in pure Rust.** Files
written by h5py with `compression="lzf"`, or with hdf5plugin's