perf(format): back large read buffers with transparent huge pages

A full read of a contiguous dataset is one memcpy from the mapped file,
yet ran at a quarter of h5py's speed on one thread: the fresh output Vec
took a page fault and a kernel page clear for every 4 KiB page written,
16384 per 64 MiB, costing several times the copy (the benchmark spent
6.2 s of 8 s in the kernel, 4.3M minor faults). numpy, so h5py, madvises
MADV_HUGEPAGE on allocations of 4 MiB or more; the typed readers' output,
the raw contiguous read and the chunk assembly buffer now do the same
(Linux only, libc as a Linux-only dependency; no-op otherwise).

New h5py comparison tests cover full and selection reads of contiguous
data for every 1-8-byte integer and float type, both byte orders, ranks
1-4, empty selections, and datasets past the 4 MiB threshold.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-26 08:12:55 -05:00
co-authored by Claude Opus 5.5
parent 63648c7000
commit 78c769f179
7 changed files with 521 additions and 11 deletions
+4
View File
@@ -30,6 +30,10 @@ ruzstd = { version = "0.9", optional = true }
bzip2 = { version = "0.6", optional = true }
snap = { version = "1", optional = true }
[target.'cfg(target_os = "linux")'.dependencies]
# madvise(MADV_HUGEPAGE) for large read buffers (see src/bulk_alloc.rs).
libc = { version = "0.2", default-features = false }
[dev-dependencies]
half = { workspace = true }
serde_json = "1"