perf(format): back large read buffers with transparent huge pages

A full read of a contiguous dataset is one memcpy from the mapped file, yet ran at a quarter of h5py's speed on one thread: the fresh output Vec took a page fault and a kernel page clear for every 4 KiB page written, 16384 per 64 MiB, costing several times the copy (the benchmark spent 6.2 s of 8 s in the kernel, 4.3M minor faults). numpy, so h5py, madvises MADV_HUGEPAGE on allocations of 4 MiB or more; the typed readers' output, the raw contiguous read and the chunk assembly buffer now do the same (Linux only, libc as a Linux-only dependency; no-op otherwise). New h5py comparison tests cover full and selection reads of contiguous data for every 1-8-byte integer and float type, both byte orders, ranks 1-4, empty selections, and datasets past the 4 MiB threshold. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
2026-09-26 08:12:55 -05:00
parent 63648c7000
commit 78c769f179
7 changed files with 521 additions and 11 deletions
@@ -277,6 +277,8 @@ pub(crate) fn alloc_output(len: usize) -> Result<Vec<u8>, FormatError> {
    if ptr.is_null() {
        return Err(failed());
    }
+    // Before anything writes to it, so a large buffer faults in huge pages.
+    crate::bulk_alloc::advise_huge_pages(ptr, len);
    // SAFETY: `ptr` came from the global allocator with the layout of
    // `[u8; len]`, which is exactly what `Vec<u8>` with capacity `len` frees;
    // all `len` bytes are initialised (zero).