perf(format): back large read buffers with transparent huge pages
A full read of a contiguous dataset is one memcpy from the mapped file, yet ran at a quarter of h5py's speed on one thread: the fresh output Vec took a page fault and a kernel page clear for every 4 KiB page written, 16384 per 64 MiB, costing several times the copy (the benchmark spent 6.2 s of 8 s in the kernel, 4.3M minor faults). numpy, so h5py, madvises MADV_HUGEPAGE on allocations of 4 MiB or more; the typed readers' output, the raw contiguous read and the chunk assembly buffer now do the same (Linux only, libc as a Linux-only dependency; no-op otherwise). New h5py comparison tests cover full and selection reads of contiguous data for every 1-8-byte integer and float type, both byte orders, ranks 1-4, empty selections, and datasets past the 4 MiB threshold. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
@@ -30,6 +30,10 @@ ruzstd = { version = "0.9", optional = true }
|
||||
bzip2 = { version = "0.6", optional = true }
|
||||
snap = { version = "1", optional = true }
|
||||
|
||||
[target.'cfg(target_os = "linux")'.dependencies]
|
||||
# madvise(MADV_HUGEPAGE) for large read buffers (see src/bulk_alloc.rs).
|
||||
libc = { version = "0.2", default-features = false }
|
||||
|
||||
[dev-dependencies]
|
||||
half = { workspace = true }
|
||||
serde_json = "1"
|
||||
|
||||
Reference in New Issue
Block a user