perf(format): back large read buffers with transparent huge pages

A full read of a contiguous dataset is one memcpy from the mapped file,
yet ran at a quarter of h5py's speed on one thread: the fresh output Vec
took a page fault and a kernel page clear for every 4 KiB page written,
16384 per 64 MiB, costing several times the copy (the benchmark spent
6.2 s of 8 s in the kernel, 4.3M minor faults). numpy, so h5py, madvises
MADV_HUGEPAGE on allocations of 4 MiB or more; the typed readers' output,
the raw contiguous read and the chunk assembly buffer now do the same
(Linux only, libc as a Linux-only dependency; no-op otherwise).

New h5py comparison tests cover full and selection reads of contiguous
data for every 1-8-byte integer and float type, both byte orders, ranks
1-4, empty selections, and datasets past the 4 MiB threshold.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

This commit is contained in:

osobh

2026-09-26 08:12:55 -05:00

co-authored by

Claude Opus 5.5

parent 63648c7000

commit 78c769f179

7 changed files with 521 additions and 11 deletions

									
										crates/clawhdf5-format/src/lib.rs
									
		+1
		
												View File
												
				@@ -61,6 +61,7 @@ pub mod attribute;

				pub mod attribute_info;

				pub mod btree_v1;

				pub mod btree_v2;

				mod bulk_alloc;

				pub mod checksum;

				pub mod chunk_cache;

				mod chunk_grid;