Merge branch 'perf/p2-contiguous-reads' into feat/p2-perf-coverage

# Conflicts:
#	CHANGELOG.md
#	docs/known-issues.md
This commit is contained in:
osobh
2026-09-26 09:10:26 -05:00
12 changed files with 1108 additions and 181 deletions
+38
View File
@@ -16,6 +16,44 @@
its lookups cost a few percent at 16 threads. Throughput with the default
pool is unchanged, and still short of an h5py process pool.
### Contiguous read speed (2026-09-26)
- **Large read buffers are backed by transparent huge pages.** A full read
of a contiguous dataset was one `memcpy` from the mapped file, yet ran at
a quarter of h5py's speed on one thread: the fresh output `Vec` took a
page fault (and a kernel page clear) for every 4 KiB page it was written
to, 16384 of them for 64 MiB, and those cost several times the copy.
numpy, and so h5py, asks for transparent huge pages on every allocation of
4 MiB or more; clawhdf5-format's read buffers now do too
(`madvise(MADV_HUGEPAGE)` on Linux, `libc` added as a Linux-only
dependency; a no-op elsewhere or when THP is disabled). It applies to the
typed readers' output (`read_f32`, `read_f64`, `read_i32`, `read_i64`,
`read_u64`, both byte orders), the raw contiguous read and the chunk
assembly buffer. Values are unchanged; new h5py comparison
`crates/clawhdf5/tests/contiguous_read_interop.rs` covers every 1-8-byte
integer and float type in both byte orders, ranks 1-4, and datasets past
the 4 MiB threshold.
- **Hyperslab and point reads of contiguous data copy runs, not elements.**
A 256 x 256 hyperslab of a contiguous `f32` dataset read at an eighth of
h5py's speed: the selection's bounding box was copied out of the file,
then walked element by element (a recursive call and two bounds checks per
element) into a second buffer, which `read_f32_selection` converted into
a third. Selections of contiguous data are now copied straight from the
file, one `memcpy` per run of elements that is contiguous in the file
(a block along the last dimension, blocks that touch, and whole rows when
the inner dimensions are selected in full, merged), with no zero-filled
intermediate; a selection covering most of the dataset no longer makes a
full copy first. The typed selection readers (`read_f32_selection`,
`read_f64_selection`, `read_i32_selection`, `read_i64_selection`) copy
directly into their output when the dataset stores that type natively,
and convert as before otherwise (big-endian, other widths). The chunked
paths use the same run-based extraction. New public
`clawhdf5_format::data_read::read_selection_native` and the sealed
`NativeElement` trait (also used by the `read_as_*` fast paths, which
gained one for native `u64`). Values are unchanged: checked against h5py
by `contiguous_read_interop.rs` (strided, blocked, adjacent-block and
whole-row hyperslabs, points, empty selections; every type, both byte
orders, ranks 1-4).
### Plugin filters (2026-09-26)
- **LZF, bitshuffle, bzip2 and Blosc read and write, in pure Rust.** Files
written by h5py with `compression="lzf"`, or with hdf5plugin's