Merge branch 'perf/p2-contiguous-reads' into feat/p2-perf-coverage
# Conflicts: # CHANGELOG.md # docs/known-issues.md
This commit is contained in:
@@ -16,6 +16,44 @@
|
||||
its lookups cost a few percent at 16 threads. Throughput with the default
|
||||
pool is unchanged, and still short of an h5py process pool.
|
||||
|
||||
### Contiguous read speed (2026-09-26)
|
||||
- **Large read buffers are backed by transparent huge pages.** A full read
|
||||
of a contiguous dataset was one `memcpy` from the mapped file, yet ran at
|
||||
a quarter of h5py's speed on one thread: the fresh output `Vec` took a
|
||||
page fault (and a kernel page clear) for every 4 KiB page it was written
|
||||
to, 16384 of them for 64 MiB, and those cost several times the copy.
|
||||
numpy, and so h5py, asks for transparent huge pages on every allocation of
|
||||
4 MiB or more; clawhdf5-format's read buffers now do too
|
||||
(`madvise(MADV_HUGEPAGE)` on Linux, `libc` added as a Linux-only
|
||||
dependency; a no-op elsewhere or when THP is disabled). It applies to the
|
||||
typed readers' output (`read_f32`, `read_f64`, `read_i32`, `read_i64`,
|
||||
`read_u64`, both byte orders), the raw contiguous read and the chunk
|
||||
assembly buffer. Values are unchanged; new h5py comparison
|
||||
`crates/clawhdf5/tests/contiguous_read_interop.rs` covers every 1-8-byte
|
||||
integer and float type in both byte orders, ranks 1-4, and datasets past
|
||||
the 4 MiB threshold.
|
||||
- **Hyperslab and point reads of contiguous data copy runs, not elements.**
|
||||
A 256 x 256 hyperslab of a contiguous `f32` dataset read at an eighth of
|
||||
h5py's speed: the selection's bounding box was copied out of the file,
|
||||
then walked element by element (a recursive call and two bounds checks per
|
||||
element) into a second buffer, which `read_f32_selection` converted into
|
||||
a third. Selections of contiguous data are now copied straight from the
|
||||
file, one `memcpy` per run of elements that is contiguous in the file
|
||||
(a block along the last dimension, blocks that touch, and whole rows when
|
||||
the inner dimensions are selected in full, merged), with no zero-filled
|
||||
intermediate; a selection covering most of the dataset no longer makes a
|
||||
full copy first. The typed selection readers (`read_f32_selection`,
|
||||
`read_f64_selection`, `read_i32_selection`, `read_i64_selection`) copy
|
||||
directly into their output when the dataset stores that type natively,
|
||||
and convert as before otherwise (big-endian, other widths). The chunked
|
||||
paths use the same run-based extraction. New public
|
||||
`clawhdf5_format::data_read::read_selection_native` and the sealed
|
||||
`NativeElement` trait (also used by the `read_as_*` fast paths, which
|
||||
gained one for native `u64`). Values are unchanged: checked against h5py
|
||||
by `contiguous_read_interop.rs` (strided, blocked, adjacent-block and
|
||||
whole-row hyperslabs, points, empty selections; every type, both byte
|
||||
orders, ranks 1-4).
|
||||
|
||||
### Plugin filters (2026-09-26)
|
||||
- **LZF, bitshuffle, bzip2 and Blosc read and write, in pure Rust.** Files
|
||||
written by h5py with `compression="lzf"`, or with hdf5plugin's
|
||||
|
||||
Reference in New Issue
Block a user