perf: copy contiguous hyperslab and point reads run by run
A 256 x 256 hyperslab of a contiguous f32 dataset read at an eighth of h5py's speed: partial_read copied the bounding box out of the file, the extractor then walked it element by element (a recursive call and two bounds checks per element) into a second buffer, and read_f32_selection converted that into a third. Selections of contiguous data are now copied straight from the file, one memcpy per run of elements contiguous in the file (gather.rs: a block along the last dimension, touching blocks as one range, whole rows merged), with no zero-filled intermediate and no full copy for large selections. The typed selection readers copy into their Vec<T> directly when the dataset stores T natively (new data_read::read_selection_native and sealed NativeElement trait, which the read_as_* fast paths now share; read_as_u64 gains one) and convert as before otherwise. The general extractor used by the chunked paths runs on the same run walker, keeping its old handling of unvalidated selections. Checked against h5py (contiguous_read_interop.rs) for strided, blocked, adjacent-block and whole-row hyperslabs, points and empty selections of every 1-8-byte type in both byte orders, ranks 1-4. Also keeps the huge-page threshold constant out of no_std builds, where it was unused. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
@@ -18,6 +18,27 @@
|
||||
`crates/clawhdf5/tests/contiguous_read_interop.rs` covers every 1-8-byte
|
||||
integer and float type in both byte orders, ranks 1-4, and datasets past
|
||||
the 4 MiB threshold.
|
||||
- **Hyperslab and point reads of contiguous data copy runs, not elements.**
|
||||
A 256 x 256 hyperslab of a contiguous `f32` dataset read at an eighth of
|
||||
h5py's speed: the selection's bounding box was copied out of the file,
|
||||
then walked element by element (a recursive call and two bounds checks per
|
||||
element) into a second buffer, which `read_f32_selection` converted into
|
||||
a third. Selections of contiguous data are now copied straight from the
|
||||
file, one `memcpy` per run of elements that is contiguous in the file
|
||||
(a block along the last dimension, blocks that touch, and whole rows when
|
||||
the inner dimensions are selected in full, merged), with no zero-filled
|
||||
intermediate; a selection covering most of the dataset no longer makes a
|
||||
full copy first. The typed selection readers (`read_f32_selection`,
|
||||
`read_f64_selection`, `read_i32_selection`, `read_i64_selection`) copy
|
||||
directly into their output when the dataset stores that type natively,
|
||||
and convert as before otherwise (big-endian, other widths). The chunked
|
||||
paths use the same run-based extraction. New public
|
||||
`clawhdf5_format::data_read::read_selection_native` and the sealed
|
||||
`NativeElement` trait (also used by the `read_as_*` fast paths, which
|
||||
gained one for native `u64`). Values are unchanged: checked against h5py
|
||||
by `contiguous_read_interop.rs` (strided, blocked, adjacent-block and
|
||||
whole-row hyperslabs, points, empty selections; every type, both byte
|
||||
orders, ranks 1-4).
|
||||
|
||||
### Plugin filters (2026-09-26)
|
||||
- **LZF, bitshuffle, bzip2 and Blosc read and write, in pure Rust.** Files
|
||||
|
||||
Reference in New Issue
Block a user