perf: copy contiguous hyperslab and point reads run by run
A 256 x 256 hyperslab of a contiguous f32 dataset read at an eighth of h5py's speed: partial_read copied the bounding box out of the file, the extractor then walked it element by element (a recursive call and two bounds checks per element) into a second buffer, and read_f32_selection converted that into a third. Selections of contiguous data are now copied straight from the file, one memcpy per run of elements contiguous in the file (gather.rs: a block along the last dimension, touching blocks as one range, whole rows merged), with no zero-filled intermediate and no full copy for large selections. The typed selection readers copy into their Vec<T> directly when the dataset stores T natively (new data_read::read_selection_native and sealed NativeElement trait, which the read_as_* fast paths now share; read_as_u64 gains one) and convert as before otherwise. The general extractor used by the chunked paths runs on the same run walker, keeping its old handling of unvalidated selections. Checked against h5py (contiguous_read_interop.rs) for strided, blocked, adjacent-block and whole-row hyperslabs, points and empty selections of every 1-8-byte type in both byte orders, ranks 1-4. Also keeps the huge-page threshold constant out of no_std builds, where it was unused. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
@@ -9,7 +9,8 @@ deleting it.
|
||||
|
||||
## Concurrent and contiguous read performance (measured 2026-09-26)
|
||||
|
||||
**Status:** open. Measured on tank with `concurrent_read` against h5py
|
||||
**Status:** open for chunked full reads; the contiguous item is fixed
|
||||
(2026-09-26). Measured on tank with `concurrent_read` against h5py
|
||||
3.16 / HDF5 2.0 (`BENCHMARKS.md`, "Concurrent reads"):
|
||||
- Full reads of chunked datasets from several threads through one `File`
|
||||
stop scaling at about 4 threads (880 MB/s on deflate data vs 4424 MB/s
|
||||
@@ -17,6 +18,12 @@ deleting it.
|
||||
scale to 1244 MB/s, so the `File`'s shared chunk cache is the suspect.
|
||||
- Contiguous datasets read 4x slower than h5py on one thread (2.5 vs
|
||||
9.8 GB/s full, 0.12x for 256 x 256 hyperslabs).
|
||||
**Fixed 2026-09-26** (not yet re-measured for `BENCHMARKS.md`): full
|
||||
reads were dominated by 4 KiB page faults on the fresh output buffer,
|
||||
which is now backed by transparent huge pages as numpy's is; hyperslab
|
||||
reads copied the selection three times, element by element, and now copy
|
||||
each contiguous run once, straight from the file into the output (see
|
||||
`CHANGELOG.md`). The chunked-read scaling item above is still open.
|
||||
Values are correct; this is speed only.
|
||||
|
||||
## Silent wrong data found by the 2026-09-25 HDF5 audit
|
||||
|
||||
Reference in New Issue
Block a user