perf(py): read an index list one group of chunks at a time

Each run of consecutive indices was its own uncached hyperslab read, so
a list over a compressed chunked dataset decoded the same chunk once per
run (d[range(0, 200000, 40)] over 20 gzip chunks: 8 s, h5py 0.014 s).
Plan::reads now groups the indices — a group ends only where a whole
chunk holds no selected index, or, unchunked, at a gap over 64 KiB — and
the selected rows are gathered from each group's block in Rust. Now
3.8 ms (h5py 4.1 ms, release, tank). The new test (1-D, 2-D and
contiguous, compared with h5py, 2 s bound) took 5.8 s before.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-26 08:57:16 -05:00
co-authored by Claude Opus 5.5
parent 8c51b05b9c
commit b43bd2e67f
5 changed files with 259 additions and 36 deletions
+13 -2
View File
@@ -26,8 +26,8 @@
read the whole dataset and slice it in numpy, and knew six dtypes. Now
integers (negative from the end), slices with positive steps, `...`, one
increasing list of integers per key and compound field names map onto the
facade's hyperslab selection (a list becomes one hyperslab per run of
consecutive indices), with h5py's results (numpy scalar for an all-integer
facade's hyperslab selection (a list is read one group of neighbouring
chunks at a time and picked from in memory), with h5py's results (numpy scalar for an all-integer
key, 0-d array for `scalar[...]`) and h5py's errors for everything else
(negative steps, `None`, boolean masks, out-of-range indices).
`Dataset.dtype` is the numpy dtype h5py reports, for every integer and
@@ -69,6 +69,17 @@
from the file (h5py's, zero for files it wrote) and stays zero-copy.
The h5py comparisons now also compare every byte of structured values
(`test_compound_padding_bytes_match_h5py` and `assert_same`).
- **Index lists no longer decode the same chunks once per run.** A list
index was one uncached hyperslab read per run of consecutive indices, so
on a chunked, compressed dataset every run decoded its chunk again:
`d[list(range(0, 200000, 40))]` over 20 gzip chunks took 8 s (h5py:
0.014 s). The list is now read in groups — for a chunked dataset a group
ends only where a whole chunk holds no selected index, so each chunk is
decoded once; otherwise at a gap of more than 64 KiB — and the selected
rows are picked from each group in Rust. The same read now takes 3.8 ms
(h5py 4.1 ms; release build on tank, best of 5).
`test_a_long_index_list_decodes_each_chunk_once` compares 1-D, 2-D and
contiguous cases with h5py under a 2 s bound (5.8 s before, debug build).
- **CI builds and tests the Python package.** It was excluded from CI.
`scripts/ci-test.sh` now lints `clawhdf5-py`, builds the wheel with
maturin, unpacks it under `target/` and runs the pytest suite; skipped