perf(py): read an index list one group of chunks at a time
Each run of consecutive indices was its own uncached hyperslab read, so a list over a compressed chunked dataset decoded the same chunk once per run (d[range(0, 200000, 40)] over 20 gzip chunks: 8 s, h5py 0.014 s). Plan::reads now groups the indices — a group ends only where a whole chunk holds no selected index, or, unchunked, at a gap over 64 KiB — and the selected rows are gathered from each group's block in Rust. Now 3.8 ms (h5py 4.1 ms, release, tank). The new test (1-D, 2-D and contiguous, compared with h5py, 2 s bound) took 5.8 s before. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
+13
-2
@@ -26,8 +26,8 @@
|
||||
read the whole dataset and slice it in numpy, and knew six dtypes. Now
|
||||
integers (negative from the end), slices with positive steps, `...`, one
|
||||
increasing list of integers per key and compound field names map onto the
|
||||
facade's hyperslab selection (a list becomes one hyperslab per run of
|
||||
consecutive indices), with h5py's results (numpy scalar for an all-integer
|
||||
facade's hyperslab selection (a list is read one group of neighbouring
|
||||
chunks at a time and picked from in memory), with h5py's results (numpy scalar for an all-integer
|
||||
key, 0-d array for `scalar[...]`) and h5py's errors for everything else
|
||||
(negative steps, `None`, boolean masks, out-of-range indices).
|
||||
`Dataset.dtype` is the numpy dtype h5py reports, for every integer and
|
||||
@@ -69,6 +69,17 @@
|
||||
from the file (h5py's, zero for files it wrote) and stays zero-copy.
|
||||
The h5py comparisons now also compare every byte of structured values
|
||||
(`test_compound_padding_bytes_match_h5py` and `assert_same`).
|
||||
- **Index lists no longer decode the same chunks once per run.** A list
|
||||
index was one uncached hyperslab read per run of consecutive indices, so
|
||||
on a chunked, compressed dataset every run decoded its chunk again:
|
||||
`d[list(range(0, 200000, 40))]` over 20 gzip chunks took 8 s (h5py:
|
||||
0.014 s). The list is now read in groups — for a chunked dataset a group
|
||||
ends only where a whole chunk holds no selected index, so each chunk is
|
||||
decoded once; otherwise at a gap of more than 64 KiB — and the selected
|
||||
rows are picked from each group in Rust. The same read now takes 3.8 ms
|
||||
(h5py 4.1 ms; release build on tank, best of 5).
|
||||
`test_a_long_index_list_decodes_each_chunk_once` compares 1-D, 2-D and
|
||||
contiguous cases with h5py under a 2 s bound (5.8 s before, debug build).
|
||||
- **CI builds and tests the Python package.** It was excluded from CI.
|
||||
`scripts/ci-test.sh` now lints `clawhdf5-py`, builds the wheel with
|
||||
maturin, unpacks it under `target/` and runs the pytest suite; skipped
|
||||
|
||||
Reference in New Issue
Block a user