docs: say when a selection read decodes more than the selection

The READMEs said ds[...] reads only the selected elements, and the
facade's read_selection docs that only intersecting chunks are
decompressed. The bounding-box path runs only when the box covers at
most half the dataset; larger boxes (any strided slice across the
dataset), compact, virtual and unwritten datasets and chunked ones with
a non-default fill value decode the whole dataset. The READMEs, the
facade and format docs, the bindings' docstrings and known-issues now
say so, and how index lists are read.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-26 09:04:48 -05:00
co-authored by Claude Opus 5.5
parent 17edfe2cf0
commit 45d617c39e
8 changed files with 72 additions and 16 deletions
+10 -3
View File
@@ -73,8 +73,9 @@ breaking change, are in [CHANGELOG.md](CHANGELOG.md).
- Default fusion weights are now the measured 0.4 / 0.6 (v2.5.0). Re-ranking had
been discarding the retrieval score, costing the Markdown backend 40.6pp of
Hit@1; fixed in v2.6.0.
- Selection reads decode only the chunks they touch (a 64×64 window: 105 ms to
0.39 ms), and full reads are 1.2–1.9× faster (v2.5.0).
- Selection reads whose bounding box covers at most half the dataset decode
only the chunks they touch (a 64×64 window: 105 ms to 0.39 ms), and full
reads are 1.2–1.9× faster (v2.5.0).
**Memory**
- A loaded store holds ~30% less (embeddings stored once, v2.6.0), and the
@@ -428,7 +429,7 @@ with clawhdf5.File("data.h5", "r") as f:
print(list(f.keys())) # sorted member names, like h5py
ds = f["group/temperatures"] # relative or absolute ("/group/...") paths
print(ds.shape, ds.dtype) # dtype is the numpy dtype h5py reports
block = ds[100:200, ::4] # reads only the selected elements
block = ds[100:200, ::4] # a small selection reads only its chunks
row = ds[-1] # integers drop the axis
picked = ds[[1, 5, 9], :] # one increasing index list per key
units = ds.attrs["units"] # attributes come back as h5py returns them
@@ -444,6 +445,12 @@ sequences, opaque, HDF5 array types and compounds; other types (references,
bitfields, ...) raise `TypeError` instead of returning guessed data. Keys
follow h5py (negative steps, `None` and boolean masks are refused). The
read itself runs with the GIL released, so Python threads read in parallel.
A selection whose bounding box covers at most half the dataset decodes only
the chunks (or contiguous rows) that box overlaps; a larger one — including
a strided slice across the whole dataset — decodes the whole dataset, as
do datasets that are compact, virtual, unwritten, or chunked with a
non-default fill value (`docs/known-issues.md`). An index list is read one
group of neighbouring chunks at a time.
Writing (`File(path, "w")`, `create_dataset`, `create_group`, `attrs[...] =`)
covers `float64`, `float32`, `int64`, `int32` and `uint8` arrays. The tests
in `crates/clawhdf5-py/tests` compare every read with h5py; run them with