docs: say when a selection read decodes more than the selection
The READMEs said ds[...] reads only the selected elements, and the facade's read_selection docs that only intersecting chunks are decompressed. The bounding-box path runs only when the box covers at most half the dataset; larger boxes (any strided slice across the dataset), compact, virtual and unwritten datasets and chunked ones with a non-default fill value decode the whole dataset. The READMEs, the facade and format docs, the bindings' docstrings and known-issues now say so, and how index lists are read. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
@@ -73,8 +73,9 @@ breaking change, are in [CHANGELOG.md](CHANGELOG.md).
|
||||
- Default fusion weights are now the measured 0.4 / 0.6 (v2.5.0). Re-ranking had
|
||||
been discarding the retrieval score, costing the Markdown backend 40.6pp of
|
||||
Hit@1; fixed in v2.6.0.
|
||||
- Selection reads decode only the chunks they touch (a 64×64 window: 105 ms to
|
||||
0.39 ms), and full reads are 1.2–1.9× faster (v2.5.0).
|
||||
- Selection reads whose bounding box covers at most half the dataset decode
|
||||
only the chunks they touch (a 64×64 window: 105 ms to 0.39 ms), and full
|
||||
reads are 1.2–1.9× faster (v2.5.0).
|
||||
|
||||
**Memory**
|
||||
- A loaded store holds ~30% less (embeddings stored once, v2.6.0), and the
|
||||
@@ -428,7 +429,7 @@ with clawhdf5.File("data.h5", "r") as f:
|
||||
print(list(f.keys())) # sorted member names, like h5py
|
||||
ds = f["group/temperatures"] # relative or absolute ("/group/...") paths
|
||||
print(ds.shape, ds.dtype) # dtype is the numpy dtype h5py reports
|
||||
block = ds[100:200, ::4] # reads only the selected elements
|
||||
block = ds[100:200, ::4] # a small selection reads only its chunks
|
||||
row = ds[-1] # integers drop the axis
|
||||
picked = ds[[1, 5, 9], :] # one increasing index list per key
|
||||
units = ds.attrs["units"] # attributes come back as h5py returns them
|
||||
@@ -444,6 +445,12 @@ sequences, opaque, HDF5 array types and compounds; other types (references,
|
||||
bitfields, ...) raise `TypeError` instead of returning guessed data. Keys
|
||||
follow h5py (negative steps, `None` and boolean masks are refused). The
|
||||
read itself runs with the GIL released, so Python threads read in parallel.
|
||||
A selection whose bounding box covers at most half the dataset decodes only
|
||||
the chunks (or contiguous rows) that box overlaps; a larger one — including
|
||||
a strided slice across the whole dataset — decodes the whole dataset, as
|
||||
do datasets that are compact, virtual, unwritten, or chunked with a
|
||||
non-default fill value (`docs/known-issues.md`). An index list is read one
|
||||
group of neighbouring chunks at a time.
|
||||
Writing (`File(path, "w")`, `create_dataset`, `create_group`, `attrs[...] =`)
|
||||
covers `float64`, `float32`, `int64`, `int32` and `uint8` arrays. The tests
|
||||
in `crates/clawhdf5-py/tests` compare every read with h5py; run them with
|
||||
|
||||
Reference in New Issue
Block a user