docs: say when a selection read decodes more than the selection

The READMEs said ds[...] reads only the selected elements, and the
facade's read_selection docs that only intersecting chunks are
decompressed. The bounding-box path runs only when the box covers at
most half the dataset; larger boxes (any strided slice across the
dataset), compact, virtual and unwritten datasets and chunked ones with
a non-default fill value decode the whole dataset. The READMEs, the
facade and format docs, the bindings' docstrings and known-issues now
say so, and how index lists are read.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

CHANGELOG.md

@@ -22,8 +22,9 @@
   tests imported `rustyhdf5`, so they failed at collection. Distribution,
   module and tests now all say `clawhdf5`, and the module has
   `__version__`.
-- **h5py-style reads that read only what is selected.** `ds[...]` used to
+- **h5py-style reads that read the selection, not the dataset.** `ds[...]`
-  read the whole dataset and slice it in numpy, and knew six dtypes. Now
+  used to read the whole dataset and slice it in numpy, and knew six
+  dtypes. Now
   integers (negative from the end), slices with positive steps, `...`, one
   increasing list of integers per key and compound field names map onto the
   facade's hyperslab selection (a list is read one group of neighbouring
@@ -102,6 +103,12 @@
   `test_errors_match_h5py` now also requires that every key h5py reads
   reads here too, with the same result, and covers more keys (0-d arrays,
   repeated and empty lists, `()`, `...`).
+- **Docs say when a selection reads more than itself.** The README and
+  the package README said `ds[...]` reads only the selected elements,
+  without condition. The library decodes the whole dataset when the
+  selection's bounding box covers more than half of it, and for compact,
+  virtual, unwritten and non-default-fill chunked datasets; the READMEs,
+  the facade's `read_selection` docs and `docs/known-issues.md` now say so.
 - **CI builds and tests the Python package.** It was excluded from CI.
   `scripts/ci-test.sh` now lints `clawhdf5-py`, builds the wheel with
   maturin, unpacks it under `target/` and runs the pytest suite; skipped

README.md

@@ -73,8 +73,9 @@ breaking change, are in [CHANGELOG.md](CHANGELOG.md).
 - Default fusion weights are now the measured 0.4 / 0.6 (v2.5.0). Re-ranking had
   been discarding the retrieval score, costing the Markdown backend 40.6pp of
   Hit@1; fixed in v2.6.0.
-- Selection reads decode only the chunks they touch (a 64×64 window: 105 ms to
+- Selection reads whose bounding box covers at most half the dataset decode
-  0.39 ms), and full reads are 1.2–1.9× faster (v2.5.0).
+  only the chunks they touch (a 64×64 window: 105 ms to 0.39 ms), and full
+  reads are 1.2–1.9× faster (v2.5.0).
 
 **Memory**
 - A loaded store holds ~30% less (embeddings stored once, v2.6.0), and the
@@ -428,7 +429,7 @@ with clawhdf5.File("data.h5", "r") as f:
     print(list(f.keys()))              # sorted member names, like h5py
     ds = f["group/temperatures"]       # relative or absolute ("/group/...") paths
     print(ds.shape, ds.dtype)          # dtype is the numpy dtype h5py reports
-    block = ds[100:200, ::4]           # reads only the selected elements
+    block = ds[100:200, ::4]           # a small selection reads only its chunks
     row = ds[-1]                       # integers drop the axis
     picked = ds[[1, 5, 9], :]          # one increasing index list per key
     units = ds.attrs["units"]          # attributes come back as h5py returns them
@@ -444,6 +445,12 @@ sequences, opaque, HDF5 array types and compounds; other types (references,
 bitfields, ...) raise `TypeError` instead of returning guessed data. Keys
 follow h5py (negative steps, `None` and boolean masks are refused). The
 read itself runs with the GIL released, so Python threads read in parallel.
+A selection whose bounding box covers at most half the dataset decodes only
+the chunks (or contiguous rows) that box overlaps; a larger one — including
+a strided slice across the whole dataset — decodes the whole dataset, as
+do datasets that are compact, virtual, unwritten, or chunked with a
+non-default fill value (`docs/known-issues.md`). An index list is read one
+group of neighbouring chunks at a time.
 Writing (`File(path, "w")`, `create_dataset`, `create_group`, `attrs[...] =`)
 covers `float64`, `float32`, `int64`, `int32` and `uint8` arrays. The tests
 in `crates/clawhdf5-py/tests` compare every read with h5py; run them with

crates/clawhdf5-format/src/data_read.rs

@@ -286,9 +286,11 @@ pub fn read_raw_data_indexed(
 
 /// Read raw bytes for only the selected elements of a dataset.
 ///
-/// For chunked layouts, only chunks that intersect the selection are read
+/// When the selection's bounding box covers at most half the dataset, only
-/// and decompressed. For compact/contiguous layouts, the full data is read
+/// that box is materialised — the overlapping rows of a contiguous dataset,
-/// and then the selection is extracted.
+/// the overlapping chunks of a chunked one, whatever its chunk index (see
+/// [`crate::partial_read`]). Otherwise, and for compact and virtual
+/// layouts, the whole dataset is decoded and the selection extracted.
 #[allow(clippy::too_many_arguments)]
 pub fn read_raw_data_selection(
     file_data: &[u8],

crates/clawhdf5-py/README.md

@@ -29,7 +29,7 @@ with clawhdf5.File("data.h5", "r") as f:
     f.keys(), f["group"].items(), "group/data" in f
     ds = f["group/data"]           # or f["/group/data"], f["group"]["data"]
     ds.shape, ds.dtype, ds.attrs["units"]
-    ds[10:20, ::2]                 # only the selected elements are read
+    ds[10:20, ::2]                 # a small selection reads only its chunks
     ds[-1], ds[..., 0], ds[[1, 4, 7]]
     np.asarray(ds)
     f["table"]["id"]               # a compound field
@@ -45,9 +45,19 @@ with clawhdf5.File("data.h5", "r") as f:
   increasing list of integers, compound field names. Each maps onto a
   hyperslab selection. `None`, negative steps and boolean masks are refused
   with h5py's errors.
+- What is read from the file: a selection whose bounding box covers at
+  most half the dataset decodes only the chunks (or contiguous rows) the box
+  overlaps. The library decodes the whole dataset for a larger box
+  (including a strided slice such as `ds[::100]` across a chunked dataset),
+  and for compact, virtual and unwritten datasets and chunked ones with a
+  non-default fill value. An index list is read one group of neighbouring
+  chunks at a time (a new group only past a chunk with no selected index),
+  so each chunk is decoded once. `ds[()]`, `ds[...]` and `np.asarray(ds)`
+  use the file's chunk cache; other selections do not.
 - The bytes the library reads become the numpy array's buffer without a
   copy, and the read runs with the GIL released, so threads read in
-  parallel.
+  parallel. A bug in the library (a Rust panic) raises
+  `clawhdf5.InternalError`, a `RuntimeError`.
 - Attributes return what h5py returns; `clawhdf5.Empty` stands for a null
   dataspace (h5py's `Empty`).
 

crates/clawhdf5-py/src/dataset.rs

@@ -1,7 +1,9 @@
 //! PyDataset — h5py-style read access to HDF5 datasets.
 //!
 //! `ds[key]` parses the key into hyperslab selections (see `select`) and
-//! reads only those elements through the facade's `read_selection`; the
+//! reads them through the facade's `read_selection`, which decodes only the
+//! chunks a small selection touches (see its docs for when it decodes the
+//! whole dataset instead); the
 //! bytes it returns become the numpy array's buffer without a copy (see
 //! `convert`). All file access and decoding runs with the GIL released, so
 //! Python threads reading the same or different datasets run in parallel.
@@ -24,7 +26,7 @@ use crate::{PyEmpty, node, to_py_err};
 /// ```python
 /// ds = f['group/dataset']
 /// ds.shape, ds.dtype, ds.attrs['units']
-/// block = ds[10:20, ::2]      # reads only the selected elements
+/// block = ds[10:20, ::2]      # a small selection reads only its chunks
 /// ```
 #[pyclass(name = "Dataset")]
 pub struct PyDataset {
@@ -299,7 +301,8 @@ impl PyDataset {
 
     /// Read with h5py indexing: integers, slices with positive steps,
     /// `...`, one increasing list of integers, and compound field names.
-    /// Only the selected elements are read from the file.
+    /// A selection whose bounding box covers at most half the dataset reads
+    /// only the chunks (or contiguous rows) it overlaps.
     fn __getitem__<'py>(
         &self,
         py: Python<'py>,

crates/clawhdf5-py/src/select.rs

@@ -1,5 +1,7 @@
 //! h5py-style indexing (`ds[1, 2:10:3, ...]`) mapped onto hyperslab
-//! selections, so only the selected elements are read.
+//! selections, so the library reads the selection rather than the whole
+//! dataset (it still decodes everything for large selections; see the
+//! facade's `Dataset::read_selection`).
 //!
 //! The rules and error messages follow h5py's `selections.py`: integers
 //! (negative from the end) drop their axis, slices must have a positive

crates/clawhdf5/src/reader.rs

@@ -525,8 +525,16 @@ impl<'f> Dataset<'f> {
 
     /// Read selected elements as raw bytes.
     ///
-    /// Only the elements matching the [`clawhdf5_format::selection::Selection`] are returned. For chunked
+    /// Only the elements matching the [`clawhdf5_format::selection::Selection`] are returned.
-    /// datasets, only intersecting chunks are decompressed.
+    ///
+    /// What is read to get them: when the selection's bounding box covers at
+    /// most half the dataset, only that box — the chunks overlapping it, or
+    /// the rows of a contiguous dataset. The whole dataset is decoded instead
+    /// when the box covers more than half (a strided selection spanning the
+    /// dataset does), for compact and virtual layouts, for a dataset with no
+    /// storage, and for a chunked dataset with a non-default fill value.
+    /// [`Selection::All`](clawhdf5_format::selection::Selection::All) goes
+    /// through the file's chunk cache; other selections do not.
     pub fn read_selection(
         &self,
         selection: &clawhdf5_format::selection::Selection,

docs/known-issues.md

@@ -7,6 +7,23 @@ deleting it.
 
 ---
 
+## Selection reads that decode more than the selection
+
+**Status:** open (documented 2026-09-26). `Dataset::read_selection` (and so
+the Python `ds[...]`) materialises only the selection's bounding box when
+that box covers at most half the dataset (`partial_read`). It decodes the
+whole dataset and extracts the selection instead when:
+- the bounding box covers more than half the dataset — which a strided
+  selection across a chunked dataset (`ds[::100]`) always does, although
+  it may touch few chunks;
+- the dataset is compact or virtual, or has no storage;
+- it is chunked with a non-default fill value (the box path does not fill
+  unallocated chunks, so the fill-aware full read is used).
+Values are correct in every case; this is cost only. Selections other than
+`Selection::All` also bypass the file's chunk cache. The bounding-box
+heuristic's other cost is measured under "Concurrent and contiguous read
+performance" below.
+
 ## Concurrent and contiguous read performance (measured 2026-09-26)
 
 **Status:** open. Measured on tank with `concurrent_read` against h5py