docs: say when a selection read decodes more than the selection

The READMEs said ds[...] reads only the selected elements, and the
facade's read_selection docs that only intersecting chunks are
decompressed. The bounding-box path runs only when the box covers at
most half the dataset; larger boxes (any strided slice across the
dataset), compact, virtual and unwritten datasets and chunked ones with
a non-default fill value decode the whole dataset. The READMEs, the
facade and format docs, the bindings' docstrings and known-issues now
say so, and how index lists are read.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

crates/clawhdf5-format/src/data_read.rs

@@ -286,9 +286,11 @@ pub fn read_raw_data_indexed(
 
 /// Read raw bytes for only the selected elements of a dataset.
 ///
-/// For chunked layouts, only chunks that intersect the selection are read
-/// and decompressed. For compact/contiguous layouts, the full data is read
-/// and then the selection is extracted.
+/// When the selection's bounding box covers at most half the dataset, only
+/// that box is materialised — the overlapping rows of a contiguous dataset,
+/// the overlapping chunks of a chunked one, whatever its chunk index (see
+/// [`crate::partial_read`]). Otherwise, and for compact and virtual
+/// layouts, the whole dataset is decoded and the selection extracted.
 #[allow(clippy::too_many_arguments)]
 pub fn read_raw_data_selection(
     file_data: &[u8],

crates/clawhdf5-py/README.md

@@ -29,7 +29,7 @@ with clawhdf5.File("data.h5", "r") as f:
     f.keys(), f["group"].items(), "group/data" in f
     ds = f["group/data"]           # or f["/group/data"], f["group"]["data"]
     ds.shape, ds.dtype, ds.attrs["units"]
-    ds[10:20, ::2]                 # only the selected elements are read
+    ds[10:20, ::2]                 # a small selection reads only its chunks
     ds[-1], ds[..., 0], ds[[1, 4, 7]]
     np.asarray(ds)
     f["table"]["id"]               # a compound field
@@ -45,9 +45,19 @@ with clawhdf5.File("data.h5", "r") as f:
   increasing list of integers, compound field names. Each maps onto a
   hyperslab selection. `None`, negative steps and boolean masks are refused
   with h5py's errors.
+- What is read from the file: a selection whose bounding box covers at
+  most half the dataset decodes only the chunks (or contiguous rows) the box
+  overlaps. The library decodes the whole dataset for a larger box
+  (including a strided slice such as `ds[::100]` across a chunked dataset),
+  and for compact, virtual and unwritten datasets and chunked ones with a
+  non-default fill value. An index list is read one group of neighbouring
+  chunks at a time (a new group only past a chunk with no selected index),
+  so each chunk is decoded once. `ds[()]`, `ds[...]` and `np.asarray(ds)`
+  use the file's chunk cache; other selections do not.
 - The bytes the library reads become the numpy array's buffer without a
   copy, and the read runs with the GIL released, so threads read in
-  parallel.
+  parallel. A bug in the library (a Rust panic) raises
+  `clawhdf5.InternalError`, a `RuntimeError`.
 - Attributes return what h5py returns; `clawhdf5.Empty` stands for a null
   dataspace (h5py's `Empty`).
 

crates/clawhdf5-py/src/dataset.rs

@@ -1,7 +1,9 @@
 //! PyDataset — h5py-style read access to HDF5 datasets.
 //!
 //! `ds[key]` parses the key into hyperslab selections (see `select`) and
-//! reads only those elements through the facade's `read_selection`; the
+//! reads them through the facade's `read_selection`, which decodes only the
+//! chunks a small selection touches (see its docs for when it decodes the
+//! whole dataset instead); the
 //! bytes it returns become the numpy array's buffer without a copy (see
 //! `convert`). All file access and decoding runs with the GIL released, so
 //! Python threads reading the same or different datasets run in parallel.
@@ -24,7 +26,7 @@ use crate::{PyEmpty, node, to_py_err};
 /// ```python
 /// ds = f['group/dataset']
 /// ds.shape, ds.dtype, ds.attrs['units']
-/// block = ds[10:20, ::2]      # reads only the selected elements
+/// block = ds[10:20, ::2]      # a small selection reads only its chunks
 /// ```
 #[pyclass(name = "Dataset")]
 pub struct PyDataset {
@@ -299,7 +301,8 @@ impl PyDataset {
 
     /// Read with h5py indexing: integers, slices with positive steps,
     /// `...`, one increasing list of integers, and compound field names.
-    /// Only the selected elements are read from the file.
+    /// A selection whose bounding box covers at most half the dataset reads
+    /// only the chunks (or contiguous rows) it overlaps.
     fn __getitem__<'py>(
         &self,
         py: Python<'py>,

crates/clawhdf5-py/src/select.rs

@@ -1,5 +1,7 @@
 //! h5py-style indexing (`ds[1, 2:10:3, ...]`) mapped onto hyperslab
-//! selections, so only the selected elements are read.
+//! selections, so the library reads the selection rather than the whole
+//! dataset (it still decodes everything for large selections; see the
+//! facade's `Dataset::read_selection`).
 //!
 //! The rules and error messages follow h5py's `selections.py`: integers
 //! (negative from the end) drop their axis, slices must have a positive

crates/clawhdf5/src/reader.rs

@@ -525,8 +525,16 @@ impl<'f> Dataset<'f> {
 
     /// Read selected elements as raw bytes.
     ///
-    /// Only the elements matching the [`clawhdf5_format::selection::Selection`] are returned. For chunked
-    /// datasets, only intersecting chunks are decompressed.
+    /// Only the elements matching the [`clawhdf5_format::selection::Selection`] are returned.
+    ///
+    /// What is read to get them: when the selection's bounding box covers at
+    /// most half the dataset, only that box — the chunks overlapping it, or
+    /// the rows of a contiguous dataset. The whole dataset is decoded instead
+    /// when the box covers more than half (a strided selection spanning the
+    /// dataset does), for compact and virtual layouts, for a dataset with no
+    /// storage, and for a chunked dataset with a non-default fill value.
+    /// [`Selection::All`](clawhdf5_format::selection::Selection::All) goes
+    /// through the file's chunk cache; other selections do not.
     pub fn read_selection(
         &self,
         selection: &clawhdf5_format::selection::Selection,