diff --git a/CHANGELOG.md b/CHANGELOG.md index e175c02..b6fc54e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -22,8 +22,9 @@ tests imported `rustyhdf5`, so they failed at collection. Distribution, module and tests now all say `clawhdf5`, and the module has `__version__`. -- **h5py-style reads that read only what is selected.** `ds[...]` used to - read the whole dataset and slice it in numpy, and knew six dtypes. Now +- **h5py-style reads that read the selection, not the dataset.** `ds[...]` + used to read the whole dataset and slice it in numpy, and knew six + dtypes. Now integers (negative from the end), slices with positive steps, `...`, one increasing list of integers per key and compound field names map onto the facade's hyperslab selection (a list is read one group of neighbouring @@ -102,6 +103,12 @@ `test_errors_match_h5py` now also requires that every key h5py reads reads here too, with the same result, and covers more keys (0-d arrays, repeated and empty lists, `()`, `...`). +- **Docs say when a selection reads more than itself.** The README and + the package README said `ds[...]` reads only the selected elements, + without condition. The library decodes the whole dataset when the + selection's bounding box covers more than half of it, and for compact, + virtual, unwritten and non-default-fill chunked datasets; the READMEs, + the facade's `read_selection` docs and `docs/known-issues.md` now say so. - **CI builds and tests the Python package.** It was excluded from CI. `scripts/ci-test.sh` now lints `clawhdf5-py`, builds the wheel with maturin, unpacks it under `target/` and runs the pytest suite; skipped diff --git a/README.md b/README.md index c266f3b..86da933 100644 --- a/README.md +++ b/README.md @@ -73,8 +73,9 @@ breaking change, are in [CHANGELOG.md](CHANGELOG.md). - Default fusion weights are now the measured 0.4 / 0.6 (v2.5.0). Re-ranking had been discarding the retrieval score, costing the Markdown backend 40.6pp of Hit@1; fixed in v2.6.0. -- Selection reads decode only the chunks they touch (a 64×64 window: 105 ms to - 0.39 ms), and full reads are 1.2–1.9× faster (v2.5.0). +- Selection reads whose bounding box covers at most half the dataset decode + only the chunks they touch (a 64×64 window: 105 ms to 0.39 ms), and full + reads are 1.2–1.9× faster (v2.5.0). **Memory** - A loaded store holds ~30% less (embeddings stored once, v2.6.0), and the @@ -428,7 +429,7 @@ with clawhdf5.File("data.h5", "r") as f: print(list(f.keys())) # sorted member names, like h5py ds = f["group/temperatures"] # relative or absolute ("/group/...") paths print(ds.shape, ds.dtype) # dtype is the numpy dtype h5py reports - block = ds[100:200, ::4] # reads only the selected elements + block = ds[100:200, ::4] # a small selection reads only its chunks row = ds[-1] # integers drop the axis picked = ds[[1, 5, 9], :] # one increasing index list per key units = ds.attrs["units"] # attributes come back as h5py returns them @@ -444,6 +445,12 @@ sequences, opaque, HDF5 array types and compounds; other types (references, bitfields, ...) raise `TypeError` instead of returning guessed data. Keys follow h5py (negative steps, `None` and boolean masks are refused). The read itself runs with the GIL released, so Python threads read in parallel. +A selection whose bounding box covers at most half the dataset decodes only +the chunks (or contiguous rows) that box overlaps; a larger one — including +a strided slice across the whole dataset — decodes the whole dataset, as +do datasets that are compact, virtual, unwritten, or chunked with a +non-default fill value (`docs/known-issues.md`). An index list is read one +group of neighbouring chunks at a time. Writing (`File(path, "w")`, `create_dataset`, `create_group`, `attrs[...] =`) covers `float64`, `float32`, `int64`, `int32` and `uint8` arrays. The tests in `crates/clawhdf5-py/tests` compare every read with h5py; run them with diff --git a/crates/clawhdf5-format/src/data_read.rs b/crates/clawhdf5-format/src/data_read.rs index d10f381..f9e0ac1 100644 --- a/crates/clawhdf5-format/src/data_read.rs +++ b/crates/clawhdf5-format/src/data_read.rs @@ -286,9 +286,11 @@ pub fn read_raw_data_indexed( /// Read raw bytes for only the selected elements of a dataset. /// -/// For chunked layouts, only chunks that intersect the selection are read -/// and decompressed. For compact/contiguous layouts, the full data is read -/// and then the selection is extracted. +/// When the selection's bounding box covers at most half the dataset, only +/// that box is materialised — the overlapping rows of a contiguous dataset, +/// the overlapping chunks of a chunked one, whatever its chunk index (see +/// [`crate::partial_read`]). Otherwise, and for compact and virtual +/// layouts, the whole dataset is decoded and the selection extracted. #[allow(clippy::too_many_arguments)] pub fn read_raw_data_selection( file_data: &[u8], diff --git a/crates/clawhdf5-py/README.md b/crates/clawhdf5-py/README.md index 9cff9ab..95cae12 100644 --- a/crates/clawhdf5-py/README.md +++ b/crates/clawhdf5-py/README.md @@ -29,7 +29,7 @@ with clawhdf5.File("data.h5", "r") as f: f.keys(), f["group"].items(), "group/data" in f ds = f["group/data"] # or f["/group/data"], f["group"]["data"] ds.shape, ds.dtype, ds.attrs["units"] - ds[10:20, ::2] # only the selected elements are read + ds[10:20, ::2] # a small selection reads only its chunks ds[-1], ds[..., 0], ds[[1, 4, 7]] np.asarray(ds) f["table"]["id"] # a compound field @@ -45,9 +45,19 @@ with clawhdf5.File("data.h5", "r") as f: increasing list of integers, compound field names. Each maps onto a hyperslab selection. `None`, negative steps and boolean masks are refused with h5py's errors. +- What is read from the file: a selection whose bounding box covers at + most half the dataset decodes only the chunks (or contiguous rows) the box + overlaps. The library decodes the whole dataset for a larger box + (including a strided slice such as `ds[::100]` across a chunked dataset), + and for compact, virtual and unwritten datasets and chunked ones with a + non-default fill value. An index list is read one group of neighbouring + chunks at a time (a new group only past a chunk with no selected index), + so each chunk is decoded once. `ds[()]`, `ds[...]` and `np.asarray(ds)` + use the file's chunk cache; other selections do not. - The bytes the library reads become the numpy array's buffer without a copy, and the read runs with the GIL released, so threads read in - parallel. + parallel. A bug in the library (a Rust panic) raises + `clawhdf5.InternalError`, a `RuntimeError`. - Attributes return what h5py returns; `clawhdf5.Empty` stands for a null dataspace (h5py's `Empty`). diff --git a/crates/clawhdf5-py/src/dataset.rs b/crates/clawhdf5-py/src/dataset.rs index c20f293..bcfd852 100644 --- a/crates/clawhdf5-py/src/dataset.rs +++ b/crates/clawhdf5-py/src/dataset.rs @@ -1,7 +1,9 @@ //! PyDataset — h5py-style read access to HDF5 datasets. //! //! `ds[key]` parses the key into hyperslab selections (see `select`) and -//! reads only those elements through the facade's `read_selection`; the +//! reads them through the facade's `read_selection`, which decodes only the +//! chunks a small selection touches (see its docs for when it decodes the +//! whole dataset instead); the //! bytes it returns become the numpy array's buffer without a copy (see //! `convert`). All file access and decoding runs with the GIL released, so //! Python threads reading the same or different datasets run in parallel. @@ -24,7 +26,7 @@ use crate::{PyEmpty, node, to_py_err}; /// ```python /// ds = f['group/dataset'] /// ds.shape, ds.dtype, ds.attrs['units'] -/// block = ds[10:20, ::2] # reads only the selected elements +/// block = ds[10:20, ::2] # a small selection reads only its chunks /// ``` #[pyclass(name = "Dataset")] pub struct PyDataset { @@ -299,7 +301,8 @@ impl PyDataset { /// Read with h5py indexing: integers, slices with positive steps, /// `...`, one increasing list of integers, and compound field names. - /// Only the selected elements are read from the file. + /// A selection whose bounding box covers at most half the dataset reads + /// only the chunks (or contiguous rows) it overlaps. fn __getitem__<'py>( &self, py: Python<'py>, diff --git a/crates/clawhdf5-py/src/select.rs b/crates/clawhdf5-py/src/select.rs index 7c52442..7d11f3c 100644 --- a/crates/clawhdf5-py/src/select.rs +++ b/crates/clawhdf5-py/src/select.rs @@ -1,5 +1,7 @@ //! h5py-style indexing (`ds[1, 2:10:3, ...]`) mapped onto hyperslab -//! selections, so only the selected elements are read. +//! selections, so the library reads the selection rather than the whole +//! dataset (it still decodes everything for large selections; see the +//! facade's `Dataset::read_selection`). //! //! The rules and error messages follow h5py's `selections.py`: integers //! (negative from the end) drop their axis, slices must have a positive diff --git a/crates/clawhdf5/src/reader.rs b/crates/clawhdf5/src/reader.rs index 2e86dfe..c7d1fe6 100644 --- a/crates/clawhdf5/src/reader.rs +++ b/crates/clawhdf5/src/reader.rs @@ -525,8 +525,16 @@ impl<'f> Dataset<'f> { /// Read selected elements as raw bytes. /// - /// Only the elements matching the [`clawhdf5_format::selection::Selection`] are returned. For chunked - /// datasets, only intersecting chunks are decompressed. + /// Only the elements matching the [`clawhdf5_format::selection::Selection`] are returned. + /// + /// What is read to get them: when the selection's bounding box covers at + /// most half the dataset, only that box — the chunks overlapping it, or + /// the rows of a contiguous dataset. The whole dataset is decoded instead + /// when the box covers more than half (a strided selection spanning the + /// dataset does), for compact and virtual layouts, for a dataset with no + /// storage, and for a chunked dataset with a non-default fill value. + /// [`Selection::All`](clawhdf5_format::selection::Selection::All) goes + /// through the file's chunk cache; other selections do not. pub fn read_selection( &self, selection: &clawhdf5_format::selection::Selection, diff --git a/docs/known-issues.md b/docs/known-issues.md index bf8f77a..320ed2f 100644 --- a/docs/known-issues.md +++ b/docs/known-issues.md @@ -7,6 +7,23 @@ deleting it. --- +## Selection reads that decode more than the selection + +**Status:** open (documented 2026-09-26). `Dataset::read_selection` (and so +the Python `ds[...]`) materialises only the selection's bounding box when +that box covers at most half the dataset (`partial_read`). It decodes the +whole dataset and extracts the selection instead when: +- the bounding box covers more than half the dataset — which a strided + selection across a chunked dataset (`ds[::100]`) always does, although + it may touch few chunks; +- the dataset is compact or virtual, or has no storage; +- it is chunked with a non-default fill value (the box path does not fill + unallocated chunks, so the fill-aware full read is used). +Values are correct in every case; this is cost only. Selections other than +`Selection::All` also bypass the file's chunk cache. The bounding-box +heuristic's other cost is measured under "Concurrent and contiguous read +performance" below. + ## Concurrent and contiguous read performance (measured 2026-09-26) **Status:** open. Measured on tank with `concurrent_read` against h5py