docs: say when a selection read decodes more than the selection
The READMEs said ds[...] reads only the selected elements, and the facade's read_selection docs that only intersecting chunks are decompressed. The bounding-box path runs only when the box covers at most half the dataset; larger boxes (any strided slice across the dataset), compact, virtual and unwritten datasets and chunked ones with a non-default fill value decode the whole dataset. The READMEs, the facade and format docs, the bindings' docstrings and known-issues now say so, and how index lists are read. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
@@ -7,6 +7,23 @@ deleting it.
|
||||
|
||||
---
|
||||
|
||||
## Selection reads that decode more than the selection
|
||||
|
||||
**Status:** open (documented 2026-09-26). `Dataset::read_selection` (and so
|
||||
the Python `ds[...]`) materialises only the selection's bounding box when
|
||||
that box covers at most half the dataset (`partial_read`). It decodes the
|
||||
whole dataset and extracts the selection instead when:
|
||||
- the bounding box covers more than half the dataset — which a strided
|
||||
selection across a chunked dataset (`ds[::100]`) always does, although
|
||||
it may touch few chunks;
|
||||
- the dataset is compact or virtual, or has no storage;
|
||||
- it is chunked with a non-default fill value (the box path does not fill
|
||||
unallocated chunks, so the fill-aware full read is used).
|
||||
Values are correct in every case; this is cost only. Selections other than
|
||||
`Selection::All` also bypass the file's chunk cache. The bounding-box
|
||||
heuristic's other cost is measured under "Concurrent and contiguous read
|
||||
performance" below.
|
||||
|
||||
## Concurrent and contiguous read performance (measured 2026-09-26)
|
||||
|
||||
**Status:** open. Measured on tank with `concurrent_read` against h5py
|
||||
|
||||
Reference in New Issue
Block a user