feat(py): h5py-style reads of only the selected elements, GIL released

ds[key] read the whole dataset and sliced it in numpy, and knew six
dtypes. Keys (ints, positive-step slices, Ellipsis, one increasing index
list, compound field names) now map onto hyperslab selections, and the
facade's read_selection bytes become the numpy buffer without a copy
(PyArray::from_vec viewed as the dtype). dtype mapping follows h5py for
all integer/IEEE float widths and byte orders, bool, enum, complex, fixed
and variable-length strings, vlen sequences, opaque, array types and
(nested, padded) compounds; anything it cannot describe exactly is a
TypeError. Attributes return what h5py returns; groups and files gain
the rest of the h5py mapping interface. Reads run under py.detach.

tests/test_read_vs_h5py.py compares >500 reads with h5py 3.16 on an
h5py-written file, checks errors match, that a damaged chunk outside the
selection is never touched, and 8 threads reading at once.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-26 08:19:48 -05:00
co-authored by Claude Opus 5.5
parent 006bf3b131
commit 2d4b211523
12 changed files with 2305 additions and 385 deletions
+32
View File
@@ -9,6 +9,38 @@
tests imported `rustyhdf5`, so they failed at collection. Distribution,
module and tests now all say `clawhdf5`, and the module has
`__version__`.
- **h5py-style reads that read only what is selected.** `ds[...]` used to
read the whole dataset and slice it in numpy, and knew six dtypes. Now
integers (negative from the end), slices with positive steps, `...`, one
increasing list of integers per key and compound field names map onto the
facade's hyperslab selection (a list becomes one hyperslab per run of
consecutive indices), with h5py's results (numpy scalar for an all-integer
key, 0-d array for `scalar[...]`) and h5py's errors for everything else
(negative steps, `None`, boolean masks, out-of-range indices).
`Dataset.dtype` is the numpy dtype h5py reports, for every integer and
IEEE float width (incl. `float16`) in either byte order, `bool`, enums
(base integer with `metadata['enum']`), complex (`r`/`i` compounds),
fixed strings (`S<n>`), variable-length strings (`object` of `bytes`, as
h5py), variable-length sequences (`object` of arrays), opaque (`V<n>`),
HDF5 array types and compounds (numpy structured, offsets and padding
kept, nested). The bytes the library returns become the numpy array's
buffer without a copy. Types the mapping cannot describe exactly
(references, bitfields, time, non-IEEE floats, integers with padding
bits, variable-length members inside compounds) raise `TypeError` rather
than return guessed data. Attributes come back as h5py returns them
(numpy scalars and arrays with the stored dtype, `str` for
variable-length strings, `numpy.bytes_` for fixed ones — **a change**:
string attributes written by this package are fixed-length and used to
come back as `str` — and `clawhdf5.Empty` for a null dataspace, which
datasets return too). `Group`/`File` gain `get`, `values`, `items`,
iteration, `len`, `name`, absolute and relative paths (`g['/a/b']`,
`g['c/d']`, `f['/']`); `Dataset` gains `ndim`, `size`, `maxshape`,
`name`, `len()` and `numpy.asarray(ds)`. File access and decoding run
with the GIL released, so Python threads read in parallel.
`crates/clawhdf5-py/tests/test_read_vs_h5py.py` compares every read with
h5py 3.16 (HDF5 2.0) on a file h5py writes. One difference is h5py's:
it returns variable-length sequences of big-endian floats unswapped; this
package returns the stored values.
### Plugin filters (2026-09-26)
- **LZF, bitshuffle, bzip2 and Blosc read and write, in pure Rust.** Files