feat(py): h5py-style reads of only the selected elements, GIL released
ds[key] read the whole dataset and sliced it in numpy, and knew six dtypes. Keys (ints, positive-step slices, Ellipsis, one increasing index list, compound field names) now map onto hyperslab selections, and the facade's read_selection bytes become the numpy buffer without a copy (PyArray::from_vec viewed as the dtype). dtype mapping follows h5py for all integer/IEEE float widths and byte orders, bool, enum, complex, fixed and variable-length strings, vlen sequences, opaque, array types and (nested, padded) compounds; anything it cannot describe exactly is a TypeError. Attributes return what h5py returns; groups and files gain the rest of the h5py mapping interface. Reads run under py.detach. tests/test_read_vs_h5py.py compares >500 reads with h5py 3.16 on an h5py-written file, checks errors match, that a damaged chunk outside the selection is never touched, and 8 threads reading at once. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
@@ -9,6 +9,38 @@
|
||||
tests imported `rustyhdf5`, so they failed at collection. Distribution,
|
||||
module and tests now all say `clawhdf5`, and the module has
|
||||
`__version__`.
|
||||
- **h5py-style reads that read only what is selected.** `ds[...]` used to
|
||||
read the whole dataset and slice it in numpy, and knew six dtypes. Now
|
||||
integers (negative from the end), slices with positive steps, `...`, one
|
||||
increasing list of integers per key and compound field names map onto the
|
||||
facade's hyperslab selection (a list becomes one hyperslab per run of
|
||||
consecutive indices), with h5py's results (numpy scalar for an all-integer
|
||||
key, 0-d array for `scalar[...]`) and h5py's errors for everything else
|
||||
(negative steps, `None`, boolean masks, out-of-range indices).
|
||||
`Dataset.dtype` is the numpy dtype h5py reports, for every integer and
|
||||
IEEE float width (incl. `float16`) in either byte order, `bool`, enums
|
||||
(base integer with `metadata['enum']`), complex (`r`/`i` compounds),
|
||||
fixed strings (`S<n>`), variable-length strings (`object` of `bytes`, as
|
||||
h5py), variable-length sequences (`object` of arrays), opaque (`V<n>`),
|
||||
HDF5 array types and compounds (numpy structured, offsets and padding
|
||||
kept, nested). The bytes the library returns become the numpy array's
|
||||
buffer without a copy. Types the mapping cannot describe exactly
|
||||
(references, bitfields, time, non-IEEE floats, integers with padding
|
||||
bits, variable-length members inside compounds) raise `TypeError` rather
|
||||
than return guessed data. Attributes come back as h5py returns them
|
||||
(numpy scalars and arrays with the stored dtype, `str` for
|
||||
variable-length strings, `numpy.bytes_` for fixed ones — **a change**:
|
||||
string attributes written by this package are fixed-length and used to
|
||||
come back as `str` — and `clawhdf5.Empty` for a null dataspace, which
|
||||
datasets return too). `Group`/`File` gain `get`, `values`, `items`,
|
||||
iteration, `len`, `name`, absolute and relative paths (`g['/a/b']`,
|
||||
`g['c/d']`, `f['/']`); `Dataset` gains `ndim`, `size`, `maxshape`,
|
||||
`name`, `len()` and `numpy.asarray(ds)`. File access and decoding run
|
||||
with the GIL released, so Python threads read in parallel.
|
||||
`crates/clawhdf5-py/tests/test_read_vs_h5py.py` compares every read with
|
||||
h5py 3.16 (HDF5 2.0) on a file h5py writes. One difference is h5py's:
|
||||
it returns variable-length sequences of big-endian floats unswapped; this
|
||||
package returns the stored values.
|
||||
|
||||
### Plugin filters (2026-09-26)
|
||||
- **LZF, bitshuffle, bzip2 and Blosc read and write, in pure Rust.** Files
|
||||
|
||||
Reference in New Issue
Block a user