feat(py): h5py-style reads of only the selected elements, GIL released

ds[key] read the whole dataset and sliced it in numpy, and knew six
dtypes. Keys (ints, positive-step slices, Ellipsis, one increasing index
list, compound field names) now map onto hyperslab selections, and the
facade's read_selection bytes become the numpy buffer without a copy
(PyArray::from_vec viewed as the dtype). dtype mapping follows h5py for
all integer/IEEE float widths and byte orders, bool, enum, complex, fixed
and variable-length strings, vlen sequences, opaque, array types and
(nested, padded) compounds; anything it cannot describe exactly is a
TypeError. Attributes return what h5py returns; groups and files gain
the rest of the h5py mapping interface. Reads run under py.detach.

tests/test_read_vs_h5py.py compares >500 reads with h5py 3.16 on an
h5py-written file, checks errors match, that a damaged chunk outside the
selection is never touched, and 8 threads reading at once.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-26 08:19:48 -05:00
co-authored by Claude Opus 5.5
parent 006bf3b131
commit 2d4b211523
12 changed files with 2305 additions and 385 deletions
+3 -3
View File
@@ -87,7 +87,7 @@ def test_dataset_dtype(sample_read_file):
def test_read_root_attrs(sample_read_file):
with clawhdf5.File(sample_read_file, "r") as f:
assert f.attrs["version"] == 1
assert f.attrs["description"] == "test file"
assert f.attrs["description"] == b"test file" # fixed-length string: numpy.bytes_, as in h5py
def test_attrs_len(sample_read_file):
@@ -132,7 +132,7 @@ def test_read_group_dataset(grouped_read_file):
def test_read_group_attrs(grouped_read_file):
with clawhdf5.File(grouped_read_file, "r") as f:
grp = f["sensors"]
assert grp.attrs["location"] == "lab"
assert grp.attrs["location"] == b"lab"
def test_nested_path_access(grouped_read_file):
@@ -176,7 +176,7 @@ def test_write_with_attrs(tmp_h5):
f.attrs["author"] = "test"
f.attrs["count"] = 42
with clawhdf5.File(tmp_h5, "r") as f:
assert f.attrs["author"] == "test"
assert f.attrs["author"] == b"test"
assert f.attrs["count"] == 42