Merge branch 'feat/p2-python-bindings' into feat/p2-perf-coverage

# Conflicts:
#	CHANGELOG.md
#	README.md
This commit is contained in:
osobh
2026-09-26 09:10:57 -05:00
26 changed files with 3541 additions and 568 deletions
+113
View File
@@ -230,6 +230,119 @@
are not byte-identical to earlier versions). Regression test:
`crates/clawhdf5/tests/writer_groups_interop.rs`.
### Python bindings (2026-09-26)
- **Panic: selections of v4 implicit-index chunked datasets** (pre-existing,
facade `Dataset::read_selection`, Rust callers too). A hyperslab whose
bounding box covered more than half of a chunked dataset with the implicit
index (`libver='latest'`, early allocation, no filters) panicked with
"index out of bounds" in `generate_implicit_chunks`: the fallback in
`data_read::read_raw_data_selection` passed the layout's chunk dimensions,
element-size dimension included, and then decoded the whole dataset
anyway. That arm now decodes and extracts directly, for every chunk index.
`crates/clawhdf5/tests/v4_chunk_index_selection.rs` reads small and large
hyperslabs of all five v4 indexes (single chunk, implicit, fixed array,
extensible array, B-tree v2) and compares them with h5py; it panicked
before. The Python bindings made this easy to reach (`ds[0:3]` on
libhdf5's `h5fc_ext*.h5` test files).
- **`pip install` / `maturin develop` now gives `import clawhdf5`.** The
distribution in `crates/clawhdf5-py/pyproject.toml` was still called
`rustyhdf5` while the extension module was `clawhdf5`, and the package's
tests imported `rustyhdf5`, so they failed at collection. Distribution,
module and tests now all say `clawhdf5`, and the module has
`__version__`.
- **h5py-style reads that read the selection, not the dataset.** `ds[...]`
used to read the whole dataset and slice it in numpy, and knew six
dtypes. Now
integers (negative from the end), slices with positive steps, `...`, one
increasing list of integers per key and compound field names map onto the
facade's hyperslab selection (a list is read one group of neighbouring
chunks at a time and picked from in memory), with h5py's results (numpy scalar for an all-integer
key, 0-d array for `scalar[...]`) and h5py's errors for everything else
(negative steps, `None`, boolean masks, out-of-range indices).
`Dataset.dtype` is the numpy dtype h5py reports, for every integer and
IEEE float width (incl. `float16`) in either byte order, `bool`, enums
(base integer with `metadata['enum']`), complex (`r`/`i` compounds),
fixed strings (`S<n>`), variable-length strings (`object` of `bytes`, as
h5py), variable-length sequences (`object` of arrays), opaque (`V<n>`),
HDF5 array types and compounds (numpy structured, offsets and padding
kept, nested). The bytes the library returns become the numpy array's
buffer without a copy. Types the mapping cannot describe exactly
(references, bitfields, time, non-IEEE floats, integers with padding
bits, variable-length members inside compounds) raise `TypeError` rather
than return guessed data. Attributes come back as h5py returns them
(numpy scalars and arrays with the stored dtype, `str` for
variable-length strings, `numpy.bytes_` for fixed ones — **a change**:
string attributes written by this package are fixed-length and used to
come back as `str` — and `clawhdf5.Empty` for a null dataspace, which
datasets return too). `Group`/`File` gain `get`, `values`, `items`,
iteration, `len`, `name`, absolute and relative paths (`g['/a/b']`,
`g['c/d']`, `f['/']`); `Dataset` gains `ndim`, `size`, `maxshape`,
`name`, `len()` and `numpy.asarray(ds)`. File access and decoding run
with the GIL released, so Python threads read in parallel.
`crates/clawhdf5-py/tests/test_read_vs_h5py.py` compares every read with
h5py 3.16 (HDF5 2.0) on a file h5py writes. One difference is h5py's:
it returns variable-length sequences of big-endian floats unswapped; this
package returns the stored values.
- **A panic in the library is an ordinary Python exception.** PyO3 turns a
Rust panic into `PanicException`, a `BaseException` that `except
Exception` does not catch. Every call from the bindings into the library
is now guarded and a panic becomes `clawhdf5.InternalError` (a
`RuntimeError`) naming the object; with the implicit-index panic above
restored, `ds[0:30]` raises it.
- **Wrong data: uninitialised padding in compound results of index lists.**
`ds[[0, 3, 6]]` joined one read per run with `np.concatenate`, which
copies structured dtypes field by field into an `np.empty` result, so the
padding bytes held whatever was in memory (pointers were seen) and leaked
through `tobytes()`, hashes and write-backs. The runs' bytes are now joined
in Rust, whole elements at a time, so the result carries the bytes read
from the file (h5py's, zero for files it wrote) and stays zero-copy.
The h5py comparisons now also compare every byte of structured values
(`test_compound_padding_bytes_match_h5py` and `assert_same`).
- **Index lists no longer decode the same chunks once per run.** A list
index was one uncached hyperslab read per run of consecutive indices, so
on a chunked, compressed dataset every run decoded its chunk again:
`d[list(range(0, 200000, 40))]` over 20 gzip chunks took 8 s (h5py:
0.014 s). The list is now read in groups — for a chunked dataset a group
ends only where a whole chunk holds no selected index, so each chunk is
decoded once; otherwise at a gap of more than 64 KiB — and the selected
rows are picked from each group in Rust. The same read now takes 3.8 ms
(h5py 4.1 ms; release build on tank, best of 5).
`test_a_long_index_list_decodes_each_chunk_once` compares 1-D, 2-D and
contiguous cases with h5py under a 2 s bound (5.8 s before, debug build).
- **Groups and datasets remember where they are.** Every `ds[...]`, and
every `g[k]`, resolved its path from the root again (two or three times
per open), and in a large group each resolution scans the group's links,
so visiting a group was quadratic: 4000 scalar datasets in one group took
39 s (`libver='earliest'`) and 131 s (`'latest'`) to list, read and
re-read in `test_big_groups_are_not_quadratic`; now 0.3 s each (debug
build). A `Dataset` keeps its object's address, and a `Group` (and the
file's root) its address and, once listed, its link table. New facade
API: `File::dataset_at(address)` opens a dataset without resolving a
path. libhdf5's `h5stat_newgrat.h5` (35001 members in the root): listing
takes 0.03 s and 2000 opens 1 ms (h5py: 0.022 s).
- **`ds[np.array(1)]` is an integer index**, as in h5py; a 0-d integer
array went down the index-list path and raised a confusing `TypeError`.
The h5py comparison keys now include 0-d arrays on every axis.
- **Tests that would notice a held GIL, and our extra errors.**
`test_reads_release_the_gil` times a Python thread spinning while another
reads: with the read made to hold the GIL it stalls for the whole read
(0.062 s of a 0.064 s read) and the test fails; released, its longest
stall is about 3 ms. (The existing threads test only checked values.)
`test_errors_match_h5py` now also requires that every key h5py reads
reads here too, with the same result, and covers more keys (0-d arrays,
repeated and empty lists, `()`, `...`).
- **Docs say when a selection reads more than itself.** The README and
the package README said `ds[...]` reads only the selected elements,
without condition. The library decodes the whole dataset when the
selection's bounding box covers more than half of it, and for compact,
virtual, unwritten and non-default-fill chunked datasets; the READMEs,
the facade's `read_selection` docs and `docs/known-issues.md` now say so.
- **CI builds and tests the Python package.** It was excluded from CI.
`scripts/ci-test.sh` now lints `clawhdf5-py`, builds the wheel with
maturin, unpacks it under `target/` and runs the pytest suite; skipped
without maturin/pytest in `$CLAWHDF5_PYTHON`, a failure then under
`CLAWHDF5_REQUIRE_INTEROP=1`. The CI interop venv installs both.
### Plugin filters (2026-09-26)
- **LZF, bitshuffle, bzip2 and Blosc read and write, in pure Rust.** Files
written by h5py with `compression="lzf"`, or with hdf5plugin's