Merge branch 'feat/p2-python-bindings' into feat/p2-perf-coverage
# Conflicts: # CHANGELOG.md # README.md
This commit is contained in:
+113
@@ -230,6 +230,119 @@
|
||||
are not byte-identical to earlier versions). Regression test:
|
||||
`crates/clawhdf5/tests/writer_groups_interop.rs`.
|
||||
|
||||
### Python bindings (2026-09-26)
|
||||
- **Panic: selections of v4 implicit-index chunked datasets** (pre-existing,
|
||||
facade `Dataset::read_selection`, Rust callers too). A hyperslab whose
|
||||
bounding box covered more than half of a chunked dataset with the implicit
|
||||
index (`libver='latest'`, early allocation, no filters) panicked with
|
||||
"index out of bounds" in `generate_implicit_chunks`: the fallback in
|
||||
`data_read::read_raw_data_selection` passed the layout's chunk dimensions,
|
||||
element-size dimension included, and then decoded the whole dataset
|
||||
anyway. That arm now decodes and extracts directly, for every chunk index.
|
||||
`crates/clawhdf5/tests/v4_chunk_index_selection.rs` reads small and large
|
||||
hyperslabs of all five v4 indexes (single chunk, implicit, fixed array,
|
||||
extensible array, B-tree v2) and compares them with h5py; it panicked
|
||||
before. The Python bindings made this easy to reach (`ds[0:3]` on
|
||||
libhdf5's `h5fc_ext*.h5` test files).
|
||||
- **`pip install` / `maturin develop` now gives `import clawhdf5`.** The
|
||||
distribution in `crates/clawhdf5-py/pyproject.toml` was still called
|
||||
`rustyhdf5` while the extension module was `clawhdf5`, and the package's
|
||||
tests imported `rustyhdf5`, so they failed at collection. Distribution,
|
||||
module and tests now all say `clawhdf5`, and the module has
|
||||
`__version__`.
|
||||
- **h5py-style reads that read the selection, not the dataset.** `ds[...]`
|
||||
used to read the whole dataset and slice it in numpy, and knew six
|
||||
dtypes. Now
|
||||
integers (negative from the end), slices with positive steps, `...`, one
|
||||
increasing list of integers per key and compound field names map onto the
|
||||
facade's hyperslab selection (a list is read one group of neighbouring
|
||||
chunks at a time and picked from in memory), with h5py's results (numpy scalar for an all-integer
|
||||
key, 0-d array for `scalar[...]`) and h5py's errors for everything else
|
||||
(negative steps, `None`, boolean masks, out-of-range indices).
|
||||
`Dataset.dtype` is the numpy dtype h5py reports, for every integer and
|
||||
IEEE float width (incl. `float16`) in either byte order, `bool`, enums
|
||||
(base integer with `metadata['enum']`), complex (`r`/`i` compounds),
|
||||
fixed strings (`S<n>`), variable-length strings (`object` of `bytes`, as
|
||||
h5py), variable-length sequences (`object` of arrays), opaque (`V<n>`),
|
||||
HDF5 array types and compounds (numpy structured, offsets and padding
|
||||
kept, nested). The bytes the library returns become the numpy array's
|
||||
buffer without a copy. Types the mapping cannot describe exactly
|
||||
(references, bitfields, time, non-IEEE floats, integers with padding
|
||||
bits, variable-length members inside compounds) raise `TypeError` rather
|
||||
than return guessed data. Attributes come back as h5py returns them
|
||||
(numpy scalars and arrays with the stored dtype, `str` for
|
||||
variable-length strings, `numpy.bytes_` for fixed ones — **a change**:
|
||||
string attributes written by this package are fixed-length and used to
|
||||
come back as `str` — and `clawhdf5.Empty` for a null dataspace, which
|
||||
datasets return too). `Group`/`File` gain `get`, `values`, `items`,
|
||||
iteration, `len`, `name`, absolute and relative paths (`g['/a/b']`,
|
||||
`g['c/d']`, `f['/']`); `Dataset` gains `ndim`, `size`, `maxshape`,
|
||||
`name`, `len()` and `numpy.asarray(ds)`. File access and decoding run
|
||||
with the GIL released, so Python threads read in parallel.
|
||||
`crates/clawhdf5-py/tests/test_read_vs_h5py.py` compares every read with
|
||||
h5py 3.16 (HDF5 2.0) on a file h5py writes. One difference is h5py's:
|
||||
it returns variable-length sequences of big-endian floats unswapped; this
|
||||
package returns the stored values.
|
||||
- **A panic in the library is an ordinary Python exception.** PyO3 turns a
|
||||
Rust panic into `PanicException`, a `BaseException` that `except
|
||||
Exception` does not catch. Every call from the bindings into the library
|
||||
is now guarded and a panic becomes `clawhdf5.InternalError` (a
|
||||
`RuntimeError`) naming the object; with the implicit-index panic above
|
||||
restored, `ds[0:30]` raises it.
|
||||
- **Wrong data: uninitialised padding in compound results of index lists.**
|
||||
`ds[[0, 3, 6]]` joined one read per run with `np.concatenate`, which
|
||||
copies structured dtypes field by field into an `np.empty` result, so the
|
||||
padding bytes held whatever was in memory (pointers were seen) and leaked
|
||||
through `tobytes()`, hashes and write-backs. The runs' bytes are now joined
|
||||
in Rust, whole elements at a time, so the result carries the bytes read
|
||||
from the file (h5py's, zero for files it wrote) and stays zero-copy.
|
||||
The h5py comparisons now also compare every byte of structured values
|
||||
(`test_compound_padding_bytes_match_h5py` and `assert_same`).
|
||||
- **Index lists no longer decode the same chunks once per run.** A list
|
||||
index was one uncached hyperslab read per run of consecutive indices, so
|
||||
on a chunked, compressed dataset every run decoded its chunk again:
|
||||
`d[list(range(0, 200000, 40))]` over 20 gzip chunks took 8 s (h5py:
|
||||
0.014 s). The list is now read in groups — for a chunked dataset a group
|
||||
ends only where a whole chunk holds no selected index, so each chunk is
|
||||
decoded once; otherwise at a gap of more than 64 KiB — and the selected
|
||||
rows are picked from each group in Rust. The same read now takes 3.8 ms
|
||||
(h5py 4.1 ms; release build on tank, best of 5).
|
||||
`test_a_long_index_list_decodes_each_chunk_once` compares 1-D, 2-D and
|
||||
contiguous cases with h5py under a 2 s bound (5.8 s before, debug build).
|
||||
- **Groups and datasets remember where they are.** Every `ds[...]`, and
|
||||
every `g[k]`, resolved its path from the root again (two or three times
|
||||
per open), and in a large group each resolution scans the group's links,
|
||||
so visiting a group was quadratic: 4000 scalar datasets in one group took
|
||||
39 s (`libver='earliest'`) and 131 s (`'latest'`) to list, read and
|
||||
re-read in `test_big_groups_are_not_quadratic`; now 0.3 s each (debug
|
||||
build). A `Dataset` keeps its object's address, and a `Group` (and the
|
||||
file's root) its address and, once listed, its link table. New facade
|
||||
API: `File::dataset_at(address)` opens a dataset without resolving a
|
||||
path. libhdf5's `h5stat_newgrat.h5` (35001 members in the root): listing
|
||||
takes 0.03 s and 2000 opens 1 ms (h5py: 0.022 s).
|
||||
- **`ds[np.array(1)]` is an integer index**, as in h5py; a 0-d integer
|
||||
array went down the index-list path and raised a confusing `TypeError`.
|
||||
The h5py comparison keys now include 0-d arrays on every axis.
|
||||
- **Tests that would notice a held GIL, and our extra errors.**
|
||||
`test_reads_release_the_gil` times a Python thread spinning while another
|
||||
reads: with the read made to hold the GIL it stalls for the whole read
|
||||
(0.062 s of a 0.064 s read) and the test fails; released, its longest
|
||||
stall is about 3 ms. (The existing threads test only checked values.)
|
||||
`test_errors_match_h5py` now also requires that every key h5py reads
|
||||
reads here too, with the same result, and covers more keys (0-d arrays,
|
||||
repeated and empty lists, `()`, `...`).
|
||||
- **Docs say when a selection reads more than itself.** The README and
|
||||
the package README said `ds[...]` reads only the selected elements,
|
||||
without condition. The library decodes the whole dataset when the
|
||||
selection's bounding box covers more than half of it, and for compact,
|
||||
virtual, unwritten and non-default-fill chunked datasets; the READMEs,
|
||||
the facade's `read_selection` docs and `docs/known-issues.md` now say so.
|
||||
- **CI builds and tests the Python package.** It was excluded from CI.
|
||||
`scripts/ci-test.sh` now lints `clawhdf5-py`, builds the wheel with
|
||||
maturin, unpacks it under `target/` and runs the pytest suite; skipped
|
||||
without maturin/pytest in `$CLAWHDF5_PYTHON`, a failure then under
|
||||
`CLAWHDF5_REQUIRE_INTEROP=1`. The CI interop venv installs both.
|
||||
|
||||
### Plugin filters (2026-09-26)
|
||||
- **LZF, bitshuffle, bzip2 and Blosc read and write, in pure Rust.** Files
|
||||
written by h5py with `compression="lzf"`, or with hdf5plugin's
|
||||
|
||||
Reference in New Issue
Block a user