Fast contiguous and concurrent reads, VL data, nested groups and links, Python bindings #15

Merged
osobh merged 41 commits from feat/p2-perf-coverage into main 2026-09-26 14:57:01 +00:00
3 changed files with 107 additions and 14 deletions
Showing only changes of commit c3850a0b66 - Show all commits
+42
View File
@@ -407,6 +407,48 @@ let values = ds.read_f64()?;
assert_eq!(values, vec![22.5, 23.1, 21.8]); assert_eq!(values, vec![22.5, 23.1, 21.8]);
``` ```
### Python
`crates/clawhdf5-py` is a Python package (PyO3 + numpy) that reads HDF5 with
an h5py-shaped API and no libhdf5. It is not on PyPI; build it with
[maturin](https://www.maturin.rs) into a virtualenv:
```bash
python -m venv .venv && . .venv/bin/activate
pip install maturin numpy
maturin develop --release -m crates/clawhdf5-py/Cargo.toml
python -c "import clawhdf5; print(clawhdf5.__version__)"
```
```python
import numpy as np
import clawhdf5
with clawhdf5.File("data.h5", "r") as f:
print(list(f.keys())) # sorted member names, like h5py
ds = f["group/temperatures"] # relative or absolute ("/group/...") paths
print(ds.shape, ds.dtype) # dtype is the numpy dtype h5py reports
block = ds[100:200, ::4] # reads only the selected elements
row = ds[-1] # integers drop the axis
picked = ds[[1, 5, 9], :] # one increasing index list per key
units = ds.attrs["units"] # attributes come back as h5py returns them
everything = np.asarray(ds)
records = f["table"] # compound -> numpy structured array
ids = records["id"] # one field
```
Reads cover integers and IEEE floats of every width in either byte order,
`bool`, enums, complex, fixed and variable-length strings, variable-length
sequences, opaque, HDF5 array types and compounds; other types (references,
bitfields, ...) raise `TypeError` instead of returning guessed data. Keys
follow h5py (negative steps, `None` and boolean masks are refused). The
read itself runs with the GIL released, so Python threads read in parallel.
Writing (`File(path, "w")`, `create_dataset`, `create_group`, `attrs[...] =`)
covers `float64`, `float32`, `int64`, `int32` and `uint8` arrays. The tests
in `crates/clawhdf5-py/tests` compare every read with h5py; run them with
`pip install pytest h5py && pytest crates/clawhdf5-py/tests`.
### Agent Memory ### Agent Memory
```rust ```rust
+56 -8
View File
@@ -3,23 +3,71 @@
[![crates.io](https://img.shields.io/crates/v/clawhdf5-py.svg)](https://crates.io/crates/clawhdf5-py) [![crates.io](https://img.shields.io/crates/v/clawhdf5-py.svg)](https://crates.io/crates/clawhdf5-py)
[![docs.rs](https://docs.rs/clawhdf5-py/badge.svg)](https://docs.rs/clawhdf5-py) [![docs.rs](https://docs.rs/clawhdf5-py/badge.svg)](https://docs.rs/clawhdf5-py)
Python bindings for clawhdf5 — a pure-Rust HDF5 library. Python bindings for clawhdf5 — a pure-Rust HDF5 library. The package is
`clawhdf5` (`import clawhdf5`); it needs numpy and no libhdf5.
## Features ## Install
- h5py-compatible API (`File`, `Group`, `Dataset`) Not on PyPI yet. Build it into a virtualenv with [maturin](https://www.maturin.rs):
- NumPy array integration
- Read and write HDF5 files from Python with no C dependencies
## Usage ```bash
pip install maturin numpy
cd crates/clawhdf5-py
maturin develop --release
python -c "import clawhdf5; print(clawhdf5.__version__)"
```
## Reading
The read API follows h5py:
```python ```python
import numpy as np
import clawhdf5 import clawhdf5
with clawhdf5.File('data.h5', 'r') as f: with clawhdf5.File("data.h5", "r") as f:
data = f['/dataset'][:] f.keys(), f["group"].items(), "group/data" in f
ds = f["group/data"] # or f["/group/data"], f["group"]["data"]
ds.shape, ds.dtype, ds.attrs["units"]
ds[10:20, ::2] # only the selected elements are read
ds[-1], ds[..., 0], ds[[1, 4, 7]]
np.asarray(ds)
f["table"]["id"] # a compound field
``` ```
- `Dataset.dtype` is the numpy dtype h5py reports: integers and IEEE floats
of every width in either byte order, `bool`, enums (with
`dtype.metadata['enum']`), complex, `S<n>` fixed strings, `object` for
variable-length strings (`bytes` values) and sequences (array values),
`V<n>` opaque, array types, and compounds as structured dtypes.
Other types raise `TypeError`.
- Keys are h5py's: integers, slices with a positive step, `...`, one
increasing list of integers, compound field names. Each maps onto a
hyperslab selection. `None`, negative steps and boolean masks are refused
with h5py's errors.
- The bytes the library reads become the numpy array's buffer without a
copy, and the read runs with the GIL released, so threads read in
parallel.
- Attributes return what h5py returns; `clawhdf5.Empty` stands for a null
dataspace (h5py's `Empty`).
## Writing
`clawhdf5.File(path, "w")` with `create_dataset(name, data=array,
chunks=..., compression="gzip")`, `create_group` and `attrs[...] = ...`
writes `float64`, `float32`, `int64`, `int32` and `uint8` arrays; the file is
written on `close()`.
## Tests
```bash
pip install pytest h5py
pytest crates/clawhdf5-py/tests
```
`tests/test_read_vs_h5py.py` compares every read with h5py on a file h5py
writes. `scripts/ci-test.sh` builds the wheel and runs these in CI.
## License ## License
MIT MIT
+9 -6
View File
@@ -395,19 +395,22 @@ clawhdf5 --path agent.h5 snapshot backup_2026-03-19.h5
Read HDF5 files from Python without libhdf5: Read HDF5 files from Python without libhdf5:
```bash ```bash
pip install clawhdf5 # coming soon — build from source for now # Not on PyPI yet: build from source into a virtualenv
cd crates/clawhdf5-py && maturin develop pip install maturin numpy
cd crates/clawhdf5-py && maturin develop --release
``` ```
```python ```python
import clawhdf5 import clawhdf5
# Read # Read (h5py-style)
f = clawhdf5.open("data.h5") with clawhdf5.File("data.h5", "r") as f:
temps = f.read_f64("temperatures") temps = f["temperatures"][:]
print(temps) # [22.5, 23.1, 21.8] print(temps) # [22.5 23.1 21.8]
``` ```
See `crates/clawhdf5-py/README.md` for the supported types and indexing.
--- ---
## Common Patterns ## Common Patterns