From c3850a0b66748dbbf1291820682261302818c2b3 Mon Sep 17 00:00:00 2001 From: osobh Date: Sat, 26 Sep 2026 08:21:48 -0500 Subject: [PATCH] =?UTF-8?q?docs:=20the=20Python=20package=20=E2=80=94=20in?= =?UTF-8?q?stall=20with=20maturin,=20h5py-style=20reading?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit README gains a Python section (maturin develop into a venv, a reading example that was run against an h5py-written file, the supported types and keys, what writing covers). The crate README says the same in more detail. QUICKSTART showed clawhdf5.open()/read_f64(), which never existed; it now shows File(...)[...]. Co-Authored-By: Claude Opus 5.5 (1M context) --- README.md | 42 +++++++++++++++++++++++ crates/clawhdf5-py/README.md | 64 +++++++++++++++++++++++++++++++----- docs/QUICKSTART.md | 15 +++++---- 3 files changed, 107 insertions(+), 14 deletions(-) diff --git a/README.md b/README.md index b9e3aa1..c266f3b 100644 --- a/README.md +++ b/README.md @@ -407,6 +407,48 @@ let values = ds.read_f64()?; assert_eq!(values, vec![22.5, 23.1, 21.8]); ``` +### Python + +`crates/clawhdf5-py` is a Python package (PyO3 + numpy) that reads HDF5 with +an h5py-shaped API and no libhdf5. It is not on PyPI; build it with +[maturin](https://www.maturin.rs) into a virtualenv: + +```bash +python -m venv .venv && . .venv/bin/activate +pip install maturin numpy +maturin develop --release -m crates/clawhdf5-py/Cargo.toml +python -c "import clawhdf5; print(clawhdf5.__version__)" +``` + +```python +import numpy as np +import clawhdf5 + +with clawhdf5.File("data.h5", "r") as f: + print(list(f.keys())) # sorted member names, like h5py + ds = f["group/temperatures"] # relative or absolute ("/group/...") paths + print(ds.shape, ds.dtype) # dtype is the numpy dtype h5py reports + block = ds[100:200, ::4] # reads only the selected elements + row = ds[-1] # integers drop the axis + picked = ds[[1, 5, 9], :] # one increasing index list per key + units = ds.attrs["units"] # attributes come back as h5py returns them + everything = np.asarray(ds) + + records = f["table"] # compound -> numpy structured array + ids = records["id"] # one field +``` + +Reads cover integers and IEEE floats of every width in either byte order, +`bool`, enums, complex, fixed and variable-length strings, variable-length +sequences, opaque, HDF5 array types and compounds; other types (references, +bitfields, ...) raise `TypeError` instead of returning guessed data. Keys +follow h5py (negative steps, `None` and boolean masks are refused). The +read itself runs with the GIL released, so Python threads read in parallel. +Writing (`File(path, "w")`, `create_dataset`, `create_group`, `attrs[...] =`) +covers `float64`, `float32`, `int64`, `int32` and `uint8` arrays. The tests +in `crates/clawhdf5-py/tests` compare every read with h5py; run them with +`pip install pytest h5py && pytest crates/clawhdf5-py/tests`. + ### Agent Memory ```rust diff --git a/crates/clawhdf5-py/README.md b/crates/clawhdf5-py/README.md index 6554f32..9cff9ab 100644 --- a/crates/clawhdf5-py/README.md +++ b/crates/clawhdf5-py/README.md @@ -3,23 +3,71 @@ [![crates.io](https://img.shields.io/crates/v/clawhdf5-py.svg)](https://crates.io/crates/clawhdf5-py) [![docs.rs](https://docs.rs/clawhdf5-py/badge.svg)](https://docs.rs/clawhdf5-py) -Python bindings for clawhdf5 — a pure-Rust HDF5 library. +Python bindings for clawhdf5 — a pure-Rust HDF5 library. The package is +`clawhdf5` (`import clawhdf5`); it needs numpy and no libhdf5. -## Features +## Install -- h5py-compatible API (`File`, `Group`, `Dataset`) -- NumPy array integration -- Read and write HDF5 files from Python with no C dependencies +Not on PyPI yet. Build it into a virtualenv with [maturin](https://www.maturin.rs): -## Usage +```bash +pip install maturin numpy +cd crates/clawhdf5-py +maturin develop --release +python -c "import clawhdf5; print(clawhdf5.__version__)" +``` + +## Reading + +The read API follows h5py: ```python +import numpy as np import clawhdf5 -with clawhdf5.File('data.h5', 'r') as f: - data = f['/dataset'][:] +with clawhdf5.File("data.h5", "r") as f: + f.keys(), f["group"].items(), "group/data" in f + ds = f["group/data"] # or f["/group/data"], f["group"]["data"] + ds.shape, ds.dtype, ds.attrs["units"] + ds[10:20, ::2] # only the selected elements are read + ds[-1], ds[..., 0], ds[[1, 4, 7]] + np.asarray(ds) + f["table"]["id"] # a compound field ``` +- `Dataset.dtype` is the numpy dtype h5py reports: integers and IEEE floats + of every width in either byte order, `bool`, enums (with + `dtype.metadata['enum']`), complex, `S` fixed strings, `object` for + variable-length strings (`bytes` values) and sequences (array values), + `V` opaque, array types, and compounds as structured dtypes. + Other types raise `TypeError`. +- Keys are h5py's: integers, slices with a positive step, `...`, one + increasing list of integers, compound field names. Each maps onto a + hyperslab selection. `None`, negative steps and boolean masks are refused + with h5py's errors. +- The bytes the library reads become the numpy array's buffer without a + copy, and the read runs with the GIL released, so threads read in + parallel. +- Attributes return what h5py returns; `clawhdf5.Empty` stands for a null + dataspace (h5py's `Empty`). + +## Writing + +`clawhdf5.File(path, "w")` with `create_dataset(name, data=array, +chunks=..., compression="gzip")`, `create_group` and `attrs[...] = ...` +writes `float64`, `float32`, `int64`, `int32` and `uint8` arrays; the file is +written on `close()`. + +## Tests + +```bash +pip install pytest h5py +pytest crates/clawhdf5-py/tests +``` + +`tests/test_read_vs_h5py.py` compares every read with h5py on a file h5py +writes. `scripts/ci-test.sh` builds the wheel and runs these in CI. + ## License MIT diff --git a/docs/QUICKSTART.md b/docs/QUICKSTART.md index 2ede177..34b5325 100644 --- a/docs/QUICKSTART.md +++ b/docs/QUICKSTART.md @@ -395,19 +395,22 @@ clawhdf5 --path agent.h5 snapshot backup_2026-03-19.h5 Read HDF5 files from Python without libhdf5: ```bash -pip install clawhdf5 # coming soon — build from source for now -cd crates/clawhdf5-py && maturin develop +# Not on PyPI yet: build from source into a virtualenv +pip install maturin numpy +cd crates/clawhdf5-py && maturin develop --release ``` ```python import clawhdf5 -# Read -f = clawhdf5.open("data.h5") -temps = f.read_f64("temperatures") -print(temps) # [22.5, 23.1, 21.8] +# Read (h5py-style) +with clawhdf5.File("data.h5", "r") as f: + temps = f["temperatures"][:] + print(temps) # [22.5 23.1 21.8] ``` +See `crates/clawhdf5-py/README.md` for the supported types and indexing. + --- ## Common Patterns