Files

T

osobhandClaude Opus 5.5 3c89a31df0 clawhdf5: FileEditor modifies existing files in place

New clawhdf5::FileEditor opens an HDF5 file (h5py-written at any libver,
HDF5 2.0 format included, or clawhdf5-written) under an exclusive flock
and changes only what an edit touches:
- write_selection/write_all/write_values: compact, contiguous (also
  late-allocated) and chunked datasets, any selection. Chunks are decoded,
  updated and re-encoded; a filtered chunk that no longer fits moves to the
  end of the file unless it is the file's last structure, which grows in
  place. New chunks go into v1 B-tree, Extensible Array (paged data blocks
  included), Fixed Array and single-chunk indexes, created on first use.
- resize: grow chunked datasets up to maxshape.
- set_attr: add/replace compact attributes, in a NIL slot or a new
  continuation chunk.
Each edit is planned in an in-memory image and refused whole
(Error::Unsupported) when any part is unsupported (v2 B-tree / implicit
new chunks, shrinking, vlen/reference data, dense or order-tracked
attributes, cache images, paged/persistent free space). Commit writes and
syncs new space before patching existing bytes. Layout v5 (HDF5 2.0)
array indexes use 8-byte filtered chunk sizes, as libhdf5 does.

Error gains Unsupported/InvalidArgument/Locked and is #[non_exhaustive];
the Python bindings map them. build_attr_message is public.

Tests (h5py, h5dump, h5rs check --data after every round; h5py r+
afterwards): appends crossing EA super/data blocks and B-tree splits, the
same B-tree node counts and EA statistics as libhdf5 for the same writes
(in order, reversed and shuffled; paged blocks), every layout and chunk
index overwritten under random selections, attributes to continuation
chunks, random operations against a model, refused edits leave the file
byte-identical, locking.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

2026-09-26 13:35:06 -05:00

src

clawhdf5: FileEditor modifies existing files in place

2026-09-26 13:35:06 -05:00

tests

test(py): detect a held GIL, and errors h5py does not raise

2026-09-26 09:03:53 -05:00

build.rs

Merge pull request 'docs(clawhdf5): document DType variants, fix unresolved doc links' (#17 ) from sdlc-docs/clawhdf5-types-20260514-165210 into main

2026-05-14 23:54:48 +00:00

Cargo.toml

fix(py): one name, clawhdf5, for the Python distribution and module

2026-09-26 08:09:54 -05:00

pyproject.toml

fix(py): one name, clawhdf5, for the Python distribution and module

2026-09-26 08:09:54 -05:00

README.md

docs: say when a selection read decodes more than the selection

2026-09-26 09:04:48 -05:00

README.md

clawhdf5-py

Python bindings for clawhdf5 — a pure-Rust HDF5 library. The package is clawhdf5 (import clawhdf5); it needs numpy and no libhdf5.

Install

Not on PyPI yet. Build it into a virtualenv with maturin:

pip install maturin numpy
cd crates/clawhdf5-py
maturin develop --release
python -c "import clawhdf5; print(clawhdf5.__version__)"

Reading

The read API follows h5py:

import numpy as np
import clawhdf5

with clawhdf5.File("data.h5", "r") as f:
    f.keys(), f["group"].items(), "group/data" in f
    ds = f["group/data"]           # or f["/group/data"], f["group"]["data"]
    ds.shape, ds.dtype, ds.attrs["units"]
    ds[10:20, ::2]                 # a small selection reads only its chunks
    ds[-1], ds[..., 0], ds[[1, 4, 7]]
    np.asarray(ds)
    f["table"]["id"]               # a compound field

Dataset.dtype is the numpy dtype h5py reports: integers and IEEE floats of every width in either byte order, bool, enums (with dtype.metadata['enum']), complex, S<n> fixed strings, object for variable-length strings (bytes values) and sequences (array values), V<n> opaque, array types, and compounds as structured dtypes. Other types raise TypeError.
Keys are h5py's: integers, slices with a positive step, ..., one increasing list of integers, compound field names. Each maps onto a hyperslab selection. None, negative steps and boolean masks are refused with h5py's errors.
What is read from the file: a selection whose bounding box covers at most half the dataset decodes only the chunks (or contiguous rows) the box overlaps. The library decodes the whole dataset for a larger box (including a strided slice such as ds[::100] across a chunked dataset), and for compact, virtual and unwritten datasets and chunked ones with a non-default fill value. An index list is read one group of neighbouring chunks at a time (a new group only past a chunk with no selected index), so each chunk is decoded once. ds[()], ds[...] and np.asarray(ds) use the file's chunk cache; other selections do not.
The bytes the library reads become the numpy array's buffer without a copy, and the read runs with the GIL released, so threads read in parallel. A bug in the library (a Rust panic) raises clawhdf5.InternalError, a RuntimeError.
Attributes return what h5py returns; clawhdf5.Empty stands for a null dataspace (h5py's Empty).

Writing

clawhdf5.File(path, "w") with create_dataset(name, data=array, chunks=..., compression="gzip"), create_group and attrs[...] = ... writes float64, float32, int64, int32 and uint8 arrays; the file is written on close().

Tests

pip install pytest h5py
pytest crates/clawhdf5-py/tests

tests/test_read_vs_h5py.py compares every read with h5py on a file h5py writes. scripts/ci-test.sh builds the wheel and runs these in CI.

License

MIT