Merge branch 'feat/p2-python-bindings' into feat/p2-perf-coverage
# Conflicts: # CHANGELOG.md # README.md
This commit is contained in:
@@ -73,8 +73,9 @@ breaking change, are in [CHANGELOG.md](CHANGELOG.md).
|
||||
- Default fusion weights are now the measured 0.4 / 0.6 (v2.5.0). Re-ranking had
|
||||
been discarding the retrieval score, costing the Markdown backend 40.6pp of
|
||||
Hit@1; fixed in v2.6.0.
|
||||
- Selection reads decode only the chunks they touch (a 64×64 window: 105 ms to
|
||||
0.39 ms), and full reads are 1.2–1.9× faster (v2.5.0).
|
||||
- Selection reads whose bounding box covers at most half the dataset decode
|
||||
only the chunks they touch (a 64×64 window: 105 ms to 0.39 ms), and full
|
||||
reads are 1.2–1.9× faster (v2.5.0).
|
||||
|
||||
**Memory**
|
||||
- A loaded store holds ~30% less (embeddings stored once, v2.6.0), and the
|
||||
@@ -432,6 +433,54 @@ b.write("groups.h5")?;
|
||||
A group holds at most 65 535 links; more is an error, as is a link over
|
||||
65 515 bytes (a very long soft-link target) in a group of more than 8 links.
|
||||
|
||||
### Python
|
||||
|
||||
`crates/clawhdf5-py` is a Python package (PyO3 + numpy) that reads HDF5 with
|
||||
an h5py-shaped API and no libhdf5. It is not on PyPI; build it with
|
||||
[maturin](https://www.maturin.rs) into a virtualenv:
|
||||
|
||||
```bash
|
||||
python -m venv .venv && . .venv/bin/activate
|
||||
pip install maturin numpy
|
||||
maturin develop --release -m crates/clawhdf5-py/Cargo.toml
|
||||
python -c "import clawhdf5; print(clawhdf5.__version__)"
|
||||
```
|
||||
|
||||
```python
|
||||
import numpy as np
|
||||
import clawhdf5
|
||||
|
||||
with clawhdf5.File("data.h5", "r") as f:
|
||||
print(list(f.keys())) # sorted member names, like h5py
|
||||
ds = f["group/temperatures"] # relative or absolute ("/group/...") paths
|
||||
print(ds.shape, ds.dtype) # dtype is the numpy dtype h5py reports
|
||||
block = ds[100:200, ::4] # a small selection reads only its chunks
|
||||
row = ds[-1] # integers drop the axis
|
||||
picked = ds[[1, 5, 9], :] # one increasing index list per key
|
||||
units = ds.attrs["units"] # attributes come back as h5py returns them
|
||||
everything = np.asarray(ds)
|
||||
|
||||
records = f["table"] # compound -> numpy structured array
|
||||
ids = records["id"] # one field
|
||||
```
|
||||
|
||||
Reads cover integers and IEEE floats of every width in either byte order,
|
||||
`bool`, enums, complex, fixed and variable-length strings, variable-length
|
||||
sequences, opaque, HDF5 array types and compounds; other types (references,
|
||||
bitfields, ...) raise `TypeError` instead of returning guessed data. Keys
|
||||
follow h5py (negative steps, `None` and boolean masks are refused). The
|
||||
read itself runs with the GIL released, so Python threads read in parallel.
|
||||
A selection whose bounding box covers at most half the dataset decodes only
|
||||
the chunks (or contiguous rows) that box overlaps; a larger one — including
|
||||
a strided slice across the whole dataset — decodes the whole dataset, as
|
||||
do datasets that are compact, virtual, unwritten, or chunked with a
|
||||
non-default fill value (`docs/known-issues.md`). An index list is read one
|
||||
group of neighbouring chunks at a time.
|
||||
Writing (`File(path, "w")`, `create_dataset`, `create_group`, `attrs[...] =`)
|
||||
covers `float64`, `float32`, `int64`, `int32` and `uint8` arrays. The tests
|
||||
in `crates/clawhdf5-py/tests` compare every read with h5py; run them with
|
||||
`pip install pytest h5py && pytest crates/clawhdf5-py/tests`.
|
||||
|
||||
### Agent Memory
|
||||
|
||||
```rust
|
||||
|
||||
Reference in New Issue
Block a user