osobhandClaude Opus 4.8
908af40282
A virtual layout previously returned UnsupportedVersion. Implement reading for the common 1-D, same-file case, reverse-engineered and validated against HDF5 2.0. - Rewrite parse_vds_mappings to the real global-heap block format (version(1) · nused(length_size) · entries · checksum(4)), where each entry is source-file(null) · source-dataset(null) · source-selection · virtual-selection. Block version 1 encodes a same-file source as a single 0x04 marker in place of the file name; version 0 stores an explicit file name. The selections are H5S-serialized and self-describing in length, so they are decoded to find entry boundaries. The previous parser used a guessed layout that did not match real files. - Extend Selection with decode_serialized() (H5S_select_serialize: ALL, NONE, and version-3 regular hyperslabs) and iter_linear_1d(). - Add read_virtual_data: resolve the mapping block from the global heap, read each same-file source dataset, and scatter its selected elements into the virtual buffer; unmapped regions stay at the zero fill value. External-file sources and N-D selections return a clean unsupported error. Tests: real-file integration test (vds_same_file.h5: partial source slice + fill gap), selection decoder unit tests built from the fixture bytes, and same-file/external mapping-parser unit tests. Co-Authored-By: Claude Opus 4.8 <[email protected]>
119 lines
6.5 KiB
Markdown
119 lines
6.5 KiB
Markdown
# Changelog
|
|
|
|
## Unreleased
|
|
|
|
### New Features
|
|
- `clawhdf5-format`: decode the **scale-offset filter** (id 6) — both the
|
|
integer variant (`H5Z_SO_INT`) and the floating-point **D-scale** variant
|
|
(`H5Z_SO_FLOAT_DSCALE`). Handles signed/unsigned int sizes, f32/f64, negative
|
|
minima, decimal scale factors and fill values; reverse-engineered against
|
|
HDF5 2.0 and validated end-to-end. The float E-scale variant remains
|
|
unsupported.
|
|
- `clawhdf5-format`: decode the **N-Bit filter** (id 5) — atomic, **compound**
|
|
and **array** layouts (the full recursive type tree, nestable to any depth),
|
|
previously unsupported. Signed and unsigned reduced-precision integers and
|
|
float members all read end-to-end, validated against HDF5 2.0.
|
|
|
|
### New Features
|
|
- `clawhdf5-format`: assemble **1-D, same-file Virtual Datasets (VDS)**.
|
|
Previously a virtual layout returned `UnsupportedVersion`. The reader now
|
|
decodes the global-heap mapping block (reverse-engineered against HDF5 2.0:
|
|
`version · nused · [source-file · source-dataset · source-selection ·
|
|
virtual-selection]* · checksum`, including the block-version-1 same-file
|
|
marker), decodes the `H5S` source/virtual dataspace **selections** (ALL,
|
|
NONE, and version-3 regular hyperslabs), reads each same-file source dataset,
|
|
and scatters its selected elements into the virtual buffer; unmapped regions
|
|
are left at the zero fill value. External-file sources and N-dimensional
|
|
selections return a clean unsupported error. The previous `parse_vds_mappings`
|
|
used a guessed layout that did not match real files and is replaced.
|
|
|
|
### Bug Fixes
|
|
- `clawhdf5-format`: read **paged Fixed Array** chunk indexes. A filtered,
|
|
fixed-dimension dataset with more than one data-block page (>1024 chunks by
|
|
default) previously failed with "paged Fixed Array data blocks not yet
|
|
supported". The reader now walks the page-init bitmap (MSB-first), skips
|
|
uninitialized pages, and resolves each page's fixed full-size slot (including
|
|
the short final page). Reverse-engineered and validated end-to-end against an
|
|
HDF5 2.0 file.
|
|
- `clawhdf5-format`: read **array-typed datatypes** (e.g. an array-typed
|
|
compound member) via `read_as_i32/i64/u64/f32/f64` — previously a
|
|
`TypeMismatch`. The array is read as a flat sequence of its base elements
|
|
(recursing for nested arrays), applying base-type precision rules.
|
|
- `clawhdf5-format`: **sign-extend reduced-precision fixed-point integers** on
|
|
read. A signed integer whose datatype precision is smaller than its storage
|
|
size is stored zero-filled, so e.g. a 16-bit-precision `-1` previously read as
|
|
`65535`. The integer read paths now extract the precision field and
|
|
sign-extend (full-width types are unchanged). Completes signed N-Bit reads and
|
|
also fixes un-filtered reduced-precision integer datasets.
|
|
- `clawhdf5-format`: read datasets written by modern HDF5 (1.14+/2.0, i.e.
|
|
`libver=latest`). Compound (class 6) and array (class 10) datatype **version 5**
|
|
messages and data layout **version 5** messages were rejected as invalid; they
|
|
reuse the v3/v4 binary structure, so they are now accepted. This unblocks
|
|
reading compound types and — critically — every chunked/compressed dataset
|
|
written by HDF5 2.0. Found by running the h5py interop tests against
|
|
h5py 3.16 / HDF5 2.0.
|
|
|
|
### Performance
|
|
- `clawhdf5-format`: chunked writes now compress all chunks up front via
|
|
`compress_all_chunks`, running across rayon threads under the `parallel`
|
|
feature when there are more than 4 filtered chunks. On-disk layout is
|
|
unchanged. Speeds up compressed embedding writes in `clawhdf5-agent` (which
|
|
enables `parallel`).
|
|
|
|
### Documentation
|
|
- Fix stale package names across all 13 per-crate READMEs (`rustyhdf5-*` /
|
|
`edgehdf5-*` → `clawhdf5-*`, usage versions → 2.1.0).
|
|
- Correct README workspace/test/crate stats and the CLAUDE.md CLI subcommand
|
|
list; document the `hnsw` and format compression/checksum feature flags and
|
|
the `entity_extract` / `async_memory` modules.
|
|
|
|
## v2.1.0 (2026-06-03)
|
|
|
|
### New Features
|
|
- `clawhdf5-agent`: HNSW now backs the vector stage of `hybrid_search`. The
|
|
`hnsw` feature is **on by default**, so semantic search uses the approximate
|
|
`clawhdf5-ann` index instead of a linear cosine scan. The index mirrors the
|
|
memory cache (node id == cache index) and self-heals — it rebuilds whenever it
|
|
drifts from the cache length, so no mutation path can desync it. Non-indexable
|
|
stores (no/zero-dim/mixed embeddings) and dimension-mismatched queries fall
|
|
back to the exact linear scan. Disable with
|
|
`--no-default-features --features float16` for exact search.
|
|
- `clawhdf5-ann`: HNSW is now a live, mutable index — added `insert`,
|
|
`mark_deleted` (soft-delete bitset; deleted nodes are traversed for
|
|
connectivity but never returned), `compact` (drops deleted vectors and
|
|
renumbers survivors), and `new` (empty index). Serialization gains a format
|
|
version tag (`HNSW_FORMAT_VERSION` = 2) and persists the deleted bitset;
|
|
pre-existing v1 files still load.
|
|
- `clawhdf5-agent`: `hybrid::merge_vector_keyword` exposes the shared
|
|
normalize-and-fuse step used by both the linear and HNSW vector paths.
|
|
- Expose `max_dimensions()` API on Dataset, MmapDataset, and LazyDataset
|
|
- NetCDF-4 unlimited dimension detection now works correctly
|
|
- Python bindings (`clawhdf5-py`) build and link on macOS with system Python
|
|
|
|
### Bug Fixes
|
|
- `clawhdf5-py`: upgrade PyO3 and numpy `0.23` → `0.28` so the bindings build on
|
|
Python 3.14 (PyO3 0.23 capped at 3.13 and hard-failed `cargo build
|
|
--workspace`). Updated for the removed `PyObject` alias (`Py<PyAny>`) and the
|
|
`Python::allow_threads` → `Python::detach` rename.
|
|
- Fix GPU L2 distance test (squared vs actual L2 mismatch in test helper)
|
|
- Mark Android JNI functions as `unsafe` for Rust 2024 edition compliance
|
|
- Add `# Safety` documentation to all public unsafe extern functions
|
|
- Fix all clippy warnings: needless_range_loop, manual_strip, ptr_arg, etc.
|
|
- Rename `RelationType::from_str` to `from_label` to avoid trait confusion
|
|
- Isolate h5py interop tests with `#[ignore]` when h5py unavailable
|
|
|
|
### Code Quality
|
|
- Full rustfmt pass across workspace (61 files)
|
|
- Refine inner unsafe blocks for Rust 2024 edition style
|
|
- Zero clippy warnings, zero clippy errors across entire workspace
|
|
- 1,546 tests passing, 0 failures
|
|
|
|
## v2.0.0 (2026-03-19)
|
|
|
|
- Unified rustyhdf5 (11 crates) and edgehdf5 (4 crates) into a single workspace
|
|
- All crates renamed to clawhdf5-* prefix
|
|
- Version bumped to 2.0.0 across all crates
|
|
- Git dependencies replaced with in-workspace path dependencies
|
|
- Added `agent` feature flag to clawhdf5-agent
|
|
|