Implements decompression for the N-Bit filter (id 5), atomic integer/float
variant — previously returned UnsupportedFilter(5). N-Bit packs each element's
significant `precision` bits MSB-first with no header; decode reads those bits
per element and places them at the datatype's bit offset in a zero-filled
`size`-byte element, reproducing HDF5's canonical reduced-precision layout
(verified byte-for-byte against the equivalent un-filtered dataset).
Reverse-engineered and validated against HDF5 2.0 / h5py: unsigned
reduced-precision datasets now read end-to-end with exact values. Signed
reduced-precision values are restored to their canonical (zero-filled) bytes;
sign-extending them to the application width is the datatype reader's job — a
pre-existing concern shared with un-filtered reduced-precision data. Recursive
compound/array N-Bit layouts remain unsupported. Adds h5py-free unit tests from
real captured chunks.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Extends the scale-offset filter (id 6) decoder to the floating-point D-scale
variant (H5Z_SO_FLOAT_DSCALE) alongside the integer variant. Shares the header
parsing and MSB-first code unpacking; reconstruction is
`value = minval + code / 10^scale_factor`, where `minval` is the minimum float
stored in the header and the all-ones code is the (defined) fill value.
Reverse-engineered and validated against HDF5 2.0 / h5py across f32 and f64,
negatives, decimal scale factors D=1..5 and multi-chunk datasets (decoded values
match h5py to full precision). The float E-scale variant remains unsupported.
Adds h5py-free unit tests from real captured chunks.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Implements decompression for the scale-offset filter (id 6), integer mode
(H5Z_SO_INT) — previously returned UnsupportedFilter(6). The on-disk format was
reverse-engineered against HDF5 2.0 / h5py and verified across signed/unsigned
element sizes, negative minima, multi-chunk datasets and fill-value handling:
minbits (u32 LE) | 0x08 | minval (8 bytes LE) | 8 reserved bytes |
MSB-first packed codes (nelmts * minbits bits)
Each code is `value - minval`; the all-ones code is reserved for the (defined)
fill value. The floating-point variants (D-scale/E-scale) use a different
algorithm and remain reported as unsupported.
Validated end-to-end (a 200-element chunked scale-offset dataset, plus negative
and unsigned datasets, now decode to the exact h5py values). Adds h5py-free unit
tests using real captured compressed chunks.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Follow-up to the v5 compound fix, found by an interop sweep over diverse
h5py/HDF5 2.0 (libver=latest) datasets:
- Array datatype (class 10) version 5 was rejected. v3/v4/v5 share the same
array encoding, so the parser now accepts 3-5.
- Data Layout message version 5 was rejected, which broke EVERY chunked/
compressed dataset written by modern HDF5. v5 reuses the v4 message
structure, so it now routes through parse_v4.
Validated end-to-end: a gzip-compressed, Fixed-Array-indexed v5 chunked dataset
now decodes to the correct values. Adds h5py-free regression tests using the
real v5 array-datatype and chunked-layout bytes, and updates the layout
invalid-version test to use v6.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
clawhdf5 rejected datatype message version 5 for the compound class with
"invalid datatype version 5 for class 6", so it could not read compound
datasets written by modern HDF5 / h5py with libver=latest. v5 reuses the same
compact member encoding as v3/v4 (name, variable-width offset, member type), so
the parser now accepts versions 3-5 for compound.
Found by running the previously-ignored h5py interop tests against h5py 3.16 /
HDF5 2.0. Adds an h5py-free regression test using the real v5 datatype bytes.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
build_chunked_data_at_ext now compresses all chunks via compress_all_chunks
(previously dead code) before laying them out, so compression runs across
rayon threads under the `parallel` feature when there are >4 filtered chunks.
Layout is unchanged — compression preserves chunk order, so on-disk bytes are
identical to the sequential path. The agent crate enables `parallel`, so this
speeds up compressed embedding writes.
Removes the #[allow(dead_code)] on compress_all_chunks and gates
PARALLEL_COMPRESS_THRESHOLD behind the `parallel` feature.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Sweep of the docs after the v2.0.0 rename and recent changes:
- Per-crate READMEs (13 files): rename leftover rustyhdf5-*/edgehdf5-*
package names and badges to clawhdf5-*, bump usage versions to 2.1.0.
- README: update stale test badge (417 -> 1500+), workspace stats
(15 crates/72K -> 17 crates/84K), agent crate stats (20.7K/32 modules),
and add the missing clawhdf5-napi and clawhdf5-bench crates to the tree.
- CLAUDE.md: correct the CLI subcommand list (inspect/dump/index/search ->
the actual create/save/search/recall/stats/flush-wal/agents-md/export/snapshot).
No code changes. Verified there are zero todo!()/unimplemented!() macros and
no TODO/FIXME comments in the tree.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Bumps [workspace.package] and all 17 crate package versions (and internal
path-dependency requirements) from 2.0.0 to 2.1.0 for a coordinated release.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Lands the changes from PR #1, whose server-side squash merge landed as an
empty commit (Gitea merge working-tree error). See branch
feat/hnsw-agent-integration-py314 for the full history.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Resolves two gaps found in a project-state review:
1. Python build was broken: PyO3/numpy 0.23 caps at Python 3.13 but the
environment has 3.14. Bumped to 0.28 and updated the two breaking APIs
(PyObject -> Py<PyAny>, allow_threads -> detach). The extension module now
imports and round-trips under Python 3.14, unblocking cargo build --workspace.
2. The "HNSW vector search over agent memories" headline was unwired:
clawhdf5-ann had zero dependents and the agent used a linear cosine+BM25 scan.
- clawhdf5-ann is now a live index: insert, mark_deleted (soft delete with a
deleted bitset, traversed but never returned), compact, and a format
version tag (v2) with backward-compatible load of v1 files.
- clawhdf5-agent wires HNSW behind the `hnsw` feature (ON by default). The
index mirrors the cache (node id == cache index) and self-heals: it rebuilds
whenever hnsw_synced_len drifts from cache.len(), so unhooked pushes can't
desync it. Non-indexable stores (no/zero-dim/mixed embeddings) and queries
whose dim doesn't match fall back to the exact linear scan.
- hybrid.rs gains merge_vector_keyword, shared by the linear and HNSW paths.
- tests/hnsw_integration.rs validates recall vs a brute-force oracle plus
insert/delete/batch behaviour.
Disable HNSW for exact search with `--no-default-features --features float16`.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
