osobhandClaude Opus 5.5
f99587c27d
Checked with h5py on a patched file:
- a VL string with an embedded NUL reads up to the NUL (libhdf5 converts
VL strings to C strings); read_vl_strings returned "a\0b";
- an element whose global heap object is not length x base size bytes is
an error ("Expected global heap object size does not match"); we
returned the object cut to the length;
- a heap address of 0 is a null element whatever its length.
vl_data::VlResolver does this, caching each parsed heap collection:
read_vl_strings parsed the whole collection again for every element.
read_vl_strings and read_vl_bytes use it; check_element_size refuses a VL
type whose stored element size is not 4 + offset size + 4. The
conformance probe resolves VL values through VlResolver instead of its
own lenient copy (575 of 697, unchanged).
The new unit tests fail against the old read_vl_strings.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
1681 lines
108 KiB
Markdown
1681 lines
108 KiB
Markdown
# Changelog
|
||
|
||
## Unreleased
|
||
|
||
### Variable-length data (2026-09-26)
|
||
- **VL values in files with 4-byte offsets** (`sizeof_addr = 4`). A VL
|
||
string attribute came back as `AttrValue::Raw`, a VL member of a compound
|
||
failed with `GlobalHeapObjectNotFound`, and VL datasets failed with a
|
||
size mismatch. Two causes: `Datatype::type_size()` reported 16 for every
|
||
VL type (the element is 4 + offset size + 4 bytes: 12 here), and the
|
||
global heap was parsed without the padding libhdf5 puts after its
|
||
collection and object headers (`H5HG_SIZEOF_HDR`/`H5HG_SIZEOF_OBJHDR`
|
||
round up to 8), so with 4-byte lengths every object was looked up 4
|
||
bytes early. `Datatype::VariableLength` now carries the element `size`
|
||
stored in the datatype message (**breaking** for code that builds or
|
||
exhaustively destructures that variant; patterns with `..` are
|
||
unaffected), and it is written back as stored. Tested against h5py
|
||
(`crates/clawhdf5/tests/vl_offset4_interop.rs`).
|
||
- **Wrong data: VL strings with an embedded NUL, and VL elements whose heap
|
||
object has the wrong size.** libhdf5 hands VL strings over as C strings,
|
||
so h5py reads `"a\0b"` as `"a"`; `read_vl_strings` returned the NUL and
|
||
what followed. An element whose heap object is not exactly
|
||
`length × base size` bytes is refused by libhdf5 ("Expected global heap
|
||
object size does not match"); we returned the object cut or padded to
|
||
the length. Both now behave as libhdf5, and a heap address of 0 is a null
|
||
element (empty) whatever its length. The new
|
||
`clawhdf5_format::vl_data::VlResolver` does this and parses each global
|
||
heap collection once per read: `read_vl_strings` parsed the whole
|
||
collection again for every element. `vl_data::check_element_size` refuses
|
||
a VL datatype whose stored size is not 4 + offset size + 4 (libhdf5
|
||
ignores the stored size). The conformance probe resolves VL elements
|
||
with `VlResolver` too; conformance unchanged at 575 of 697.
|
||
|
||
### Plugin filters (2026-09-26)
|
||
- **LZF, bitshuffle, bzip2 and Blosc read and write, in pure Rust.** Files
|
||
written by h5py with `compression="lzf"`, or with hdf5plugin's
|
||
`Bitshuffle`, `BZip2` and `Blosc`, failed with `UnsupportedFilter`. New
|
||
`clawhdf5-format`/`clawhdf5` features: `lzf` (32000, **on by default**, no
|
||
dependencies), `bitshuffle` (32008: transpose only, LZ4 and Zstandard
|
||
modes), `bzip2` (307), `blosc` (32001: Blosc 1 frames with BloscLZ,
|
||
LZ4/LZ4HC, Snappy, Zlib and Zstandard codecs and byte/bit shuffle;
|
||
BloscLZ is decoded by a port of c-blosc 1.21's decoder, and cannot be
|
||
written), and `plugin-filters` for all four. None compiles C: Zstandard is
|
||
ruzstd, bzip2 is libbz2-rs-sys. Write with `DatasetBuilder::with_lzf()`,
|
||
`with_bitshuffle(..)`, `with_bzip2(..)`, `with_blosc(..)` or
|
||
`with_plugin_filter(PluginFilter::..)`; `ChunkOptions` gains a `plugin`
|
||
field (**breaking** for code that builds `ChunkOptions` with a struct
|
||
literal and no `..Default::default()`). Tested both ways against h5py 3.16
|
||
+ hdf5plugin 7.1 over 1-3-D shapes with partial edge chunks, 1-8-byte
|
||
types in both byte orders and incompressible data
|
||
(`crates/clawhdf5/tests/plugin_filters_interop.rs`). Conformance: 573 of
|
||
697 files ok (was 569) — h5ex_d_lzf/bshuf/bzip2/blosc.
|
||
- **Filter registry.** Filters are looked up by ID in
|
||
`clawhdf5_format::filter_registry` instead of a `match`: the built-in
|
||
table (per build), then codecs registered at run time with
|
||
`register_filter(id, codec)` — a decoding closure or a `FilterCodec` that
|
||
can also encode. Built-in IDs cannot be overridden; a registered decoder's
|
||
output is held to the chunk-size bound. Unknown IDs still fail with
|
||
`UnsupportedFilter(id)`, whose message now names known filters and the
|
||
missing feature ("unsupported filter: 32026 (Blosc2, not implemented by
|
||
clawhdf5)").
|
||
- **Not implemented:** Blosc2 (32026) and ZFP (32013) remain a clear error.
|
||
- **Wrong data: a chunk that decodes short read as zeros** (pre-existing, every
|
||
filter). HDF5 stores every chunk at the full chunk size, so a filter
|
||
pipeline that decodes to fewer bytes means a corrupt chunk; every chunk
|
||
reader (full, cached, selection, parallel, partial) padded it with zeros.
|
||
It is now an error naming the chunk ("chunk at [16] decoded to 16 bytes,
|
||
expected 32"), via the new `filters::decompress_chunk_exact`. libhdf5
|
||
returns the rest of such a chunk uninitialised, or fails when the filter
|
||
checks. A Blosc frame declaring no data for a non-empty chunk is an error
|
||
too. Legitimate edge chunks are unaffected (they are stored full-size,
|
||
filtered or not); conformance is unchanged at 573 of 697, with no file
|
||
changing class.
|
||
- **Crash: a hostile Blosc chunk panicked** in builds with overflow checks
|
||
(debug builds, `cargo test`, `maturin develop`): a frame size below the
|
||
16-byte header underflowed. It is now an error. Every new decoder (LZF,
|
||
bitshuffle, bzip2, Blosc/BloscLZ) is fuzzed with random and mutated frames
|
||
in the unit tests.
|
||
- **`register_filter(32023, ..)` works with the `pcodec` feature.** 32023 is
|
||
Granular BitRound's ID; the built-in entry there only reads clawhdf5
|
||
<= 2.7.0's pcodec chunks (filter name `"pcodec"`), so a registered codec now
|
||
handles every other chunk with that ID, and writes. It was refused as
|
||
"built in".
|
||
|
||
### Upgrade Notes
|
||
- **HDF5 correctness audit (2026-09-25).** A sweep of 686 public files (the
|
||
libhdf5 test files, the HDF Group's CVE reproducers, pyfive, netcdf-c,
|
||
netcdf4-python, h5wasm, h5py and xarray corpora), a 567-case read matrix and
|
||
a 96-case write matrix against HDF5 1.10–2.0 found bugs that returned wrong
|
||
values with no error, and files we wrote that libhdf5 rejects. The fixes are
|
||
listed under Correctness and Interop. What changes for callers:
|
||
- **Chunked datasets whose max shape is larger than their current shape**,
|
||
or whose unlimited dimension is not the first, were indexed by the current
|
||
shape instead of the max shape, both when read and when written. Files from
|
||
libhdf5 now read correctly. Files clawhdf5 wrote with such a max shape were
|
||
laid out wrongly and now read the way libhdf5 always read them — rewrite
|
||
them. Agent stores and ClawBrainHub files have no max shape and are
|
||
unaffected.
|
||
- Integer reads (`read_i32`/`read_i64`/`read_u64`/...) of float data now
|
||
convert (truncate toward zero, saturate at the type's range, NaN reads as
|
||
0) instead of returning the IEEE bit pattern, and out-of-range integers
|
||
saturate instead of keeping the low bits.
|
||
- `FileWriter::finish()` now returns an error instead of writing a corrupt
|
||
file for: a header message over 64 KiB (e.g. an attribute larger than
|
||
~64 KiB), a group/dataset/link name that is empty, `.` or contains `/`
|
||
(nested paths were written as one literal link), a max shape smaller than
|
||
the shape, a page size outside 512 B–1 GiB, and more than 65 535 chunks in
|
||
a dataset with several unlimited dimensions.
|
||
- **Breaking (format crate):** `ObjectHeaderWriter::serialize`,
|
||
`BatchObjectHeaderWriter::compute_sizes`/`serialize_all` and
|
||
`build_chunked_data_from_precompressed` return `Result`;
|
||
`read_fixed_array_chunks`/`read_extensible_array_chunks` take `max_dims`;
|
||
`build_fixed_array_at`/`ea_writer::build_extensible_array_at` take one
|
||
`Option<WrittenChunk>` per index slot; `fill_value::dataset_fill_value`
|
||
returns `UnresolvedSharedMessage` for a shared message it cannot resolve
|
||
instead of `None`. `FillTime::default()` is `IfSet` (libhdf5's default;
|
||
default files are byte-identical).
|
||
- **ZeroClaw does not use clawhdf5.** The project described itself as
|
||
ZeroClaw's memory backend ("imported as a `clawhdf5` Cargo feature"). Checked
|
||
against ZeroClaw v0.8.5 (the latest release), the `osobh/zeroclaw` fork and
|
||
their full history: no such feature or backend has ever existed. And
|
||
`clawhdf5-migrate`'s "ZeroClaw layout" (`memory_chunks`, `sessions`,
|
||
`entities`, `relations`) is not ZeroClaw's schema — ZeroClaw uses a single
|
||
`memories` table — so the migrator cannot read a ZeroClaw database. The
|
||
claims are withdrawn; the migrator's layout is documented as its own.
|
||
- **OpenClaw is not supported, and never was.** The docs described a
|
||
"drop-in" OpenClaw memory backend enabled with `memory.backend = "clawhdf5"`.
|
||
That config was never valid in any OpenClaw release (v2026.2–v2026.7
|
||
accepted only `builtin`/`qmd` and rejected unknown keys, so a Gateway given
|
||
it refuses to start; OpenClaw 2.0 removed the key), no plugin was ever built,
|
||
and `@redclaw/clawhdf5` was never published. The integration docs
|
||
(`openclaw-integration.md`, `openclaw-config.md`, `migration-guide.md`) are
|
||
removed; `docs/openclaw.md` explains the status and what a real plugin would
|
||
need against OpenClaw v2026.9.6. `ClawhdfBackend` stays as a library API.
|
||
- **Breaking:** `MemoryError` is now `#[non_exhaustive]` and gained
|
||
`SigningKeyRequired`; a `match` on it needs a wildcard arm. Future variants
|
||
will no longer be breaking.
|
||
- **Breaking:** `clawhdf5-agent`'s `agent` feature is removed. It enabled
|
||
nothing — the agent layer is always built — but the README and guides told
|
||
people to pass it; drop `agent` from `features = [...]`.
|
||
- **`clawhdf5-migrate` now writes a real agent store.** Its output used to be
|
||
a layout of its own (`/chunks`, `/sessions`, `/entities`, `/relations`, no
|
||
`/meta`) that `HDF5Memory::open` rejected, so a migrated file could not be
|
||
used as agent memory. Files it wrote before this release are not agent
|
||
stores; re-run the migration. Also: embeddings default to `float16` like
|
||
any new store (`--f32` opts out; `--float16` is a hidden no-op); a row with
|
||
the wrong embedding length is an error instead of being truncated or
|
||
padded; `--incremental` now matches rows by content against an existing
|
||
store and follows the source's deleted flags; a source with no memory rows
|
||
needs `--embedding-dim`. The per-dataset SHA-256 provenance attributes of
|
||
the old layout are gone (the agent schema has no place for them).
|
||
- **Files written by clawhdf5 now open in h5py and libhdf5.** Every `f32`
|
||
dataset we wrote — including every agent store's embeddings — was refused
|
||
with "sign bit position out of bounds", and every empty dataset with
|
||
"invalid dataset size". Both were write-side bugs present in every release;
|
||
clawhdf5's own reader was unaffected. An agent store is rewritten in full at
|
||
each checkpoint, so it becomes readable at its next checkpoint on this
|
||
version; other files with `f32` or empty datasets need rewriting. Details in
|
||
`docs/known-issues.md`.
|
||
- **New stores store embeddings as half precision by default.**
|
||
`MemoryConfig::float16` was persisted and otherwise ignored; it now writes
|
||
`float16` embeddings (48% smaller files at 100K) and rounds each embedding
|
||
to half precision as it is saved — and it defaults to `true` for new
|
||
stores. On the full LongMemEval haystack with real MiniLM embeddings every
|
||
retrieval metric matched `f32`. **Existing stores are unaffected**: every
|
||
agent store has recorded `float16 = false`, and keeps it (a v2.5.0 fixture
|
||
guards this). A store that already had `float16 = true` rounds its
|
||
embeddings when next opened and writes them as `float16` at its next
|
||
checkpoint. Opt out with `float16 = false` or `create --f32`; the CLI's
|
||
`--float16` is still accepted and now a no-op. Values beyond ±65504 are
|
||
refused, so keep `f32` for unnormalised vectors.
|
||
- **Breaking:** `MemoryError` gained `InvalidEntry`, returned when a
|
||
`float16` store is given an embedding value beyond ±65504. Exhaustive
|
||
matches need the new arm.
|
||
- **The default build no longer compiles any C.** Deflate now defaults to the
|
||
pure-Rust zlib-rs instead of zlib-ng, so building the core crates needs
|
||
neither cmake nor a C compiler. Speed on HDF5 reads and writes is within 6%
|
||
of zlib-ng, and compressed output is byte-identical. To keep zlib-ng, enable
|
||
`fast-deflate` (on `clawhdf5`, `clawhdf5-format` or `clawhdf5-filters`); it
|
||
overrides zlib-rs wherever it is on.
|
||
- **A truncated deflate chunk is now an error.** It used to read back short,
|
||
with no error.
|
||
- **Minimum supported Rust is 1.92**, now declared in every crate's
|
||
`rust-version` and checked in CI.
|
||
- **New stores use the int8 vector index by default.**
|
||
`MemoryConfig::quantized_index` now defaults to `true`: a quarter of the
|
||
index memory, builds 1.8x (x86-64) and 2.3x (Raspberry Pi 5) faster, and
|
||
searches 1.63x and 1.18x faster at equal recall, measured on every
|
||
configuration tested. **Existing stores are unaffected** — a store written
|
||
with v2.6.0 or later keeps its persisted setting, and one written before the
|
||
setting existed opens as `false` and keeps its f32 index. Set
|
||
`quantized_index = false`, or pass `create --f32-index` to the CLI, to opt
|
||
out. The CLI's `--quantized-index` is still accepted but is now a no-op.
|
||
|
||
### Tools
|
||
- New crate **`clawhdf5-tools`** with the binary **`h5rs`**: HDF5
|
||
command-line tools without libhdf5, built only on the `clawhdf5` facade
|
||
and `clawhdf5-format` (no C, so it also builds as a static musl binary).
|
||
- `h5rs ls [-r] [-v] FILE[/path]` lists objects like h5ls (its first two
|
||
columns are h5ls's text on the test files) plus the datatype; `-v` adds
|
||
address, link count, layout and chunk index, chunk size, storage,
|
||
filters, datatype and attributes.
|
||
- `h5rs dump [--json] [-A] [-p] [-d PATH] FILE` prints DDL text that is
|
||
byte-identical to h5dump 1.14.6's (and to Debian's 1.14.5, which CI
|
||
uses) on the test files (all layouts and
|
||
chunk indexes, v1/v2 groups, compound, enum, strings, links, named
|
||
types, attributes; null-padded strings show their NULs at any depth),
|
||
or JSON in the HDF Group's hdf5-json layout (schema in the crate
|
||
README). Nested compounds print inline and `long double` values as
|
||
errors (exit 1); both are listed in the README.
|
||
- `h5rs stat FILE` reports h5stat's object, link, rank, layout, filter,
|
||
attribute, raw-data and file-size figures (equal to h5stat's on the test
|
||
files); metadata space is one figure, not broken down.
|
||
- `h5rs diff [-r] [-q] [-n N] [-d D] [-p R] [--follow-symlinks] A B [OBJ1
|
||
[OBJ2]]` (option names as h5diff's: `-c` is `--compare`, the count is
|
||
`-n`/`--count=N`) compares objects, kinds, datatypes, shapes, attributes, values and link
|
||
targets; exit status 0/1/2 as h5diff's. Soft links are compared by
|
||
target path, as h5diff's default, or with `--follow-symlinks` by the
|
||
objects they lead to (external links are never followed). Every path is
|
||
compared, including every name of a hard-linked object and the members
|
||
of a hard-linked group; with a `-d`/`-p` tolerance, integers are
|
||
compared exactly in integer arithmetic (no loss above 2^53), and a `-p`
|
||
below the f64 epsilon compares exactly, as h5diff's. Objects that cannot
|
||
be compared count as a difference (h5diff exits 0 for them), and NaN
|
||
equals NaN.
|
||
- `h5rs check [--data] FILE` is a structural validator: it walks every
|
||
object, parses every header message, verifies the checksums of every
|
||
version 2+ structure it meets (superblock, object headers and
|
||
continuation chunks, v2 B-tree nodes, fractal heap headers and — which
|
||
the library's reads do not — every direct and indirect heap block, and
|
||
extensible/fixed array chunk indexes), checks each chunk index against
|
||
its dataset (aligned, in-extent, unique, plausibly sized chunks), and
|
||
that raw data lies inside the file without overlaps. Every problem is
|
||
printed with its address; exit 1 when there are any. libhdf5's h5check
|
||
reads only the 1.8 format. On the conformance corpus it passes all 418
|
||
files that both clawhdf5 and h5py read in full, and `check --data` flags
|
||
134 of the 150 CVE and fuzzer files of the `cve_hdf5` corpus (tank,
|
||
2026-09-26). `--data` also follows variable-length data into its global
|
||
heap collections and reports a damaged one at its address. It inherits
|
||
the library's tolerance, though: 9 of the 16 it passes are files h5dump
|
||
1.14.6 rejects (see `docs/known-issues.md`, header checks).
|
||
- Values over `--max-bytes` (default 1 GiB) are reported instead of read;
|
||
a panic is caught and reported as an internal error (exit 3).
|
||
`scripts/h5rs-fuzz.sh` runs every subcommand over a corpus (default the
|
||
CVE reproducers, optionally with byte-flipped copies) with overflow
|
||
checks, a timeout and a memory limit, and fails on any panic, crash or
|
||
hang; `scripts/h5rs-check-ok-files.sh` runs `check --data` over the
|
||
fully-read conformance files.
|
||
- Because the library does not verify fractal heap block checksums when
|
||
it reads a dense group's links or dense attributes, `h5rs` verifies a
|
||
heap's blocks before reading from it and refuses a damaged one, as
|
||
libhdf5 does, instead of printing what the damaged block holds.
|
||
|
||
### Signing
|
||
- `clawhdf5-agent`: **Ed25519-signed checkpoints** — the README's
|
||
"cryptographically verifiable memory", now true. With
|
||
`HDF5Memory::set_signing_key(key)`, every checkpoint stores a signed
|
||
manifest: a SHA-256 per record (text, embedding as stored, channel,
|
||
timestamp, session, tags, deleted flag, activation) in a Merkle tree, plus
|
||
hashes of the settings (and WAL mark), sessions and knowledge graph, with
|
||
the per-record hashes in `/integrity/record_hashes`.
|
||
`HDF5Memory::verify(path, &public_key)` recomputes everything from the file
|
||
and reports which part changed and which records (`changed_records`); a
|
||
forged manifest fails the signature. The key is never persisted; a signed
|
||
store refuses to checkpoint without it (`MemoryError::SigningKeyRequired`),
|
||
and `remove_signature()` is the deliberate way back to unsigned. Saves still
|
||
in the WAL are not covered (`wal_entries_unsigned`). Tests include every
|
||
kind of edit, and an edit made with h5py in place, which verify pinpoints.
|
||
Cost: ~20% of a checkpoint, 32 bytes per record (`BENCHMARKS.md`, "Signed
|
||
checkpoints"). New dependencies `ed25519-dalek`, `sha2`, `rand_core` — pure
|
||
Rust; the no-C check still passes.
|
||
- `clawhdf5-cli`: `keygen --out <file>` (owner-only key file),
|
||
`--signing-key <file>` / `CLAWHDF5_SIGNING_KEY` on writing commands
|
||
(`create` signs immediately), `verify --public-key <hex|file>` (JSON report;
|
||
exit status 2 if not valid), and `signed` in `create`/`stats` output.
|
||
|
||
### Migration
|
||
- `clawhdf5-migrate`: writes through the agent's own API (`HDF5Memory::create`
|
||
/ `open`, `save_batch`, the session cache and knowledge graph), so there is
|
||
no second copy of the schema. Sessions and entities/relations carry over;
|
||
deleted rows become deleted records (or are left out with
|
||
`--skip-deleted`). Every source row is checked before the output is created,
|
||
so a source that cannot be migrated leaves an existing store untouched.
|
||
Validation reads the result back with `HDF5Memory::open_read_only`, compares
|
||
every field (embeddings bit for bit — `round_to_f16` of the source for a
|
||
`float16` store) and checks that a migrated record is found by search. The
|
||
`half`-based conversion is gone; `clawhdf5_format::float16` is the only one.
|
||
42 tests, including h5py opening a migrated store; an adversarial review's
|
||
two blocker and four major findings are fixed with regression tests.
|
||
- `clawhdf5-agent`: `HDF5Memory::sessions()` / `sessions_mut()`,
|
||
`HDF5Memory::delete_batch(&[usize])` (one save, all-or-nothing, never
|
||
auto-compacts), `SessionCache::add_at`, and `SessionCache` / `SessionEntry`
|
||
re-exported from the crate root.
|
||
|
||
### Search
|
||
- `clawhdf5-agent`: **`HDF5Memory::search` with `SearchOptions`** — source
|
||
filtering, re-ranking and confidence rejection in the store's own search
|
||
path. Re-ranking and confidence rejection used to be reachable only
|
||
through the OpenClaw backend, which now calls `search` with both on.
|
||
- `with_sources([..])` restricts a search to records from those source
|
||
channels. It applies before ranking, so a filtered search still returns up
|
||
to `k` results, normalised over what it can return. Measured at 100K: the
|
||
exact filtered top 10 for filters keeping 50%, 10% and 1% of the store and
|
||
for records far from the query, and never slower than an unfiltered search
|
||
(2.3 ms for a 1% filter vs 4.6 ms unfiltered). See `BENCHMARKS.md`,
|
||
"Search options".
|
||
- `with_rerank(ReRankConfig)` re-ranks a pool of `max(3k, 10)` candidates
|
||
(`rerank_pool` to change it) by relevance, recency, source authority and
|
||
activation; `with_confidence(ConfidenceConfig)` drops low-confidence
|
||
results; `at_time(now)` pins the clock for recency. About 3% on latency.
|
||
- `hybrid_search` and `hybrid_search_with` are unchanged (tested bit for
|
||
bit against `search` with default options).
|
||
- `clawhdf5-agent`: the OpenClaw backend's search now boosts the Hebbian
|
||
activation of the `k` results it returns, not of the whole `3k` candidate
|
||
pool it re-ranks.
|
||
|
||
### Documentation
|
||
- OpenClaw claims withdrawn across the README, QUICKSTART, USE_CASES, ROADMAP
|
||
(Track 7 marked withdrawn) and the `openclaw` module docs; the dead
|
||
`github.com/redclawsystems/openclaw` link is gone. The Node package is
|
||
marked unpublished and broken (now `"private": true` so it cannot be
|
||
published by accident), with its bugs recorded in `docs/known-issues.md`.
|
||
|
||
### Benchmarks
|
||
- Every undated or pre-September section of `BENCHMARKS.md` re-run on one
|
||
machine on one day (tank, 2026-09-24, commit 5c8323c), with the command for
|
||
each and every number traced back to the raw output by a separate check.
|
||
Where a figure moved, the section says so. Two apparent regressions were
|
||
isolated rather than published: knowledge-graph traversal (a real bug,
|
||
fixed above) and the write path, which measures the same at v2.3.0 on this
|
||
machine — the old 18 µs / 6.17 ms figures came from an undated run on other
|
||
hardware; `float16` adds ~2 µs per save and the int8 index nothing.
|
||
- New `multimodal_bench`: cross-modal search at 1K and 10K records, which the
|
||
README claimed but nothing measured.
|
||
- `footprint_bench` reports whether it built `float16` or `f32` stores and
|
||
takes `--f32`; it had kept printing "f32" after the default changed.
|
||
- New `concurrent_read` harness, with an h5py counterpart
|
||
(`crates/clawhdf5-bench/scripts/concurrent_read_h5py.py`, threads or
|
||
processes) and `compare_concurrent_read.py`: decoded read throughput and
|
||
scaling efficiency at 1-16 threads on one open file, full reads of distinct
|
||
datasets and random hyperslabs of one dataset, deflate and contiguous, warm
|
||
or `--cold` page cache, JSON output. Not yet measured — `BENCHMARKS.md`
|
||
("Concurrent reads") has the commands and no numbers.
|
||
|
||
### Interop
|
||
- **h5py could not open chunked datasets we wrote with a chunk dimension
|
||
from 65 536 to 16 777 215.** A version-4 layout must store its chunk
|
||
dimensions in the fewest bytes that hold the largest (3 for 70 000);
|
||
the writer rounded 3 up to 4, and HDF5 2.0.0 (h5py 3.16) refuses that
|
||
("stored chunk dimension encoding length does not match value calculated
|
||
from chunk dimensions"). Newer libhdf5 and clawhdf5 read those files; new
|
||
files use the exact width. Test: `we_write_chunk_dimensions_in_the_fewest_bytes`.
|
||
- **Conformance sweep in the repo** (`conformance/`, report in
|
||
`CONFORMANCE.md`). `conformance/run.sh` fetches eight public HDF5 corpora
|
||
pinned by commit (libhdf5's test files, the HDF Group's CVE reproducers,
|
||
pyfive, netcdf-c, netcdf4-python, h5wasm, h5py, xarray-data) into a
|
||
gitignored cache, reads every file with clawhdf5 and with h5py/libhdf5 (and
|
||
the CVE files with h5dump) under a timeout and memory limit, compares them
|
||
object by object and regenerates the report — about 30 s once the corpus is
|
||
cached. A nightly Gitea job (`.gitea/workflows/conformance.yml`) runs it and
|
||
fails on any panic, hang, crash or out-of-memory, or when a file in
|
||
`conformance/baseline.json` stops reading identically. First report, on
|
||
42b81d9: 467 of 697 files identical to h5py, 123 our-error, 15 mismatch
|
||
(2 of them an h5py bug), 92 that libhdf5 cannot read, no panics, hangs or
|
||
crashes. Compared with the ad-hoc audit sweep, the probe now compares
|
||
N-Bit floats (and integers with a bit offset) as the values libhdf5
|
||
converts them to rather than raw file bytes — 8 files that were reported as
|
||
mismatches read identically — and the reference side no longer flips
|
||
between runs when libhdf5 aborts while freeing h5py objects.
|
||
- `clawhdf5-format`: **every `f32` dataset was unreadable by h5py and
|
||
libhdf5.** The float datatype encoder hard-coded the sign bit's position to
|
||
63, correct only for `f64`; libhdf5 validates it and refused the dataset. It
|
||
is now derived from the type (15 / 31 / 63). Our reader ignores the field,
|
||
and the interop suites only wrote `f64`, which is how it went unnoticed.
|
||
- `clawhdf5-format`: **every empty dataset was unreadable by h5py and
|
||
libhdf5.** It was written with a real address and zero bytes, which trips
|
||
libhdf5's `addr + size <= addr` overflow check. An empty contiguous dataset
|
||
now gets the undefined address, as libhdf5 writes it. This affected every
|
||
agent store without sessions or a knowledge graph.
|
||
- New interop tests: `f32` and `float16` datasets in both directions (our
|
||
`float16` rounding matches numpy's bit for bit on 4 020 probe values,
|
||
including ties, subnormals and the overflow boundary), and an agent store —
|
||
`f32` and `float16` — opened by h5py with every dataset decoded.
|
||
- `clawhdf5-format` filters, checked against libhdf5 + hdf5plugin:
|
||
- **LZ4 (32004) now uses the registered HDF5 LZ4 format** (8-byte BE size,
|
||
4-byte BE block size, BE-length-prefixed blocks). Our old framing (4-byte
|
||
LE size + one block) was readable only by clawhdf5, and we could not read
|
||
libhdf5's (`h5ex_d_lz4.h5`). Old clawhdf5 LZ4 chunks still read; they are
|
||
told apart unambiguously (a registered chunk starts with four zero bytes).
|
||
- **Zstd (32015) frames now record the content size**, which libhdf5's zstd
|
||
plugin needs; h5py could not read our zstd datasets.
|
||
- **Pcodec moved from filter ID 32023 to 480.** 32023 is registered to
|
||
Granular BitRound, whose decode is a pass-through — libhdf5 with that
|
||
plugin would have returned compressed bytes as data. Pcodec has no
|
||
registered ID; 480 is in the registry's private range (256–511) and only
|
||
clawhdf5 can read it. Chunks written under 32023 with the filter name
|
||
`pcodec` (clawhdf5 ≤ 2.7.0) still read.
|
||
- **SZIP decode matches libhdf5.** It returned garbage or zeros with no
|
||
error for libhdf5-written files (the 4-byte size prefix, 32/64-bit
|
||
byte-plane interleaving, reference interval, scanline padding and byte
|
||
order were all handled wrongly) and rejected 64-bit data.
|
||
- N-Bit honours libhdf5's "need not compress" flag (multi-filter pipelines
|
||
such as `tfilters.h5` failed) and reads enum/no-op members.
|
||
- Scale-offset `float` decode uses libhdf5's single-precision arithmetic
|
||
(was 1 ULP off for some values).
|
||
- A pipeline with Fletcher32 ahead of the compressor (h5py
|
||
`set_fletcher32()` then `set_deflate()`) no longer fails with "deflate:
|
||
output exceeds size limit".
|
||
- `clawhdf5-format`: **HDF5 1.4/1.6-era files are readable.** Data Layout
|
||
message versions 1 and 2 (compact, contiguous, and chunked through the
|
||
version-1 B-tree) failed with `InvalidLayoutVersion` — 84 of the 686 files in
|
||
the 2026-09-25 audit sweep, 205 datasets. They now read as libhdf5 does;
|
||
checked byte for byte against h5py on HDF5's own test files
|
||
(`tests/legacy_format_interop.rs`).
|
||
|
||
### Storage
|
||
- `clawhdf5-format`: **half-precision datasets.**
|
||
`DatasetBuilder::with_f16_data` writes IEEE binary16 (numpy `float16`),
|
||
rounding to nearest-even; `make_f16_type`, and `clawhdf5_format::float16`
|
||
with the conversions, which are checked against the `half` crate on 16.7M
|
||
values and round-trip all 65 536 half values. Reading `float16` as `f32`
|
||
gained a little-endian fast path.
|
||
- `clawhdf5-agent`: **`MemoryConfig::float16` stores embeddings as half
|
||
precision.** At 100K x 384 the file goes from 154.0 to 80.8 MiB (−48%), a
|
||
checkpoint from 752 to 512 ms and open from 300 to 252 ms, with the same
|
||
vector recall@10 against an exact scan (0.999 vs 0.994) and the same
|
||
`hybrid_search` latency; at 10K open is 3 ms slower. On the full
|
||
LongMemEval haystack with real MiniLM embeddings every retrieval metric is
|
||
identical to `f32` (`longmemeval_bench --float16`). The cache rounds each
|
||
embedding as it is saved, so memory and file agree bit for bit and a store
|
||
returns the same results before and after a reopen (tested). Out-of-range
|
||
values are refused with `MemoryError::InvalidEntry` rather than stored as
|
||
infinity; batches are all or nothing. CLI: `create --float16`. See
|
||
`BENCHMARKS.md`, "float16 embedding storage".
|
||
|
||
### Browser (WebAssembly)
|
||
- **New crate `clawhdf5-wasm`:** the reader compiled to
|
||
`wasm32-unknown-unknown` with a wasm-bindgen JavaScript API —
|
||
`open(bytes)`, `list`, `info`, `attrs`, `read`, `readHyperslab` — returning
|
||
typed arrays of the stored width (`BigInt64Array` for 64-bit integers),
|
||
string arrays for strings and enums, and a thrown `Error` for types with no
|
||
typed-array form (compound, reference, opaque, VL sequences) or filters the
|
||
build lacks (Zstd, SZIP). Read-only; the file is held in memory.
|
||
- **`examples/wasm-viewer/`:** a drop-a-file HDF5/NetCDF-4 viewer page (tree,
|
||
type/shape/attributes, values paged as hyperslabs; `?file=&path=` opens a
|
||
URL). `build.sh` produces the package; `test/run.sh` checks it under Node
|
||
(251 checks against values h5py/libhdf5 read back from an h5py- and a
|
||
netCDF4-written file) and renders the page in headless Chromium. Size,
|
||
measured 2026-09-26 on tank (`gzip -9 -n`): 627,501 B of wasm, 191,639 B
|
||
gzipped, plus 21,826 B (4,487 B) of JS glue; h5wasm 0.10.3's embedded wasm
|
||
is 3,544,184 B (907,096 B) — full libhdf5, so not equal functionality. See
|
||
`examples/wasm-viewer/README.md`.
|
||
- The facade's read path already built for `wasm32-unknown-unknown` (nothing
|
||
needed gating); `ci-test.sh` now builds it (`--no-default-features`) and
|
||
lints `clawhdf5-wasm` for that target, and CI installs the target. The Node
|
||
and browser tests run in `ci-test.sh` only where `node` and `wasm-bindgen`
|
||
exist (not the CI container); CI checks the same expectations natively
|
||
(`clawhdf5-wasm`'s `h5py_interop` test).
|
||
- `Dataset::raw_datatype()` (facade) returns the full stored datatype, for
|
||
decoding `read_selection` bytes with `clawhdf5_format::data_read`.
|
||
|
||
### Build
|
||
- **Pure-Rust default.** `clawhdf5-format`, `clawhdf5-filters` and the
|
||
`clawhdf5` facade default to the `zlib-rs` deflate backend; `fast-deflate`
|
||
(zlib-ng) is opt-in. No crate in the default dependency tree of the core
|
||
crates compiles C, and `ci-test.sh` now fails if one appears. The facade's
|
||
`fast-deflate` was on by default and is now off. See `BENCHMARKS.md`,
|
||
"Deflate backend".
|
||
- `zlib-rs` also enables flate2's `runtime_detection`. Without it zlib-rs has
|
||
no `std`, cannot detect SIMD at runtime, and inflates 3.5x slower; the
|
||
workspace builds flate2 with `default-features = false`, which had been
|
||
switching it off.
|
||
- `rust-version = "1.92"` for the whole workspace (the floor: `wgpu` requires
|
||
it), and CI checks the workspace on exactly that toolchain.
|
||
- CI keeps zlib-ng building and tested; the arm64 job no longer needs cmake.
|
||
|
||
### Correctness
|
||
- **Corrupt files libhdf5 refuses are now refused instead of read.** On the
|
||
HDF Group's CVE reproducers, 18 objects that libhdf5 (HDF5 2.0, through
|
||
h5py) refuses to open were read by clawhdf5, some as wrong data (a chunk
|
||
dimension of 0 read as all fill values; chunks read at offsets off the
|
||
chunk grid). The
|
||
parser now makes libhdf5's checks, with libhdf5's error text:
|
||
- object headers (`FormatError::InvalidObjectHeader`): every message of a
|
||
v1 chunk is read and more than the prefix's count is refused (the rest
|
||
used to be dropped); v1 message sizes must be multiples of 8 and a v1
|
||
chunk cannot end in a gap; a message running past its chunk is an error
|
||
(it used to end the chunk quietly); contradictory message flags; a
|
||
message of a class that cannot be shared flagged shareable; a
|
||
reference-count message in a v1 header; malformed continuation,
|
||
reference-count and modification-time messages; unknown v2 header
|
||
flags.
|
||
- datatypes (`FormatError::InvalidDatatype`): size 0; integer bits outside
|
||
the type; float exponent/mantissa outside the type, empty or
|
||
overlapping; a compound with no members, a member outside the compound,
|
||
a duplicate name or overlapping members; an enum whose size differs from
|
||
its base type's or with an empty name; array rank over 32 or a zero
|
||
dimension; an opaque tag length that is not a multiple of 8; in a
|
||
version-1 (unchecksummed) header, a numeric type that leaves more than
|
||
half its bits unused (`Datatype::parse_in_header`,
|
||
`Datatype::check_unused_bits`). A v1/v2 float's class bit 6 was read as
|
||
VAX byte order; libhdf5 ignores it before version 3, and so does this.
|
||
The overlap check measures each earlier member by its stored size, as
|
||
libhdf5 does, so a variable-length member (4 + offset size + 4 bytes)
|
||
in a file with 4-byte offsets does not overlap the member after it.
|
||
- chunked layouts (`FormatError::InvalidChunkDimensions`): a zero chunk
|
||
dimension, a chunk rank that does not match the dataspace, a chunk of
|
||
4 GiB or more indexed by a v1 B-tree (layout version 3 or earlier;
|
||
0x80000000-sized chunks hung the reader — layout versions 4 and 5 allow
|
||
larger chunks, and HDF5 2.0 writes them), an element size in the
|
||
layout that differs from the datatype's stored size (the chunks were
|
||
laid out with the wrong element size), and v1 B-tree
|
||
chunk keys whose offsets are not multiples of the chunk dimensions,
|
||
including the keys that only bound a node
|
||
(`chunked_read::collect_chunk_info_checked`).
|
||
- truncated files (`FormatError::TruncatedFile`, `Superblock::data_end`):
|
||
a file shorter than the end of file its superblock records is refused
|
||
("truncated file"), and nothing past that end is read. Every reader
|
||
does this: `File`, `LazyFile` and `MmapFile`, and in `clawhdf5-io`
|
||
`NativeVol` (at `open`, and on read for `from_bytes`),
|
||
`AsyncHDF5File` and `MpiVol` (the MPI path is not built in CI: it
|
||
needs an MPI installation).
|
||
- the writer: `FileWriter::finish()` / `FileBuilder::finish()` refuse a
|
||
datatype the reader would refuse (`FormatError::SerializationError`,
|
||
"datatype cannot be written: ..."), such as a compound with a repeated
|
||
field name or no fields, or an enum member with an empty name
|
||
(`CompoundTypeBuilder` and `EnumTypeBuilder` build them without
|
||
complaint). These were never valid HDF5 — h5py refuses them — and
|
||
clawhdf5 wrote them, which made files it could not read back.
|
||
|
||
Checks newer libhdf5 releases make but HDF5 2.0 does not (bit-field
|
||
offsets, the variable-length kind, array sizes) are left out, so files
|
||
h5py opens still open. Two libhdf5 checks are skipped on purpose because
|
||
clawhdf5 up to v2.7.0 wrote files that fail them without being wrong:
|
||
the sign bit of every float at position 63, and a size-0 string type for
|
||
an empty-string attribute (new fixtures written by v2.7.0 guard this).
|
||
Conformance: 569 -> 571 ok (h5stat_err_refcount.h5,
|
||
h5clear_fsm_persist_less.h5), and 17 of the 18 CVE objects now fail as in
|
||
libhdf5 (see `docs/known-issues.md` for the one left), as do 10 files
|
||
h5py refuses as truncated. Tests:
|
||
`header_validation_interop.rs` (h5py writes, the test damages a copy, both
|
||
libraries must refuse it), `legacy_writer_files.rs`, and unit tests next
|
||
to each check. **Breaking (format crate):** `FormatError` gained
|
||
`InvalidObjectHeader`, `InvalidDatatype`, `InvalidChunkDimensions` and
|
||
`TruncatedFile`; an exhaustive `match` on it needs the new arms.
|
||
- **Chunked datasets whose chunk dimensions take 3, 5, 6 or 7 bytes did not
|
||
open.** A version-4 layout (`libver="latest"`) stores each chunk dimension
|
||
in the fewest bytes that hold the largest one, so a chunk dimension from
|
||
65 536 to 16 777 215 (e.g. h5py `chunks=(70000,)`) takes 3 bytes; only 1, 2,
|
||
4 and 8 were read, and the rest failed with `UnexpectedEof`. Widths 1-8 are
|
||
read now, and 0 or more than 8 is refused as libhdf5 refuses it. A width
|
||
larger than needed is accepted: HDF5 2.0.0 refuses one ("stored chunk
|
||
dimension encoding length does not match"), but libhdf5 since
|
||
HDFGroup/hdf5@e124c36 (2026-06-05) reads it, and clawhdf5 itself wrote such
|
||
layouts.
|
||
- `clawhdf5-format` VDS: variable-length and reference data from a source in
|
||
another file is refused. Those elements are global-heap IDs and object
|
||
addresses in the source file; copied into the virtual dataset they would
|
||
be decoded against the wrong file and name another object.
|
||
- `clawhdf5-agent`: a store whose `/meta` has an attribute that cannot be
|
||
decoded fails to open (`MemoryError::Schema`). With `attrs()` now leaving
|
||
unreadable attributes out, it would otherwise have opened with defaults in
|
||
place of its settings (`float16`, `compression`, the WAL mark, ...).
|
||
- `clawhdf5-format` reader: an old-style group whose local heap has a free
|
||
list pointing outside the heap was listed with names read from the broken
|
||
heap (garbage names on `cve-2021-36977.h5` once its user block was
|
||
applied). libhdf5 refuses such a heap ("bad heap free list"); so do we now,
|
||
with `FormatError::InvalidLocalHeapFreeList`. As in libhdf5 the free list
|
||
is checked when the first name is read (`LocalHeap::validate_free_list`,
|
||
new), so an empty group with a damaged heap still lists as empty.
|
||
- **Files with a user block** (`h5py.File(..., userblock_size=N)`, `h5jam`;
|
||
the superblock at 512, 1024, …) could not be read: every address in the
|
||
file is relative to the superblock, but it was applied from byte 0
|
||
(`InvalidObjectHeaderVersion` on the root group). `File` (mmap, buffered,
|
||
`from_bytes`), `MmapFile`, `LazyFile`, `AsyncHDF5File`, the VOL readers,
|
||
the HNSW loader and external VDS sources now view the file from the
|
||
superblock on, using the signature's position as the base address as
|
||
libhdf5 does; `user_block_size()` reports the user block (h5py's
|
||
`userblock_size`), and `as_bytes()` returns the bytes from the superblock
|
||
on. **Breaking (format crate):** `Superblock::parse` refuses a non-zero
|
||
signature offset with `FormatError::UserBlockNotStripped`, since the
|
||
addresses it returns would be applied to the wrong bytes; pass the slice
|
||
from `signature::split_user_block` (new) and parse at offset 0.
|
||
- `clawhdf5-format` reader: version-1 shared messages (HDF5 1.6-era files,
|
||
e.g. a dataset using a committed datatype in libhdf5's `tcompound.h5`)
|
||
read the heap-offset field of the embedded symbol-table entry as the
|
||
target address and failed with `InvalidObjectHeaderVersion`. The address
|
||
is now read after it, as libhdf5 does. **Breaking (format crate):**
|
||
`shared_message::parse_shared_ref` takes `length_size`. A reference whose
|
||
target header has no message of the referenced type is now
|
||
`FormatError::SharedMessageTargetMissing` instead of returning the first
|
||
other message found there (which decoded as garbage).
|
||
- `clawhdf5-format` reader: array members of version-1 compound datatypes
|
||
(HDF5 1.6-era files, e.g. libhdf5's `tcompound.h5`) were read as a single
|
||
element: a `[4] i32` member came back as one `i32`, with the wrong size.
|
||
The legacy per-member dimension fields are now decoded into an array type,
|
||
as libhdf5 does; more than four dimensions, or a zero-sized one, is an
|
||
error.
|
||
- `clawhdf5-format` virtual datasets (VDS), checked against HDF5 2.0 through
|
||
h5py (`crates/clawhdf5/tests/vds_interop.rs`):
|
||
- **Wrong data:** elements no mapping supplies — unmapped regions, and
|
||
mappings whose source file or dataset is missing — read as 0 instead of
|
||
the virtual dataset's fill value (e.g. h5py `fillvalue=-1`). Assembly moved
|
||
to the new `vds` module: `vds::read_virtual_dataset` takes the fill value
|
||
and a resolver that can refuse a name (`VdsFileResolver`), and `File`
|
||
passes the dataset's fill value. A missing source *dataset* read as an
|
||
error; it is fill now, as in libhdf5. Source datasets are read with their
|
||
own fill value for unallocated chunks, and a source whose datatype differs
|
||
from the virtual dataset's is an error (libhdf5 converts; we do not).
|
||
`File` now refuses a source name that leaves the virtual file's directory
|
||
(`../x.h5`, absolute paths), or any external source of a `File::from_bytes`
|
||
file, with an error — these used to read as fill.
|
||
**Behaviour change:** the raw-read API (`read_raw_data_full*`), which has
|
||
no fill value, now returns an error for a virtual dataset with unmapped
|
||
elements instead of zeros.
|
||
- Unlimited and printf-style mappings are supported (all 7 VDS files in the
|
||
libhdf5 test set are such mappings, e.g. Eiger/Percival detector layouts).
|
||
`%b` in a source file or dataset name is the block number and `%%` a
|
||
literal `%` (other `%` sequences are an error, as in libhdf5); block `j`
|
||
is read from the source named with `j`, probing from 0 up to the first
|
||
missing source. Unlimited source/virtual selections cover as much as the
|
||
source's current extent fills, including a partial last block. As
|
||
libhdf5 does on `H5Dget_space`, the extent is recomputed from the sources
|
||
present (default "last available" view, printf gap 0) —
|
||
`vds::virtual_dataset_extent`, used by `Dataset::shape()` — so e.g.
|
||
`vds-eiger.h5` is `[5, 10, 10]`, not its stored `[20, 10, 10]`. A source
|
||
stored in the other byte order is byte-swapped (libhdf5 converts);
|
||
other type conversions remain an error.
|
||
- Hyperslab selection versions 1 and 2 were refused ("only version-3
|
||
hyperslab selections are supported"). Version 1 is what libhdf5 writes for
|
||
every VDS created with the default format bounds (h5py's default), so
|
||
those could not be read at all; version 2 is its encoding of an unlimited
|
||
selection. Both are decoded now, as are irregular hyperslabs (a union of
|
||
blocks, read in row-major order as libhdf5 iterates them).
|
||
`SerializedSelection` exposes the raw form, including unlimited counts.
|
||
- The version-1 mapping list HDF5 2.0 writes (low version bound 2.0) was
|
||
misparsed: each entry's flags byte was read as the start of the source
|
||
file name, and names shared with an earlier entry (stored as that entry's
|
||
index) were not followed. Now decoded as `H5D__virtual_load_layout` does.
|
||
- `clawhdf5-format` reader — **values returned wrong with no error:**
|
||
- Fixed Array and Extensible Array chunk indexes were laid out by the
|
||
dataset's current shape instead of its max shape (23 libhdf5 test files,
|
||
and any h5py file with e.g. `maxshape=(10, None)` or `(20, 10)` under
|
||
`libver='latest'`).
|
||
- Files with 4-byte offsets: unfiltered chunked datasets read as zeros.
|
||
Chunk B-tree keys store offsets in 8 bytes whatever the file's offset
|
||
size.
|
||
- A chunk's filter mask skipped the whole pipeline when any bit was set;
|
||
only the flagged filters are skipped now.
|
||
- Float data read as an integer returned the bit pattern; narrowing integer
|
||
reads kept the low bits; bfloat16 was decoded as IEEE half. Floats are now
|
||
decoded from their datatype fields (bf16, FP8 E4M3/E5M2, IEEE half, single
|
||
and double).
|
||
- `vl_data::read_vl_bytes` truncated sequences of non-byte base types.
|
||
- A shared fill-value message read as zero fill; it is resolved now,
|
||
including from the file's shared-message (SOHM) table, which could never
|
||
resolve because its index version byte was skipped.
|
||
- Two threads reading two chunked datasets through one `File` could get each
|
||
other's chunks (the shared chunk cache was switched between datasets
|
||
across separate lock acquisitions). The cache is now keyed by dataset.
|
||
- Compound datatype version 1 members with legacy array dimensions (HDF5
|
||
before 1.4, which had no array class) were read as a single scalar at
|
||
the member's offset; they are now array members, as in libhdf5
|
||
(`tarrold.h5`, `tcompound.h5`). Only reachable once layout versions 1/2
|
||
were readable, since the files that use it are that old.
|
||
- `clawhdf5-format` reader — errors on valid files: a version-1 shared
|
||
message (a committed datatype in HDF5 1.4/1.6-era files) was read as if the
|
||
object header address followed the reserved bytes; it follows a link-name
|
||
offset (the reference is an old-style symbol table entry), so the reader
|
||
followed the name offset and failed with `InvalidObjectHeaderVersion`
|
||
(`tcompound.h5`). New `shared_message::parse_shared_ref_sized` takes the
|
||
superblock's length size; `parse_shared_ref` assumes it equals the offset
|
||
size.
|
||
- `clawhdf5-format` reader — errors on valid files: enum and bool datasets
|
||
through the numeric readers; the "don't filter partial edge chunks" layout
|
||
flag; Fletcher32 ahead of deflate (NetCDF-4's order). Unknown-message flags
|
||
follow libhdf5 (`tbogus.h5`): "fail if unknown" is refused, "fail if unknown
|
||
and writing" is ignored by a reader.
|
||
- `clawhdf5-format` reader — dense groups and attributes (links or
|
||
attributes kept in a fractal heap indexed by a v2 B-tree):
|
||
- A link heap larger than the root indirect block's direct rows (512 KiB
|
||
with libhdf5's defaults: a few thousand long link names, or ~20 000 short
|
||
ones) could not be listed: child indirect blocks were given the wrong
|
||
number of rows, so every link stored in one was unreachable.
|
||
- v2 B-trees of depth 3 or more (a dense group of ~22 000+ links) were
|
||
misparsed: internal-node child pointers were read with widths from an
|
||
estimate instead of libhdf5's per-depth record capacities, and the
|
||
listing failed. The same B-tree code indexes dense attributes, shared
|
||
messages and chunks.
|
||
- Fractal-heap "huge" objects (larger than the heap's managed-object
|
||
limit, 4 KiB by default — e.g. an 8 KiB dense attribute or a link with a
|
||
very long name) and "tiny" objects are now read; the ID type was taken
|
||
from the wrong bits (6-7, the version, instead of 4-5), so a huge object
|
||
failed and took every attribute on its object down with it (NetCDF-4
|
||
files such as netcdf4-python's `issue671.nc`). Huge objects are found
|
||
directly from the ID or through the huge-object v2 B-tree, filtered or
|
||
not.
|
||
- Heaps with an I/O filter pipeline (a group created with a filter on its
|
||
creation property list compresses its link heap) are now read: the
|
||
header's pipeline was skipped with the wrong size, so its checksum was
|
||
looked for in the wrong place, and filtered direct blocks were read raw.
|
||
- A user-defined link (link class 65-255, e.g. 187 in libhdf5's
|
||
`tall.h5`/`tudlink.h5`) made its whole group unlistable. Such links
|
||
cannot be followed without the application that registered the class, so
|
||
they are now left out of `datasets()`/`groups()` and path lookup, as h5py
|
||
leaves out links it cannot open; reserved link types are still an error.
|
||
- `clawhdf5` — soft links are listed, as h5py lists them: `datasets()` and
|
||
`groups()` on `Group`/`MmapGroup`/`LazyGroup` include each soft link under
|
||
its own name as the kind of object it resolves to, and `dataset(name)` /
|
||
`group(name)` open through it. Relative targets resolve from the group
|
||
holding the link. Dangling or cyclic soft links, external links and
|
||
user-defined links are left out (h5py lists their names but cannot open
|
||
them). Previously soft links were missing from the listings, and in
|
||
old-style (symbol table) groups a soft link made the listing fail. New
|
||
`group_v2::resolve_group_children` / `resolve_path_from` and
|
||
`group_v1::v1_soft_links` in `clawhdf5-format`.
|
||
- `clawhdf5` — one unreadable attribute no longer fails `attrs()` for every
|
||
attribute on its object: it is left out of the map, and the new
|
||
`attrs_with_errors()` (on every group and dataset handle) returns the map
|
||
plus one error per attribute left out. Returned values are always complete.
|
||
An error in the attribute index itself (attribute info message, dense heap
|
||
header or B-tree) still fails the call. `clawhdf5-format` gains
|
||
`attribute::extract_attributes_tolerant`; `extract_attributes_full` stays
|
||
strict.
|
||
- `clawhdf5-format` reader — files with shared object header messages
|
||
(SOHM, `H5Pset_shared_mesg_index`): a datatype, dataspace, filter pipeline
|
||
or attribute stored in the file's SOHM heap failed with "invalid shared
|
||
message version: 2" — only shared fill values loaded the SOHM table — so
|
||
such files' datasets and attributes could not be read.
|
||
`shared_message::resolve_shared_message` now loads the table when a
|
||
reference needs it (36 cases of the audit's read matrix).
|
||
- `clawhdf5-format` writer — **files libhdf5 rejects or reads wrong:**
|
||
- Extensible Array (one unlimited dimension): chunks from index 244 on were
|
||
written but never indexed and read as 0, by libhdf5 and by us.
|
||
- Fixed Array: more than 1 024 chunks gave checksum errors (data blocks
|
||
were never paged).
|
||
- A finite max shape larger than the shape gave libhdf5 "addr overflow"; an
|
||
unlimited dimension that is not the first scrambled the data; several
|
||
unlimited dimensions (`(None, None)`) broke the whole file. These now
|
||
write the index libhdf5 writes (swizzled Extensible Array, or a B-tree v2
|
||
index for several unlimited dimensions).
|
||
- Header messages over 64 KiB (the size field is 16 bits) and compact
|
||
datasets at 65 534–65 535 bytes produced corrupt files.
|
||
- Reference, Opaque, BitField and Time datatypes were written as empty
|
||
messages; they now encode as HDF5 2.0 does.
|
||
- `with_page_size` wrote a nonexistent superblock version 4; it now writes
|
||
the v3 superblock and File Space Info message libhdf5 writes.
|
||
- `FillTime` values were rotated on disk (NEVER was written as ALLOC, and so
|
||
on). New `DatasetBuilder::with_fill_value`.
|
||
- An empty-string attribute got a zero-size datatype, which made every
|
||
attribute on the object unreadable in libhdf5.
|
||
- `maxshape` equal to the shape no longer forces chunked layout.
|
||
- `clawhdf5-format`: **a truncated deflate chunk read back short, with no
|
||
error.** The deflate filter used flate2's streaming reader, which returns the
|
||
bytes it has when the input runs out before the end-of-stream marker. It now
|
||
decodes in one pass into a buffer sized to the chunk and reports a
|
||
truncated stream as `DecompressionError`. Same fix in `clawhdf5-filters`,
|
||
where output longer than the stated size was also silently cut off; it is
|
||
now an error.
|
||
|
||
### Defaults
|
||
- `clawhdf5-agent`: `MemoryConfig::float16` defaults to `true` for new stores,
|
||
measured rather than assumed: identical LongMemEval retrieval on real
|
||
embeddings, 48% smaller files and faster checkpoints and opens at 100K.
|
||
`clawhdf5-cli create --f32` opts out; like `--f32-index`, it only ever
|
||
switches the default off.
|
||
- `clawhdf5-agent`: `MemoryConfig::quantized_index` defaults to `true` for new
|
||
stores. The reason it had been off — that int8 search was slower on ARM —
|
||
did not survive measurement (see Corrections). Stores that predate the
|
||
setting still load it as `false`, so reopening one never changes how its
|
||
index is held; a store written by the v2.5.0 CLI is now a test fixture that
|
||
guards exactly that, and the test fails if the load default is changed.
|
||
- `clawhdf5-cli`: `create --f32-index` opts out. `create` used to assign
|
||
`--quantized-index` straight into the config, which under the new default
|
||
would have forced every CLI-created store back to f32 unless the caller
|
||
knew to ask; it now only ever switches the default off.
|
||
|
||
### Performance
|
||
- `clawhdf5-agent`: consolidation's novelty scoring (each `add_memory` against
|
||
the whole working tier) computes the new record's norm once, takes each
|
||
comparison in one vectorised pass instead of three, and splits a working
|
||
tier of 4 096+ records across threads — same results, tested against the
|
||
old formula. It had made `consolidation_efficiency` stall at 100K; the
|
||
complete run now takes 8 min and fills in the 100K cycle row (46.66 ms) and
|
||
the memory-reduction table.
|
||
- `clawhdf5-bench`: `consolidation_efficiency` no longer prints a record-count
|
||
ratio as a "BM25 Speedup" (it was never measured), nor claims cycle time
|
||
grows sub-linearly (its own numbers grow slightly faster than linearly).
|
||
- `clawhdf5-agent`: **knowledge-graph traversal was 6.5x slower than it
|
||
should be.** `bfs_neighbors` and `spreading_activation` built an adjacency
|
||
index over the whole graph on every call (1efd82c), so a 2-hop BFS over 1K
|
||
entities took 155 µs. The index is now cached on `KnowledgeCache` and
|
||
checked against a fingerprint of the graph on each use — one pass over
|
||
entity ids and relation endpoints, no allocation — so any change, including
|
||
direct edits of its public `Vec`s, still rebuilds it (tested). BFS over 1K
|
||
entities: 155.1 -> 23.1 µs; spreading activation over 100: 22.8 -> 10.1 µs.
|
||
- `clawhdf5-format`, `clawhdf5-filters`: both deflate paths hand the codec the
|
||
whole chunk in one call, into a buffer allocated once, instead of streaming
|
||
it through a 32 KiB buffer: about 5% on chunked writes and 10% on zlib-ng's
|
||
1 MB inflate.
|
||
- `clawhdf5-accel`: **`dot_i8` has aarch64 kernels** — `SDOT` for CPUs with
|
||
the ARMv8.2 dot-product extension (Cortex-A76 and later, Neoverse-N1, every
|
||
Apple Silicon generation) and plain NEON (`vmull_s8` + `vpadalq_s16`) for
|
||
the rest, selected at runtime. `SDOT` is issued through inline assembly,
|
||
because the `vdotq_s32` intrinsic is still behind the unstable
|
||
`stdarch_neon_dotprod` feature. On a Raspberry Pi 5 at N = 100 000 and
|
||
equal recall, the quantised index answers **1.18x the queries per second**
|
||
of f32 (7 267 vs 6 164) and builds **2.3x faster** (14 464 vs 33 413 ms).
|
||
Both kernels are tested bit-for-bit against scalar on real hardware, each
|
||
explicitly — dispatch only ever takes one path on a given CPU, so testing
|
||
through it alone would have left the plain-NEON fallback unexercised on any
|
||
machine with `SDOT`.
|
||
|
||
### Corrections
|
||
- The v2.7.0 entry for `dot_i8` said `quantized_index` stayed off by default
|
||
because "aarch64 falls back to the scalar loop", implying the ~13% search
|
||
penalty measured on x86 applied on ARM too. It did not. That figure came
|
||
from scalar int8 against hand-written AVX2 f32 kernels on x86, whose
|
||
portable baseline is SSE2; on aarch64 NEON is the baseline, and measured on
|
||
a Pi 5 the scalar int8 loop already matched f32 for search while building
|
||
1.76x faster. The claim was extrapolated rather than measured.
|
||
|
||
## v2.7.0 (2026-09-20)
|
||
|
||
### Upgrade Notes
|
||
- **Two read-path bugs fixed, one of them silent.** Datasets indexed by an
|
||
Extensible Array (any dataset with one unlimited dimension) returned data
|
||
from the wrong chunks past their first few dozen. If you have readings taken
|
||
from such a dataset with an earlier release, they may be wrong; re-read them.
|
||
- **A corrupt chunk index is now an error.** Fixed and Extensible Array
|
||
structures carry checksums that were previously ignored, so damage surfaced
|
||
as plausible data from the wrong offset. Code that read a damaged file and
|
||
got numbers will now get `ChecksumMismatch` instead. That is the point.
|
||
- **Breaking:** `MemoryConfig` gained `hnsw_m`, `hnsw_ef_construction` and
|
||
`hnsw_ef_search`, so literal constructions need updating;
|
||
`..Default::default()` does not. All three default to the previous
|
||
behaviour.
|
||
|
||
### Correctness
|
||
- `clawhdf5-format`: **datasets indexed by an Extensible Array returned wrong
|
||
data beyond their first few dozen chunks.** One unlimited dimension gives a
|
||
dataset an Extensible Array chunk index, whose first elements (4 by default)
|
||
sit inline in the index block and whose rest live in data blocks sized by a
|
||
formula the reader got wrong. In the default layout everything through the
|
||
36th chunk happened to line up and the 37th onwards did not: a 400-chunk
|
||
dataset silently returned wrong values from chunk 37, and datasets past
|
||
about a thousand chunks failed outright with "invalid Extensible Array data
|
||
block signature". **Reads were wrong, not
|
||
merely refused** — the caller got plausible numbers from the wrong chunks.
|
||
Four separate layout errors, each checked against files written by HDF5 2.0
|
||
and against the library source:
|
||
- the number of data blocks in super block `u` is `2^(u/2)`, not `2^u`;
|
||
- each holds `2^((u+1)/2) * data_blk_min_elmts` elements, which doubles
|
||
every *other* level rather than every level;
|
||
- a super block carries a block-offset field before its data block
|
||
addresses, which was not skipped;
|
||
- the page-init bitmap belongs to the super block, one bit per page packed
|
||
across all its data blocks (MSB first), and was being read from inside the
|
||
data block instead; a paged data block also ends its prefix with a
|
||
checksum before the first page.
|
||
Covered now by interop tests at 4, 37, 400, 5 000 and 200 000 chunks (the
|
||
last large enough for paged data blocks), plus sparse, gzip-filtered and
|
||
2-D cases. Writing is unaffected; this is a read-path bug.
|
||
- `clawhdf5-format`: the sibling Fixed Array index (fixed dimensions written
|
||
with `libver='latest'`) was checked against the same range and is correct,
|
||
including paged data blocks and sparse datasets — it really does keep its
|
||
page-init bitmap in the data block, where the Extensible Array does not.
|
||
It had no real-file coverage above the inline sizes either, so it now has
|
||
the same tests.
|
||
|
||
### Security
|
||
- `clawhdf5-format`: **a crafted file could crash any reader through B-tree v2
|
||
traversal.** Recursion was bounded only by the depth the file claimed (a
|
||
`u16`), and child addresses were never checked for sharing. A node listing
|
||
itself as its own child under a header claiming 65 535 levels — under 100
|
||
bytes — overflowed the stack and **aborted the process** (SIGABRT, not a
|
||
catchable error). Levels whose children all point at one shared node below
|
||
reached it fan-out^depth times: 29.5 million records from ~5 KB, and one
|
||
more level would exhaust memory. Both are now errors, returned in under a
|
||
millisecond: depth is capped at 64 (as the fractal heap already was), and
|
||
traversal stops once it has produced more records than the file has bytes
|
||
to hold. Every B-tree v2 user goes through this path — dense attributes,
|
||
v2 groups, shared messages and chunk indexes. Valid files are unaffected,
|
||
including a depth-2 HDF5 2.0 chunk index with 40 000 records, now covered by
|
||
an interop test.
|
||
|
||
### Integrity
|
||
- `clawhdf5-format`: **Fixed and Extensible Array chunk indexes now verify
|
||
their checksums** (the `checksum` feature, on by default). Every structure
|
||
in both — header, index block, super block, data block and each data block
|
||
page — carries a Jenkins lookup3 checksum that was parsed past and ignored.
|
||
The consequence of skipping it is not a missing warning but wrong data: a
|
||
single flipped bit in a chunk address still parses, still points inside the
|
||
file, and the reader hands back whatever bytes now sit there as the chunk's
|
||
contents. Verified in both directions — the checksums accept files written
|
||
by HDF5 2.0 at 100 to 200 000 chunks, dense, sparse, filtered and paged,
|
||
and an interop test corrupts an address to confirm the read now fails
|
||
instead of returning data (it does return data when the check is removed).
|
||
|
||
### Performance
|
||
- `clawhdf5-agent`: **opening a store is ~28% faster** (455 ms -> 327 ms at
|
||
100k x 384). `read_from_disk` memory-mapped the file and then copied the
|
||
entire mapping into a `Vec` for `File::from_bytes`, when `File::open`
|
||
memory-maps it directly — so every open paid a full-file memcpy for nothing.
|
||
Process peak memory is unchanged: the peak falls after the parse, during the
|
||
index build, so the transient never reached the high-water mark. The
|
||
footprint harness now reports that peak next to the retained figure, which
|
||
is how this was checked rather than assumed.
|
||
- `clawhdf5-accel`: **`dot_i8`, a runtime-dispatched int8 dot product** (AVX2:
|
||
sign-extend each half to `i16`, then `madd_epi16`; scalar fallback
|
||
elsewhere). The quantised HNSW index used a scalar loop while the `f32` path
|
||
it was measured against ran AVX2, so the ~13% throughput cost recorded for
|
||
`MemoryConfig::quantized_index` was a missing kernel rather than a property
|
||
of int8. With the kernel, at N = 100 000 x 384 and equal recall, the
|
||
quantised index answers **1.63x as many queries per second** (21 848 vs
|
||
13 399 at ef=64, recall 0.9940 vs 0.9945) and builds **1.8x faster** (1778
|
||
vs 3197 ms) — on top of holding a quarter of the vectors. Medians of three
|
||
alternating runs. It remains off by default only because the kernel is
|
||
AVX2-only and aarch64 falls back to the scalar loop. Integer arithmetic, so
|
||
the SIMD path is tested to agree with scalar bit for bit.
|
||
|
||
### Tuning
|
||
- `clawhdf5-agent`: **the HNSW parameters are configurable** —
|
||
`MemoryConfig::hnsw_m`, `hnsw_ef_construction` and `hnsw_ef_search`
|
||
(defaults 16, 64, and 0 meaning "scale with `k`", i.e. today's behaviour).
|
||
They were constants, so a deployment could not trade recall against memory
|
||
or query speed at all. All three are persisted with the store. Values are
|
||
clamped where the index requires it: `clawhdf5-ann` asserts a graph degree
|
||
of at least 2, so a configured 0 — from a file, or from a caller who took 0
|
||
to mean "default" — used to abort the process inside the builder. Lowering
|
||
`ef_search` also no longer narrows the candidate pool that fusion sees.
|
||
**Breaking:** `MemoryConfig` gained fields, so literal constructions need
|
||
updating; `..Default::default()` does not.
|
||
|
||
### Documentation
|
||
- `clawhdf5-agent`: `BM25Index::search` claimed to use Block-Max WAND for early
|
||
termination. It never did; it scores every match exhaustively. It now says
|
||
so, and why no pruning would help the store: `hybrid_search` uses `scores()`,
|
||
since fusion normalises over every match.
|
||
|
||
## v2.6.0 (2026-09-20)
|
||
|
||
### Upgrade Notes
|
||
- **Re-ranked results change, substantially for the better.** `RerankInput`
|
||
and `ReRankConfig` gained fields (`relevance`, `relevance_weight`), so
|
||
literal constructions need updating; `..Default::default()` does not. Any
|
||
caller that re-ranked was previously getting results ordered by age with the
|
||
retrieval score discarded — see below.
|
||
- **Breaking:** `MemoryCache::embeddings` is a `cache::Embeddings` rather than
|
||
a `Vec<Vec<f32>>` (indexing still yields a `&[f32]` row); `embeddings_flat`
|
||
is gone, replaced by `flat_embeddings()`; `rebuild_flat()` is a deprecated
|
||
no-op.
|
||
- `MemoryConfig` gained `quantized_index` (default `false`, so behaviour is
|
||
unchanged unless you opt in); literal constructions need the field.
|
||
|
||
### Retrieval quality
|
||
- `clawhdf5-agent`: **re-ranking discarded the retrieval score.**
|
||
`reranker::rerank` built its combined score from temporal decay, source
|
||
authority and Hebbian activation only — `RerankInput` had no relevance field
|
||
— so re-ranking a candidate pool reordered it by age and threw the
|
||
retriever's ordering away. The OpenClaw backend re-ranked every search, so
|
||
this was its shipping behaviour: measured over the full LongMemEval haystack
|
||
it cost **40.6pp of Hit@1** (11.0% vs 51.6%) and two thirds of MRR (0.183 vs
|
||
0.643). `RerankInput::relevance` and `ReRankConfig::relevance_weight` (1.0 by
|
||
default) fix it: relevance leads and the metadata signals break near-ties,
|
||
which restores retrieval (Hit@1 +0.4pp vs no re-ranking) and improves
|
||
recency discrimination by 6–7pp. **Breaking:** `RerankInput` and
|
||
`ReRankConfig` gained fields, so literal constructions need updating;
|
||
`..Default::default()` does not.
|
||
- `clawhdf5-bench`: the LongMemEval harness feeds the dataset's real session
|
||
dates to the store instead of a synthetic counter (decay needs true
|
||
intervals, not just the right order), and reports `newest_gold_first` — on a
|
||
`knowledge-update` question, did the newest gold session outrank the stale
|
||
one it supersedes? Plain recall cannot see this, because both are labelled
|
||
gold. New `--rerank-sweep`.
|
||
|
||
### Memory
|
||
- `clawhdf5-agent`: **`MemoryConfig::quantized_index`** stores the vector
|
||
index's own copy of the embeddings as `i8` rather than `f32`, which at 100k
|
||
384-dim entries takes the index from 266 to 123 MiB and the whole reopened
|
||
store from 399 to 256 MiB (2.72x -> **1.74x** the raw vectors). Quantised
|
||
distances are approximate and `ef` cannot compensate — recall@10 tops out at
|
||
0.967 against f32's 0.9995 — so the query path re-scores the candidate pool
|
||
against the exact embeddings the store already holds, which restores recall
|
||
(0.9940 vs 0.9945 at ef=64) for about 13% of QPS. **Off by default**: it
|
||
trades query speed for memory, and which side is worth more depends on the
|
||
deployment. The setting is persisted, so a reopened store does not silently
|
||
revert to four times the index memory.
|
||
- `clawhdf5-ann`: `Storage::Int8` and the `build_with` / `new_with` /
|
||
`from_graph_bytes_with` constructors that select it. The scale is per row,
|
||
not global — a fixed `[-1, 1]` scale spends fewer than 12 of the 255 levels
|
||
on a unit-length 128-dim vector and is unusable (0.35 top-10 overlap against
|
||
an exact ranking, versus 0.99 per row). `compact()` keeps the storage it was
|
||
given; serialized indexes still carry f32 vectors, so a quantised index is
|
||
rebuilt rather than loaded.
|
||
- `clawhdf5-agent`: **a loaded store holds ~30% less memory** (100k 384-dim
|
||
entries: 505 -> 357 MiB, 3.44x -> 2.43x the raw vectors). The cache kept
|
||
every embedding twice — a `Vec<Vec<f32>>` and a flattened copy for the
|
||
batched kernels, maintained in lock-step — so it now stores only the flat
|
||
buffer and indexes into it. Recall and query latency are unchanged.
|
||
**Breaking:** `MemoryCache::embeddings` is a `cache::Embeddings` rather than
|
||
a `Vec<Vec<f32>>` (indexing still yields a `&[f32]` row); `embeddings_flat`
|
||
is gone, replaced by `flat_embeddings()`; `rebuild_flat()` is a deprecated
|
||
no-op. Rows are now always exactly `dim` long — shorter ones are
|
||
zero-padded — which makes the ragged-row case that used to silently
|
||
misalign the flattened copy unrepresentable.
|
||
- `clawhdf5-bench`: `search_harness --footprint` reports live heap use per
|
||
stage, measured with a counting allocator (RSS cannot see a structure freed
|
||
into the allocator's own pool).
|
||
|
||
### Testing
|
||
- The Python interop suites honour **`CLAWHDF5_PYTHON`**, and `ci-test.sh`
|
||
picks up a `.venv/bin/python` automatically. On a PEP 668 "externally
|
||
managed" system h5py cannot be installed into the system interpreter at all,
|
||
so every interop suite — the h5py writer round-trips, the facade, netCDF4
|
||
and the reference files — was skipping silently. A silent skip here is
|
||
exactly how the v5 compound-datatype bug reached a release.
|
||
`CLAWHDF5_REQUIRE_INTEROP=1` still turns a skip into a failure.
|
||
|
||
## v2.5.0 (2026-09-19)
|
||
|
||
### Upgrade Notes
|
||
- **Retrieval rankings change, for the better.** The default fusion weights
|
||
move from `0.7/0.3` to `0.4/0.6` (`hybrid::DEFAULT_FUSION`), measured over the
|
||
full LongMemEval haystack: turn-level Hit@1 51.6% vs 44.2%, MRR 0.643 vs
|
||
0.586. `unified_search` and the OpenClaw backend pick this up automatically;
|
||
callers passing weights to `hybrid_search` explicitly are unaffected.
|
||
- **Out-of-range selections are now errors.** `read_*_selection` used to return
|
||
data for a selection that ran past a dataset edge — a hyperslab came back
|
||
zero-padded, and a point with an out-of-range coordinate wrapped into the
|
||
next row. Both are now `FormatError::SelectionOutOfBounds`. Code relying on
|
||
the old (wrong) values will start seeing errors.
|
||
- **Large compressed datasets written without explicit chunk dimensions get a
|
||
different layout.** They used to be stored as one chunk; they are now split
|
||
to ~1 MiB chunks. The files stay standard and h5py-readable, and explicit
|
||
`with_chunks` is unaffected.
|
||
- `rayon` is now a default dependency of `clawhdf5-agent` (the parallel index
|
||
build). Opt out with `--no-default-features --features float16,hnsw`.
|
||
- `clawhdf5-ann` search results no longer shrink when records near the query
|
||
have been deleted, so a search that previously returned fewer than `k`
|
||
results now returns `k`.
|
||
|
||
### Retrieval quality
|
||
- `clawhdf5-agent`: optional keyword stemming — `bm25::TokenFilter::Stemmed`
|
||
and `HDF5Memory::set_token_filter`, so "training" and "trains" match. **Off
|
||
by default**, on measurement rather than principle: over the full LongMemEval
|
||
haystack it buys depth and costs the top rank (BM25 alone: Hit@5 +2.8pp,
|
||
Hit@10 +2.4pp, Hit@1 −1.8pp, MRR unchanged), and on the shipping hybrid
|
||
configuration the trade is narrower still. See `BENCHMARKS.md`.
|
||
- `clawhdf5-agent`: **`QueryExpander::expand` panicked on ordinary non-ASCII
|
||
input** — `"İ AI"` was enough. It searched a lowercased copy of the query and
|
||
then sliced the *original* with those offsets, which only works while
|
||
lowercasing preserves byte length (Turkish `İ` is 2 bytes and lowercases to
|
||
3). Depending on where the offsets drifted it either corrupted the output
|
||
("İstanbul AI trip" lost a character) or panicked. Matching now walks the
|
||
original string.
|
||
- `clawhdf5-agent`: query expansion no longer rewrites text inside words.
|
||
`replace_word_case_insensitive` did a plain substring replace despite its
|
||
name, so "training" became "trArtificial Intelligencening" and "programming"
|
||
became "Pull Requestogramming" — every acronym expansion of ordinary prose
|
||
was corrupt. Matches now require word boundaries; genuine acronyms
|
||
(`API`, `database`) still expand.
|
||
- `clawhdf5-agent`: **the default fusion weights are now the measured ones.**
|
||
A sweep of every 0.1 step over the full LongMemEval haystack (500 questions,
|
||
real MiniLM embeddings) shows the long-standing `0.7/0.3` default is
|
||
*strictly dominated* by `0.4/0.6` — turn-level Hit@1 51.6% vs 44.2%, Hit@5
|
||
81.4% vs 79.2%, Hit@10 87.8% vs 85.8%, MRR 0.643 vs 0.586, and better at
|
||
session level too. The finding was recorded in `BENCHMARKS.md` but had never
|
||
been applied: `unified_search` and the OpenClaw backend both hardcoded
|
||
`0.7/0.3`. They now use `hybrid::DEFAULT_FUSION`. **Callers passing weights
|
||
to `hybrid_search` explicitly are unaffected** — pass `0.4`/`0.6` (or use
|
||
`hybrid_search_with`) to get the tuned behaviour.
|
||
- `clawhdf5-agent`: fusion is now selectable. New `hybrid::Fusion`
|
||
(`Weighted { vector, keyword }` or `Rrf { k }`), `hybrid::fuse`,
|
||
`hybrid::hybrid_search_fused` and `HDF5Memory::hybrid_search_with`.
|
||
Reciprocal rank fusion existed but was unreachable from the store, so it had
|
||
never been measured against the weighted sum; the LongMemEval bench now has
|
||
an `RRF` mode.
|
||
|
||
### HDF5 Read Path
|
||
- **Selection reads cost what the selection costs.** `read_*_selection` decoded
|
||
the *entire* dataset and then picked elements out, so a 64 x 64 window of a
|
||
64 MB compressed dataset took 105 ms - about as long as reading all of it.
|
||
Now only the rows (contiguous) or chunks that overlap the selection's
|
||
bounding box are read and decompressed: that window takes 0.39 ms, one row
|
||
2.7 ms, one column 5.2 ms. Results are identical to the full-read path
|
||
(equivalence-tested over random hyperslabs and point lists, ranks 1-3,
|
||
contiguous / chunked / deflate). New `read_harness` bench binary.
|
||
- **Faster full reads** (same-moment A/B, 64 MB `f64`): chunked + deflate
|
||
110 -> 69 ms, chunked 72 -> 60 ms, contiguous 56 -> 30 ms. The facade's
|
||
cached read path now decompresses cache misses in parallel batches (it was
|
||
sequential; only the uncached reader was parallel) and caches only datasets
|
||
that fit the chunk cache; unfiltered chunks are copied straight from the file
|
||
bytes; a contiguous dataset is converted straight from the file bytes; and
|
||
the native-endian conversions no longer zero a buffer before overwriting it.
|
||
- **Datasets indexed by a version-2 B-tree now read** (layout v4, chunk index
|
||
type 5 — what `libver='latest'` uses for two or more unlimited dimensions;
|
||
previously "unsupported chunked layout"). The four copies of the chunk-index
|
||
dispatch are now one shared function, so every read path gets it.
|
||
- **`H5T_STD_REF` references** (HDF5 1.12+, datatype message version 4) parse:
|
||
`ReferenceType` gains `Object2`, `DatasetRegion2` and `Attribute`, and
|
||
`read_object_references` decodes the new object references. Previously any
|
||
dataset of this type failed with `InvalidReferenceType(2)`. Tested against a
|
||
file written by HDF5 2.0 itself (fixture + generator script committed).
|
||
- **Automatic chunk sizes.** Asking for compression (or any filter) without
|
||
`with_chunks` used to store the whole dataset as one chunk, so any read had
|
||
to decompress everything and nothing could be decoded in parallel. Datasets up
|
||
to 1 MiB stay a single chunk, as before; larger ones are split by halving the
|
||
dimensions in turn until a chunk is at most 1 MiB (the approach h5py takes).
|
||
**Behaviour change:** large compressed datasets written without explicit
|
||
chunk dimensions get a different (standard, h5py-readable) layout. Explicit
|
||
`with_chunks` is unaffected.
|
||
- **Out-of-range selections are errors.** They used to return data: a hyperslab
|
||
past an edge came back padded with zeros, and a point whose column was out of
|
||
range wrapped into the next row and returned that element. Now
|
||
`FormatError::SelectionOutOfBounds` (also for a rank mismatch or overlapping
|
||
blocks).
|
||
|
||
### Search
|
||
- `clawhdf5-ann`: **faster index builds.** Back-link pruning is 90% of a
|
||
build's distance evaluations; the bulk build now inserts in batches and
|
||
prunes each overflowing neighbour list once per batch (10K: 1676 -> 1074 ms).
|
||
With the `parallel` feature, planning and pruning run on a thread pool (10K:
|
||
388 ms, 100K: ~21 s -> 5.9 s on 16 cores). The graph is deterministic and
|
||
identical with or without the feature. `clawhdf5-agent`'s `parallel` feature
|
||
enables it for the agent's index and is now **on by default** (adds `rayon`
|
||
to the default dependency set; build with `--no-default-features --features
|
||
float16,hnsw` to opt out).
|
||
- `clawhdf5-ann`: `HnswIndex::search` returned fewer than `k` results — often
|
||
none — when the records nearest the query had been deleted: it collected `ef`
|
||
candidates, *then* dropped the deleted ones, *then* took `k`. Deleted nodes
|
||
are now traversed as waypoints but never occupy a result slot, so a search
|
||
returns the `k` nearest live records. Matters for any store that deletes or
|
||
supersedes memories without compacting straight away.
|
||
|
||
## v2.4.0 (2026-09-19)
|
||
|
||
### Upgrade Notes
|
||
- **Search results improve on upgrade.** The HNSW index now reaches true
|
||
neighbours it previously could not (recall@10 0.31 -> 0.98 at 100K records on
|
||
clustered data), so `hybrid_search` rankings change for the better. The agent
|
||
rebuilds its index from the store automatically; a standalone `HnswIndex`
|
||
persisted with `to_hdf5_bytes` keeps its old graph until rebuilt.
|
||
- **`hybrid_search` no longer writes the store.** Hebbian activation boosts are
|
||
persisted by the next checkpoint (any flushing write, `flush_wal`, or when
|
||
the `HDF5Memory` is dropped) instead of inside every query; a crash before
|
||
then forgets only the boosts since the last checkpoint. Activation weights
|
||
are now capped at 16.
|
||
- A new sidecar file, `<store>.h5.ann`, holds the vector index graph. It is
|
||
derived data: safe to delete (the index is rebuilt), copied by `snapshot()`,
|
||
and worth including when copying a store by hand to avoid a rebuild.
|
||
- `BM25Index` no longer caches IDF and gained `add_document`,
|
||
`remove_document`, `pad_to`, `scores`, `len` and `is_empty`; results are now
|
||
deterministic (ties break by record id).
|
||
|
||
### Search
|
||
- `clawhdf5-ann`: **HNSW recall fix.** Neighbours were chosen as the plain
|
||
closest-M, which on clustered data (what embeddings look like) turns each
|
||
cluster into an island: recall@10 was 0.87 / 0.67 / 0.31 at 1K / 10K / 100K
|
||
vectors and did not improve with `ef`. The index now uses the HNSW paper's
|
||
diversity heuristic (Algorithm 4 with kept pruned connections) when linking a
|
||
new node and when pruning back-links: recall@10 at `ef = 64` is 1.00 / 1.00 /
|
||
0.98 and responds to `ef`. Builds are slower (~3.5x at 10K). Existing
|
||
persisted indexes keep their old graph until rebuilt; the agent rebuilds its
|
||
index from the cache, so stores pick this up automatically.
|
||
- `clawhdf5-agent`: **`hybrid_search` is 23-39x faster in steady state** (p50
|
||
5.5 -> 0.24 ms at 1K records, 49 -> 2.1 ms at 10K, 884 -> 23 ms at 100K).
|
||
Every query used to rebuild the BM25 index from scratch and rewrite the whole
|
||
`.h5` file. The keyword index now lives for the life of the store and is
|
||
updated incrementally (add / remove / in-place update, exactly equivalent to
|
||
a fresh build - property-tested), and a query no longer writes the store.
|
||
**Behaviour change:** Hebbian activation boosts are persisted by the next
|
||
checkpoint (any flushing write, `flush_wal`, or drop) rather than
|
||
immediately; a crash in between forgets only the boosts since the last
|
||
checkpoint. Activation weights are now capped (16.0) - they grew without
|
||
bound.
|
||
- `clawhdf5-agent`: **the vector index is persisted**, so `open()` no longer
|
||
rebuilds it on the first search (first query after open: 2627 -> 15 ms at 10K
|
||
records, 36 s -> 159 ms at 100K). The HNSW graph — not the vectors, which the
|
||
store already holds — is written to `<store>.h5.ann` at each checkpoint and
|
||
tied to it by a generation id in `/meta`; a missing, stale, damaged or
|
||
structurally invalid sidecar is ignored and the index rebuilt. Records
|
||
replayed from the WAL join the loaded index incrementally; a replayed update
|
||
or delete invalidates it. `snapshot()` copies it. Batch saves no longer force
|
||
a full index rebuild.
|
||
- `clawhdf5-ann`: faster HNSW build and search with identical recall. The
|
||
cosine metric stores unit vectors and compares them with a plain dot product
|
||
(it re-derived both norms on every distance evaluation), and the per-call
|
||
`HashSet` of visited nodes is a reusable epoch-stamped array. Build 2.75 ->
|
||
1.89 s at 10K and ~38 -> 21 s at 100K; QPS at `ef = 64` 22.7K -> 39K at 10K.
|
||
Distances returned by `search` are unchanged (1 - cosine). Indexes loaded
|
||
from older HDF5 files are normalised on load.
|
||
- `clawhdf5-accel`: the SIMD backend is detected once per process instead of
|
||
on every kernel call.
|
||
- `clawhdf5-ann`: `HnswIndex::graph_to_bytes` / `from_graph_bytes` — graph-only
|
||
serialization (checksummed, every neighbour id and level validated on load).
|
||
- `clawhdf5-agent`: a further 4-5x on `hybrid_search` with **identical
|
||
rankings** (p50 now 0.07 / 0.49 / 4.65 ms at 1K / 10K / 100K — 79x / 100x /
|
||
190x faster than v2.3.0). Fusion needs every keyword score but not their
|
||
ranking: new `BM25Index::scores` returns them unsorted from a dense
|
||
accumulator (it hashed every posting, then sorted every match), and
|
||
`merge_vector_keyword` selects its top k instead of sorting every candidate.
|
||
Capping the keyword candidate pool was measured and rejected: it changes the
|
||
top-10 for most queries (`search_harness --fusion-study`).
|
||
- `clawhdf5-agent`: BM25 results are deterministic (ties break by record id),
|
||
top-k uses a bounded heap, and the "WAND early termination" that computed a
|
||
bound and then ignored it is gone. IDF is computed per query.
|
||
- `clawhdf5-bench`: new `search_harness` binary — HNSW recall@10 / QPS / latency
|
||
per `ef` against an exact scan, and end-to-end `hybrid_search` timings, on
|
||
deterministic clustered (or `--uniform`) data. Baseline in `BENCHMARKS.md`.
|
||
|
||
## v2.3.0 (2026-09-19)
|
||
|
||
### Upgrade Notes
|
||
- **A memory store now has a single writer.** `HDF5Memory::create`/`open` take
|
||
an exclusive lock (`<store>.h5.lock`); a second open of the same store — in
|
||
the same or another process — returns `MemoryError::Locked`. Code that opened
|
||
a second handle just to read should use `HDF5Memory::open_read_only`.
|
||
- **Unsigned array attributes arrive as `AttrValue::U64Array`**, not
|
||
`I64Array`, and `attrs()` may now return `AttrValue::Raw`. Exhaustive matches
|
||
on `AttrValue` need the two new arms.
|
||
- **WAL header version 3 → 4.** v3 files are read and upgraded in place, but a
|
||
store written by 2.3.0 with a pending WAL cannot be opened by 2.2.0 or
|
||
earlier (it is refused, not corrupted). Checkpoint first
|
||
(`flush_wal`) if you need to downgrade.
|
||
- `MemoryConfig::compression` now uses deflate unless the agent's new `zstd`
|
||
feature is enabled; it previously failed outright in a default build.
|
||
- `MemoryError` gained `Locked`; `FormatError` gained `UnresolvedSharedMessage`,
|
||
`ExternalDataFilesUnsupported` and `ExternalLinkUnsupported`; `MessageType`
|
||
gained `ExternalDataFiles`.
|
||
|
||
### Bug Fixes
|
||
- `clawhdf5-format`: compound datatypes written with **default libver bounds**
|
||
(datatype message version 1 — what plain `h5py.File(path, 'w')` produces)
|
||
were mis-parsed. The v1 member layout carries 28 bytes of legacy array
|
||
fields after the byte offset (the parser skipped 24), and v2 pads member
|
||
names to 8 bytes and has no array fields at all (the parser did neither), so
|
||
every member after the first byte offset was read from the wrong position —
|
||
typically surfacing as `Overflow("compound member ...")` on read. Found by
|
||
adding a default-libver axis to the h5py interop tests; byte-level regression
|
||
tests for v1 and v2 added.
|
||
- `clawhdf5-gpu`: `gpu_tests` could hang forever under the default parallel
|
||
test runner — every test created its own wgpu instance and device at once.
|
||
Tests now serialise GPU access, and GPU→CPU readback waits are bounded
|
||
(30 s) so a wedged driver returns `GpuError::BufferMap` instead of blocking.
|
||
- `clawhdf5-agent`: `benches/bench.rs` and `benches/memory_bench.rs` no longer
|
||
compiled against the current `strategy`/`consolidation` APIs.
|
||
|
||
### HDF5 Compatibility
|
||
- `clawhdf5-format`/`clawhdf5`: datasets and attributes that use a **committed
|
||
(named) datatype** now read correctly. They store a shared-message reference;
|
||
the facade parsed the reference bytes as the datatype (`Time { size: 0 }`,
|
||
unreadable data) and silently dropped such attributes. The shared-reference
|
||
parser itself was wrong for real files: version 2 has no reserved bytes, and
|
||
the version 3 types were inverted (1 = SOHM heap, 2 = committed).
|
||
- **Fill values are applied on read.** There was no Fill Value message parser:
|
||
the holes of a sparse chunked dataset read as zeros even when the fill value
|
||
was not zero (silently wrong data), and a dataset that was created but never
|
||
written failed with `NoDataAllocated` where h5py returns a filled array.
|
||
Messages v1–v3 and the old 0x0004 form are parsed; the fill value is written
|
||
into exactly the chunk-grid cells missing from the chunk index.
|
||
- **Soft links are followed** during path resolution, in old- and new-style
|
||
groups (absolute/relative targets, links to groups, links through links),
|
||
with a depth limit so a link cycle is an error rather than a hang. A dangling
|
||
link reports the target it could not find.
|
||
- Things the reader does not follow are now explicit errors instead of wrong
|
||
answers: an external link is `ExternalLinkUnsupported { filename,
|
||
object_path }` (was `PathNotFound`), and a dataset whose raw data lives in
|
||
external files (message 0x0007, now a known `MessageType`) is
|
||
`ExternalDataFilesUnsupported` (it would otherwise read as fill values).
|
||
- **`attrs()` no longer drops attributes.** Any attribute whose datatype had
|
||
no `AttrValue` variant was omitted with no error — including every Python
|
||
`bool` (h5py stores `attrs["flag"] = True` as an enum), complex numbers,
|
||
compound values and object references. Now:
|
||
- numpy/h5py-style booleans (an enum of exactly `FALSE`=0 / `TRUE`=1) decode
|
||
as `I64` / `I64Array` of 0/1;
|
||
- new `AttrValue::U64Array` keeps unsigned arrays unsigned (they were cast to
|
||
`I64Array`, so values above `i64::MAX` came back negative). **Behaviour
|
||
change:** code matching `I64Array` for an unsigned attribute must also
|
||
match `U64Array` (the netCDF-4 CF helpers and Python bindings do);
|
||
- new `AttrValue::Raw { datatype, shape, data }` carries everything else
|
||
verbatim, decodable with `clawhdf5_format::data_read` against `datatype`.
|
||
Both new variants are writable, so an attribute can be copied between files
|
||
unchanged. Python receives `Raw` as `{"dtype", "shape", "data"}`.
|
||
- All of the above are covered by h5py interop tests under both default and
|
||
`libver='latest'` bounds, compared against h5py's own readback.
|
||
|
||
### Security
|
||
- `clawhdf5`: virtual-dataset source file names are untrusted input but were
|
||
joined straight onto the opened file's directory, so a crafted file could
|
||
make the reader open any path the process can reach (absolute path, or `..`
|
||
components). Only plain relative paths inside that directory are accepted.
|
||
|
||
### Durability & Integrity
|
||
- `clawhdf5-agent`: a crash between writing a checkpoint and truncating the WAL
|
||
no longer **duplicates every pending entry** on the next open. Each
|
||
checkpoint records a `WalMark` (byte length + chained CRC of the WAL prefix it
|
||
folded in) in `/meta`; `open()` skips exactly that prefix when it is still
|
||
present. No WAL format change for this; older files behave as before.
|
||
- `clawhdf5-agent`: checkpoints and snapshots are durable as a unit — the temp
|
||
file is synced before the rename and the directory after it. Individual WAL
|
||
appends remain unsynced by design (documented in `CLAUDE.md`).
|
||
- `clawhdf5-agent`: `save_or_update` hits are logged as a new `Update` WAL
|
||
record, so replay updates in place instead of appending a duplicate. WAL
|
||
header version 3 → 4 (so older builds refuse the file rather than truncating
|
||
a record they can't parse); v3 files are read and upgraded in place.
|
||
- `clawhdf5-agent`: loading validates every per-record dataset length (a
|
||
truncated store is now `MemoryError::Schema`, not a later panic), fixes the
|
||
`n.len() == n.len()` tautology that trusted a norms dataset of any length,
|
||
and rejects `embedding_dim == 0` with records present.
|
||
- `clawhdf5-agent`: eight behavioural `MemoryConfig` fields are now persisted in
|
||
`/meta`. Previously they reset to defaults on every open — a compressed store
|
||
was rewritten uncompressed, `wal_enabled = false` flipped back to `true`.
|
||
- `clawhdf5-agent`: `compression = true` never worked in a default build (it
|
||
requested Zstd without enabling the feature, so every checkpoint failed with
|
||
`unsupported filter: 32015`). Default builds now use deflate; Zstd is the new
|
||
opt-in `zstd` feature.
|
||
- `clawhdf5-agent`: **single-writer lock** (`<store>.h5.lock`,
|
||
`MemoryError::Locked`) — two handles on one store used to silently destroy
|
||
each other's data. New `HDF5Memory::open_read_only` gives a lock-free,
|
||
never-writing view; the CLI's read-only subcommands use it.
|
||
- `clawhdf5-agent`: an unreadable WAL (torn header / bad magic) is quarantined
|
||
(`HDF5Memory::quarantined_wal()`) instead of blocking `open()` of a healthy
|
||
store. A WAL from an unknown newer version still fails and is left intact.
|
||
- `clawhdf5-agent`: provenance records are renumbered on compaction (they
|
||
weren't, so every later `save_or_update` raised a false High integrity
|
||
alert); pending anomaly alerts and tracked sessions are bounded;
|
||
`snapshot()` includes entries still in the WAL.
|
||
- `clawhdf5-agent`: hybrid ranking is deterministic (index tie-breaks instead
|
||
of `HashMap` order); a set of identical positive scores — including a single
|
||
candidate — normalises to 1.0 rather than 0.0; the Hebbian boost no longer
|
||
reinforces zero-score filler results.
|
||
- `clawhdf5-format`: chunked/VDS/hyperslab reads size their buffers with
|
||
overflow-checked arithmetic and fallible allocation, so crafted dimensions
|
||
are `FormatError::Overflow` instead of a wrapped size or a process abort;
|
||
`parallel_read` bounds checks use `checked_add`.
|
||
- `clawhdf5`: a malformed filter-pipeline message is an error instead of being
|
||
treated as "no filters" (which returned compressed bytes as data);
|
||
`FileBuilder::write` is atomic and synced instead of truncating the
|
||
destination first.
|
||
|
||
### CI / Testing
|
||
- CI now lints every target (`cargo clippy --all-targets`) plus
|
||
`clawhdf5-format`'s optional features, compiles all benches, and tests the
|
||
format feature matrix. Previously test/bench code and feature-gated modules
|
||
were never linted; the accumulated clippy backlog is fixed.
|
||
- CI installs python3 + h5py/numpy/netCDF4/xarray and sets
|
||
`CLAWHDF5_REQUIRE_INTEROP=1`, which turns a missing interop dependency into a
|
||
test **failure**. Until now every h5py/netCDF4 interop test silently skipped
|
||
in CI, which is how the HDF5 2.0 compound bug fixed in v2.2.0 reached a user.
|
||
The `#[ignore]`d `writer_h5py_tests` suite is run explicitly.
|
||
- h5py-generated-file tests now cover default libver bounds as well as
|
||
`libver='latest'` (HDF5 2.0 raised the default low bound to 1.8).
|
||
- `clawhdf5-agent`: WAL property tests (round trip; after any corruption the
|
||
entries read back are an exact prefix of what was written — 1500 seeded
|
||
cases), a crash-recovery matrix (an on-disk image after every operation, the
|
||
checkpoint window, and the WAL torn at every byte length, each reopened and
|
||
checked against a model), and a WAL fuzz target.
|
||
- Optional fuzz smoke run (`CLAWHDF5_FUZZ_SECONDS=N scripts/ci-test.sh`); new
|
||
datatype corpus seeds for v1 compound and native complex messages.
|
||
|
||
## v2.2.0 (2026-09-18)
|
||
|
||
### Security
|
||
- `clawhdf5-format`: bounded decompression output (`MAX_DECOMPRESS_SIZE`) for
|
||
deflate/lz4/zstd/pcodec so a crafted compressed chunk can't drive an
|
||
unbounded allocation (memory-exhaustion DoS).
|
||
- `clawhdf5-format`: `chunked_read.rs`/`data_read.rs`/`local_heap.rs` bounds
|
||
audit — added `ensure_len` overflow guards at every plain-arithmetic
|
||
offset+size check, a recursion-depth guard against a crafted
|
||
self-referencing/cyclic B-tree chunk index, a fix for an unguarded
|
||
compound-datatype `byte_offset` overrun in `read_compound_fields`, and an
|
||
`ndims - 1` underflow guard for degenerate zero-dimension chunked layouts.
|
||
Added a new `fuzz_dataset_read` cargo-fuzz target (walks every dataset in a
|
||
parsed file and exercises the contiguous/chunked/compact raw-data read
|
||
paths) which found and fixed 3 real crash bugs — an integer-multiply
|
||
overflow in `copy_chunk_to_output`'s N-D assembly path, the `ndims - 1`
|
||
underflow above, and an overflow in `local_heap.rs` — within the first few
|
||
fuzzing runs.
|
||
- `clawhdf5-format`: `btree_v1.rs` overflow-safe bounds checks via a local
|
||
`ensure_len` helper, closing a `usize`-overflow panic reachable from a
|
||
crafted near-`usize::MAX` B-tree offset.
|
||
- `clawhdf5-agent`: WAL length-prefix caps (`MAX_WAL_FIELD_LEN`, 64 MiB) reject
|
||
a corrupted/truncated length claim before allocating. Followed by a full
|
||
per-entry CRC32 trailer (`WAL_VERSION` bumped to 2) — a bit-flip inside an
|
||
entry now stops replay cleanly instead of silently accepting corrupted
|
||
data. Old-format WAL files are still read correctly and migrated to the new
|
||
format on next open.
|
||
- `clawhdf5-android`: validate `embedding_len`/`query_embedding_len` against
|
||
the handle's configured `embedding_dim` (and reject null pointers) before
|
||
constructing a slice from a raw pointer in `edgehdf5_save` /
|
||
`edgehdf5_hybrid_search`.
|
||
- `clawhdf5-py`: bump pyo3/numpy `0.28` → `0.29`, clearing two RUSTSEC
|
||
advisories (OOB read in `PyList`/`PyTuple` iterator; missing `Sync` bound on
|
||
`PyCFunction::new_closure`).
|
||
- Clarified that the integrity hashes in `clawhdf5-agent::provenance`
|
||
(FNV-1a) and `clawhdf5-format::provenance` (SHA-256) are unkeyed and detect
|
||
only accidental corruption, not tampering — doc-only change, no behavior
|
||
change.
|
||
|
||
### Performance
|
||
- `clawhdf5-format`: chunk cache lookup is now O(1) (`slot_index: HashMap`)
|
||
instead of a linear scan, and cache hits return a shared `Arc` instead of
|
||
cloning the decompressed buffer — the hottest path in chunked reads.
|
||
- `clawhdf5-ann`: optional `parallel` feature (rayon) parallelizes HNSW's
|
||
`prune_connections` neighbor-distance computation. The outer build/insert
|
||
loop is deliberately left sequential — it has genuine cross-iteration data
|
||
dependencies and needs its own correctness-focused design pass.
|
||
- `clawhdf5-format/chunked_read.rs`: removed 12 unnecessary
|
||
`chunk_dimensions[..rank].to_vec()` allocations where callees already
|
||
accept `&[u32]`.
|
||
|
||
### Architecture
|
||
- Added `.gitea/workflows/ci.yml`, actually wiring the long-existing
|
||
`scripts/ci-test.sh` (fmt, clippy, tests, no_std check) into CI on every
|
||
push/PR to `main`. Fixed stale package names in `ci-test.sh`/
|
||
`check-nostd.sh` that had been silently no-op'ing the `clawhdf5-py`
|
||
exclusion and the no_std check.
|
||
- Fixed a genuine no_std build break in `clawhdf5-format` (uncovered once the
|
||
no_std CI check actually started running): `core::sync::atomic::AtomicU64`
|
||
doesn't exist on `thumbv7em-none-eabihf` (switched to `portable-atomic`),
|
||
missing `alloc` imports for `Box`/`Vec`/`format!` on a few no_std paths, and
|
||
`f64::powi` (std/libm-only) replaced with a local exponentiation-by-squaring
|
||
helper in the scale-offset filter.
|
||
- Added `[workspace.dependencies]` for `tempfile`/`criterion`/`half`/`serde`,
|
||
fixing a real version skew on `half` (`2` vs `2.7` across crates).
|
||
- Fixed version skew: `clawhdf5-py` (`pyproject.toml`) and
|
||
`packages/clawhdf5-node` (`package.json`) were both behind the actual crate
|
||
version (2.1.0).
|
||
- Documented that the `mpi-io` feature's read/write paths are root-read
|
||
+broadcast / gather-to-rank-0, not true collective I/O.
|
||
|
||
### Documentation
|
||
- BENCHMARKS.md: re-ran the previously-undated "LongMemEval Results", "SIMD &
|
||
Parallelism", and "Vector Search Latency"/"Comparison to MemX" sections on
|
||
a second machine (tank, Ryzen 7 7800X3D) with explicit dates and reproduce
|
||
commands. Found and corrected a methodology issue in the SIMD/Parallelism
|
||
benchmark selection (several originally-compared benchmarks didn't actually
|
||
isolate the scalar/SIMD/parallel axis).
|
||
- README.md / ROADMAP.md / CLAUDE.md: corrected several stale facts —
|
||
the `clawhdf5-types` crate (removed earlier) was still listed in the
|
||
README crate map; the LongMemEval numbers in the README badge and table
|
||
didn't match the actual (much better) benchmark results in BENCHMARKS.md;
|
||
total line-of-code and test-count figures were stale; `clawhdf5-gpu`'s
|
||
CubeCL→wgpu correction; documented the new `clawhdf5-ann` `parallel`
|
||
feature flag, which had no entry in the Feature Flags table.
|
||
|
||
### New Features
|
||
- `clawhdf5-migrate`: substantial engine improvements:
|
||
- **Real content validation** — the post-migration check now reads the written
|
||
HDF5 back and compares actual content (chunk text, embeddings, and every
|
||
session/entity/relation field) against the source, not just row counts. A
|
||
representative sample of chunk rows is verified by default; `--validate-full`
|
||
checks every row. A corrupt migration that preserves counts no longer passes.
|
||
- **Configurable schema** — table names are no longer hardcoded; queries are
|
||
built from a `SchemaConfig` (table + ordered column names, defaulting to the
|
||
ZeroClaw layout) with `--chunks-table` / `--sessions-table` /
|
||
`--entities-table` / `--relations-table` overrides.
|
||
- **Streaming count pass** — `--dry-run` now does a `COUNT(*)`-only pass per
|
||
table instead of loading every row into memory.
|
||
- **Incremental migration** — `--incremental` reads the existing output, reads
|
||
only source chunks newer than the last migrated id, and appends them
|
||
(refreshing the metadata groups), instead of re-migrating everything.
|
||
- `clawhdf5-format`: read **IEEE-754 half-precision (f16)** floats. `read_as_f32`
|
||
/ `read_as_f64` previously only handled 4- and 8-byte floats; 2-byte floats
|
||
(e.g. float16-stored embeddings) now decode via a no_std-safe bit conversion.
|
||
- `clawhdf5-format`: **write multi-block fractal heaps** (root indirect block).
|
||
Dense attribute and dense link storage previously capped at a single direct
|
||
block (~64 KiB of heap data — a few thousand attributes/links). When the
|
||
objects exceed one direct block, the heap now lays out a root indirect block
|
||
(FHIB) over multiple direct blocks sized by the doubling table, distributing
|
||
objects across blocks with correct per-block heap offsets. Validated
|
||
end-to-end: a 2,500-attribute object and a 2,500-link group round-trip
|
||
through our reader and are read correctly by h5py. (Objects still may not
|
||
span a block — no huge-object path.)
|
||
- `clawhdf5-format`: **write dense group link storage** (fractal heap + v2
|
||
B-tree). A group with more than 8 links (libhdf5's compact `max_compact`
|
||
default) is now written densely — its links live in a fractal heap indexed by
|
||
a v2 B-tree of type 5 (link-name index) referenced from the group's LinkInfo
|
||
message — instead of as inline Link messages. This matches libhdf5's
|
||
compact→dense switchover and keeps large groups out of the object header.
|
||
Reverse-engineered against libhdf5: link heaps use `heap_id_length` 7 /
|
||
`max_heap_size` 32 (vs 8 / 40 for attributes). The shared single-direct-block
|
||
fractal-heap builder is now parameterized and used by both dense attributes
|
||
and dense links. Validated end-to-end: our reader round-trips, and h5py reads
|
||
the dense groups we write. (Single direct block — up to ~a couple thousand
|
||
links per group; beyond that needs indirect blocks, still unsupported.)
|
||
|
||
### Robustness
|
||
- `clawhdf5-format`: harden the readers added this cycle against malformed /
|
||
hostile input — they parse untrusted bytes and must return errors, never
|
||
panic, OOM, or recurse without bound. Fixed concrete vectors found by audit
|
||
and locked in with adversarial tests:
|
||
- **Paged Fixed Array**: `1 << max_nelmts_bits` shift overflow (a `u8` ≥ 64);
|
||
element/page offset multiplications now checked; element count bounded by
|
||
file size.
|
||
- **H5S selection decoder**: `ALL`/`NONE` no longer claim 16 bytes they don't
|
||
have; hyperslab `rank` capped at 32 (`H5S_MAX_RANK`) to stop a giant
|
||
allocation; `iter_linear` coordinate/stride/product arithmetic is checked.
|
||
- **VDS mapping parser**: no pre-allocation from the untrusted `nused`; all
|
||
selection slicing is bounds-checked.
|
||
- **scale-offset / N-Bit filters**: `1 << minbits` overflow at `minbits == 64`;
|
||
N-Bit `bit_offset + precision` overflow; N-Bit type-tree recursion depth
|
||
capped (no stack overflow from a crafted nested tree); element counts
|
||
bounded by the chunk's expected decompressed size so a bogus count can't
|
||
drive a huge allocation.
|
||
- **Virtual Dataset assembly**: a virtual dataset whose source is itself
|
||
virtual (a cycle) now errors instead of recursing into a stack overflow.
|
||
|
||
### New Features
|
||
- `clawhdf5-agent`: **compress fixed-length string datasets** (memory text
|
||
chunks, session summaries, ids, tags, entity/relation names, …). These were
|
||
always stored uncompressed with a "chunked compound not yet supported" note
|
||
that was simply stale — chunked writes work for fixed-size string/compound
|
||
datatypes like any other. `write_string_dataset` now chunks + deflates a
|
||
string dataset once its payload reaches 4 KiB, so large, highly-redundant
|
||
NullPad content shrinks substantially while tiny metadata stays contiguous
|
||
(no chunk-overhead bloat).
|
||
- `clawhdf5-format`: decode the **scale-offset filter** (id 6) — both the
|
||
integer variant (`H5Z_SO_INT`) and the floating-point **D-scale** variant
|
||
(`H5Z_SO_FLOAT_DSCALE`). Handles signed/unsigned int sizes, f32/f64, negative
|
||
minima, decimal scale factors and fill values; reverse-engineered against
|
||
HDF5 2.0 and validated end-to-end. The float E-scale variant remains
|
||
unsupported.
|
||
- `clawhdf5-format`: decode the **N-Bit filter** (id 5) — atomic, **compound**
|
||
and **array** layouts (the full recursive type tree, nestable to any depth),
|
||
previously unsupported. Signed and unsigned reduced-precision integers and
|
||
float members all read end-to-end, validated against HDF5 2.0.
|
||
|
||
### New Features
|
||
- `clawhdf5` / `clawhdf5-format`: read **external-file Virtual Datasets (VDS)**.
|
||
The format layer gains `read_raw_data_full_with_resolver` and a
|
||
`VdsSourceResolver` callback (`Fn(&str) -> Option<Vec<u8>>`) that maps a
|
||
stored source file name to its bytes, so the pure-byte reader can pull in
|
||
external sources without a filesystem of its own. The `clawhdf5` `File` API
|
||
wires a default resolver that reads sibling source files relative to the
|
||
opened file's directory, so `File::open(...).dataset(...).read_*()` now
|
||
transparently assembles cross-file VDS. A source file the resolver cannot
|
||
supply leaves its region at the fill value (matching HDF5); an external
|
||
source with no resolver at all is a clean error. In-memory files
|
||
(`File::from_bytes`) have no directory, so only same-file VDS resolves there.
|
||
- `clawhdf5-format`: assemble **same-file Virtual Datasets (VDS)** of any rank.
|
||
Previously a virtual layout returned `UnsupportedVersion`. The reader now
|
||
decodes the global-heap mapping block (reverse-engineered against HDF5 2.0:
|
||
`version · nused · [source-file · source-dataset · source-selection ·
|
||
virtual-selection]* · checksum`, including the block-version-1 same-file
|
||
marker), decodes the `H5S` source/virtual dataspace **selections** (ALL,
|
||
NONE, and version-3 regular hyperslabs), reads each same-file source dataset,
|
||
and scatters its selected elements into the virtual buffer in row-major order
|
||
(so multi-dimensional block mappings land correctly); unmapped regions are
|
||
left at the zero fill value. External-file sources return a clean unsupported
|
||
error. The previous `parse_vds_mappings` used a guessed layout that did not
|
||
match real files and is replaced.
|
||
|
||
### Tests
|
||
- `clawhdf5-format`: regression test for **scale-offset float E-scale**
|
||
datasets. The HDF5 library does not implement E-scale encoding — when asked
|
||
for it (`cd_values[0] = 1`) it stores the chunk raw and sets the chunk filter
|
||
mask to skip the filter — so these files read back verbatim purely by
|
||
honoring the per-chunk filter mask. The test locks in that behavior against a
|
||
fixture produced via the HDF5 low-level API; no E-scale decoder is needed.
|
||
|
||
### Bug Fixes
|
||
- `clawhdf5-format`: **read multi-direct-block fractal heaps**. The reader split
|
||
direct vs indirect block rows using the FRHP "Starting # of Rows in Root
|
||
Indirect Block" field (a constant, typically 1), so any heap whose data spans
|
||
more than one direct block — common in libhdf5 files with a large group or
|
||
many dense attributes — was misread as having indirect blocks and failed with
|
||
`InvalidFractalHeapSignature`. The split is now derived from the heap geometry
|
||
(`max_direct_rows = log2(max_direct / start) + 2`). Validated against an
|
||
h5py-written 400-dense-attribute group (root indirect block, 4 rows, 13 direct
|
||
blocks).
|
||
- `clawhdf5-format`: scope the per-file **chunk cache by dataset**. The shared
|
||
`ChunkCache` built its chunk index once and reused it for every chunked
|
||
dataset in the file, keyed only by chunk coordinate with no dataset
|
||
discrimination. With a single chunked dataset per file this was latent; once a
|
||
file holds two chunked datasets of different rank (e.g. a 1-D compressed
|
||
string array and the 2-D embeddings matrix), the first dataset's index was
|
||
reused for the second, panicking with an out-of-bounds chunk coordinate. The
|
||
cache now rebinds (dropping its index, chunk-index map, layout, and
|
||
decompressed slots) whenever the dataset being read changes, while still
|
||
caching repeated/sequential access to the same dataset.
|
||
- `clawhdf5-format`: read **paged Fixed Array** chunk indexes. A filtered,
|
||
fixed-dimension dataset with more than one data-block page (>1024 chunks by
|
||
default) previously failed with "paged Fixed Array data blocks not yet
|
||
supported". The reader now walks the page-init bitmap (MSB-first), skips
|
||
uninitialized pages, and resolves each page's fixed full-size slot (including
|
||
the short final page). Reverse-engineered and validated end-to-end against an
|
||
HDF5 2.0 file.
|
||
- `clawhdf5-format`: read **array-typed datatypes** (e.g. an array-typed
|
||
compound member) via `read_as_i32/i64/u64/f32/f64` — previously a
|
||
`TypeMismatch`. The array is read as a flat sequence of its base elements
|
||
(recursing for nested arrays), applying base-type precision rules.
|
||
- `clawhdf5-format`: **sign-extend reduced-precision fixed-point integers** on
|
||
read. A signed integer whose datatype precision is smaller than its storage
|
||
size is stored zero-filled, so e.g. a 16-bit-precision `-1` previously read as
|
||
`65535`. The integer read paths now extract the precision field and
|
||
sign-extend (full-width types are unchanged). Completes signed N-Bit reads and
|
||
also fixes un-filtered reduced-precision integer datasets.
|
||
- `clawhdf5-format`: read datasets written by modern HDF5 (1.14+/2.0, i.e.
|
||
`libver=latest`). Compound (class 6) and array (class 10) datatype **version 5**
|
||
messages and data layout **version 5** messages were rejected as invalid; they
|
||
reuse the v3/v4 binary structure, so they are now accepted. This unblocks
|
||
reading compound types and — critically — every chunked/compressed dataset
|
||
written by HDF5 2.0. Found by running the h5py interop tests against
|
||
h5py 3.16 / HDF5 2.0.
|
||
Independently reported (with a patch) against the v2.1.0 tag by
|
||
M. Scot Breitenfeld (The HDF Group) — v2.1.0 predates this fix.
|
||
- `clawhdf5-format`: parse HDF5 2.0 native complex datatypes (class 11,
|
||
datatype version 5, e.g. `H5T_COMPLEX_IEEE_F64LE`). The properties are a
|
||
single base floating-point datatype, not a compound-style member list; the
|
||
old parser read the base type's bytes as member names, producing a garbage
|
||
datatype, and failed with `UnexpectedEof` when a complex type was nested in
|
||
a compound. It is now surfaced as the equivalent `{r, i}` compound (the
|
||
shape h5py writes for numpy complex dtypes), with a size check against the
|
||
base type. Validated end-to-end against an HDF5 2.0-written file.
|
||
|
||
### Performance
|
||
- `clawhdf5-format`: chunked writes now compress all chunks up front via
|
||
`compress_all_chunks`, running across rayon threads under the `parallel`
|
||
feature when there are more than 4 filtered chunks. On-disk layout is
|
||
unchanged. Speeds up compressed embedding writes in `clawhdf5-agent` (which
|
||
enables `parallel`).
|
||
|
||
### Documentation
|
||
- Fix stale package names across all 13 per-crate READMEs (`rustyhdf5-*` /
|
||
`edgehdf5-*` → `clawhdf5-*`, usage versions → 2.1.0).
|
||
- Correct README workspace/test/crate stats and the CLAUDE.md CLI subcommand
|
||
list; document the `hnsw` and format compression/checksum feature flags and
|
||
the `entity_extract` / `async_memory` modules.
|
||
|
||
## v2.1.0 (2026-06-03)
|
||
|
||
### New Features
|
||
- `clawhdf5-agent`: HNSW now backs the vector stage of `hybrid_search`. The
|
||
`hnsw` feature is **on by default**, so semantic search uses the approximate
|
||
`clawhdf5-ann` index instead of a linear cosine scan. The index mirrors the
|
||
memory cache (node id == cache index) and self-heals — it rebuilds whenever it
|
||
drifts from the cache length, so no mutation path can desync it. Non-indexable
|
||
stores (no/zero-dim/mixed embeddings) and dimension-mismatched queries fall
|
||
back to the exact linear scan. Disable with
|
||
`--no-default-features --features float16` for exact search.
|
||
- `clawhdf5-ann`: HNSW is now a live, mutable index — added `insert`,
|
||
`mark_deleted` (soft-delete bitset; deleted nodes are traversed for
|
||
connectivity but never returned), `compact` (drops deleted vectors and
|
||
renumbers survivors), and `new` (empty index). Serialization gains a format
|
||
version tag (`HNSW_FORMAT_VERSION` = 2) and persists the deleted bitset;
|
||
pre-existing v1 files still load.
|
||
- `clawhdf5-agent`: `hybrid::merge_vector_keyword` exposes the shared
|
||
normalize-and-fuse step used by both the linear and HNSW vector paths.
|
||
- Expose `max_dimensions()` API on Dataset, MmapDataset, and LazyDataset
|
||
- NetCDF-4 unlimited dimension detection now works correctly
|
||
- Python bindings (`clawhdf5-py`) build and link on macOS with system Python
|
||
|
||
### Bug Fixes
|
||
- `clawhdf5-py`: upgrade PyO3 and numpy `0.23` → `0.28` so the bindings build on
|
||
Python 3.14 (PyO3 0.23 capped at 3.13 and hard-failed `cargo build
|
||
--workspace`). Updated for the removed `PyObject` alias (`Py<PyAny>`) and the
|
||
`Python::allow_threads` → `Python::detach` rename.
|
||
- Fix GPU L2 distance test (squared vs actual L2 mismatch in test helper)
|
||
- Mark Android JNI functions as `unsafe` for Rust 2024 edition compliance
|
||
- Add `# Safety` documentation to all public unsafe extern functions
|
||
- Fix all clippy warnings: needless_range_loop, manual_strip, ptr_arg, etc.
|
||
- Rename `RelationType::from_str` to `from_label` to avoid trait confusion
|
||
- Isolate h5py interop tests with `#[ignore]` when h5py unavailable
|
||
|
||
### Code Quality
|
||
- Full rustfmt pass across workspace (61 files)
|
||
- Refine inner unsafe blocks for Rust 2024 edition style
|
||
- Zero clippy warnings, zero clippy errors across entire workspace
|
||
- 1,546 tests passing, 0 failures
|
||
|
||
## v2.0.0 (2026-03-19)
|
||
|
||
- Unified rustyhdf5 (11 crates) and edgehdf5 (4 crates) into a single workspace
|
||
- All crates renamed to clawhdf5-* prefix
|
||
- Version bumped to 2.0.0 across all crates
|
||
- Git dependencies replaced with in-workspace path dependencies
|
||
- Added `agent` feature flag to clawhdf5-agent
|
||
|