The docs described a "drop-in" OpenClaw memory backend enabled with
`memory.backend = "clawhdf5"`. Checked against OpenClaw's source and
docs (v2026.2.26 through v2026.9.6): that config was never valid —
v2026.2-v2026.7 accepted only "builtin"/"qmd" and rejected unknown
keys, so a Gateway given it refuses to start, and v2026.8.1 (OpenClaw
2.0) removed the key. No plugin was ever built (no manifest, no
registration, no tools), nothing was tested against OpenClaw, the
linked github.com/redclawsystems/openclaw is a 404, and
@redclaw/clawhdf5 was never published.
Decision (2026-09-25): not pursuing an OpenClaw plugin for now; ZeroClaw
is the integration target.
- Remove openclaw-integration.md, openclaw-config.md and
migration-guide.md; add docs/openclaw.md: the status, what a memory
plugin needs against v2026.9.6 (plugins.slots.memory, manifest with
kind "memory", registerMemoryCapability / MemorySearchManager,
prebuilt native packages), and what this repo has as building blocks.
- README, QUICKSTART, USE_CASES, ROADMAP (Track 7 withdrawn), CLAUDE.md
and the `openclaw` module docs describe ClawhdfBackend as what it is:
a Markdown-oriented library backend, not an OpenClaw plugin. The
QUICKSTART example is corrected (the old one called a three-argument
create that does not exist) and states its limits.
- packages/clawhdf5-node: marked unpublished and broken, "private": true
so it cannot be published by accident; its bugs (snake_case vs
camelCase fields, wrong addon path, no way to store an embedding,
wrong WAL name) are recorded in docs/known-issues.md.
- Two broken rustdoc links fixed along the way.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Makes the README's "cryptographically verifiable memory" true.
With HDF5Memory::set_signing_key(key), every checkpoint stores a signed
manifest of the store: a SHA-256 per memory record (text, embedding as
stored, channel, timestamp, session, tags, deleted flag, activation) in
a Merkle tree, plus hashes of the settings (and WAL mark), sessions and
knowledge graph. The signature, public key and manifest hashes go in
/meta; the per-record hashes in /integrity/record_hashes, so
HDF5Memory::verify(path, &public_key) can say which records changed, not
just that something did. A forged manifest fails the signature.
Decisions, as agreed:
- the key is set on the open store and never persisted;
- a signed store refuses to checkpoint without its key
(MemoryError::SigningKeyRequired); remove_signature() is the
deliberate way back to unsigned;
- checkpoints only: saves still in the WAL are not covered, and verify
reports how many there are.
The hashes cover exactly what the file persists, in the form the loader
returns it (strings lose trailing NULs; an empty WAL mark is not
written), so untouched stores verify across any number of reopen and
checkpoint cycles. MemoryError becomes #[non_exhaustive] (it already
gains variants in this unreleased version).
CLI: keygen (owner-only key file), --signing-key / CLAWHDF5_SIGNING_KEY
on writing commands (create signs immediately), verify --public-key
(JSON; exit 2 if not valid), `signed` in create/stats output.
Tests: reopen/checkpoint cycles with awkward strings (f16 and f32),
refusal without the key, wrong and rotated keys, eight kinds of edit
each detected and located, a forged manifest, unsigned stores, NULs in
text, and an edit made in place with h5py that verify pinpoints.
Cost on tank (search_harness --signing-study --full, 3 runs): ~20% of a
checkpoint (+9 ms at 10K, +89-112 ms at 100K), verify 18.6 ms / 247 ms,
32 bytes per record in the file. New deps ed25519-dalek, sha2,
rand_core: pure Rust, the no-C check passes.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
It enabled nothing — the agent layer is always built — yet the README,
QUICKSTART and USE_CASES told people to pass it. Removed, with those
snippets fixed: they now depend on the git repository (nothing is on
crates.io, so `version = "2.0"` never resolved) and USE_CASES no longer
presents the `float16` feature as half-precision storage (that is
MemoryConfig::float16, on by default for new stores).
Breaking for anyone passing `features = ["agent"]`: drop it.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
consolidation_efficiency never finished: stopped after 19 minutes on one
core while building its 100K case. Not the consolidation cycle (linear:
17 us at 100 records, 2.16 ms at 10K) but the setup — every add_memory
scores the new record's novelty against the whole working tier, the
benchmark lets that tier reach 50K, and each comparison recomputed both
norms: ~5e9 comparisons of three passes each.
ImportanceScorer::score_surprise now computes the new record's norm
once, takes each comparison in one fused, 8-lane pass (dot product and
the other norm together), and splits a working tier of 4096+ records
across threads with the `parallel` feature. Same results: tested against
the old cosine formula, including shorter, empty and zero vectors and
the parallel path. The work stays quadratic in the working-tier size by
design; with regular consolidation the tier stays near
working_capacity (100) and inserts are cheap.
The complete run takes 8 min 10 s on tank and fills in the 100K cycle
row (46.66 ms) and the memory-reduction table, which had never been
published. The binary no longer prints a record-count ratio as a
"BM25 Speedup" (never measured; Part 1 measures search latency) or
claims sub-linear cycle scaling (its own numbers grow slightly faster
than linearly).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Every undated or pre-September section re-run on one machine on one day
(tank, AMD Ryzen 7 7800X3D, 2026-09-24, commit 5c8323c), 24 commands run
serially with the load average checked before each, with the command
recorded for each section. A separate check traced every changed number
back to the raw output; its corrections are applied (e.g. the on-disk
~820 B/record is float16 plus always-deflated text on a synthetic corpus
of 40 distinct texts, not float16 alone).
Two apparent regressions were isolated rather than published:
- knowledge-graph traversal: a real bug, fixed in the previous commit;
- the write path: v2.3.0 built and run on the same machine measures the
same as today, so the old 18 us / 6.17 ms figures (undated, other
hardware) are not reproducible; float16 adds ~2 us per save and the
int8 index nothing (both isolated by switching the bench's config).
Also:
- new multimodal_bench: cross-modal search at 1K/10K records, which the
README claimed but nothing measured;
- footprint_bench reports whether it built float16 or f32 stores and
takes --f32 (it kept printing "f32" after the default changed);
- README: performance tables, the "Why" table figures and the SQLite
migration section (from the previous migrate commit);
- CHANGELOG for this branch.
Not re-run: consolidation_efficiency's 100K row and its memory-reduction
part (stopped for time), and cross_platform.sh.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
bfs_neighbors and spreading_activation built an adjacency index over the
whole graph on every call (1efd82c), so a 2-hop BFS over 1K entities
paid to index every entity and relation first: 155 us, 6.5x the 24 us
the README quoted. Found by the dated benchmark re-run.
The index is now cached on KnowledgeCache and checked against a
fingerprint of the graph on each use — one pass over entity ids and
relation endpoints, no allocation — so any change, including direct
edits of the public entities/relations Vecs (schema.rs's load path
pushes to them), still triggers a rebuild. A test edits the graph
directly in every way (push, in-place rewire, pop + push at equal
length) between traversals.
tank, 2026-09-24: BFS 1K entities 155.1 -> 23.1 us, 100 entities
17.5 -> 5.23 us, spreading activation 100 22.8 -> 10.1 us.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
clawhdf5-migrate wrote a layout of its own (/chunks, /sessions,
/entities, /relations, root attributes, no /meta or schema_version) that
HDF5Memory::open rejects, so a "migrated" SQLite database could not be
used as agent memory — contrary to the README.
It now writes through the agent's own API (HDF5Memory::create/open,
save_batch, the session cache and the knowledge graph), so there is no
second copy of the schema:
- sessions and entities/relations carry over; deleted rows become
deleted records (or are left out with --skip-deleted);
- embeddings follow the library default (float16), --f32 opts out and
--float16 is a hidden no-op, as in clawhdf5-cli; the `half`-based
conversion is gone;
- every source row is checked before the output is created: a wrong
embedding length, an empty embedding, a dimension that differs from
an existing store's, or a float16 value beyond +-65504 is an error
naming the chunk id, and an existing store is left untouched;
- --incremental opens the existing store, adds only rows it does not
hold (matched by content) and follows the source's deleted flags;
- a source with no memory rows needs --embedding-dim;
- validation reads the result back with HDF5Memory::open_read_only,
compares every field (embeddings bit for bit, round_to_f16 of the
source for float16) and checks a migrated record is found by search.
clawhdf5-agent gains HDF5Memory::sessions()/sessions_mut(),
HDF5Memory::delete_batch (one save, all-or-nothing, no auto-compact),
SessionCache::add_at, and re-exports SessionCache/SessionEntry.
The old layout's per-dataset SHA-256 provenance attributes have no place
in the agent schema and are gone. An adversarial review found two
blockers (silent truncation of long embeddings; an --incremental
dimension check that could never fire) and four majors (a failed run
wiping the existing store, dim-0 stores, deleted-flag drift); all are
fixed with regression tests. 42 migrate tests, incl. h5py opening a
migrated store.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
MemoryConfig::float16 now defaults to true for new stores, on
measurement: on the full LongMemEval haystack with real MiniLM
embeddings every retrieval metric matched f32 (previous commit), and at
100K the file is 48% smaller with faster checkpoints and opens.
Existing stores are unaffected: every agent store has recorded
`float16 = false` in /meta and keeps it. A test opens the v2.5.0
fixture, saves and checkpoints, and checks the embeddings are still f32
with the old rows bit-identical; another checks a new store is float16.
CLI: `create --f32` opts out; like `--f32-index` it only ever switches
the default off. `--float16` is still accepted and now a no-op.
Values beyond +-65504 are refused, so f32 remains the choice for
unnormalised vectors — the upgrade note says so.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
`longmemeval_bench --float16` builds every per-question store with
MemoryConfig::float16, so the vector stage searches half-rounded
embeddings exactly as such a store holds them.
Full longmemeval_s (500 questions, ~494 turns each) with real
all-MiniLM-L6-v2 embeddings, f32 vs float16, on tank (CUDA): identical
at every Hit@k and MRR, turn and session level, in all eight modes —
bar RRF session MRR 0.9253 vs 0.9254 and one or two flips out of ~320
in which gold session ranks first. The f32 run reproduces the published
hybrid numbers exactly. The earlier float16 evidence was synthetic
clustered data only; this is the real-embedding check.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
`HDF5Memory::search(query_embedding, query_text, &SearchOptions)` is the
store's full search path. `SearchOptions::new(k)` is plain hybrid search
with the tuned default fusion; each further stage is opt-in:
- `with_sources([..])`: only records from these source channels. The
filter applies before ranking, so a filtered search still returns up
to k results, normalised over what it can return. The HNSW pool is
over-fetched in proportion to what the filter removes, and the allowed
records are scanned exactly whenever that costs fewer distance
evaluations than the index would (~pool x M) — and as the fallback if
the pool comes back short. Keyword matches are filtered too.
- `with_rerank(ReRankConfig)` re-ranks a max(3k, 10) candidate pool by
relevance, recency, source authority and activation;
`with_confidence(ConfidenceConfig)` drops low-confidence results;
`at_time(now)` pins the recency clock.
These were reachable only through the OpenClaw backend, which is now
`search` with both on. Its Hebbian boost now goes to the k results it
returns rather than the whole 3k candidate pool. `hybrid_search` and
`hybrid_search_with` are wrappers and unchanged (tested bit for bit).
Measured on tank (search_harness --options-study --full, 3 runs): at
100K every filter — 50%, 10%, 1% of the store, and records far from the
query — returns the exact filtered top 10, and none is slower than an
unfiltered search (1%: 2.3 ms vs 4.6 ms). Re-rank + confidence costs
about 3%. A first version decided between index and exact scan by pool
size vs store size; it measured 0.976 recall at 12.3 ms on the
far-from-query filter, which is why the rule compares costs instead.
Tests: tests/search_options.rs (filter correctness and full pages via
both paths, far-from-query fallback, edge cases, equality with
hybrid_search_with, re-rank recency, confidence, boost scope).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The setting was persisted in /meta and otherwise ignored: embeddings
were always written as f32. It now does what it says.
clawhdf5-format:
- `DatasetBuilder::with_f16_data` writes IEEE binary16 (numpy float16),
rounding to nearest-even, and `make_f16_type`.
- `clawhdf5_format::float16` holds the f32 <-> f16 conversions, the one
implementation the writer, the reader and the agent all use. Checked
against the `half` crate on 16.7M f32 values and round-trips all 65536
half values; the h5py interop tests confirm the rounding matches
numpy's bit for bit (4020 values incl. ties, subnormals, overflow).
- Reading little-endian float16 as f32 has a fast path.
clawhdf5-agent:
- A float16 store writes /memory/embeddings as half precision, and
`MemoryCache::half_precision` rounds each embedding as it enters the
cache (save, update, WAL replay, and on load of a store still f32 on
disk), so memory and file agree bit for bit and a store searches the
same before and after a reopen (tested).
- Values beyond +-65504 are refused with the new
`MemoryError::InvalidEntry` rather than stored as infinity, on every
save path; batches are all or nothing, and a rejected ephemeral entry
stays in the ephemeral tier. Breaking for exhaustive matches.
- CLI: `create --float16`. Off by default.
Measured on tank, 384-dim, six runs alternating order, medians
(search_harness --float16-study --full): at 100K the file goes from
154.0 to 80.8 MiB (-48%), checkpoint 752 -> 512 ms, open 300 -> 252 ms;
vector recall@10 against an exact scan and hybrid_search latency do not
change. At 10K open is 3 ms slower. Also a test that h5py opens a whole
agent store, f32 and float16, and decodes every dataset.
Docs: README, BENCHMARKS.md ("float16 embedding storage"), CHANGELOG
(including the h5py interop fixes in the previous commit), CLAUDE.md.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Two write-side bugs, both present in every release (the first at least
since v2.1.0), made libhdf5 refuse files written by clawhdf5. Our own
reader ignores both fields, and the interop suites only ever wrote f64
from our side, so nothing here caught them.
- Every f32 dataset: "sign bit position out of bounds". The float
datatype encoder hard-coded the sign bit's position (bits 8-15 of the
class bit field) to 63, which is right only for f64. It is now derived
from the type: bit_offset + bit_precision - 1. This covered every
agent store's embeddings, norms and activation weights.
- Every empty dataset: "invalid dataset size, likely file corruption".
It was written with a real address and size 0, which trips libhdf5's
`addr + size <= addr` overflow check. An empty contiguous dataset now
gets the undefined address, as libhdf5 writes it. This covered every
agent store without sessions or a knowledge graph.
Agent stores are rewritten in full at each checkpoint, so they become
readable at their next checkpoint on a fixed build; other files with f32
or empty datasets need rewriting. Both are recorded in
docs/known-issues.md.
Tests: the sign position byte for f32/f64, and h5py reading our f32
datasets (plain and chunked + deflate) bit for bit.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
rust-version = "1.92" in [workspace.package], inherited by every crate.
1.92 is the floor: wgpu (clawhdf5-gpu) requires it, and the whole
workspace, Python bindings included, checks cleanly on it. ci-test.sh
reads the version from Cargo.toml and checks the workspace on exactly
that toolchain, so the manifests and the README badge cannot drift from
what actually builds. The badge said 1.75, below edition 2024's own
floor.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The core crates (clawhdf5, -agent, -format, -io, -filters, -ann, -accel,
-netcdf4, -cli) now build no C by default: deflate defaults to zlib-rs,
a pure-Rust port of zlib-ng, and zlib-ng becomes the opt-in
`fast-deflate`, which overrides zlib-rs wherever it is enabled. A default
build no longer needs cmake or a C compiler.
Measured on tank, both builds run alternately, three rounds, medians:
zlib-rs is within 6% of zlib-ng on every HDF5 read and write (512x512
deflate-6 chunked write 1.458 vs 1.484 ms; 64 MB compressed read 64.4
vs 65.2 ms), and compressed output is byte-identical. Details in
BENCHMARKS.md, "Deflate backend".
Getting there took two fixes the first measurement exposed:
- zlib-rs needs `std` to detect SIMD at runtime. flate2 enables it via
its default `runtime_detection`, which `default-features = false` had
switched off, leaving zlib-rs 3.5x slower on inflate. The `zlib-rs`
features now enable it.
- Both deflate paths streamed through flate2's 32 KiB read/write
wrappers. They now hand the codec the whole chunk in one call, into a
buffer sized up front (~5% on chunked writes). This also fixes a
silent short read: the streaming reader returned a truncated stream's
bytes without an error; a truncated chunk is now DecompressionError.
In clawhdf5-filters, output longer than the stated size is now an
error rather than silently cut off.
CI: ci-test.sh lints and tests the zlib-ng path, and fails if a
C-building crate (*-sys, cc, cmake) enters a core crate's default
dependency tree. The arm64 job no longer installs cmake.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The README had fallen behind v2.3.0-v2.7.0, and parts of it were not
true. Checked every claim against the code and BENCHMARKS.md:
- Three of the six Quick Start snippets no longer compiled (Agent
Memory, Consolidation, OpenClaw); all six now do.
- Hybrid search was described as RRF throughout. The default has been
weighted 0.4/0.6 fusion since v2.5.0; re-ranking and confidence
rejection run only in the OpenClaw backend.
- The `float16` feature does not halve embedding storage (the store
always writes f32), `--features agent` enables nothing, "Source
Isolation" is not wired in, and nothing backs "billion-scale" IVF-PQ.
- "Cryptographically verifiable" overstated an unkeyed, session-scoped
FNV-1a ledger; "Zero C dependencies" was false while zlib-ng was the
default deflate backend.
- Stale numbers: tests (1,650 -> 1,868), Rust badge (1.75 is below
edition 2024's floor), 6.5 KB/record on disk (BENCHMARKS.md: 1.7 KB),
consolidation and hybrid-search latency, and a feature-flag table
broken by a paragraph pasted into it.
- The file schema, module table and crate map now match the code.
Adds a "What's new (v2.2 -> v2.7)" section for collaborators, leading
with the silent Extensible Array read bug fixed in v2.7.0. Footer links
point at git.redclaw.dev. CLAUDE.md: clawhdf5-migrate is the SQLite
migration tool, and MemoryConfig::compression is off by default.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Two jobs, which runners serve them, and the two constraints that kept
the x86 job failing on every push until today: no JavaScript actions
(`rust:latest` has no `node`, and GitHub is not reachable from every
runner) and `cmake` for libz-ng-sys. Also notes that the Docker Hub
`latest` tag for the runner is frozen at 0.6.1, so it is not a way to
stay current.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Read from the job logs of the first run with the arm64 job, rather than
guessed:
- The x86 `test` job has been failing on every push, in about three
seconds. It runs in `rust:latest` and starts with actions/checkout, a
JavaScript action, and `rust:latest` has no `node`: exit 127 before a
line of code was built. It is replaced with a plain git checkout (the
image has git), and actions/cache, JavaScript for the same reason, is
dropped.
- `test-arm64` got through checkout, toolchain, the aarch64 check and
clippy, then failed building libz-ng-sys — pulled in by
clawhdf5-format's default `fast-deflate` — because the host runner had
no cmake. The x86 image lacks it too, so the x86 job would have hit the
same wall one step later.
Both jobs now install cmake where they can (the Docker job and the x86
container run as root) and say plainly when they cannot (a host runner),
instead of failing inside a build script. vision-01 now has cmake.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The NEON kernels in clawhdf5-accel — `dot_i8` including its SDOT path,
and the f32 NEON kernels that predate it — are cfg'd out on x86, so the
existing job has never compiled, linted or tested a line of them. They
were verified once, by hand, on a Raspberry Pi 5.
`test-arm64` runs on `linux_arm64`, which two runners serve in different
ways: vision-01 executes steps on the host with Rust preinstalled, and
vision-02 executes them in docker.gitea.com/runner-images. The job is
written to work in both: no `container:`, no JavaScript actions (those
are fetched from GitHub, which not every runner reliably reaches), and
an explicit `+stable` toolchain so a host's default — vision-01's is a
January nightly — is neither relied on nor changed. Fetches retry, since
one runner's outbound network was seen failing intermittently.
It lints the accel crate and tests accel, ann and format. It reports
rather than requires the dot-product extension: on a core without it the
plain-NEON kernel is the one that runs, and the tests cover whichever is
present.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
quantized_index defaults to true for new stores — smaller and faster at
equal recall on every configuration measured. Existing stores keep
their setting, and stores predating it stay f32, guarded by a real
v2.5.0 store committed as a test fixture.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
`MemoryConfig::quantized_index` now defaults to `true`. It holds a
quarter of the index memory and, with the exact re-score, is faster at
equal recall on every configuration measured: 1.63x the queries per
second on x86-64 (AVX2) and 1.18x on a Raspberry Pi 5 (NEON SDOT), with
builds 1.8x and 2.3x faster. The one argument for keeping it off — that
int8 search was slower on ARM — did not survive being measured.
Existing stores do not change. A store written with v2.6.0 or later
keeps its persisted setting. One written before the setting existed has
no stored value, and it loads as `false` rather than as the new default,
so reopening it never changes how its index is held. That case is
guarded by a real store written with the v2.5.0 CLI, committed as
`tests/fixtures/store_v2_5_0.h5` (6.8 KB): the test asserts it reopens
with an f32 index and still searches, and it fails if the load default
is changed to `true`.
The CLI needed more than a new default. `create --quantized-index`
assigned its value straight into the config, so under the new default
every CLI-created store would have been forced back to f32 unless the
caller knew to ask for int8. It is replaced by `--f32-index`, which only
ever switches the default off; `--quantized-index` is still accepted,
hidden, as a no-op, and the two conflict.
The whole agent suite passes under the new default, including the
brute-force recall oracle, now running on int8 plus re-score without
being asked to.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
SDOT and plain-NEON kernels for dot_i8, tested bit-exact against scalar
on real ARM. At equal recall the quantised index is 1.18x f32 on a Pi 5
and builds 2.3x faster. Also corrects an unmeasured claim that it was
slower than f32 on ARM.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
On a Raspberry Pi 5 at N = 100 000 and equal recall (0.9940 vs 0.9945),
medians of three runs:
f32 33 413 ms build 6 164 QPS
int8 scalar 18 950 ms ~6 190 QPS (what v2.7.0 shipped)
int8 NEON ~17 000 ms 6 640 QPS
int8 SDOT 14 464 ms 7 267 QPS 1.18x f32, 2.3x build
The docs said quantised search stayed off by default because aarch64
"falls back to the scalar loop, where the original trade still
applies" — that it was ~13% slower than f32 there, as on x86. That was
extrapolated rather than measured, and it was wrong: x86's portable
baseline is SSE2 against hand-written AVX2 f32 kernels, but on aarch64
NEON is the baseline and the scalar loop vectorises well, so it already
matched f32. Corrected in BENCHMARKS.md, README.md and CLAUDE.md; the
released v2.7.0 changelog entry is left as it was and the correction is
recorded in a new one.
Labelled as Pi 5 figures throughout — a Pi's memory bandwidth and cache
are far below an M-series or flagship phone, so the ratios will move.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
`dot_i8` had an AVX2 kernel and a scalar fallback, so on aarch64 the
quantised HNSW index ran the scalar loop. It now dispatches to one of
two NEON kernels:
- `dot_i8_dotprod`: the ARMv8.2 dot-product instruction, `SDOT`, which
multiplies and accumulates sixteen i8 pairs into four i32 lanes per
instruction. Present on Cortex-A76 and later (Raspberry Pi 5, current
Android phones), Neoverse-N1 (Graviton2, Ampere Altra) and every Apple
Silicon generation. Issued as inline assembly because the `vdotq_s32`
intrinsic is still behind the unstable `stdarch_neon_dotprod` feature;
inline asm is stable on aarch64.
- `dot_i8`: plain NEON for cores without the extension — `vmull_s8`
widens to i16 (even -128 * -128 fits) and `vpadalq_s16` folds adjacent
pairs into i32 accumulators, so nothing overflows.
Selected at runtime with `is_aarch64_feature_detected!("dotprod")`.
Verified on a Raspberry Pi 5 (Cortex-A76, `asimddp` present), not just
compiled — the aarch64 code is cfg'd out on x86, so x86 CI never builds
or lints it:
- both kernels bit-exact against scalar at every length, tails and
extremes included. Each is tested directly rather than through
dispatch, because dispatch only takes one path on a given CPU: on the
Pi, testing through it alone would never have run the plain-NEON
fallback at all.
- mutation-checked: dropping the SDOT kernel's second accumulator fails
at length 32, and using the low half twice in the NEON kernel fails at
length 16 — the first lengths that exercise each.
- the ANN suite passes, including int8 recall against ground truth.
- clippy clean with -D warnings on aarch64.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Graph degree and both candidate-list sizes were constants, so recall
could not be traded against memory or query speed. Now MemoryConfig
fields, persisted with the store, defaulting to today's behaviour.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Graph degree and the build- and query-time candidate list sizes were
constants, so a deployment had no way to trade recall against memory or
query speed. They are now `MemoryConfig::hnsw_m`,
`hnsw_ef_construction` and `hnsw_ef_search`, persisted with the store
and defaulting to exactly the previous behaviour (16, 64, and a query
list that scales with `k`).
Two things the straightforward version would have got wrong:
`clawhdf5-ann` asserts a graph degree of at least 2, so a configured 0 —
from a file, or from a caller reading 0 as "use the default" — aborted
the process inside the index builder. The store clamps instead, and a
test covers it: removing the clamp makes that test panic rather than
fail.
`ef_search` and the candidate pool handed to score fusion were the same
number. Tying the pool to the new setting would mean lowering `ef` for
speed also narrows what fusion sees, quietly degrading hybrid results
through a knob that looks like it only costs time. They are now
independent.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
read_from_disk mapped the file and then copied the whole mapping for
File::from_bytes, which maps it itself. Store open is ~28% faster
(455 ms -> 327 ms at 100k x 384); peak memory is unchanged, because the
peak falls after the parse during the index build.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
`read_from_disk` memory-mapped the file and then copied the entire
mapping into a `Vec` to hand to `File::from_bytes` — but `File::open`
memory-maps it itself whenever the facade's `mmap` feature is on, which
it is by default. So every open mapped the file, memcpy'd all of it, and
parsed the copy.
Store open at 100k x 384: 455 ms -> 327 ms, about 28% faster (two runs
after the change, 326.8 and 328.1 ms).
Peak memory is unchanged, which is worth saying because the opposite is
the natural assumption. The footprint harness now tracks a high-water
mark next to the retained figure, and it shows the peak falling after
the parse, during the index build — so a buffer allocated and freed
inside the parse never reaches it. Confirmed rather than assumed:
holding a deliberate extra copy of the whole file across the parse
leaves the peak exactly where it was, which is also what proved the
instrument was working before trusting its answer.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Fixed and Extensible Array structures all carry a Jenkins checksum that
was ignored. A single flipped bit in a chunk address parses cleanly and
points inside the file, so without the check the reader returns another
chunk's bytes as data.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Every structure in both chunk indexes — header, index block, super
block, data block and each data block page — carries a Jenkins lookup3
checksum, and all of them were parsed past and ignored.
What that costs is not a warning but correct data. Flip one low bit of a
chunk address and the index still has the right shape, the address still
lands inside the file, and the reader returns whatever bytes now sit
there as that chunk's contents. Nothing else in the parse can tell.
Verified in both directions. The checksums accept files written by
HDF5 2.0 from 100 to 200 000 chunks — dense, sparse, gzip-filtered and
paged — which also confirms the block layouts byte for byte, since a
wrong offset would fail every file. And an interop test corrupts an
address to check the read fails instead of returning data: removing the
verification makes that test fail with "corruption produced data instead
of an error", which is what it is there to prove.
The first version of that test passed with verification disabled — it
corrupted a byte a structural check already rejected, so it proved
nothing. Worth recording, since a test that passes for the wrong reason
looks exactly like coverage.
Hand-built fixtures now stamp real checksums, as HDF5 writers do, and
the Extensible Array ones no longer describe the superseded layout.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The quantised HNSW index was measured against f32 with a scalar loop on
one side and clawhdf5-accel's AVX2 kernels on the other, so the ~13%
throughput cost attributed to quantisation was a missing kernel.
With clawhdf5_accel::dot_i8 in place, at equal recall the quantised
index answers 1.63x the queries per second and builds 1.8x faster,
holding a quarter of the vectors. Still off by default, now because the
kernel is AVX2-only and aarch64 falls back to scalar.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Measured with `clawhdf5_accel::dot_i8` in place, medians of three
alternating runs at N = 100 000 x 384, same binary:
build f32 3197 ms int8 1778 ms int8+re-score 1826 ms
ef = 64 f32 13 399 QPS @ 0.9945 int8+re-score 21 848 QPS @ 0.9940
So at equal recall the quantised index is 1.63x the queries per second
and 1.8x the build speed, holding a quarter of the vectors. The earlier
"~13% of QPS" figure compared a scalar int8 loop against hand-written
AVX2 f32 kernels and was measuring the missing kernel; it is kept in
BENCHMARKS.md with that explanation rather than quietly replaced.
Still off by default, now for portability rather than performance: the
kernel is AVX2-only and aarch64 falls back to scalar, where the original
trade applies. A NEON kernel would settle it.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The int8-quantised HNSW index compared vectors with a scalar loop that
the compiler vectorised for the x86-64 baseline (SSE2), while the f32
path it was measured against goes through `clawhdf5-accel` and runs
AVX2. So the ~13% throughput cost recorded for `quantized_index` was a
missing kernel rather than a property of int8.
`clawhdf5_accel::dot_i8` adds a scalar fallback and an AVX2 path:
sign-extend each 16-byte half to i16, then `madd_epi16`, which
multiplies and sums adjacent pairs straight into i32 lanes. It is
dispatched through the same detected backend as the f32 kernels, and
the index now calls it.
Integer arithmetic, so the SIMD path must agree with scalar bit for
bit — tested at lengths that are and are not multiples of the block,
and at the -128 extreme for overflow.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Two read-path bugs found by auditing the improvement plan against the
code, both reaching every release up to v2.6.0.
Datasets indexed by an Extensible Array — any dataset with one
unlimited dimension — returned data from the wrong chunks past their
first few dozen, silently. The only fixture in the suite had three
chunks, inside the inline limit, so no test had ever read a data block.
A crafted file could abort the process through unbounded B-tree v2
recursion, or exhaust memory through shared subtrees. Both now refuse
in under a millisecond, and the fuzz target that should have caught it
(it only fuzzed header parsing) now walks the tree.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The Extensible Array bug reached a release because no fixture had more
chunks than fit inline, so its data blocks were never read. The Fixed
Array index had the same blind spot: nothing exercised it above a
handful of chunks, and nothing reached the paged layout at all.
Checked at 100, 5 000 and 200 000 chunks plus a sparse dataset that
leaves whole pages uninitialised. It is correct throughout — it does
keep its page-init bitmap inside the data block, which is the
difference from the Extensible Array that made assuming otherwise a
bug. Adding the tests so that stays true.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
A dataset with exactly one unlimited dimension — the ordinary
append-only case — is indexed by an Extensible Array. Only its first
few chunk entries (4 by default) sit inline in the index block, and
everything past them was read with the wrong layout. In the default
shape the 37th chunk onward came back from the wrong place: a
400-chunk dataset returned 364 wrong values while reporting success,
and beyond about a thousand chunks the read failed outright. Silently
wrong data is the worse half of that.
It survived because the only Extensible Array fixture in the suite had
three chunks — inside the inline limit — so no test ever reached a data
block.
Four layout errors, each confirmed against files written by HDF5 2.0 and
against the library source rather than inferred:
- super block `u` owns 2^(u/2) data blocks, not 2^u;
- each holds 2^((u+1)/2) * data_blk_min_elmts elements — the two
quantities double every *other* level, a half step apart;
- a super block carries a block-offset field before its data block
addresses, which was not skipped;
- the page-init bitmap belongs to the super block, one bit per page
packed across all of its data blocks and read MSB-first, rather than
living inside the data block; a paged data block also ends its prefix
with a checksum before the first page.
Where the spec left room for doubt the file settled it: decoding a
paged block's elements and reading the chunk values they address
identifies the mapping exactly, and the bitmap's 68 set bits matched
the 34 data blocks x 2 pages that 200 000 elements need, which only
holds MSB-first.
New interop tests cross every boundary — 4, 37, 400, 5 000 and 200 000
chunks, the last with paged data blocks — plus sparse (uninitialised
pages taking fill values), gzip-filtered elements and a 2-D dataset.
All three fail against the old traversal.
Writing is untouched; this was a read-path bug.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The existing target only called `BTreeV2Header::parse`, so the
recursive walk behind it — where a node that is its own child overflowed
the stack — was never fuzzed at all. Parsing also requires a valid
Jenkins checksum, which random input essentially never produces, so
almost every input stopped at the first branch.
The target now walks the tree after a successful parse, and also builds
a header straight from the input bytes so the traversal is reachable
without forging a checksum.
Checked both ways: against the unfixed traversal libFuzzer finds the
stack overflow (ASan: stack-overflow), and against the fix that same
input executes in 0 ms and 34.7 million further runs produce no crash,
timeout or OOM.
Corpora and crash artifacts stay out of the repository; the two crafted
inputs are covered by unit tests instead.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Traversal recursed one frame per level with the depth taken from the
file (a u16), and followed child addresses without asking whether they
were shared. Two crafted inputs, both reproduced before fixing:
- A node listing itself as its own child, under a header claiming 65 535
levels, overflowed the stack and aborted the process — SIGABRT, not an
error a caller can handle — from under 100 bytes.
- Levels whose children all point at one shared node below reached it
fan-out^depth times: 29.5 million records in 8 s from ~5 KB, and one
more level would exhaust memory.
Depth is now capped at 64, as the fractal heap already was; no real tree
approaches it, since even at the minimum fan-out of two that is over
2^64 records. And traversal stops once it has produced more records
than the file has bytes to hold them — a valid tree stores each record
once in its own bytes, so this bounds shared subtrees without trusting
the header's own `total_records`. Both inputs now fail in under a
millisecond.
Every B-tree v2 user goes through this collector: dense attributes, v2
groups, shared messages and chunk indexes. To show the budget never
refuses a real file, a new interop test has HDF5 2.0 write a depth-2
chunk index with 40 000 records and reads back all 160 000 values; it
fails when the budget is deliberately made too tight.
Also corrects `BM25Index::search`, which claimed to use Block-Max WAND.
It scores exhaustively, and pruning would not help the store:
`hybrid_search` needs every score because fusion normalises over them.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The workflow already installed h5py into /opt/interop and set
CLAWHDF5_REQUIRE_INTEROP=1, but it reached the tests only by appending
that venv to $GITHUB_PATH, which Gitea's runner does not reliably
propagate into test subprocesses. If `python3` resolved to the system
interpreter instead, every interop suite would skip. Setting
CLAWHDF5_PYTHON outright removes the question: together with
REQUIRE_INTEROP the suites either run or the build goes red.
Verified both directions locally — with a venv the four suites run 94
tests green; with a bogus interpreter and REQUIRE_INTEROP=1 the facade
and netCDF4 suites fail 22 tests rather than skipping.
Also documents creating the local `.venv` that `ci-test.sh` detects.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Cuts a loaded store's memory from 2.72x to 1.74x the raw vectors at 100k
x 384 via MemoryConfig::quantized_index, with recall held at the f32
index's level by re-scoring candidates against the exact embeddings the
store already holds. Off by default: it trades ~13% of QPS for the
memory.
Also restores the Python interop suites, which had been skipping
silently on this machine because no interpreter has h5py and PEP 668
blocks installing it into the system one.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Every Python interop suite had stopped running on this machine: the h5py
writer round-trips, the facade suite, netCDF4 and the reference files.
`python3` is 3.14, nothing on the box has h5py, and PEP 668 refuses to
install it into a system interpreter at all — so the availability probes
all returned false and each suite skipped without failing.
A silent skip here is exactly how the v5 compound-datatype bug reached a
release, so the probes now read `CLAWHDF5_PYTHON` and `ci-test.sh` picks
up `.venv/bin/python` on its own. The detection sits at the top of the
script rather than beside the interop step, because the non-ignored
suites run in the earlier `cargo test` step and would otherwise still
miss it. `CLAWHDF5_REQUIRE_INTEROP=1` continues to turn a skip into a
failure.
Verified against a venv with h5py 3.16 / HDF5 2.0.0: 94 interop tests
across the four suites, all passing.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
`MemoryConfig::quantized_index` stores the HNSW index's own copy of the
embeddings as i8 rather than f32. At 100k x 384 that takes the index from
266 to 123 MiB and the whole reopened store from 399 to 256 MiB — 2.72x
to 1.74x the raw vectors, the largest remaining item in the footprint.
Quantised distances are approximate and `ef` cannot compensate, because
the loss is in the distances rather than in the graph: recall@10 tops out
at 0.967 against f32's 0.9995 and does not move between ef=128 and
ef=256. The store already holds the exact embeddings, though, so when the
index is quantised the query path re-scores the candidate pool against
them before fusion. That restores recall (0.9940 vs 0.9945 at ef=64) and
costs about 13% of QPS.
Off by default: it trades query speed for memory and which side is worth
more depends on the deployment. The flag is persisted in `/meta`, so a
reopened store does not silently revert to four times the index memory,
and the sidecar graph is rehydrated into the configured storage.
Also on the CLI as `create --quantized-index`.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
osobh