The docs described a "drop-in" OpenClaw memory backend enabled with
`memory.backend = "clawhdf5"`. Checked against OpenClaw's source and
docs (v2026.2.26 through v2026.9.6): that config was never valid —
v2026.2-v2026.7 accepted only "builtin"/"qmd" and rejected unknown
keys, so a Gateway given it refuses to start, and v2026.8.1 (OpenClaw
2.0) removed the key. No plugin was ever built (no manifest, no
registration, no tools), nothing was tested against OpenClaw, the
linked github.com/redclawsystems/openclaw is a 404, and
@redclaw/clawhdf5 was never published.
Decision (2026-09-25): not pursuing an OpenClaw plugin for now; ZeroClaw
is the integration target.
- Remove openclaw-integration.md, openclaw-config.md and
migration-guide.md; add docs/openclaw.md: the status, what a memory
plugin needs against v2026.9.6 (plugins.slots.memory, manifest with
kind "memory", registerMemoryCapability / MemorySearchManager,
prebuilt native packages), and what this repo has as building blocks.
- README, QUICKSTART, USE_CASES, ROADMAP (Track 7 withdrawn), CLAUDE.md
and the `openclaw` module docs describe ClawhdfBackend as what it is:
a Markdown-oriented library backend, not an OpenClaw plugin. The
QUICKSTART example is corrected (the old one called a three-argument
create that does not exist) and states its limits.
- packages/clawhdf5-node: marked unpublished and broken, "private": true
so it cannot be published by accident; its bugs (snake_case vs
camelCase fields, wrong addon path, no way to store an embedding,
wrong WAL name) are recorded in docs/known-issues.md.
- Two broken rustdoc links fixed along the way.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
`HDF5Memory::search(query_embedding, query_text, &SearchOptions)` is the
store's full search path. `SearchOptions::new(k)` is plain hybrid search
with the tuned default fusion; each further stage is opt-in:
- `with_sources([..])`: only records from these source channels. The
filter applies before ranking, so a filtered search still returns up
to k results, normalised over what it can return. The HNSW pool is
over-fetched in proportion to what the filter removes, and the allowed
records are scanned exactly whenever that costs fewer distance
evaluations than the index would (~pool x M) — and as the fallback if
the pool comes back short. Keyword matches are filtered too.
- `with_rerank(ReRankConfig)` re-ranks a max(3k, 10) candidate pool by
relevance, recency, source authority and activation;
`with_confidence(ConfidenceConfig)` drops low-confidence results;
`at_time(now)` pins the recency clock.
These were reachable only through the OpenClaw backend, which is now
`search` with both on. Its Hebbian boost now goes to the k results it
returns rather than the whole 3k candidate pool. `hybrid_search` and
`hybrid_search_with` are wrappers and unchanged (tested bit for bit).
Measured on tank (search_harness --options-study --full, 3 runs): at
100K every filter — 50%, 10%, 1% of the store, and records far from the
query — returns the exact filtered top 10, and none is slower than an
unfiltered search (1%: 2.3 ms vs 4.6 ms). Re-rank + confidence costs
about 3%. A first version decided between index and exact scan by pool
size vs store size; it measured 0.976 recall at 12.3 ms on the
far-from-query filter, which is why the rule compares costs instead.
Tests: tests/search_options.rs (filter correctness and full pages via
both paths, far-from-query fallback, edge cases, equality with
hybrid_search_with, re-rank recency, confidence, boost scope).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
MemoryCache held every embedding in two places: a `Vec<Vec<f32>>` and a
flattened copy for the batched kernels, kept in lock-step on every push,
update and compaction. A store loaded from disk therefore carried the corpus
twice, plus one heap allocation per entry.
A new `cache::Embeddings` owns just the flat `[N x dim]` buffer and indexes
into it, so `embeddings[i]` still reads as a `&[f32]` row. The batch kernels
take a `VectorSet` (implemented for both `Embeddings` and `Vec<Vec<f32>>`)
instead of `&[Vec<f32>]`, so their callers and tests are unchanged. Loading no
longer unflattens what it just read.
100k 384-dim entries, reopened from disk: 505 -> 357 MiB, 3.44x -> 2.43x the
raw vectors. Recall (1.0000 at ef=64) and query latency are unchanged.
Rows are now always exactly `dim` long, shorter ones zero-padded. The old
representation allowed ragged rows, which silently misaligned the flattened
copy — every row after a wrong-length embedding — and `update` carried a
comment about falling back to a rebuild to avoid exactly that. It is now
unrepresentable. A record saved without an embedding holds a zero row and is
told apart by its norm, which is what `total_embeddings` now counts.
Measured with a counting allocator rather than RSS: freeing a structure
returns its pages to the allocator's pool, not the OS, so an RSS reading from
inside the process showed the two representations as identical.
Breaking: MemoryCache::embeddings changes type, embeddings_flat is replaced by
flat_embeddings(), rebuild_flat() is a deprecated no-op.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
BENCHMARKS.md has recorded since the weight sweep that the 0.7/0.3 default is
strictly dominated by 0.4/0.6 over the full LongMemEval haystack, but the
shipping code never adopted it: unified_search and the OpenClaw backend both
passed 0.7/0.3. Re-running the sweep here (500 questions, real MiniLM
embeddings on a GPU) reproduces it — turn-level Hit@1 51.6% vs 44.2%, Hit@5
81.4% vs 79.2%, Hit@10 87.8% vs 85.8%, MRR 0.6430 vs 0.5856 — so both now use
hybrid::DEFAULT_FUSION, which is that operating point and carries the
reasoning. A unit test pins it.
Fusion is also selectable now. hybrid::Fusion is either Weighted { vector,
keyword } or Rrf { k }; hybrid::fuse applies either to one candidate list per
stage, and merge_vector_keyword / hybrid_search delegate to it, so the public
API is unchanged. New HDF5Memory::hybrid_search_with and
hybrid::hybrid_search_fused take a Fusion. Reciprocal rank fusion was
implemented but reachable only as a free function over a linear scan, so it
had never been compared with the weighted sum on equal terms; it is now a mode
in the LongMemEval bench (measurement to follow).
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
- clippy --all-targets plus a clawhdf5-format feature matrix (parallel, lz4,
zstd, pcodec, fast-checksum); fix the accumulated lint backlog in test,
bench and feature-gated code (no behaviour changes).
- Install python3 + h5py/numpy/netCDF4/xarray in the CI container and set
CLAWHDF5_REQUIRE_INTEROP=1, which makes a missing interop dependency a test
failure. Every h5py/netCDF4 interop test used to skip silently in CI. Run
the #[ignore]d writer_h5py_tests suite explicitly.
- cargo bench --no-run so benches can't rot; fix bench.rs and memory_bench.rs,
which no longer compiled against the current strategy/consolidation APIs.
- Optional fuzz smoke run via CLAWHDF5_FUZZ_SECONDS.
- CHANGELOG and docs/known-issues.md updated.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
