- EmbeddingAnomalyDetector: score against pre-update stats so outlier
cannot dilute its own z-score by pulling the mean toward itself.
Handle zero-variance dimensions explicitly: any meaningful deviation
from an all-identical training set is quarantined immediately.
- Android concurrent test: add `unsafe impl Sync for SendableHandle`
so Arc<SendableHandle> satisfies the Send bound required by
std::thread::spawn (Mutex inside the Handle makes this sound).
- clawhdf5-format/clawhdf5 Cargo.toml: remove fast-deflate from
default features to allow builds in environments without cmake/c++
(fast-deflate remains available as an opt-in feature).
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Eliminates the O(N × terms) rebuild on every HDF5Memory::open() call
for large corpora.
Changes:
bm25.rs — Add BM25Index::to_bytes() / from_bytes()
Compact binary format (magic "BM25" + version byte, then doc_lengths,
inverted posting lists, and idf cache, all length-prefixed LE u32/f32).
from_bytes() validates magic, version, and expected doc count so a
stale or corrupted sidecar falls back to a fresh build.
lib.rs — Wire sidecar into open() and flush()
- bm25_sidecar_path() free function returns <h5 path>.bm25
- open(): after WAL replay, tries to load the sidecar; uses it if
valid, otherwise leaves bm25_cache = None for lazy rebuild.
- flush(): if bm25_cache is Some, writes the sidecar alongside the
.h5 file. Failure is best-effort — a write error is silently
swallowed so it never disrupts the main flush path.
Tests (bm25.rs):
- sidecar_round_trip_preserves_search_results: verifies identical
doc_id and score (within 1e-5) before and after round-trip.
- sidecar_stale_doc_count_rejected: wrong expected_doc_count → None.
- sidecar_bad_magic_rejected: corrupted magic bytes → None.
- sidecar_empty_index_round_trip: zero-doc edge case.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
INT-14 — Add a dedicated `benchmark` CI job to .gitea/workflows/ci.yml.
On pushes to main it saves a Criterion baseline. On pull requests it
loads the baseline and fails the job if Criterion reports a regression.
INT-15 — Add EmbeddingAnomalyDetector to anomaly.rs.
Uses Welford's online algorithm to maintain a running mean and per-
dimension variance. Evaluates each new embedding via diagonal
Mahalanobis distance (mean squared z-score); embeddings that exceed
the threshold are returned as EmbeddingVerdict::Quarantine with a
reason string, signalling the caller to store them in a quarantine
dataset rather than the primary store.
Includes a warmup phase (always Accept) to seed statistics before
the detector becomes meaningful.
Added 5 unit tests covering warmup, in-distribution, outlier,
dimension-mismatch, and count tracking.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
INT-06 — Add WAL replay fuzz target (crates/clawhdf5-agent/fuzz/).
Writes arbitrary bytes to a temp file and runs them through
WalFile::read_entries, exercising the magic-byte check, version
dispatch, CRC32 guard, length-prefix bounds, and EOF handling.
No byte sequence should cause a panic or OOM.
INT-08 — Wrap Android JNI HDF5Memory handles in Mutex.
Handle type changed from *mut HDF5Memory to *mut Mutex<HDF5Memory>.
Every JNI entry point acquires the lock before calling into
HDF5Memory, making concurrent calls from multiple Java/Kotlin threads
safe without requiring the caller to synchronize externally.
Added concurrent_count_active_is_safe test to exercise the path.
INT-10 — Add media reference sandboxing to MediaRef::validate().
Path references are canonicalized and checked to stay within an
optional sandbox directory (preventing ../ traversal).
URL references must use a scheme from ALLOWED_URL_SCHEMES (https,
http); file://, data:, and schemeless strings are rejected.
Inline references are always accepted.
Added 9 unit tests covering the acceptance and rejection paths.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
INT-01: Change hybrid search default weights from 0.7/0.3 to 0.4/0.6 (vector/keyword)
in openclaw.rs and lib.rs call sites, and update the async_memory.rs doc comment.
LongMemEval benchmarks show 0.4/0.6 strictly dominates 0.7/0.3 on Hit@1, Hit@5,
Hit@10, and MRR at both turn and session granularity.
INT-16: Cache BM25 index in HDF5Memory to avoid O(N×terms) rebuild on every
hybrid_search call. Index is lazily built on first search and invalidated (set to
None) by every write path: save(), save_or_update(), save_batch(), delete(), compact().
Uses take()/put-back to avoid borrow conflicts with &mut self in vector_keyword_search.
INT-17: Clamp decay_factor to [0.0, 1.0) in spreading_activation. A caller passing
decay_factor >= 1.0 would cause activation to accumulate unboundedly through cycles
for the full max_steps duration. Clamping guarantees convergence.
INT-18: Add deny.toml at workspace root for cargo-deny. Enforces MIT-compatible
licenses, warns on duplicate semver-major versions, and flags unmaintained crates.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
INT-02: Add [profile.release.package.clawhdf5-format] overflow-checks=true to
root Cargo.toml — provides defense-in-depth for untrusted byte offset
arithmetic in the HDF5 format parser.
INT-03: Install cargo-audit in Gitea CI workflow and call it from ci-test.sh
with --deny warnings. The script gracefully skips the step if cargo-audit is
not installed locally, so developer machines are unaffected.
INT-05: Add three cycle-safety tests for KnowledgeCache:
- test_bfs_neighbors_cycle_terminates: A→B→C→A, verifies b and c appear once
- test_bfs_neighbors_self_loop_terminates: self-loop A→A, verifies empty result
- test_spreading_activation_cycle_converges: cyclic graph with decay_factor 0.5,
verifies finite convergence and all nodes receive activation
The BFS visited-set guard was already present; these tests lock it in as a
regression boundary so future refactors cannot silently remove it.
Bump WAL_VERSION to 2: every entry (Save and Tombstone) now ends with a
4-byte CRC32 trailer computed over its type+timestamp+payload bytes, using
the existing clawhdf5_format::checksum::crc32 (already available since
clawhdf5-agent depends on clawhdf5-format with fast-checksum enabled).
A bit-flip inside an entry is now detected and replay stops there, instead
of silently accepting corrupted data as before.
Write side needed no restructuring — append_save/append_tombstone already
buffer an entry's bytes before a single write_all, so the CRC is just
appended to that buffer first.
Read side: read_len_prefixed_str/read_embedding are generalized from
&mut File to R: Read, and a new TeeReader<R> wraps the file handle for one
entry at a time, accumulating every byte actually consumed (via read_exact)
into a buffer. This lets read_entries compute the CRC over exactly the
bytes read for a Save entry without needing to know its length up front
(its sub-fields are length-prefixed and interleaved with the length itself
only becoming known as parsing proceeds). A new read_one_entry<R: Read>
factors the per-entry-type field parsing shared by both the legacy and
current read paths.
Backward compatibility: WAL_VERSION_LEGACY_NO_CRC (1) files are still
readable via WalFile::read_entries (old field-by-file-handle path,
unchanged, no CRC expected). WalFile::open migrates a legacy file by
recreating it fresh in the current format — safe because the only two
real call sites (HDF5Memory::open/create) always call read_entries before
open, so entries are already replayed by the time migration happens.
New tests: a corrupted-payload-byte test confirming replay stops cleanly
at the corrupted entry (no prior coverage existed for mid-entry bit-flip
detection), a legacy-v1-format read test, and an open()-migration test.
Add [workspace.dependencies] to the root Cargo.toml for the four
duplicated-across-many-crates dependencies flagged by the earlier review:
tempfile (7 crates), criterion (6), half (4 — real version skew, clawhdf5-gpu
pinned 2.7 while others used bare 2), and serde (4). Update every consuming
crate to `dep = { workspace = true }`, preserving crate-local `optional =
true` where it already existed. half now resolves uniformly to 2.7.x
workspace-wide instead of two separate semver ranges.
Also fixed clawhdf5-filters/Cargo.toml's stale "rustyhdf5" description
while touching the file (same class of leftover rename as prior fixes).
Not touching rayon/byteorder/clap (no skew found, lower priority).
- clawhdf5-android: validate embedding_len/query_embedding_len against
the handle's configured embedding_dim (and reject null pointers)
before constructing a slice via from_raw_parts in edgehdf5_save and
edgehdf5_hybrid_search. Strengthen the # Safety docs to state the
now-enforced invariant and its limits. Add unit tests covering
mismatched length and null-pointer rejection.
- clawhdf5-py: bump pyo3/numpy 0.28 -> 0.29, clearing RUSTSEC-2026-0176
(OOB read in PyList/PyTuple iterator) and RUSTSEC-2026-0177 (missing
Sync bound on PyCFunction::new_closure). No source changes needed;
confirmed via cargo audit that both advisories no longer appear.
- clawhdf5-agent/wal.rs: cap read_len_prefixed_str/read_embedding's
length claims at a new MAX_WAL_FIELD_LEN (64 MiB) before allocating,
so a corrupted/truncated WAL length field fails cleanly instead of
attempting a huge allocation. Add regression tests for both.
- BENCHMARKS.md: add a top-of-file traceability note distinguishing the
dated/hardware-cited/reproducible h5bench and tank-validation sections
from the older sections that don't yet meet that bar.
- Fix version skew: clawhdf5-py (pyproject.toml 1.93.0 -> 2.1.0) and
packages/clawhdf5-node (package.json 2.0.0 -> 2.1.0) were both behind
the actual crate version.
- Correct stale ROADMAP.md claims: the TypeScript bridge already has a
complete napi-rs package (not "no package.json"); CI/CD is now wired
up via .gitea/workflows/ci.yml.
- Fix CLAUDE.md: clawhdf5-gpu uses wgpu with hand-written WGSL compute
shaders, not CubeCL.
- chunked_read.rs: drop 12 unnecessary chunk_dimensions[..rank].to_vec()
allocations — all three callees already accept &[u32].
- btree_v1.rs: add an overflow-safe ensure_len(data, offset, needed)
helper (checked_add) and use it at the two plain-arithmetic bounds
guards, closing a usize-overflow edge case reachable from a crafted
near-usize::MAX B-tree offset. Add a regression test.
- Clarify that the integrity hashes in clawhdf5-agent/provenance.rs
(FNV-1a) and clawhdf5-format/provenance.rs (SHA-256) are unkeyed and
only detect accidental corruption, not tampering — doc-only change.
- README.md: document that the mpi-io feature's read/write paths are
root-read+broadcast / gather-to-rank-0, not true collective I/O.
- Add .gitea/workflows/ci.yml running scripts/ci-test.sh (fmt, clippy,
test, no_std check) on push/PR to main.
- Fix stale rustyhdf5-py/rustyhdf5-format package names in
ci-test.sh/check-nostd.sh, which had been silently no-op'ing those
checks (cargo warns but doesn't fail on an unknown --exclude/-p
target).
- With those checks actually running, fix the real issues they surface:
- clippy: useless_conversion in chunked_write.rs, byte_char_slices in
global_heap.rs/object_header.rs.
- cargo fmt: apply formatting across the workspace (whitespace only).
- no_std (thumbv7em-none-eabihf) build errors in clawhdf5-format:
core::sync::atomic::AtomicU64 doesn't exist on that target (no
native 64-bit atomics) — switch profiling.rs's counters to
portable-atomic, which falls back to a CAS-based emulation there
and is a no-op wrapper elsewhere. Add missing alloc imports for
Box (filters.rs), Vec (filters_szip.rs), and format! (dict_encoding.rs)
on no_std paths. Replace f64::powi (std/libm-only) with a small
local exponentiation-by-squaring helper in the scale-offset filter.
- Use Zstd level 3 instead of deflate(1) for embedding dataset compression.
Auto-shuffle (already the default since the TDT pre-filter commit) is now
the only shuffle needed — the explicit .with_shuffle() call was redundant.
- Benchmark: save_without_wal_single improves 67 → 61 µs (-9%).
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Implements WAL group commit optimizations (arXiv:2507.13062):
1. Serialize each WAL entry to a local Vec<u8> before writing, reducing
write() syscalls per entry from ~8 to 1.
2. Defer header entry_count updates to every GROUP_COMMIT_SIZE (8) entries
instead of per-entry, eliminating 3 lseek() + 1 write() per entry.
3. Fix read_entries() to read until EOF instead of looping entry_count
times — the header count is now a pre-allocation hint only. This is
strictly more robust: tolerates stale counts from deferred updates AND
truncated files from crashes mid-write.
Benchmark results:
- wal_flush_100_entries: -7.8% latency improvement (469 µs)
- save_with_wal_single: -1.7% (18.2 µs)
- save_without_wal_single: -2.3% (67 µs, full HDF5 write)
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
clawhdf5-agent: fixed-length string datasets (memory text chunks, session
summaries, ids, tags, entity/relation names) were stored uncompressed behind
a stale "chunked compound not yet supported" comment. Chunked writes work for
fixed-size string/compound datatypes like any other, so write_string_dataset
now chunks + deflates once a dataset's payload reaches 4 KiB — large,
redundant NullPad content compresses well while tiny metadata stays
contiguous (no chunk-overhead bloat). The dead `compress` parameter is
removed in favor of this size heuristic.
clawhdf5-format: enabling string compression exposed a latent bug — the
per-file ChunkCache built its chunk index once and reused it for every
chunked dataset in the file, keyed only by chunk coordinate with no dataset
discrimination. With one chunked dataset per file this never surfaced; with
two of different rank (a 1-D compressed string array and the 2-D embeddings
matrix) the first dataset's rank-1 index was reused for the second, panicking
with an out-of-bounds chunk coordinate. The cache now binds to a dataset by
its chunk-index address and rebinds — dropping the index, chunk-index map,
layout, and decompressed slots — whenever the dataset being read changes,
while still caching repeated/sequential access to the same dataset.
Tests: facade regression reading a 1-D compressed string dataset and a 2-D
compressed f32 dataset through one shared File cache (verified to panic
without the fix); existing agent e2e tests (large text chunks, migration
round-trip) now pass with compression on.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Sweep of the docs after the v2.0.0 rename and recent changes:
- Per-crate READMEs (13 files): rename leftover rustyhdf5-*/edgehdf5-*
package names and badges to clawhdf5-*, bump usage versions to 2.1.0.
- README: update stale test badge (417 -> 1500+), workspace stats
(15 crates/72K -> 17 crates/84K), agent crate stats (20.7K/32 modules),
and add the missing clawhdf5-napi and clawhdf5-bench crates to the tree.
- CLAUDE.md: correct the CLI subcommand list (inspect/dump/index/search ->
the actual create/save/search/recall/stats/flush-wal/agents-md/export/snapshot).
No code changes. Verified there are zero todo!()/unimplemented!() macros and
no TODO/FIXME comments in the tree.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Bumps [workspace.package] and all 17 crate package versions (and internal
path-dependency requirements) from 2.0.0 to 2.1.0 for a coordinated release.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Lands the changes from PR #1, whose server-side squash merge landed as an
empty commit (Gitea merge working-tree error). See branch
feat/hnsw-agent-integration-py314 for the full history.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Resolves two gaps found in a project-state review:
1. Python build was broken: PyO3/numpy 0.23 caps at Python 3.13 but the
environment has 3.14. Bumped to 0.28 and updated the two breaking APIs
(PyObject -> Py<PyAny>, allow_threads -> detach). The extension module now
imports and round-trips under Python 3.14, unblocking cargo build --workspace.
2. The "HNSW vector search over agent memories" headline was unwired:
clawhdf5-ann had zero dependents and the agent used a linear cosine+BM25 scan.
- clawhdf5-ann is now a live index: insert, mark_deleted (soft delete with a
deleted bitset, traversed but never returned), compact, and a format
version tag (v2) with backward-compatible load of v1 files.
- clawhdf5-agent wires HNSW behind the `hnsw` feature (ON by default). The
index mirrors the cache (node id == cache index) and self-heals: it rebuilds
whenever hnsw_synced_len drifts from cache.len(), so unhooked pushes can't
desync it. Non-indexable stores (no/zero-dim/mixed embeddings) and queries
whose dim doesn't match fall back to the exact linear scan.
- hybrid.rs gains merge_vector_keyword, shared by the linear and HNSW paths.
- tests/hnsw_integration.rs validates recall vs a brute-force oracle plus
insert/delete/batch behaviour.
Disable HNSW for exact search with `--no-default-features --features float16`.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
