`WalFile::open` scanned the chained entries to resume the CRC chain, then
seeked to END OF FILE to append. After a crash mid-append — the ordinary way a
WAL ends up damaged — that puts the next entry BEHIND the torn bytes:
[1..N verified][torn tail][N+1, chained to N]
`read_chained_entries` stops at the torn tail, so N+1 is unreachable forever
even though its `append` returned Ok and synced. Silent loss of an acknowledged
write, in the one situation a WAL exists for.
`read_chained_entries` now also returns the byte length of the verified prefix,
and `open` truncates to it and appends there. The torn tail was never
acknowledged to any caller, so discarding it loses nothing, and the file offset
then matches the `running_crc` the chain continues from.
Verified by negative control: with the previous `seek(End(0))` the new test
fails with "got 1 entr(y/ies) — the post-crash write was silently lost".
Introduced by neither this branch nor the chaining work — v2 seeked to EOF too.
What changed is that `open` now scans and therefore KNOWS where the verified
prefix ends, which is what makes the fix a two-line consequence of information
already in hand.
Co-Authored-By: Claude Opus 5 <[email protected]>
Missed in the INT-16 flat-embedding-buffer commit — this integration
test call site lives under tests/, outside the src/ tree that was
grepped for callers.
Two related gaps in the WAL format, both closed:
1. Each entry's CRC32 covered only its own bytes, with no sequence number
or chaining — entries could be reordered, duplicated, or spliced (e.g.
a Tombstone moved before/after its target Save) while every individual
entry still passed its own CRC check, silently changing replayed cache
state. Bump to WAL_VERSION 3: each entry's CRC32 trailer is now computed
over its own bytes chained with the previous entry's stored CRC
(crc32(entry_bytes ++ prev_crc)), seeded at 0 after a truncation. Moving,
duplicating, or reordering an entry breaks the chain at that point, and
replay stops there — same handling as a bit-flip or truncation. The
previous per-entry-CRC-only format becomes WAL_VERSION_CRC_UNCHAINED (2)
and remains fully readable (not restricted, since it still verifies each
entry); WalFile::open migrates it to v3 by recreating the file fresh,
same as the existing v1 migration.
WalFile::open() on an existing v3 file scans it once to resume the CRC
chain correctly for further appends — required because a process
restart without an intervening flush reopens the same (non-truncated)
WAL and keeps appending to it, so new entries must chain against the
real last entry already on disk, not restart from 0.
2. WAL_VERSION_LEGACY_NO_CRC (v1, no integrity verification at all) was
reachable through the public WalFile::read_entries — a version byte
flipped from 2/3 down to 1 silently downgraded every entry to the
fully-unverified pre-hardening parser for any caller, not just the
one-time migration path. Split into WalFile::read_entries (rejects v1
with a typed error; still reads v2/v3) and the pub(crate)
read_entries_for_migration (accepts v1 too), used exclusively by
HDF5Memory::open's migration flow.
INT-09
check_rate_anomaly's 60s window is shared across all sessions/sources —
when it trips, the alert reported only the anonymous aggregate count,
unlike the separate cumulative max_writes_per_session check, which does
name the offending session. A session's write count can never exceed the
window's aggregate count, so whenever the window trips, name the
top-contributing session and source within it in the same alert instead
of adding a second, redundant per-session threshold check.
INT-07
check_pattern_anomaly did a plain case-folded literal-substring test, so
inserting whitespace, punctuation between letters, or a zero-width/
invisible-formatting character anywhere in a flagged phrase defeated every
one of the 15 injection patterns while the text still displays normally.
Add normalize_for_pattern_match: lowercases, drops control and
invisible-format characters (ZWSP, ZWJ, ZWNJ, bidi marks, BOM, soft
hyphen, word joiner, invisible math operators), drops punctuation
entirely (so split words rejoin instead of just being separated), and
collapses whitespace runs. Apply it to both the chunk and each configured
pattern before matching.
Scope, stated plainly: this does not add Unicode NFKC normalization or
confusable/homoglyph folding (e.g. Cyrillic а standing in for Latin a) —
that needs a per-codepoint confusable table (Unicode's confusables.txt)
beyond what's reasonable to hand-roll correctly, and no such crate is a
dependency of this crate today. A determined attacker using homoglyphs
can still evade these patterns; only the whitespace/punctuation/
zero-width bypasses are closed here.
INT-06
Two related trust-boundary gaps, both closed:
1. ConsolidationEngine::add_memory took a plain `source: MemorySource`
parameter, so any caller could claim MemorySource::System/Correction —
which get elevated importance weighting in score_correction — for
content whose actual origin the caller doesn't control or hasn't
verified. Split into add_memory(UntrustedSource) for ordinary
caller-supplied content (User/Tool/Retrieval only, no elevated variant
exists to claim) and add_trusted_memory(TrustedSource) for content whose
elevated trust the caller has independently verified (System/
Correction). Updated the one production consumer outside this crate
(clawhdf5-bench's consolidation_efficiency benchmark) and all tests.
2. The provenance/anomaly wiring added in the previous commit introduced
the same pattern: infer_memory_source mapped source_channel == "system"
or "correction" straight to the elevated MemorySource variants. Since
MemoryEntry.source_channel is unvalidated caller-supplied text, this let
a write dodge check_source_anomaly's User-flood detection by simply
self-labeling source_channel = "system". infer_memory_source now never
returns System/Correction — only Tool/Retrieval (recognized channel
names) or User (everything else, the conservative default).
INT-05
ProvenanceStore, WriteAnomalyDetector, and their check_*/verify_integrity
methods had zero callers outside their own module/tests — lib.rs only
declared the modules. The 15 injection-pattern checks, rate limiting, and
content-hash integrity verification described as shipped in ROADMAP.md
Track 5 never executed during normal library usage.
HDF5Memory::save/save_batch/save_or_update now record a MemoryProvenance
entry (content hash, inferred MemorySource, session) for every write, run
check_rate_anomaly/check_pattern_anomaly/check_source_anomaly against it,
and queue any triggered AnomalyAlert for the caller to drain via the new
take_anomaly_alerts(). save_or_update's update path additionally verifies
the existing record's content against its last recorded hash before
overwriting, catching accidental in-session corruption.
Scope notes, stated plainly rather than overclaimed:
- There is no on-disk provenance ledger (see the CLAUDE.md note added
here) — this is session-scoped bookkeeping, not a disk-integrity
control. open() starts the store empty; there's no historical hash to
verify loaded records against, so "verify on load" is implemented as
"populate the store so subsequent updates in this session are
checkable" rather than a check against nothing.
- MemorySource is inferred from source_channel via a plain string match
(infer_memory_source) — a heuristic for bookkeeping, not the gated
trust-boundary construction INT-05 asks for. That remains open.
- Alerts never block a save; this only makes detection real instead of
dead code. Whether writes should ever be blocked is a policy decision
left to the caller/a follow-up item.
INT-04
blas_cosine_batch and accelerate_cosine_batch_vecs re-flattened the
entire Vec<Vec<f32>> corpus into a fresh Vec<f32> on every single
query before running the batch matmul — an O(N·dim) copy paid per
query when fast-math/accelerate/openblas is enabled, even though a
flat fast-path (blas_cosine_batch_flat / accelerate_cosine_batch)
already existed for pre-flattened input.
Add MemoryCache::embeddings_flat, a contiguous [N × embedding_dim]
buffer maintained incrementally in push/update/compact (O(1) amortized
append, O(dim) in-place overwrite, O(n) rebuild only on compact/bulk
load). schema.rs's direct-push load path calls the new rebuild_flat()
explicitly. flat_embeddings() now just clones the already-maintained
buffer instead of rebuilding it.
Thread the flat buffer through strategy::search_with_metrics as a new
vectors_flat parameter, used only by the Blas/Accelerate arms (now
calling the *_flat variants); other strategies are unaffected. No
current caller wires search_with_metrics into the production query
path yet (only its own tests exercise it) — this fixes the identified
per-query re-flatten and makes the flat buffer available for whenever
that wiring lands.
INT-16
bfs_neighbors scanned the entire relations list per queue-popped node
(O(V·E) instead of O(V+E)) and did an O(n) linear find over entities
per discovered neighbor; spreading_activation scanned the entire
relations list per active node per step (O(max_steps·active·E)). Add
a per-call AdjacencyIndex (entity-id -> entities-index map, entity-id
-> touching-relation-indices map) built once in O(V+E) and shared by
both traversal loops, replacing the linear scans with O(degree) /
O(1) lookups.
Built fresh per call rather than cached on KnowledgeCache: entities
and relations are plain pub Vecs pushed to directly by schema.rs's
load path (bypassing add_entity/add_relation), so a persisted index
would need extra staleness bookkeeping. get_relations_from/
get_relations_to are left as plain O(E) filters — they're single-node
lookups already optimal for a standalone call; wrapping them in an
O(V+E) index build would be a regression, not a fix, and nothing in
the codebase currently calls them in a per-node loop.
Added a self-loop regression test: the index must visit a src==tgt
relation exactly once, matching the original flat-iteration behavior.
INT-13
resolve_or_create allocated a fresh lowercased String for every entity
on every call (this runs per extracted mention during entity/relation
extraction) and never short-circuited on an exact dist == 0 match,
scoring every remaining entity regardless. Add Entity::name_lower,
computed once at construction (add_entity, and schema.rs's direct-push
load path), and break out of the scan as soon as an exact match is
found.
INT-12
top_k_scores.sort_by(...) ran over the full k-sized buffer for every
matching document that beat the running threshold (twice in the full
branch), plus another full sort on first reaching k results —
O(m·k log k) for m matching documents. Replace the Vec<f32> buffer
with a BinaryHeap<Reverse<HeapScore>> min-heap of size k, giving
O(m log k). Existing wand_returns_same_results_as_exhaustive test
confirms results are unchanged.
INT-11
records.retain(|r| !evict_ids.contains(&r.id)) called Vec::contains
(linear scan) for every record against evict_ids, giving O(n·m) cost
on both Working- and Episodic-tier eviction every consolidation tick.
Build evict_ids as a HashSet for O(1) membership checks.
INT-15
score_surprise only reads r.embedding by reference, so cloning every
Working-tier record's full chunk text + embedding Vec<f32> on every
add_memory call was wasted work, discarded immediately after use.
Collect Vec<&MemoryRecord> instead and change score_surprise's
signature to take &[&MemoryRecord].
INT-14
Bump WAL_VERSION to 2: every entry (Save and Tombstone) now ends with a
4-byte CRC32 trailer computed over its type+timestamp+payload bytes, using
the existing clawhdf5_format::checksum::crc32 (already available since
clawhdf5-agent depends on clawhdf5-format with fast-checksum enabled).
A bit-flip inside an entry is now detected and replay stops there, instead
of silently accepting corrupted data as before.
Write side needed no restructuring — append_save/append_tombstone already
buffer an entry's bytes before a single write_all, so the CRC is just
appended to that buffer first.
Read side: read_len_prefixed_str/read_embedding are generalized from
&mut File to R: Read, and a new TeeReader<R> wraps the file handle for one
entry at a time, accumulating every byte actually consumed (via read_exact)
into a buffer. This lets read_entries compute the CRC over exactly the
bytes read for a Save entry without needing to know its length up front
(its sub-fields are length-prefixed and interleaved with the length itself
only becoming known as parsing proceeds). A new read_one_entry<R: Read>
factors the per-entry-type field parsing shared by both the legacy and
current read paths.
Backward compatibility: WAL_VERSION_LEGACY_NO_CRC (1) files are still
readable via WalFile::read_entries (old field-by-file-handle path,
unchanged, no CRC expected). WalFile::open migrates a legacy file by
recreating it fresh in the current format — safe because the only two
real call sites (HDF5Memory::open/create) always call read_entries before
open, so entries are already replayed by the time migration happens.
New tests: a corrupted-payload-byte test confirming replay stops cleanly
at the corrupted entry (no prior coverage existed for mid-entry bit-flip
detection), a legacy-v1-format read test, and an open()-migration test.
Add [workspace.dependencies] to the root Cargo.toml for the four
duplicated-across-many-crates dependencies flagged by the earlier review:
tempfile (7 crates), criterion (6), half (4 — real version skew, clawhdf5-gpu
pinned 2.7 while others used bare 2), and serde (4). Update every consuming
crate to `dep = { workspace = true }`, preserving crate-local `optional =
true` where it already existed. half now resolves uniformly to 2.7.x
workspace-wide instead of two separate semver ranges.
Also fixed clawhdf5-filters/Cargo.toml's stale "rustyhdf5" description
while touching the file (same class of leftover rename as prior fixes).
Not touching rayon/byteorder/clap (no skew found, lower priority).
- clawhdf5-android: validate embedding_len/query_embedding_len against
the handle's configured embedding_dim (and reject null pointers)
before constructing a slice via from_raw_parts in edgehdf5_save and
edgehdf5_hybrid_search. Strengthen the # Safety docs to state the
now-enforced invariant and its limits. Add unit tests covering
mismatched length and null-pointer rejection.
- clawhdf5-py: bump pyo3/numpy 0.28 -> 0.29, clearing RUSTSEC-2026-0176
(OOB read in PyList/PyTuple iterator) and RUSTSEC-2026-0177 (missing
Sync bound on PyCFunction::new_closure). No source changes needed;
confirmed via cargo audit that both advisories no longer appear.
- clawhdf5-agent/wal.rs: cap read_len_prefixed_str/read_embedding's
length claims at a new MAX_WAL_FIELD_LEN (64 MiB) before allocating,
so a corrupted/truncated WAL length field fails cleanly instead of
attempting a huge allocation. Add regression tests for both.
- BENCHMARKS.md: add a top-of-file traceability note distinguishing the
dated/hardware-cited/reproducible h5bench and tank-validation sections
from the older sections that don't yet meet that bar.
- Fix version skew: clawhdf5-py (pyproject.toml 1.93.0 -> 2.1.0) and
packages/clawhdf5-node (package.json 2.0.0 -> 2.1.0) were both behind
the actual crate version.
- Correct stale ROADMAP.md claims: the TypeScript bridge already has a
complete napi-rs package (not "no package.json"); CI/CD is now wired
up via .gitea/workflows/ci.yml.
- Fix CLAUDE.md: clawhdf5-gpu uses wgpu with hand-written WGSL compute
shaders, not CubeCL.
- chunked_read.rs: drop 12 unnecessary chunk_dimensions[..rank].to_vec()
allocations — all three callees already accept &[u32].
- btree_v1.rs: add an overflow-safe ensure_len(data, offset, needed)
helper (checked_add) and use it at the two plain-arithmetic bounds
guards, closing a usize-overflow edge case reachable from a crafted
near-usize::MAX B-tree offset. Add a regression test.
- Clarify that the integrity hashes in clawhdf5-agent/provenance.rs
(FNV-1a) and clawhdf5-format/provenance.rs (SHA-256) are unkeyed and
only detect accidental corruption, not tampering — doc-only change.
- README.md: document that the mpi-io feature's read/write paths are
root-read+broadcast / gather-to-rank-0, not true collective I/O.
- Add .gitea/workflows/ci.yml running scripts/ci-test.sh (fmt, clippy,
test, no_std check) on push/PR to main.
- Fix stale rustyhdf5-py/rustyhdf5-format package names in
ci-test.sh/check-nostd.sh, which had been silently no-op'ing those
checks (cargo warns but doesn't fail on an unknown --exclude/-p
target).
- With those checks actually running, fix the real issues they surface:
- clippy: useless_conversion in chunked_write.rs, byte_char_slices in
global_heap.rs/object_header.rs.
- cargo fmt: apply formatting across the workspace (whitespace only).
- no_std (thumbv7em-none-eabihf) build errors in clawhdf5-format:
core::sync::atomic::AtomicU64 doesn't exist on that target (no
native 64-bit atomics) — switch profiling.rs's counters to
portable-atomic, which falls back to a CAS-based emulation there
and is a no-op wrapper elsewhere. Add missing alloc imports for
Box (filters.rs), Vec (filters_szip.rs), and format! (dict_encoding.rs)
on no_std paths. Replace f64::powi (std/libm-only) with a small
local exponentiation-by-squaring helper in the scale-offset filter.
- Use Zstd level 3 instead of deflate(1) for embedding dataset compression.
Auto-shuffle (already the default since the TDT pre-filter commit) is now
the only shuffle needed — the explicit .with_shuffle() call was redundant.
- Benchmark: save_without_wal_single improves 67 → 61 µs (-9%).
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Implements WAL group commit optimizations (arXiv:2507.13062):
1. Serialize each WAL entry to a local Vec<u8> before writing, reducing
write() syscalls per entry from ~8 to 1.
2. Defer header entry_count updates to every GROUP_COMMIT_SIZE (8) entries
instead of per-entry, eliminating 3 lseek() + 1 write() per entry.
3. Fix read_entries() to read until EOF instead of looping entry_count
times — the header count is now a pre-allocation hint only. This is
strictly more robust: tolerates stale counts from deferred updates AND
truncated files from crashes mid-write.
Benchmark results:
- wal_flush_100_entries: -7.8% latency improvement (469 µs)
- save_with_wal_single: -1.7% (18.2 µs)
- save_without_wal_single: -2.3% (67 µs, full HDF5 write)
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
clawhdf5-agent: fixed-length string datasets (memory text chunks, session
summaries, ids, tags, entity/relation names) were stored uncompressed behind
a stale "chunked compound not yet supported" comment. Chunked writes work for
fixed-size string/compound datatypes like any other, so write_string_dataset
now chunks + deflates once a dataset's payload reaches 4 KiB — large,
redundant NullPad content compresses well while tiny metadata stays
contiguous (no chunk-overhead bloat). The dead `compress` parameter is
removed in favor of this size heuristic.
clawhdf5-format: enabling string compression exposed a latent bug — the
per-file ChunkCache built its chunk index once and reused it for every
chunked dataset in the file, keyed only by chunk coordinate with no dataset
discrimination. With one chunked dataset per file this never surfaced; with
two of different rank (a 1-D compressed string array and the 2-D embeddings
matrix) the first dataset's rank-1 index was reused for the second, panicking
with an out-of-bounds chunk coordinate. The cache now binds to a dataset by
its chunk-index address and rebinds — dropping the index, chunk-index map,
layout, and decompressed slots — whenever the dataset being read changes,
while still caching repeated/sequential access to the same dataset.
Tests: facade regression reading a 1-D compressed string dataset and a 2-D
compressed f32 dataset through one shared File cache (verified to panic
without the fix); existing agent e2e tests (large text chunks, migration
round-trip) now pass with compression on.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Sweep of the docs after the v2.0.0 rename and recent changes:
- Per-crate READMEs (13 files): rename leftover rustyhdf5-*/edgehdf5-*
package names and badges to clawhdf5-*, bump usage versions to 2.1.0.
- README: update stale test badge (417 -> 1500+), workspace stats
(15 crates/72K -> 17 crates/84K), agent crate stats (20.7K/32 modules),
and add the missing clawhdf5-napi and clawhdf5-bench crates to the tree.
- CLAUDE.md: correct the CLI subcommand list (inspect/dump/index/search ->
the actual create/save/search/recall/stats/flush-wal/agents-md/export/snapshot).
No code changes. Verified there are zero todo!()/unimplemented!() macros and
no TODO/FIXME comments in the tree.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Bumps [workspace.package] and all 17 crate package versions (and internal
path-dependency requirements) from 2.0.0 to 2.1.0 for a coordinated release.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Lands the changes from PR #1, whose server-side squash merge landed as an
empty commit (Gitea merge working-tree error). See branch
feat/hnsw-agent-integration-py314 for the full history.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Resolves two gaps found in a project-state review:
1. Python build was broken: PyO3/numpy 0.23 caps at Python 3.13 but the
environment has 3.14. Bumped to 0.28 and updated the two breaking APIs
(PyObject -> Py<PyAny>, allow_threads -> detach). The extension module now
imports and round-trips under Python 3.14, unblocking cargo build --workspace.
2. The "HNSW vector search over agent memories" headline was unwired:
clawhdf5-ann had zero dependents and the agent used a linear cosine+BM25 scan.
- clawhdf5-ann is now a live index: insert, mark_deleted (soft delete with a
deleted bitset, traversed but never returned), compact, and a format
version tag (v2) with backward-compatible load of v1 files.
- clawhdf5-agent wires HNSW behind the `hnsw` feature (ON by default). The
index mirrors the cache (node id == cache index) and self-heals: it rebuilds
whenever hnsw_synced_len drifts from cache.len(), so unhooked pushes can't
desync it. Non-indexable stores (no/zero-dim/mixed embeddings) and queries
whose dim doesn't match fall back to the exact linear scan.
- hybrid.rs gains merge_vector_keyword, shared by the linear and HNSW paths.
- tests/hnsw_integration.rs validates recall vs a brute-force oracle plus
insert/delete/batch behaviour.
Disable HNSW for exact search with `--no-default-features --features float16`.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
