hybrid_search rebuilt the BM25 index from scratch (re-tokenising every record)
and rewrote the whole .h5 file on every single query, so a query cost O(store
size) in both CPU and disk I/O. Steady-state p50 per the search harness:
5.5 -> 0.24 ms (1K), 49 -> 2.1 ms (10K), 884 -> 23 ms (100K).
- BM25Index is incremental: add_document / remove_document keep it exactly
equivalent to a fresh build over the same live documents (property test: 60
random op sequences compared against BM25Index::build after every step). IDF
moves to query time since it depends on the live document count. Top-k uses
a bounded heap, ties break by doc id (results were HashMap-ordered), and the
"WAND" code that computed a bound and then discarded it is removed.
- HDF5Memory keeps one index for its lifetime, built lazily. Appends are
picked up by ensure_bm25_fresh whatever path added them; delete and in-place
update report themselves; compaction drops the index. A test drives every
mutation and compares against a fresh build.
- A query no longer calls flush(). Activation boosts are marked dirty and
persisted by the next checkpoint, including a best-effort one on drop so a
search-only session keeps them (approved behaviour change). Activation
weights are capped at 16.0; they previously grew without bound.
The archived mission branch's BM25 cache was reviewed and not used: it was
invalidated by every write, so interleaved save/search still rebuilt per
query, and it changed the default fusion weights.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
- ProvenanceStore::remap: compaction renumbers cache indices (which are the
provenance record ids) but nothing renumbered the ledger, so after any
compaction — including the automatic one in delete() — every surviving
record's hash was filed under a different record and the next
save_or_update raised a bogus High "integrity mismatch" alert.
- Pending anomaly alerts are capped (newest 1024 kept). Alerts never block a
save, and a session over its write limit alerts on every write, so a caller
that didn't drain them grew the queue without bound.
- WriteAnomalyDetector tracks at most 4096 sessions, forgetting the
least-active half on overflow instead of leaking one entry per session id
for the life of the process.
- snapshot() copies the pending WAL next to the .h5 copy, so a snapshot is
the store as it is now rather than as of the last checkpoint (it used to
silently omit up to wal_max_entries recent saves).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
- HDF5Memory::create/open take an exclusive advisory lock on <store>.h5.lock
(std File::try_lock, no new dependency). The store lives in memory and is
rewritten wholesale at each checkpoint, so two handles on one store used to
silently destroy each other's data; a second writer now gets
MemoryError::Locked. The OS drops the lock with the descriptor, so a crash
never leaves a stale lock. Acquisition retries for ~250 ms to absorb a
previous owner that is mid-teardown; AsyncHDF5Memory::shutdown releases the
lock once its writer task has stopped.
- HDF5Memory::open_read_only: a lock-free, point-in-time view (checkpoint +
current WAL contents, replayed in memory) that never writes — it does not
repair, upgrade or move the WAL, and anything that would persist returns an
error. The CLI's recall/stats/agents-md/export use it, so a store can be
inspected while an agent has it open. Tests that reopened a store purely to
verify on-disk state now use it.
- open() no longer fails on a WAL that cannot possibly be replayed (torn
header, bad magic): it is moved to <store>.h5.wal.corrupt-<ts>, reported via
HDF5Memory::quarantined_wal(), and the healthy .h5 opens from its last
checkpoint. A well-formed header with an unknown version still fails and is
left untouched — most likely a newer build's WAL, which must not be
discarded.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
A save_or_update that hit an existing record was logged as a plain Save, so
replaying the WAL appended a duplicate instead of updating in place. It is now
logged as WalEntryType::Update (0x04) carrying the target index, and replay
applies it with cache.update().
The WAL header version goes 3 -> 4 for the benefit of older binaries: they
don't know record type 0x04, would read it as a torn tail and truncate it and
everything after it. An unknown header version makes them refuse the file
instead. The framing is otherwise identical, so v3 files are read by the same
code and upgraded in place on open (the header is outside the CRC chain).
Also drop the redundant WAL truncate that several callers ran straight after
flush(), which already truncates.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
flush() writes the new .h5 and only then truncates the WAL. A crash in that
window left a .h5 that already contained the pending entries AND a WAL that
still listed them, and open() replayed the WAL unconditionally — every pending
entry came back twice.
A checkpoint now records a WalMark in /meta (wal_applied_len/wal_applied_crc):
the byte length and chained CRC of the WAL prefix it folded in. On open, if
the WAL's v3 CRC chain passes through exactly that position, the entries up to
it are skipped; otherwise (the normal case: the WAL was truncated) everything
is replayed. No WAL format change; files without the attributes behave as
before. WalFile tracks its chain length alongside running_crc and resumes both
on reopen.
Also make the checkpoint and snapshot durable as a unit: sync the temp file
before the rename and the parent directory after it, so a power loss can't
leave an empty or partial .h5 under the final name. This is per-checkpoint
cost only; individual WAL appends remain unsynced by design.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Formatting only. cargo fmt --check was already failing on main (accel SIMD
kernels, agent, format, migrate, bench); CI now enforces it.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
- clippy --all-targets plus a clawhdf5-format feature matrix (parallel, lz4,
zstd, pcodec, fast-checksum); fix the accumulated lint backlog in test,
bench and feature-gated code (no behaviour changes).
- Install python3 + h5py/numpy/netCDF4/xarray in the CI container and set
CLAWHDF5_REQUIRE_INTEROP=1, which makes a missing interop dependency a test
failure. Every h5py/netCDF4 interop test used to skip silently in CI. Run
the #[ignore]d writer_h5py_tests suite explicitly.
- cargo bench --no-run so benches can't rot; fix bench.rs and memory_bench.rs,
which no longer compiled against the current strategy/consolidation APIs.
- Optional fuzz smoke run via CLAWHDF5_FUZZ_SECONDS.
- CHANGELOG and docs/known-issues.md updated.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Two related gaps in the WAL format, both closed:
1. Each entry's CRC32 covered only its own bytes, with no sequence number
or chaining — entries could be reordered, duplicated, or spliced (e.g.
a Tombstone moved before/after its target Save) while every individual
entry still passed its own CRC check, silently changing replayed cache
state. Bump to WAL_VERSION 3: each entry's CRC32 trailer is now computed
over its own bytes chained with the previous entry's stored CRC
(crc32(entry_bytes ++ prev_crc)), seeded at 0 after a truncation. Moving,
duplicating, or reordering an entry breaks the chain at that point, and
replay stops there — same handling as a bit-flip or truncation. The
previous per-entry-CRC-only format becomes WAL_VERSION_CRC_UNCHAINED (2)
and remains fully readable (not restricted, since it still verifies each
entry); WalFile::open migrates it to v3 by recreating the file fresh,
same as the existing v1 migration.
WalFile::open() on an existing v3 file scans it once to resume the CRC
chain correctly for further appends — required because a process
restart without an intervening flush reopens the same (non-truncated)
WAL and keeps appending to it, so new entries must chain against the
real last entry already on disk, not restart from 0.
2. WAL_VERSION_LEGACY_NO_CRC (v1, no integrity verification at all) was
reachable through the public WalFile::read_entries — a version byte
flipped from 2/3 down to 1 silently downgraded every entry to the
fully-unverified pre-hardening parser for any caller, not just the
one-time migration path. Split into WalFile::read_entries (rejects v1
with a typed error; still reads v2/v3) and the pub(crate)
read_entries_for_migration (accepts v1 too), used exclusively by
HDF5Memory::open's migration flow.
INT-09
Two related trust-boundary gaps, both closed:
1. ConsolidationEngine::add_memory took a plain `source: MemorySource`
parameter, so any caller could claim MemorySource::System/Correction —
which get elevated importance weighting in score_correction — for
content whose actual origin the caller doesn't control or hasn't
verified. Split into add_memory(UntrustedSource) for ordinary
caller-supplied content (User/Tool/Retrieval only, no elevated variant
exists to claim) and add_trusted_memory(TrustedSource) for content whose
elevated trust the caller has independently verified (System/
Correction). Updated the one production consumer outside this crate
(clawhdf5-bench's consolidation_efficiency benchmark) and all tests.
2. The provenance/anomaly wiring added in the previous commit introduced
the same pattern: infer_memory_source mapped source_channel == "system"
or "correction" straight to the elevated MemorySource variants. Since
MemoryEntry.source_channel is unvalidated caller-supplied text, this let
a write dodge check_source_anomaly's User-flood detection by simply
self-labeling source_channel = "system". infer_memory_source now never
returns System/Correction — only Tool/Retrieval (recognized channel
names) or User (everything else, the conservative default).
INT-05
ProvenanceStore, WriteAnomalyDetector, and their check_*/verify_integrity
methods had zero callers outside their own module/tests — lib.rs only
declared the modules. The 15 injection-pattern checks, rate limiting, and
content-hash integrity verification described as shipped in ROADMAP.md
Track 5 never executed during normal library usage.
HDF5Memory::save/save_batch/save_or_update now record a MemoryProvenance
entry (content hash, inferred MemorySource, session) for every write, run
check_rate_anomaly/check_pattern_anomaly/check_source_anomaly against it,
and queue any triggered AnomalyAlert for the caller to drain via the new
take_anomaly_alerts(). save_or_update's update path additionally verifies
the existing record's content against its last recorded hash before
overwriting, catching accidental in-session corruption.
Scope notes, stated plainly rather than overclaimed:
- There is no on-disk provenance ledger (see the CLAUDE.md note added
here) — this is session-scoped bookkeeping, not a disk-integrity
control. open() starts the store empty; there's no historical hash to
verify loaded records against, so "verify on load" is implemented as
"populate the store so subsequent updates in this session are
checkable" rather than a check against nothing.
- MemorySource is inferred from source_channel via a plain string match
(infer_memory_source) — a heuristic for bookkeeping, not the gated
trust-boundary construction INT-05 asks for. That remains open.
- Alerts never block a save; this only makes detection real instead of
dead code. Whether writes should ever be blocked is a policy decision
left to the caller/a follow-up item.
INT-04
Resolves two gaps found in a project-state review:
1. Python build was broken: PyO3/numpy 0.23 caps at Python 3.13 but the
environment has 3.14. Bumped to 0.28 and updated the two breaking APIs
(PyObject -> Py<PyAny>, allow_threads -> detach). The extension module now
imports and round-trips under Python 3.14, unblocking cargo build --workspace.
2. The "HNSW vector search over agent memories" headline was unwired:
clawhdf5-ann had zero dependents and the agent used a linear cosine+BM25 scan.
- clawhdf5-ann is now a live index: insert, mark_deleted (soft delete with a
deleted bitset, traversed but never returned), compact, and a format
version tag (v2) with backward-compatible load of v1 files.
- clawhdf5-agent wires HNSW behind the `hnsw` feature (ON by default). The
index mirrors the cache (node id == cache index) and self-heals: it rebuilds
whenever hnsw_synced_len drifts from cache.len(), so unhooked pushes can't
desync it. Non-indexable stores (no/zero-dim/mixed embeddings) and queries
whose dim doesn't match fall back to the exact linear scan.
- hybrid.rs gains merge_vector_keyword, shared by the linear and HNSW paths.
- tests/hnsw_integration.rs validates recall vs a brute-force oracle plus
insert/delete/batch behaviour.
Disable HNSW for exact search with `--no-default-features --features float16`.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
