Neighbours were selected as the plain closest-M, for both a new node's links
and back-link pruning. On clustered data every link of a node inside a tight
cluster then goes to that same cluster, so clusters become islands a search
entering elsewhere can never reach: recall@10 was 0.87 / 0.67 / 0.31 at
1K / 10K / 100K (384-dim) and flat in ef. Uniform random data — all the
existing tests used — does not show it.
Implement the HNSW paper's Algorithm 4 with keepPrunedConnections: accept a
candidate only if it is closer to the node than to every neighbour already
accepted, then fill spare slots with the closest rejected ones. Recall@10 at
ef=64 is now 1.00 / 1.00 / 0.98 and rises with ef; uniform data improves
slightly. Build is ~3.5x slower at 10K (extra distance evaluations), to be
recovered by the distance-kernel work. The needless rayon fan-out over <=33
distances in prune_connections is gone.
Tests: a clustered-data recall test for bulk build and incremental insert
(scores 0.43 with the old selection), and a unit test of the selection rule.
Harness gains --uniform and --ann-only; before/after in BENCHMARKS.md.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
New clawhdf5-bench binary `search_harness`, the measurement baseline for the
search hot-path work. On deterministic clustered 384-dim data it reports HNSW
build time and, per ef, recall@10 against an exact scan, QPS and p50/p99; and
for HDF5Memory: ingest, checkpoint, open, first-query-after-open and
steady-state hybrid_search latency at 1K/10K (and 100K with --full). Optional
JSON output for tracking.
Baseline recorded in BENCHMARKS.md. It shows two problems: HNSW recall@10 does
not respond to ef and falls from 0.87 (1K) to 0.31 (100K) on clustered data,
and end-to-end hybrid_search is ~1000x slower than its vector stage because
every query rebuilds BM25 and rewrites the .h5 file.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Bump all workspace crates, the node package and pyproject to 2.3.0, finalize
the changelog and add upgrade notes for the behaviour changes.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
attrs() silently omitted any attribute whose datatype had no AttrValue variant
— including every Python bool, which h5py stores as an enum — plus complex,
compound and reference attributes, and cast unsigned 64-bit arrays to
I64Array so values above i64::MAX came back negative.
- numpy/h5py-style booleans (an enum of exactly FALSE=0 / TRUE=1 over an
integer base) decode as I64 / I64Array of 0/1.
- AttrValue::U64Array keeps unsigned arrays unsigned. Behaviour change: an
unsigned array attribute no longer arrives as I64Array; the netCDF-4 CF
helpers (_FillValue, valid_range) and the Python bindings handle it.
- AttrValue::Raw { datatype, shape, data } carries any other attribute
verbatim (also used when a value fails to decode as its declared type), so
the attribute list is always complete. Decodable with data_read against the
datatype; Python receives {"dtype", "shape", "data"}.
- Both new variants are writable, so attributes round-trip between files.
h5py interop tests cover reading 13 attribute kinds and h5py reading back a
compound and a u64 attribute written by clawhdf5.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
- tests/wal_properties.rs — deterministic generator, reproducible by seed:
everything appended is read back intact (300 cases), and after ANY damage
to the file (bit flips, truncation, inserted/deleted bytes, duplicated or
rotated regions, overwritten ranges; 1500 cases) reading never panics and
yields an exact prefix of what was written — the guarantee the chained CRC
exists to give. Opening for append then repairs the tail and a new entry
lands right behind the surviving prefix.
- tests/crash_recovery.rs — builds the on-disk images a process crash can
leave and reopens each against a model of what was acknowledged: an image
after every operation (random saves, in-place updates, checkpoints, small
wal_max_entries), the checkpoint window (new .h5 + not-yet-truncated WAL)
over several rounds, and the WAL torn at every byte length, which must
recover the checkpoint plus a prefix of the operations logged since.
- fuzz/fuzz_wal_replay — arbitrary bytes as a WAL: read and open-for-append
must not panic, and open() must not change what is replayable. Based on the
target from the clawmates mission branch (4aee2fa), with the repair
property added. The CI fuzz step now covers both fuzz crates.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The VDS resolver joined the source file name stored in the HDF5 file straight
onto the opened file's directory. That name is untrusted: an absolute path
replaces the base directory outright and `..` components climb out of it, so
a crafted file could make the reader open any path the process can reach.
Only plain relative paths of normal components are accepted now; anything
else resolves to "source not found".
Co-Authored-By: Claude Fable 5.1 <[email protected]>
- Path resolution follows soft links in both old-style (symbol table, cache
type 2) and new-style (compact and dense Link message) groups: absolute and
relative targets, links to groups, links through links, with a depth limit
so a link cycle is NestingDepthExceeded rather than a hang. A dangling link
reports the target it could not find. Previously every soft link was
PathNotFound.
- An external link is FormatError::ExternalLinkUnsupported { filename,
object_path } instead of a misleading PathNotFound.
- Message 0x0007 (External Data Files) is now a known MessageType, and a
dataset carrying it is FormatError::ExternalDataFilesUnsupported. Such a
dataset has no data address in this file, so it would otherwise be read as
"never written" and answered with fill values — wrong data, no error.
- Dense link iteration is shared between hard-link listing and the new
symbolic-link lookup; entry listing behaviour is unchanged.
- h5py interop test for both libver settings.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
HDF5 allocates lazily: a chunk nobody wrote doesn't exist in the file, and a
dataset nobody wrote has no data address. Such regions must read as the
dataset's fill value. There was no Fill Value message parser at all, so:
- a sparse chunked dataset read its holes as zeros — silently wrong whenever
the fill value isn't zero (h5py `fillvalue=-1` came back as 0);
- a dataset that was created but never written failed with NoDataAllocated /
"no address for chunked layout" where h5py returns a filled array.
New clawhdf5_format::fill_value: parses Fill Value messages v1-v3 and the old
0x0004 message (validated against HDF5 2.0 output under default and latest
libver), builds a fully filled dataset when there is no storage, and writes the
fill value into exactly the chunk-grid cells absent from the chunk index —
never mistaking a stored zero for a hole, clipping edge chunks, any rank. It is
skipped entirely for the default (zero) fill value. The chunk index dispatch is
extracted from read_chunked_data into a reusable list_chunks.
The reader, lazy and mmap facades apply it on full reads; selection reads go
through a fill-aware full read when the fill value matters. h5py interop test
compares against h5py's own readback, including a sparse 2-D dataset and a
hyperslab straddling allocated and unallocated chunks.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
A dataset created from a committed (named) datatype stores only a shared-
message reference to it. The facade parsed those reference bytes as the
datatype itself, producing `Time { size: 0 }` and unreadable data, and an
attribute using a committed datatype was silently dropped.
- shared_message::parse_shared_ref had the encoding wrong: it skipped six
reserved bytes for version 2 (only version 1 has them) and had the version 3
types inverted (1 is the SOHM heap, 2 is "committed, in another object
header"). Verified against h5py 3.16 / HDF5 2.0, which writes
`02 02 <address>` under both default and latest libver bounds. Resolution
now dispatches on which field the reference carries.
- New shared_message::message_data resolves a header message through the
indirection; the reader, lazy and mmap facades use it for datatype,
dataspace and filter-pipeline messages.
- AttributeMessage honours the v2/v3 flags (bit 0 datatype shared, bit 1
dataspace shared) via the new parse_in_file, used everywhere file data is
available. Parsing a shared attribute without file access is now
FormatError::UnresolvedSharedMessage instead of a garbage datatype.
- h5py interop test covering both libver settings.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
- Dataset::filter_pipeline() (reader, lazy and mmap variants) swallowed parse
errors with `.ok()`, so a malformed pipeline message silently became "no
filters" and the still-compressed chunk bytes were returned as the data. It
now returns Result<Option<_>>; a present-but-unparseable pipeline is
Error::Format.
- FileBuilder::write used std::fs::write, which truncates the destination
first: a crash mid-write destroyed the existing file. It now writes a
sibling temp file, syncs it, renames it over the target and syncs the
directory, cleaning the temp file up on failure.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Dataspace and chunk dimensions are untrusted 64-bit fields, but the chunked
read paths computed `num_elements() as usize * elem_size` and
`chunk_dims.product() * elem_size` with plain arithmetic and fed the result to
`vec![0u8; n]`. A crafted file could wrap the product (under-sizing the output
buffer that chunks are then copied into) or request an allocation large enough
to abort the process.
- Dataspace::checked_num_elements, checked_byte_len, checked_chunk_byte_len
and alloc_output (try_reserve_exact) replace the plain products and
vec![0; n] at every chunked read site, plus the VDS and hyperslab paths.
Overflow and allocation failure are FormatError::Overflow.
- Dataspace::num_elements saturates instead of wrapping.
- A zero-element dataset returns early, which also keeps the stride products
in range when another dimension is huge.
- parallel_read.rs: the three `c_addr + size > len` bounds checks used a raw
add; they now use checked_add like the rest of the crate.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
test_hebbian_activation_boost failed intermittently. Root causes, all in the
query path:
- normalize_scores mapped a set of identical scores — including the
single-candidate case — to 0.0, so a lone perfect match contributed nothing
to the fused score. Identical positive scores now normalise to 1.0 (all
equally the best match); identical non-positive scores stay 0.0.
- merge_vector_keyword sorted a HashMap's entries by score alone and then
truncated, so which ties survived varied from run to run; hybrid_search had
the same problem in its final sort. Both now break ties by index.
- hybrid_search applied the Hebbian boost to every returned record, including
the zero-score filler that pads the list when fewer than k records match.
With random tie-breaking a filler record could collect as many boosts as the
real hit. Only records with a positive fused score are reinforced now.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
- ProvenanceStore::remap: compaction renumbers cache indices (which are the
provenance record ids) but nothing renumbered the ledger, so after any
compaction — including the automatic one in delete() — every surviving
record's hash was filed under a different record and the next
save_or_update raised a bogus High "integrity mismatch" alert.
- Pending anomaly alerts are capped (newest 1024 kept). Alerts never block a
save, and a session over its write limit alerts on every write, so a caller
that didn't drain them grew the queue without bound.
- WriteAnomalyDetector tracks at most 4096 sessions, forgetting the
least-active half on overflow instead of leaking one entry per session id
for the life of the process.
- snapshot() copies the pending WAL next to the .h5 copy, so a snapshot is
the store as it is now rather than as of the last checkpoint (it used to
silently omit up to wal_max_entries recent saves).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
- HDF5Memory::create/open take an exclusive advisory lock on <store>.h5.lock
(std File::try_lock, no new dependency). The store lives in memory and is
rewritten wholesale at each checkpoint, so two handles on one store used to
silently destroy each other's data; a second writer now gets
MemoryError::Locked. The OS drops the lock with the descriptor, so a crash
never leaves a stale lock. Acquisition retries for ~250 ms to absorb a
previous owner that is mid-teardown; AsyncHDF5Memory::shutdown releases the
lock once its writer task has stopped.
- HDF5Memory::open_read_only: a lock-free, point-in-time view (checkpoint +
current WAL contents, replayed in memory) that never writes — it does not
repair, upgrade or move the WAL, and anything that would persist returns an
error. The CLI's recall/stats/agents-md/export use it, so a store can be
inspected while an agent has it open. Tests that reopened a store purely to
verify on-disk state now use it.
- open() no longer fails on a WAL that cannot possibly be replayed (torn
header, bad magic): it is moved to <store>.h5.wal.corrupt-<ts>, reported via
HDF5Memory::quarantined_wal(), and the healthy .h5 opens from its last
checkpoint. A well-formed header with an unknown version still fails and is
left untouched — most likely a newer build's WAL, which must not be
discarded.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Eight MemoryConfig fields (float16, compression, compression_level,
compact_threshold, hebbian_boost, decay_factor, wal_enabled, wal_max_entries)
were never written to /meta, so reopening a store silently reset them to
defaults — a compressed store was rewritten uncompressed by the first
checkpoint after a reopen, and wal_enabled=false flipped back to true. They
are now stored as /meta attributes; each is optional on load so older files
keep opening with the previous defaults, and non-finite floats are ignored.
Writing the round-trip test exposed that `compression = true` never worked in
a default build: the embeddings dataset called with_zstd() unconditionally but
the agent crate never enabled the zstd feature, so every checkpoint failed
with "unsupported filter: 32015". The default build now compresses with
deflate (always available, pure Rust path); Zstd is opt-in via a new `zstd`
agent feature.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
A save_or_update that hit an existing record was logged as a plain Save, so
replaying the WAL appended a duplicate instead of updating in place. It is now
logged as WalEntryType::Update (0x04) carrying the target index, and replay
applies it with cache.update().
The WAL header version goes 3 -> 4 for the benefit of older binaries: they
don't know record type 0x04, would read it as a torn tail and truncate it and
everything after it. An unknown header version makes them refuse the file
instead. The framing is otherwise identical, so v3 files are read by the same
code and upgraded in place on open (the header is outside the CRC chain).
Also drop the redundant WAL truncate that several callers ran straight after
flush(), which already truncates.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
flush() writes the new .h5 and only then truncates the WAL. A crash in that
window left a .h5 that already contained the pending entries AND a WAL that
still listed them, and open() replayed the WAL unconditionally — every pending
entry came back twice.
A checkpoint now records a WalMark in /meta (wal_applied_len/wal_applied_crc):
the byte length and chained CRC of the WAL prefix it folded in. On open, if
the WAL's v3 CRC chain passes through exactly that position, the entries up to
it are skipped; otherwise (the normal case: the WAL was truncated) everything
is replayed. No WAL format change; files without the attributes behave as
before. WalFile tracks its chain length alongside running_crc and resumes both
on reopen.
Also make the checkpoint and snapshot durable as a unit: sync the temp file
before the rename and the parent directory after it, so a power loss can't
leave an empty or partial .h5 under the final name. This is per-checkpoint
cost only; individual WAL appends remain unsynced by design.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The norms guard was the tautology `n.len() == n.len()`, so a norms dataset
of any length was trusted and corrupted every cosine score; other per-record
datasets were not length-checked at all, so a truncated file loaded and then
panicked on the first index. Mismatches are now MemoryError::Schema, stored
norms are used only when they match the record count, and embedding_dim == 0
with records present is rejected instead of panicking in chunks(0).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Formatting only. cargo fmt --check was already failing on main (accel SIMD
kernels, agent, format, migrate, bench); CI now enforces it.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
- clippy --all-targets plus a clawhdf5-format feature matrix (parallel, lz4,
zstd, pcodec, fast-checksum); fix the accumulated lint backlog in test,
bench and feature-gated code (no behaviour changes).
- Install python3 + h5py/numpy/netCDF4/xarray in the CI container and set
CLAWHDF5_REQUIRE_INTEROP=1, which makes a missing interop dependency a test
failure. Every h5py/netCDF4 interop test used to skip silently in CI. Run
the #[ignore]d writer_h5py_tests suite explicitly.
- cargo bench --no-run so benches can't rot; fix bench.rs and memory_bench.rs,
which no longer compiled against the current strategy/consolidation APIs.
- Optional fuzz smoke run via CLAWHDF5_FUZZ_SECONDS.
- CHANGELOG and docs/known-issues.md updated.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Every test created its own wgpu instance and device (with adapter-maximum
limits) concurrently, which could wedge the driver and hang the suite
indefinitely. Tests now hold a process-wide lock while they own a device, and
GpuAccelerator readback waits are bounded at 30s so a stuck driver surfaces as
GpuError::BufferMap instead of blocking forever.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Compound datasets written with default libver bounds (datatype message
version 1, i.e. plain h5py.File(path, 'w')) could not be read: the v1 member
layout has 28 bytes of legacy array fields after the byte offset
(dimensionality 1, reserved 3, permutation 4, reserved 4, four sizes 16) and
the parser skipped 24, so every following member was read 4 bytes off. v2 was
also wrong: it keeps the 8-byte name padding and has no array fields.
Found by adding a default-libver axis to the h5py-generated-file tests (HDF5
2.0 raised the default low bound to 1.8, so "default" files are a distinct
format path from libver='latest'). Adds byte-level v1/v2 regression tests, a
truncation test, and fuzz corpus seeds for v1 compound and native complex.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Bump all workspace crates, the node package and pyproject to 2.2.0 and
finalize the changelog. The repository URL in every manifest pointed at a
GitHub location that does not resolve; use the real origin.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Class 11 (datatype version 5) properties are a single base floating-point
datatype message, not a compound-style member list. The old parser read the
base type's bytes as member names, yielding a garbage datatype, and failed
with UnexpectedEof when a complex type was nested in a compound.
Parse the base type and surface the type as the equivalent {r, i} compound
(the shape h5py writes for numpy complex dtypes), with a size check against
the base type. Covered by byte-level tests taken from HDF5 2.0 output and an
h5py end-to-end test (writer_h5py_tests is now 27/27 against HDF5 2.0.0).
Found while validating a user report of InvalidDatatypeVersion
{ class: 6, version: 5 } against v2.1.0 (already fixed on main in a13ff51,
never released). Add docs/known-issues.md recording that report, this bug,
the open reference-v4 gap and a gpu_tests parallel-run hang; credit the
reporter in the changelog.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The SIMD migration weakened the near-zero-norm guard in all four
clawhdf5-accel cosine_similarity backends (scalar/avx2/avx512/neon)
from `denom < f32::EPSILON` to `denom == 0.0`. Vectors with a tiny
but nonzero norm (denom in (0, 1.19e-7)) fell through to dot/denom
and scored as identical instead of maximally dissimilar, diverging
from the pre-SIMD scalar loop's documented fallback behavior.
Restores the epsilon threshold in all four backends so
`1.0 - cosine_similarity(...)` in hnsw.rs::compute_distance
reproduces the old fallback exactly. Adds regression tests in
clawhdf5-accel and clawhdf5-ann locking in the near-zero-norm case.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Two missions found largely disjoint ground — v3 in clawhdf5-ann, -io and
-migrate, v6 in -agent and -format — so this merge is mostly additive.
One genuine collision: BOTH runs independently implemented
`Dataset::verify_provenance` in clawhdf5/src/reader.rs, and git kept both,
producing `E0592 duplicate definitions`. They were functionally identical
apart from `self.file.data.as_bytes()` (v3) vs `self.file.as_bytes()` (v6).
Kept v6's — it is the variant that compiles against the current tree and
passed 52 suites, and its doc comment is the more honest one, stating both the
full-read cost and that an unkeyed hash stored beside its data is not a
tamper-evidence guarantee.
Verified present after the merge:
P1 clawhdf5-ann now depends on clawhdf5-accel; compute_distance calls
l2_distance / cosine_similarity instead of a scalar loop
P2 AsyncFileReader caches its handle and length behind a mutex
PR1 clawhdf5-migrate writes SHINES provenance, and validate checks it
v6 WAL torn-tail truncation, char-boundary truncate, agent hardening
52 suites pass, 0 failures.
Co-Authored-By: Claude Opus 5 <[email protected]>
`WalFile::open` scanned the chained entries to resume the CRC chain, then
seeked to END OF FILE to append. After a crash mid-append — the ordinary way a
WAL ends up damaged — that puts the next entry BEHIND the torn bytes:
[1..N verified][torn tail][N+1, chained to N]
`read_chained_entries` stops at the torn tail, so N+1 is unreachable forever
even though its `append` returned Ok and synced. Silent loss of an acknowledged
write, in the one situation a WAL exists for.
`read_chained_entries` now also returns the byte length of the verified prefix,
and `open` truncates to it and appends there. The torn tail was never
acknowledged to any caller, so discarding it loses nothing, and the file offset
then matches the `running_crc` the chain continues from.
Verified by negative control: with the previous `seek(End(0))` the new test
fails with "got 1 entr(y/ies) — the post-crash write was silently lost".
Introduced by neither this branch nor the chaining work — v2 seeked to EOF too.
What changed is that `open` now scans and therefore KNOWS where the verified
prefix ends, which is what makes the fix a two-line consequence of information
already in hand.
Co-Authored-By: Claude Opus 5 <[email protected]>
Missed in the INT-16 flat-embedding-buffer commit — this integration
test call site lives under tests/, outside the src/ tree that was
grepped for callers.
Two related gaps in the WAL format, both closed:
1. Each entry's CRC32 covered only its own bytes, with no sequence number
or chaining — entries could be reordered, duplicated, or spliced (e.g.
a Tombstone moved before/after its target Save) while every individual
entry still passed its own CRC check, silently changing replayed cache
state. Bump to WAL_VERSION 3: each entry's CRC32 trailer is now computed
over its own bytes chained with the previous entry's stored CRC
(crc32(entry_bytes ++ prev_crc)), seeded at 0 after a truncation. Moving,
duplicating, or reordering an entry breaks the chain at that point, and
replay stops there — same handling as a bit-flip or truncation. The
previous per-entry-CRC-only format becomes WAL_VERSION_CRC_UNCHAINED (2)
and remains fully readable (not restricted, since it still verifies each
entry); WalFile::open migrates it to v3 by recreating the file fresh,
same as the existing v1 migration.
WalFile::open() on an existing v3 file scans it once to resume the CRC
chain correctly for further appends — required because a process
restart without an intervening flush reopens the same (non-truncated)
WAL and keeps appending to it, so new entries must chain against the
real last entry already on disk, not restart from 0.
2. WAL_VERSION_LEGACY_NO_CRC (v1, no integrity verification at all) was
reachable through the public WalFile::read_entries — a version byte
flipped from 2/3 down to 1 silently downgraded every entry to the
fully-unverified pre-hardening parser for any caller, not just the
one-time migration path. Split into WalFile::read_entries (rejects v1
with a typed error; still reads v2/v3) and the pub(crate)
read_entries_for_migration (accepts v1 too), used exclusively by
HDF5Memory::open's migration flow.
INT-09
verify_dataset existed and was tested, but was only ever called from
clawhdf5-format's own test files — no reader path in clawhdf5-io or the
clawhdf5 facade called it, so a corrupted dataset was silently readable
even though the write-side SHA-256 hash machinery (gated on the
provenance feature) had already written what it needed to detect that.
Add Dataset::verify_provenance() to the clawhdf5 facade, gated behind a
new `provenance` feature (on by default, forwarding to
clawhdf5-format/provenance which is already default-on). It surfaces a
typed VerifyResult (Ok/Mismatch/NoHash) via the existing Error type
rather than panicking. Deliberately NOT called automatically on
open()/dataset() — it decodes and hashes the entire dataset, which would
regress every read path (including the zero-copy/mmap ones) if run
unconditionally; callers opt in per dataset where the cost is
acceptable (e.g. a periodic integrity sweep).
Also re-export clawhdf5_format::provenance from the facade crate so
VerifyResult is reachable without depending on clawhdf5-format directly.
INT-08
check_rate_anomaly's 60s window is shared across all sessions/sources —
when it trips, the alert reported only the anonymous aggregate count,
unlike the separate cumulative max_writes_per_session check, which does
name the offending session. A session's write count can never exceed the
window's aggregate count, so whenever the window trips, name the
top-contributing session and source within it in the same alert instead
of adding a second, redundant per-session threshold check.
INT-07
check_pattern_anomaly did a plain case-folded literal-substring test, so
inserting whitespace, punctuation between letters, or a zero-width/
invisible-formatting character anywhere in a flagged phrase defeated every
one of the 15 injection patterns while the text still displays normally.
Add normalize_for_pattern_match: lowercases, drops control and
invisible-format characters (ZWSP, ZWJ, ZWNJ, bidi marks, BOM, soft
hyphen, word joiner, invisible math operators), drops punctuation
entirely (so split words rejoin instead of just being separated), and
collapses whitespace runs. Apply it to both the chunk and each configured
pattern before matching.
Scope, stated plainly: this does not add Unicode NFKC normalization or
confusable/homoglyph folding (e.g. Cyrillic а standing in for Latin a) —
that needs a per-codepoint confusable table (Unicode's confusables.txt)
beyond what's reasonable to hand-roll correctly, and no such crate is a
dependency of this crate today. A determined attacker using homoglyphs
can still evade these patterns; only the whitespace/punctuation/
zero-width bypasses are closed here.
INT-06
Two related trust-boundary gaps, both closed:
1. ConsolidationEngine::add_memory took a plain `source: MemorySource`
parameter, so any caller could claim MemorySource::System/Correction —
which get elevated importance weighting in score_correction — for
content whose actual origin the caller doesn't control or hasn't
verified. Split into add_memory(UntrustedSource) for ordinary
caller-supplied content (User/Tool/Retrieval only, no elevated variant
exists to claim) and add_trusted_memory(TrustedSource) for content whose
elevated trust the caller has independently verified (System/
Correction). Updated the one production consumer outside this crate
(clawhdf5-bench's consolidation_efficiency benchmark) and all tests.
2. The provenance/anomaly wiring added in the previous commit introduced
the same pattern: infer_memory_source mapped source_channel == "system"
or "correction" straight to the elevated MemorySource variants. Since
MemoryEntry.source_channel is unvalidated caller-supplied text, this let
a write dodge check_source_anomaly's User-flood detection by simply
self-labeling source_channel = "system". infer_memory_source now never
returns System/Correction — only Tool/Retrieval (recognized channel
names) or User (everything else, the conservative default).
INT-05
ProvenanceStore, WriteAnomalyDetector, and their check_*/verify_integrity
methods had zero callers outside their own module/tests — lib.rs only
declared the modules. The 15 injection-pattern checks, rate limiting, and
content-hash integrity verification described as shipped in ROADMAP.md
Track 5 never executed during normal library usage.
HDF5Memory::save/save_batch/save_or_update now record a MemoryProvenance
entry (content hash, inferred MemorySource, session) for every write, run
check_rate_anomaly/check_pattern_anomaly/check_source_anomaly against it,
and queue any triggered AnomalyAlert for the caller to drain via the new
take_anomaly_alerts(). save_or_update's update path additionally verifies
the existing record's content against its last recorded hash before
overwriting, catching accidental in-session corruption.
Scope notes, stated plainly rather than overclaimed:
- There is no on-disk provenance ledger (see the CLAUDE.md note added
here) — this is session-scoped bookkeeping, not a disk-integrity
control. open() starts the store empty; there's no historical hash to
verify loaded records against, so "verify on load" is implemented as
"populate the store so subsequent updates in this session are
checkable" rather than a check against nothing.
- MemorySource is inferred from source_channel via a plain string match
(infer_memory_source) — a heuristic for bookkeeping, not the gated
trust-boundary construction INT-05 asks for. That remains open.
- Alerts never block a save; this only makes detection real instead of
dead code. Whether writes should ever be blocked is a policy decision
left to the caller/a follow-up item.
INT-04
blas_cosine_batch and accelerate_cosine_batch_vecs re-flattened the
entire Vec<Vec<f32>> corpus into a fresh Vec<f32> on every single
query before running the batch matmul — an O(N·dim) copy paid per
query when fast-math/accelerate/openblas is enabled, even though a
flat fast-path (blas_cosine_batch_flat / accelerate_cosine_batch)
already existed for pre-flattened input.
Add MemoryCache::embeddings_flat, a contiguous [N × embedding_dim]
buffer maintained incrementally in push/update/compact (O(1) amortized
append, O(dim) in-place overwrite, O(n) rebuild only on compact/bulk
load). schema.rs's direct-push load path calls the new rebuild_flat()
explicitly. flat_embeddings() now just clones the already-maintained
buffer instead of rebuilding it.
Thread the flat buffer through strategy::search_with_metrics as a new
vectors_flat parameter, used only by the Blas/Accelerate arms (now
calling the *_flat variants); other strategies are unaffected. No
current caller wires search_with_metrics into the production query
path yet (only its own tests exercise it) — this fixes the identified
per-query re-flatten and makes the flat buffer available for whenever
that wiring lands.
INT-16
bfs_neighbors scanned the entire relations list per queue-popped node
(O(V·E) instead of O(V+E)) and did an O(n) linear find over entities
per discovered neighbor; spreading_activation scanned the entire
relations list per active node per step (O(max_steps·active·E)). Add
a per-call AdjacencyIndex (entity-id -> entities-index map, entity-id
-> touching-relation-indices map) built once in O(V+E) and shared by
both traversal loops, replacing the linear scans with O(degree) /
O(1) lookups.
Built fresh per call rather than cached on KnowledgeCache: entities
and relations are plain pub Vecs pushed to directly by schema.rs's
load path (bypassing add_entity/add_relation), so a persisted index
would need extra staleness bookkeeping. get_relations_from/
get_relations_to are left as plain O(E) filters — they're single-node
lookups already optimal for a standalone call; wrapping them in an
O(V+E) index build would be a regression, not a fix, and nothing in
the codebase currently calls them in a per-node loop.
Added a self-loop regression test: the index must visit a src==tgt
relation exactly once, matching the original flat-iteration behavior.
INT-13
resolve_or_create allocated a fresh lowercased String for every entity
on every call (this runs per extracted mention during entity/relation
extraction) and never short-circuited on an exact dist == 0 match,
scoring every remaining entity regardless. Add Entity::name_lower,
computed once at construction (add_entity, and schema.rs's direct-push
load path), and break out of the scan as soon as an exact match is
found.
INT-12
top_k_scores.sort_by(...) ran over the full k-sized buffer for every
matching document that beat the running threshold (twice in the full
branch), plus another full sort on first reaching k results —
O(m·k log k) for m matching documents. Replace the Vec<f32> buffer
with a BinaryHeap<Reverse<HeapScore>> min-heap of size k, giving
O(m log k). Existing wand_returns_same_results_as_exhaustive test
confirms results are unchanged.
INT-11
records.retain(|r| !evict_ids.contains(&r.id)) called Vec::contains
(linear scan) for every record against evict_ids, giving O(n·m) cost
on both Working- and Episodic-tier eviction every consolidation tick.
Build evict_ids as a HashSet for O(1) membership checks.
INT-15
score_surprise only reads r.embedding by reference, so cloning every
Working-tier record's full chunk text + embedding Vec<f32> on every
add_memory call was wasted work, discarded immediately after use.
Collect Vec<&MemoryRecord> instead and change score_surprise's
signature to take &[&MemoryRecord].
INT-14
truncate() sliced source.chunk (arbitrary UTF-8 from the source SQLite
database) at a raw byte offset. A multi-byte character straddling byte
40 panics with "byte index 40 is not a char boundary" instead of
producing the mismatch diagnostic the code exists to report — and this
is the default validate_hdf5 path, not test-only. Cut on the nearest
char boundary at or before 40 instead.
INT-10
Datatype::parse recurses into itself for Compound/Enumeration/
VariableLength/Array/Complex member and base types with no depth
counter. A message data size capped at u16::MAX (65535 bytes) allows
~8000 levels of nesting in a crafted file, enough to blow the stack —
worse on the project's no_std/embedded targets with only a few KB of
stack. Thread a depth counter through a new parse_with_depth, mirroring
object_header.rs's continuation-depth guard, and reject past 64 levels
with FormatError::NestingDepthExceeded. The public Datatype::parse
signature is unchanged.
INT-03
SymbolTableNode::parse used raw offset+8 arithmetic that can overflow
on a crafted v1-group B-tree leaf with a near-u64::MAX SNOD child
pointer (group_v1.rs passes such offsets through unchecked). Switch to
checked_add, matching read_offset in the same file. Also harden the
entries_start + num_symbols*entry_size computation with checked_add
for consistency, even though num_symbols being u16 already bounds
that multiply. Add regression tests.
INT-02
Six sites used raw `offset + N > file_data.len()` arithmetic that can
overflow on a crafted file with an address field near u64::MAX,
bypassing the bounds check before the next slice op panics. Switch to
the checked_add-based ensure_len pattern already used by local_heap.rs
and other parsers in this crate. Add regression tests for offsets near
usize::MAX in both files.
INT-01
Mission: 01a00c41-bac0-7eb3-a8c8-8b7044f3086d
Phase: 01a00c41-bac2-71e3-a58b-c473421200ee
Committed by the ClawMates delivery pipeline from the agents' working tree. Authored by agents, not by the named committer.
than h5py (5e)
stable-worldmodel (arXiv 2605.21800, LeCun/Balestriero) supports HDF5 as
one of three native formats and measures generic HDF5 at 1,416-1,474
samples/s for per-frame sample loading. This measures clawhdf5 against
that shape, hardware-controlled: clawhdf5 and h5py reading the SAME file
on the SAME machine.
worldmodel_sampling example: mmap an (N,H,W,C) uint8 observation dataset,
read each frame once per pass in shuffled (dataloader) order. The file is
written by h5py (benchmarks/gen_worldmodel_frames.py) — clawhdf5 parsing
an externally-produced HDF5 file is itself the interop result — and read
by both clawhdf5 and the h5py counterpart (benchmarks/bench_worldmodel_h5py.py,
opening exactly stable-worldmodel's HDF5Dataset: swmr + 256 MB cache).
Results (tank, Ryzen 7 7800X3D, 20000x64x64x3 = 246 MB, in page cache,
median of 3):
clawhdf5 zero-copy view 593k samples/sec 8.1x
clawhdf5 materialised copy 518k samples/sec 7.1x
h5py (swmr, 256 MB cache) 73k samples/sec 1.0x
The materialised-copy row is the fair equal-work comparison (to_vec per
frame, matching h5py's numpy materialisation) and is still 7.1x faster;
that the copy costs almost nothing shows the gap is h5py's per-frame call
overhead, not data movement. Honest caveats in BENCHMARKS.md: absolute
numbers are NOT comparable to the paper's (different hardware, smaller
frames, no torch/transform), only the same-machine ratio is; this is an
in-page-cache measurement isolating read-path overhead, not disk
bandwidth.
Adds only an example, two benchmark scripts, and a BENCHMARKS.md section —
no library code. (Workspace clippy has pre-existing toolchain drift
unrelated to this change; tracked separately.)
Tier 4b reported hybrid retrieval at 0.7/0.3 and noted the weights were "the
documented default, not a searched optimum". `--sweep` searches them: 0.0 to 1.0
in 0.1 steps, reusing the one-time embedding table so eleven configurations cost
barely more than three.
The result is not a refinement. 0.7/0.3 is **strictly dominated**:
vector/keyword Hit@1 Hit@5 Hit@10 MRR sHit@5
0.0 / 1.0 53.8% 75.0% 81.6% 0.6320 93.6%
0.3 / 0.7 53.2% 78.8% 87.2% 0.6463 96.0%
0.4 / 0.6 51.6% 81.4% 87.8% 0.6429 96.8%
0.5 / 0.5 48.2% 81.4% 88.2% 0.6234 97.4%
0.7 / 0.3 44.4% 79.2% 86.0% 0.5868 95.8%
1.0 / 0.0 36.0% 71.8% 81.6% 0.5027 94.2%
0.4/0.6 beats 0.7/0.3 on every metric at both granularities — Hit@1 +7.2pp,
Hit@5 +2.2, Hit@10 +1.8, MRR +0.056. No trade is being made; the default simply
sat on the wrong side of the peak. It is now 0.4/0.6, and README's usage snippet
recommends the same.
This corrects a conclusion I published one commit ago. Measuring only 0.7/0.3, I
wrote that fusion "buys deeper recall and pays for it at rank 1" and advised
callers taking a single top hit to prefer BM25. That was an artifact of the bad
weight, not a property of fusion: at 0.3/0.7 hybrid *beats* BM25 on MRR (0.6463
vs 0.6320) and Hit@5 (78.8% vs 75.0%) while giving up 0.6pp of Hit@1. Both
BENCHMARKS.md and README carry the correction rather than a quiet edit, since
the old text told readers to configure their systems a particular way.
The three-mode ablation rows are kept at their original settings — they measure
the shape of each stage in isolation, and the operating point now comes from the
sweep instead.
The GPU path worked but was effectively hidden. cudarc's build script shells out
to `nvcc`, which ships in /usr/local/cuda/bin — a directory the reference host
had installed but never exported to the login shell, so `--features
embeddings-cuda` failed with a bare "`nvcc --version` failed" panic from a
dependency's build script, and the runtime fallback then reported only
"Embedder: CPU (...)" before spending hours on work a GPU does in minutes.
Two changes, both about making the failure legible rather than changing what the
code does:
- The CPU fallback now says why it fell back and what that costs, with the
concrete fix. A run that silently takes two orders of magnitude longer reads
as a hang, not as a configuration choice.
- BENCHMARKS.md states the build-time nvcc requirement, where the toolkit
actually installs, and that a shell file read non-interactively is the place
to export it — `~/.zshenv` rather than `~/.zshrc`, because build scripts do
not run in an interactive shell.
Host-side, the reference machine's CUDA exports lived in ~/.bashrc below its
non-interactive guard while the login shell is zsh, so they never applied to
anything. Moved to ~/.zshenv with duplicate-prepend guards; `nvcc --version`
and `cargo build --features embeddings-cuda` now both work over a plain
non-interactive ssh with no manual export.
Every LongMemEval number this project has published measured BM25 alone. The
bench passed zero-vector embeddings with vector_weight=0.0, so the HNSW/vector
stage — the thing the README credits for retrieval quality — contributed
nothing and was never tested.
An optional `embeddings` feature loads all-MiniLM-L6-v2 via candle and encodes
the corpus for real. It is off by default and nothing in the shipped crates
depends on it, so a project that advertises no heavyweight dependencies keeps
that property; without the feature the bench behaves exactly as before.
Full haystack, n=500, turn-level:
Hit@1 Hit@5 Hit@10 MRR
BM25 only 53.8% 75.0% 81.6% 0.6320
Vector only 36.0% 71.8% 81.6% 0.5027
Hybrid 0.7/0.3 44.4% 79.2% 86.0% 0.5868
Session-level, hybrid leads outright: 88.2 / 95.8 / 97.8 / 0.9158.
The hybrid claim holds for depth and not for precision@1. Hybrid is the best
configuration at Hit@5 and Hit@10 at both granularities — turn-level Hit@5 gains
4.2 points over BM25 and 7.4 over vector-only, which is the result that justifies
running two stages at all. But BM25 alone still leads turn-level Hit@1 and MRR,
so fusing buys deeper recall and pays at rank 1. Callers assembling five memories
of context want hybrid; callers taking a single top hit are better served by BM25
today. The 0.7/0.3 weights are the documented default, not a searched optimum.
omni-cortex's four-signal ablation found the same direction independently — there,
adding BM25 to a dense retriever raised nDCG@5 while lowering Hit@1 and MRR. Two
codebases, two fusion schemes, same trade.
Vector-only trailing BM25 at every turn-level cutoff except Hit@10 is stated
plainly rather than buried: LongMemEval questions share heavy vocabulary with
their evidence turns, which is close to the best case for lexical matching, and
MiniLM at 384-d is a small model.
Implementation notes:
- Texts are deduplicated before encoding. The haystack sessions are drawn from
a shared pool, so 500 questions x 493.5 turns collapses to 190,015 unique
strings — the difference between encoding the corpus once and per question.
- `embeddings-cuda` adds the GPU path, and it is not a convenience: 190k texts
take ~13 min on an RTX 5060 Ti, while the same work on 8 CPU cores was still
unfinished after 30 minutes. The device is selected at runtime with a CPU
fallback, so a machine without CUDA still works.
- Mean-pooling is masked and the output L2-normalised, which is the published
recipe for this checkpoint (not the [CLS] pooler).
One measurement wrinkle, recorded rather than smoothed over: on the oracle
variant BM25-only reads 84.2% Hit@5 with real embedding vectors present against
84.4% with zero vectors — one question of 500 changes rank, MRR identical at
0.6597. On the full haystack the two agree exactly. Weight 0.0 evidently does not
make the vector stage bit-for-bit absent from candidate selection on a small
corpus.
Verified on the Linux dev host: 49 groups / 1659 passed / 0 failed, clippy clean
under -D warnings, fmt clean, with and without the feature.
