Fusion min-max normalises over every keyword match, so hybrid_search asked
BM25 for a ranked list of the whole corpus: a hash insert per posting, then a
sort of every match, then the merge sorted every candidate again to keep k.
- BM25Index::scores returns every match unsorted, accumulated in a dense array
(contributions are strictly positive, so zero means untouched). search() is
built on it with the bounded heap.
- merge_vector_keyword partitions out its top k (select_nth) and orders only
those, with the same score-then-id order.
- Both hybrid paths use scores().
Rankings are identical (equivalence tests for both changes). p50 0.24 -> 0.07
ms (1K), 2.1 -> 0.49 ms (10K), 23 -> 4.65 ms (100K).
The harness gains --fusion-study, which measured the alternative — capping the
keyword pool — and found it changes the top-10 for most queries (overlap
0.83-0.92, different #1 for 10-35%) for only a 2x saving. Not adopted.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
- Cosine distance was 1 - dot/(|a||b|), re-deriving both norms on every
evaluation in the innermost loop of build and search. The index now stores
unit vectors (prepared at build, insert, graph load and HDF5 load; the query
once per search) and uses 1 - dot. Zero vectors stay zero, giving distance 1
as before. Returned distances are unchanged.
- search_layer allocated a HashSet of visited nodes per call. It is now an
epoch-stamped u32 array in thread-local scratch, reused across calls, so
search(&self) stays shareable between threads.
- clawhdf5-accel caches the detected SIMD backend in a OnceLock.
Recall is identical. Build 2.75 -> 1.89 s (10K), ~38 -> 21 s (100K); QPS at
ef=64 22.7K -> 39K (10K), 10.4K -> 14K (100K).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
open() marked the HNSW index dirty, so the first search of every session
rebuilt it from scratch — 36 s at 100K records with the (better, slower)
heuristic build. First query after open is now 1.7 / 15 / 159 ms at
1K / 10K / 100K; what remains is the one-off keyword index build.
- clawhdf5-ann: HnswIndex::graph_to_bytes / from_graph_bytes serialize the
graph only (levels, tombstones, adjacency as u32, CRC32). The existing HDF5
serializer embeds a full copy of every vector, which would double a store
that already holds them. Loading validates everything — counts, levels vs
layer count, connection limits, every neighbour id and the layer it must
exist on — so a damaged graph, or a hostile one with a valid checksum, is an
error rather than an out-of-bounds walk during search.
- clawhdf5-agent: each checkpoint writes the graph to <store>.h5.ann (synced,
atomic, before the .h5) and records a fresh generation id in /meta. open()
loads the sidecar only if its generation matches that checkpoint; missing,
stale, damaged or mismatched sidecars are ignored and the index rebuilt.
Records appended through WAL replay join the loaded index incrementally; a
replayed Update or Tombstone invalidates it. snapshot() copies it. Only an
index that exactly mirrors the cache is saved; otherwise a stale sidecar is
removed.
- ensure_hnsw_fresh inserts records appended since the last sync instead of
rebuilding, so save_batch no longer marks the whole index dirty.
- CheckpointMeta { wal_applied, ann_generation } with *_with_meta build/write/
read functions; the *_with_mark ones delegate.
- Harness reports the one-off cold index build separately from the first query
after a reopen.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
hybrid_search rebuilt the BM25 index from scratch (re-tokenising every record)
and rewrote the whole .h5 file on every single query, so a query cost O(store
size) in both CPU and disk I/O. Steady-state p50 per the search harness:
5.5 -> 0.24 ms (1K), 49 -> 2.1 ms (10K), 884 -> 23 ms (100K).
- BM25Index is incremental: add_document / remove_document keep it exactly
equivalent to a fresh build over the same live documents (property test: 60
random op sequences compared against BM25Index::build after every step). IDF
moves to query time since it depends on the live document count. Top-k uses
a bounded heap, ties break by doc id (results were HashMap-ordered), and the
"WAND" code that computed a bound and then discarded it is removed.
- HDF5Memory keeps one index for its lifetime, built lazily. Appends are
picked up by ensure_bm25_fresh whatever path added them; delete and in-place
update report themselves; compaction drops the index. A test drives every
mutation and compares against a fresh build.
- A query no longer calls flush(). Activation boosts are marked dirty and
persisted by the next checkpoint, including a best-effort one on drop so a
search-only session keeps them (approved behaviour change). Activation
weights are capped at 16.0; they previously grew without bound.
The archived mission branch's BM25 cache was reviewed and not used: it was
invalidated by every write, so interleaved save/search still rebuilt per
query, and it changed the default fusion weights.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Neighbours were selected as the plain closest-M, for both a new node's links
and back-link pruning. On clustered data every link of a node inside a tight
cluster then goes to that same cluster, so clusters become islands a search
entering elsewhere can never reach: recall@10 was 0.87 / 0.67 / 0.31 at
1K / 10K / 100K (384-dim) and flat in ef. Uniform random data — all the
existing tests used — does not show it.
Implement the HNSW paper's Algorithm 4 with keepPrunedConnections: accept a
candidate only if it is closer to the node than to every neighbour already
accepted, then fill spare slots with the closest rejected ones. Recall@10 at
ef=64 is now 1.00 / 1.00 / 0.98 and rises with ef; uniform data improves
slightly. Build is ~3.5x slower at 10K (extra distance evaluations), to be
recovered by the distance-kernel work. The needless rayon fan-out over <=33
distances in prune_connections is gone.
Tests: a clustered-data recall test for bulk build and incremental insert
(scores 0.43 with the old selection), and a unit test of the selection rule.
Harness gains --uniform and --ann-only; before/after in BENCHMARKS.md.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
New clawhdf5-bench binary `search_harness`, the measurement baseline for the
search hot-path work. On deterministic clustered 384-dim data it reports HNSW
build time and, per ef, recall@10 against an exact scan, QPS and p50/p99; and
for HDF5Memory: ingest, checkpoint, open, first-query-after-open and
steady-state hybrid_search latency at 1K/10K (and 100K with --full). Optional
JSON output for tracking.
Baseline recorded in BENCHMARKS.md. It shows two problems: HNSW recall@10 does
not respond to ef and falls from 0.87 (1K) to 0.31 (100K) on clustered data,
and end-to-end hybrid_search is ~1000x slower than its vector stage because
every query rebuilds BM25 and rewrites the .h5 file.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Bump all workspace crates, the node package and pyproject to 2.3.0, finalize
the changelog and add upgrade notes for the behaviour changes.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
attrs() silently omitted any attribute whose datatype had no AttrValue variant
— including every Python bool, which h5py stores as an enum — plus complex,
compound and reference attributes, and cast unsigned 64-bit arrays to
I64Array so values above i64::MAX came back negative.
- numpy/h5py-style booleans (an enum of exactly FALSE=0 / TRUE=1 over an
integer base) decode as I64 / I64Array of 0/1.
- AttrValue::U64Array keeps unsigned arrays unsigned. Behaviour change: an
unsigned array attribute no longer arrives as I64Array; the netCDF-4 CF
helpers (_FillValue, valid_range) and the Python bindings handle it.
- AttrValue::Raw { datatype, shape, data } carries any other attribute
verbatim (also used when a value fails to decode as its declared type), so
the attribute list is always complete. Decodable with data_read against the
datatype; Python receives {"dtype", "shape", "data"}.
- Both new variants are writable, so attributes round-trip between files.
h5py interop tests cover reading 13 attribute kinds and h5py reading back a
compound and a u64 attribute written by clawhdf5.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
- tests/wal_properties.rs — deterministic generator, reproducible by seed:
everything appended is read back intact (300 cases), and after ANY damage
to the file (bit flips, truncation, inserted/deleted bytes, duplicated or
rotated regions, overwritten ranges; 1500 cases) reading never panics and
yields an exact prefix of what was written — the guarantee the chained CRC
exists to give. Opening for append then repairs the tail and a new entry
lands right behind the surviving prefix.
- tests/crash_recovery.rs — builds the on-disk images a process crash can
leave and reopens each against a model of what was acknowledged: an image
after every operation (random saves, in-place updates, checkpoints, small
wal_max_entries), the checkpoint window (new .h5 + not-yet-truncated WAL)
over several rounds, and the WAL torn at every byte length, which must
recover the checkpoint plus a prefix of the operations logged since.
- fuzz/fuzz_wal_replay — arbitrary bytes as a WAL: read and open-for-append
must not panic, and open() must not change what is replayable. Based on the
target from the clawmates mission branch (4aee2fa), with the repair
property added. The CI fuzz step now covers both fuzz crates.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The VDS resolver joined the source file name stored in the HDF5 file straight
onto the opened file's directory. That name is untrusted: an absolute path
replaces the base directory outright and `..` components climb out of it, so
a crafted file could make the reader open any path the process can reach.
Only plain relative paths of normal components are accepted now; anything
else resolves to "source not found".
Co-Authored-By: Claude Fable 5.1 <[email protected]>
- Path resolution follows soft links in both old-style (symbol table, cache
type 2) and new-style (compact and dense Link message) groups: absolute and
relative targets, links to groups, links through links, with a depth limit
so a link cycle is NestingDepthExceeded rather than a hang. A dangling link
reports the target it could not find. Previously every soft link was
PathNotFound.
- An external link is FormatError::ExternalLinkUnsupported { filename,
object_path } instead of a misleading PathNotFound.
- Message 0x0007 (External Data Files) is now a known MessageType, and a
dataset carrying it is FormatError::ExternalDataFilesUnsupported. Such a
dataset has no data address in this file, so it would otherwise be read as
"never written" and answered with fill values — wrong data, no error.
- Dense link iteration is shared between hard-link listing and the new
symbolic-link lookup; entry listing behaviour is unchanged.
- h5py interop test for both libver settings.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
HDF5 allocates lazily: a chunk nobody wrote doesn't exist in the file, and a
dataset nobody wrote has no data address. Such regions must read as the
dataset's fill value. There was no Fill Value message parser at all, so:
- a sparse chunked dataset read its holes as zeros — silently wrong whenever
the fill value isn't zero (h5py `fillvalue=-1` came back as 0);
- a dataset that was created but never written failed with NoDataAllocated /
"no address for chunked layout" where h5py returns a filled array.
New clawhdf5_format::fill_value: parses Fill Value messages v1-v3 and the old
0x0004 message (validated against HDF5 2.0 output under default and latest
libver), builds a fully filled dataset when there is no storage, and writes the
fill value into exactly the chunk-grid cells absent from the chunk index —
never mistaking a stored zero for a hole, clipping edge chunks, any rank. It is
skipped entirely for the default (zero) fill value. The chunk index dispatch is
extracted from read_chunked_data into a reusable list_chunks.
The reader, lazy and mmap facades apply it on full reads; selection reads go
through a fill-aware full read when the fill value matters. h5py interop test
compares against h5py's own readback, including a sparse 2-D dataset and a
hyperslab straddling allocated and unallocated chunks.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
A dataset created from a committed (named) datatype stores only a shared-
message reference to it. The facade parsed those reference bytes as the
datatype itself, producing `Time { size: 0 }` and unreadable data, and an
attribute using a committed datatype was silently dropped.
- shared_message::parse_shared_ref had the encoding wrong: it skipped six
reserved bytes for version 2 (only version 1 has them) and had the version 3
types inverted (1 is the SOHM heap, 2 is "committed, in another object
header"). Verified against h5py 3.16 / HDF5 2.0, which writes
`02 02 <address>` under both default and latest libver bounds. Resolution
now dispatches on which field the reference carries.
- New shared_message::message_data resolves a header message through the
indirection; the reader, lazy and mmap facades use it for datatype,
dataspace and filter-pipeline messages.
- AttributeMessage honours the v2/v3 flags (bit 0 datatype shared, bit 1
dataspace shared) via the new parse_in_file, used everywhere file data is
available. Parsing a shared attribute without file access is now
FormatError::UnresolvedSharedMessage instead of a garbage datatype.
- h5py interop test covering both libver settings.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
- Dataset::filter_pipeline() (reader, lazy and mmap variants) swallowed parse
errors with `.ok()`, so a malformed pipeline message silently became "no
filters" and the still-compressed chunk bytes were returned as the data. It
now returns Result<Option<_>>; a present-but-unparseable pipeline is
Error::Format.
- FileBuilder::write used std::fs::write, which truncates the destination
first: a crash mid-write destroyed the existing file. It now writes a
sibling temp file, syncs it, renames it over the target and syncs the
directory, cleaning the temp file up on failure.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Dataspace and chunk dimensions are untrusted 64-bit fields, but the chunked
read paths computed `num_elements() as usize * elem_size` and
`chunk_dims.product() * elem_size` with plain arithmetic and fed the result to
`vec![0u8; n]`. A crafted file could wrap the product (under-sizing the output
buffer that chunks are then copied into) or request an allocation large enough
to abort the process.
- Dataspace::checked_num_elements, checked_byte_len, checked_chunk_byte_len
and alloc_output (try_reserve_exact) replace the plain products and
vec![0; n] at every chunked read site, plus the VDS and hyperslab paths.
Overflow and allocation failure are FormatError::Overflow.
- Dataspace::num_elements saturates instead of wrapping.
- A zero-element dataset returns early, which also keeps the stride products
in range when another dimension is huge.
- parallel_read.rs: the three `c_addr + size > len` bounds checks used a raw
add; they now use checked_add like the rest of the crate.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
test_hebbian_activation_boost failed intermittently. Root causes, all in the
query path:
- normalize_scores mapped a set of identical scores — including the
single-candidate case — to 0.0, so a lone perfect match contributed nothing
to the fused score. Identical positive scores now normalise to 1.0 (all
equally the best match); identical non-positive scores stay 0.0.
- merge_vector_keyword sorted a HashMap's entries by score alone and then
truncated, so which ties survived varied from run to run; hybrid_search had
the same problem in its final sort. Both now break ties by index.
- hybrid_search applied the Hebbian boost to every returned record, including
the zero-score filler that pads the list when fewer than k records match.
With random tie-breaking a filler record could collect as many boosts as the
real hit. Only records with a positive fused score are reinforced now.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
- ProvenanceStore::remap: compaction renumbers cache indices (which are the
provenance record ids) but nothing renumbered the ledger, so after any
compaction — including the automatic one in delete() — every surviving
record's hash was filed under a different record and the next
save_or_update raised a bogus High "integrity mismatch" alert.
- Pending anomaly alerts are capped (newest 1024 kept). Alerts never block a
save, and a session over its write limit alerts on every write, so a caller
that didn't drain them grew the queue without bound.
- WriteAnomalyDetector tracks at most 4096 sessions, forgetting the
least-active half on overflow instead of leaking one entry per session id
for the life of the process.
- snapshot() copies the pending WAL next to the .h5 copy, so a snapshot is
the store as it is now rather than as of the last checkpoint (it used to
silently omit up to wal_max_entries recent saves).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
- HDF5Memory::create/open take an exclusive advisory lock on <store>.h5.lock
(std File::try_lock, no new dependency). The store lives in memory and is
rewritten wholesale at each checkpoint, so two handles on one store used to
silently destroy each other's data; a second writer now gets
MemoryError::Locked. The OS drops the lock with the descriptor, so a crash
never leaves a stale lock. Acquisition retries for ~250 ms to absorb a
previous owner that is mid-teardown; AsyncHDF5Memory::shutdown releases the
lock once its writer task has stopped.
- HDF5Memory::open_read_only: a lock-free, point-in-time view (checkpoint +
current WAL contents, replayed in memory) that never writes — it does not
repair, upgrade or move the WAL, and anything that would persist returns an
error. The CLI's recall/stats/agents-md/export use it, so a store can be
inspected while an agent has it open. Tests that reopened a store purely to
verify on-disk state now use it.
- open() no longer fails on a WAL that cannot possibly be replayed (torn
header, bad magic): it is moved to <store>.h5.wal.corrupt-<ts>, reported via
HDF5Memory::quarantined_wal(), and the healthy .h5 opens from its last
checkpoint. A well-formed header with an unknown version still fails and is
left untouched — most likely a newer build's WAL, which must not be
discarded.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Eight MemoryConfig fields (float16, compression, compression_level,
compact_threshold, hebbian_boost, decay_factor, wal_enabled, wal_max_entries)
were never written to /meta, so reopening a store silently reset them to
defaults — a compressed store was rewritten uncompressed by the first
checkpoint after a reopen, and wal_enabled=false flipped back to true. They
are now stored as /meta attributes; each is optional on load so older files
keep opening with the previous defaults, and non-finite floats are ignored.
Writing the round-trip test exposed that `compression = true` never worked in
a default build: the embeddings dataset called with_zstd() unconditionally but
the agent crate never enabled the zstd feature, so every checkpoint failed
with "unsupported filter: 32015". The default build now compresses with
deflate (always available, pure Rust path); Zstd is opt-in via a new `zstd`
agent feature.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
A save_or_update that hit an existing record was logged as a plain Save, so
replaying the WAL appended a duplicate instead of updating in place. It is now
logged as WalEntryType::Update (0x04) carrying the target index, and replay
applies it with cache.update().
The WAL header version goes 3 -> 4 for the benefit of older binaries: they
don't know record type 0x04, would read it as a torn tail and truncate it and
everything after it. An unknown header version makes them refuse the file
instead. The framing is otherwise identical, so v3 files are read by the same
code and upgraded in place on open (the header is outside the CRC chain).
Also drop the redundant WAL truncate that several callers ran straight after
flush(), which already truncates.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
flush() writes the new .h5 and only then truncates the WAL. A crash in that
window left a .h5 that already contained the pending entries AND a WAL that
still listed them, and open() replayed the WAL unconditionally — every pending
entry came back twice.
A checkpoint now records a WalMark in /meta (wal_applied_len/wal_applied_crc):
the byte length and chained CRC of the WAL prefix it folded in. On open, if
the WAL's v3 CRC chain passes through exactly that position, the entries up to
it are skipped; otherwise (the normal case: the WAL was truncated) everything
is replayed. No WAL format change; files without the attributes behave as
before. WalFile tracks its chain length alongside running_crc and resumes both
on reopen.
Also make the checkpoint and snapshot durable as a unit: sync the temp file
before the rename and the parent directory after it, so a power loss can't
leave an empty or partial .h5 under the final name. This is per-checkpoint
cost only; individual WAL appends remain unsynced by design.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The norms guard was the tautology `n.len() == n.len()`, so a norms dataset
of any length was trusted and corrupted every cosine score; other per-record
datasets were not length-checked at all, so a truncated file loaded and then
panicked on the first index. Mismatches are now MemoryError::Schema, stored
norms are used only when they match the record count, and embedding_dim == 0
with records present is rejected instead of panicking in chunks(0).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Formatting only. cargo fmt --check was already failing on main (accel SIMD
kernels, agent, format, migrate, bench); CI now enforces it.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
- clippy --all-targets plus a clawhdf5-format feature matrix (parallel, lz4,
zstd, pcodec, fast-checksum); fix the accumulated lint backlog in test,
bench and feature-gated code (no behaviour changes).
- Install python3 + h5py/numpy/netCDF4/xarray in the CI container and set
CLAWHDF5_REQUIRE_INTEROP=1, which makes a missing interop dependency a test
failure. Every h5py/netCDF4 interop test used to skip silently in CI. Run
the #[ignore]d writer_h5py_tests suite explicitly.
- cargo bench --no-run so benches can't rot; fix bench.rs and memory_bench.rs,
which no longer compiled against the current strategy/consolidation APIs.
- Optional fuzz smoke run via CLAWHDF5_FUZZ_SECONDS.
- CHANGELOG and docs/known-issues.md updated.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Every test created its own wgpu instance and device (with adapter-maximum
limits) concurrently, which could wedge the driver and hang the suite
indefinitely. Tests now hold a process-wide lock while they own a device, and
GpuAccelerator readback waits are bounded at 30s so a stuck driver surfaces as
GpuError::BufferMap instead of blocking forever.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Compound datasets written with default libver bounds (datatype message
version 1, i.e. plain h5py.File(path, 'w')) could not be read: the v1 member
layout has 28 bytes of legacy array fields after the byte offset
(dimensionality 1, reserved 3, permutation 4, reserved 4, four sizes 16) and
the parser skipped 24, so every following member was read 4 bytes off. v2 was
also wrong: it keeps the 8-byte name padding and has no array fields.
Found by adding a default-libver axis to the h5py-generated-file tests (HDF5
2.0 raised the default low bound to 1.8, so "default" files are a distinct
format path from libver='latest'). Adds byte-level v1/v2 regression tests, a
truncation test, and fuzz corpus seeds for v1 compound and native complex.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Bump all workspace crates, the node package and pyproject to 2.2.0 and
finalize the changelog. The repository URL in every manifest pointed at a
GitHub location that does not resolve; use the real origin.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Class 11 (datatype version 5) properties are a single base floating-point
datatype message, not a compound-style member list. The old parser read the
base type's bytes as member names, yielding a garbage datatype, and failed
with UnexpectedEof when a complex type was nested in a compound.
Parse the base type and surface the type as the equivalent {r, i} compound
(the shape h5py writes for numpy complex dtypes), with a size check against
the base type. Covered by byte-level tests taken from HDF5 2.0 output and an
h5py end-to-end test (writer_h5py_tests is now 27/27 against HDF5 2.0.0).
Found while validating a user report of InvalidDatatypeVersion
{ class: 6, version: 5 } against v2.1.0 (already fixed on main in a13ff51,
never released). Add docs/known-issues.md recording that report, this bug,
the open reference-v4 gap and a gpu_tests parallel-run hang; credit the
reporter in the changelog.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
ann/io/migrate/agent work from three agent missions, an independent audit, and the two defects it found: the cosine near-zero guard weakened during the SIMD migration, and WAL appends after a torn tail being silently unreplayable. 52 suites green; both fixes proven by negative control.
The SIMD migration weakened the near-zero-norm guard in all four
clawhdf5-accel cosine_similarity backends (scalar/avx2/avx512/neon)
from `denom < f32::EPSILON` to `denom == 0.0`. Vectors with a tiny
but nonzero norm (denom in (0, 1.19e-7)) fell through to dot/denom
and scored as identical instead of maximally dissimilar, diverging
from the pre-SIMD scalar loop's documented fallback behavior.
Restores the epsilon threshold in all four backends so
`1.0 - cosine_similarity(...)` in hnsw.rs::compute_distance
reproduces the old fallback exactly. Adds regression tests in
clawhdf5-accel and clawhdf5-ann locking in the near-zero-norm case.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Two missions found largely disjoint ground — v3 in clawhdf5-ann, -io and
-migrate, v6 in -agent and -format — so this merge is mostly additive.
One genuine collision: BOTH runs independently implemented
`Dataset::verify_provenance` in clawhdf5/src/reader.rs, and git kept both,
producing `E0592 duplicate definitions`. They were functionally identical
apart from `self.file.data.as_bytes()` (v3) vs `self.file.as_bytes()` (v6).
Kept v6's — it is the variant that compiles against the current tree and
passed 52 suites, and its doc comment is the more honest one, stating both the
full-read cost and that an unkeyed hash stored beside its data is not a
tamper-evidence guarantee.
Verified present after the merge:
P1 clawhdf5-ann now depends on clawhdf5-accel; compute_distance calls
l2_distance / cosine_similarity instead of a scalar loop
P2 AsyncFileReader caches its handle and length behind a mutex
PR1 clawhdf5-migrate writes SHINES provenance, and validate checks it
v6 WAL torn-tail truncation, char-boundary truncate, agent hardening
52 suites pass, 0 failures.
Co-Authored-By: Claude Opus 5 <[email protected]>
`WalFile::open` scanned the chained entries to resume the CRC chain, then
seeked to END OF FILE to append. After a crash mid-append — the ordinary way a
WAL ends up damaged — that puts the next entry BEHIND the torn bytes:
[1..N verified][torn tail][N+1, chained to N]
`read_chained_entries` stops at the torn tail, so N+1 is unreachable forever
even though its `append` returned Ok and synced. Silent loss of an acknowledged
write, in the one situation a WAL exists for.
`read_chained_entries` now also returns the byte length of the verified prefix,
and `open` truncates to it and appends there. The torn tail was never
acknowledged to any caller, so discarding it loses nothing, and the file offset
then matches the `running_crc` the chain continues from.
Verified by negative control: with the previous `seek(End(0))` the new test
fails with "got 1 entr(y/ies) — the post-crash write was silently lost".
Introduced by neither this branch nor the chaining work — v2 seeked to EOF too.
What changed is that `open` now scans and therefore KNOWS where the verified
prefix ends, which is what makes the fix a two-line consequence of information
already in hand.
Co-Authored-By: Claude Opus 5 <[email protected]>
Missed in the INT-16 flat-embedding-buffer commit — this integration
test call site lives under tests/, outside the src/ tree that was
grepped for callers.
Two related gaps in the WAL format, both closed:
1. Each entry's CRC32 covered only its own bytes, with no sequence number
or chaining — entries could be reordered, duplicated, or spliced (e.g.
a Tombstone moved before/after its target Save) while every individual
entry still passed its own CRC check, silently changing replayed cache
state. Bump to WAL_VERSION 3: each entry's CRC32 trailer is now computed
over its own bytes chained with the previous entry's stored CRC
(crc32(entry_bytes ++ prev_crc)), seeded at 0 after a truncation. Moving,
duplicating, or reordering an entry breaks the chain at that point, and
replay stops there — same handling as a bit-flip or truncation. The
previous per-entry-CRC-only format becomes WAL_VERSION_CRC_UNCHAINED (2)
and remains fully readable (not restricted, since it still verifies each
entry); WalFile::open migrates it to v3 by recreating the file fresh,
same as the existing v1 migration.
WalFile::open() on an existing v3 file scans it once to resume the CRC
chain correctly for further appends — required because a process
restart without an intervening flush reopens the same (non-truncated)
WAL and keeps appending to it, so new entries must chain against the
real last entry already on disk, not restart from 0.
2. WAL_VERSION_LEGACY_NO_CRC (v1, no integrity verification at all) was
reachable through the public WalFile::read_entries — a version byte
flipped from 2/3 down to 1 silently downgraded every entry to the
fully-unverified pre-hardening parser for any caller, not just the
one-time migration path. Split into WalFile::read_entries (rejects v1
with a typed error; still reads v2/v3) and the pub(crate)
read_entries_for_migration (accepts v1 too), used exclusively by
HDF5Memory::open's migration flow.
INT-09
verify_dataset existed and was tested, but was only ever called from
clawhdf5-format's own test files — no reader path in clawhdf5-io or the
clawhdf5 facade called it, so a corrupted dataset was silently readable
even though the write-side SHA-256 hash machinery (gated on the
provenance feature) had already written what it needed to detect that.
Add Dataset::verify_provenance() to the clawhdf5 facade, gated behind a
new `provenance` feature (on by default, forwarding to
clawhdf5-format/provenance which is already default-on). It surfaces a
typed VerifyResult (Ok/Mismatch/NoHash) via the existing Error type
rather than panicking. Deliberately NOT called automatically on
open()/dataset() — it decodes and hashes the entire dataset, which would
regress every read path (including the zero-copy/mmap ones) if run
unconditionally; callers opt in per dataset where the cost is
acceptable (e.g. a periodic integrity sweep).
Also re-export clawhdf5_format::provenance from the facade crate so
VerifyResult is reachable without depending on clawhdf5-format directly.
INT-08
check_rate_anomaly's 60s window is shared across all sessions/sources —
when it trips, the alert reported only the anonymous aggregate count,
unlike the separate cumulative max_writes_per_session check, which does
name the offending session. A session's write count can never exceed the
window's aggregate count, so whenever the window trips, name the
top-contributing session and source within it in the same alert instead
of adding a second, redundant per-session threshold check.
INT-07
check_pattern_anomaly did a plain case-folded literal-substring test, so
inserting whitespace, punctuation between letters, or a zero-width/
invisible-formatting character anywhere in a flagged phrase defeated every
one of the 15 injection patterns while the text still displays normally.
Add normalize_for_pattern_match: lowercases, drops control and
invisible-format characters (ZWSP, ZWJ, ZWNJ, bidi marks, BOM, soft
hyphen, word joiner, invisible math operators), drops punctuation
entirely (so split words rejoin instead of just being separated), and
collapses whitespace runs. Apply it to both the chunk and each configured
pattern before matching.
Scope, stated plainly: this does not add Unicode NFKC normalization or
confusable/homoglyph folding (e.g. Cyrillic а standing in for Latin a) —
that needs a per-codepoint confusable table (Unicode's confusables.txt)
beyond what's reasonable to hand-roll correctly, and no such crate is a
dependency of this crate today. A determined attacker using homoglyphs
can still evade these patterns; only the whitespace/punctuation/
zero-width bypasses are closed here.
INT-06
osobh