MemoryCache held every embedding in two places: a `Vec<Vec<f32>>` and a
flattened copy for the batched kernels, kept in lock-step on every push,
update and compaction. A store loaded from disk therefore carried the corpus
twice, plus one heap allocation per entry.
A new `cache::Embeddings` owns just the flat `[N x dim]` buffer and indexes
into it, so `embeddings[i]` still reads as a `&[f32]` row. The batch kernels
take a `VectorSet` (implemented for both `Embeddings` and `Vec<Vec<f32>>`)
instead of `&[Vec<f32>]`, so their callers and tests are unchanged. Loading no
longer unflattens what it just read.
100k 384-dim entries, reopened from disk: 505 -> 357 MiB, 3.44x -> 2.43x the
raw vectors. Recall (1.0000 at ef=64) and query latency are unchanged.
Rows are now always exactly `dim` long, shorter ones zero-padded. The old
representation allowed ragged rows, which silently misaligned the flattened
copy — every row after a wrong-length embedding — and `update` carried a
comment about falling back to a rebuild to avoid exactly that. It is now
unrepresentable. A record saved without an embedding holds a zero row and is
told apart by its norm, which is what `total_embeddings` now counts.
Measured with a counting allocator rather than RSS: freeing a structure
returns its pages to the allocator's pool, not the OS, so an RSS reading from
inside the process showed the two representations as identical.
Breaking: MemoryCache::embeddings changes type, embeddings_flat is replaced by
flat_embeddings(), rebuild_flat() is a deprecated no-op.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Two gaps in the LongMemEval harness, both of which had to close before any
recency feature could be judged.
The store was fed a synthetic counter (ts += 1.0 per turn) and the dataset's
own `haystack_dates` were ignored. Session order happened to be chronological,
so ordering was right, but the intervals were fiction — and exponential decay
is a function of the interval, so anything time-aware was being measured
against made-up ages. Dates are now parsed (civil-from-days, pinned against
reference values) and turns are spread over the minutes after their session
start; an unparseable date falls back to position so order still holds.
`newest_gold_first` measures what recall cannot. On a `knowledge-update`
question LongMemEval labels *both* the stale session and the one that
supersedes it as gold, so returning either scores as a hit even though only
one answers the question. The new metric asks whether the newest gold session
outranked the older ones. The current retriever scores 43-45% on it across
every mode — chance — which is the gap a temporal signal is supposed to close.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Bump all workspace crates, the node package and pyproject to 2.5.0, fold the
two unreleased sections together and add upgrade notes for the behaviour
changes.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The keyword stage had no stemming, so "training" and "trains" were unrelated
terms. bm25::TokenFilter::Stemmed strips common English inflections (plurals,
-ing/-ed, with consonant un-doubling) from documents and queries alike;
BM25Index::build_with and HDF5Memory::set_token_filter select it, and the
index records which filter built it so a stale one is rebuilt rather than
mixed.
Measured over the full LongMemEval haystack (500 questions, real MiniLM
embeddings) rather than adopted on principle — and it is a trade, not a win:
BM25 only Hit@1 53.8% Hit@5 75.0% Hit@10 81.6% MRR 0.6320
BM25 stemmed Hit@1 52.0% Hit@5 77.8% Hit@10 84.0% MRR 0.6320
Hybrid 0.4/0.6 Hit@1 51.6% Hit@5 81.4% Hit@10 87.8% MRR 0.6430
Hybrid stemmed Hit@1 50.2% Hit@5 81.4% Hit@10 88.2% MRR 0.6394
Conflation buys depth and costs the top rank: on BM25 alone MRR is unchanged
to four decimal places, the deeper gains exactly offsetting the rank-1 loss.
On the shipping hybrid configuration the vector stage already supplies most of
that recall, so the trade is narrower and slightly negative. Default stays
Plain; Stemmed is there for callers who want Hit@5/@10 over rank-1 precision.
The stemmer is deliberately conservative — it only strips inflections, and
only when the stem stays long enough to be meaningful, since an aggressive one
also conflates unrelated words. Tests pin both the pairs that must meet and
the pairs that must not.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Probing what QueryExpander::expand actually produces for LongMemEval questions
turned up two defects in the same helper.
`replace_word_case_insensitive` did a plain substring replace despite its
name, so acronym expansion fired inside ordinary words: "training" became
"trArtificial Intelligencening" ("ai") and "programming" became
"Pull Requestogramming" ("pr"). Nearly every acronym expansion of prose was
corrupt. Matching now requires word boundaries at both ends; real acronyms
(API, database) still expand in both directions.
The same helper searched `text.to_lowercase()` and then sliced `text` with the
offsets it found. That holds only while lowercasing preserves byte length, and
it does not — Turkish 'İ' is 2 bytes and lowercases to 3. Offsets after such a
character drifted, so output was silently corrupted ("İstanbul AI trip" lost a
character) or the slice landed inside a character or past the end and
panicked: `expand("İ AI")` was enough, from a plain query string. Matching now
walks the original string, comparing case-insensitively char by char, so
offsets are always valid.
Regression tests cover both, plus whole-word matching at string edges. The
morphological rules remain crude ("during" -> "dured"); that is a quality
limit, not a correctness bug, and is now documented as a reason to measure
before enabling expansion on a retrieval path.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
BENCHMARKS.md has recorded since the weight sweep that the 0.7/0.3 default is
strictly dominated by 0.4/0.6 over the full LongMemEval haystack, but the
shipping code never adopted it: unified_search and the OpenClaw backend both
passed 0.7/0.3. Re-running the sweep here (500 questions, real MiniLM
embeddings on a GPU) reproduces it — turn-level Hit@1 51.6% vs 44.2%, Hit@5
81.4% vs 79.2%, Hit@10 87.8% vs 85.8%, MRR 0.6430 vs 0.5856 — so both now use
hybrid::DEFAULT_FUSION, which is that operating point and carries the
reasoning. A unit test pins it.
Fusion is also selectable now. hybrid::Fusion is either Weighted { vector,
keyword } or Rrf { k }; hybrid::fuse applies either to one candidate list per
stage, and merge_vector_keyword / hybrid_search delegate to it, so the public
API is unchanged. New HDF5Memory::hybrid_search_with and
hybrid::hybrid_search_fused take a Fusion. Reciprocal rank fusion was
implemented but reachable only as a free function over a linear scan, so it
had never been compared with the weighted sum on equal terms; it is now a mode
in the LongMemEval bench (measurement to follow).
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
HDF5 1.12 revised the reference datatype (class 7) in datatype message version
4: reference types 2-4 are the new H5T_STD_REF object / dataset-region /
attribute references. Datatype::parse rejected them with
InvalidReferenceType, so any dataset of that type was unreadable.
h5py cannot write this type, which is why it had never been tested. A real
file was produced by calling the libhdf5 bundled in the h5py wheel through
ctypes (H5T_STD_REF_g, H5Rcreate_object, H5Dwrite); the 2 KB result is
committed as tests/fixtures/std_ref_hdf5_2_0.h5 with its generator,
gen_std_ref.py.
- ReferenceType gains Object2, DatasetRegion2 and Attribute, accepted only
from datatype version 4.
- read_object_references decodes Object2 elements: type(1) flags(1)
token_size(1) token, zero-padded to the element size; the token is the
target's object header address. A null reference decodes to the undefined
address; an external reference, a wrong type byte or a token that doesn't
fit is an error.
The fixture test follows both references and checks they resolve to the
objects they were created from.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Requesting a filter without chunk dimensions made the whole dataset a single
chunk. Any read, even one row, then decompresses everything, and a large
dataset cannot be decoded in parallel — which also made the new partial reads
pointless for such files.
auto_chunk_dims keeps datasets up to 1 MiB as one chunk (unchanged behaviour)
and splits larger ones by halving the dimensions in turn, so chunks keep
roughly the dataset's proportions, until a chunk is at most 1 MiB — h5py's
approach. An empty (unlimited, unwritten) dimension is treated as 1024. The
writer passes the element size through resolve_chunk_dims_for; the old
resolve_chunk_dims assumes 8-byte elements. Explicit with_chunks always wins.
Interop test: h5py reads an auto-chunked 13 MB deflate dataset, sees chunks
between 128 KiB and 1 MiB, and a small dataset still has one chunk.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
shuffle_decompress — on the read path of every compressed dataset, since
shuffle is applied automatically before compression — was the naive
`result[i * es + j] = data[j * n + i]`: a multiply and two bounds checks per
byte. It now interleaves fixed-width arrays of byte planes for element sizes
2/4/8/16 (bounds checks hoisted, vectorisable), with a chunked generic
fallback. The write-side shuffle was already optimised; this was the asymmetry
the survey flagged. Modest wall-clock effect now that decode is parallel
(chunked+deflate full read ~70 -> ~66 ms). Round-trip test over element sizes
1-24 and several lengths.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Same-moment A/B on a 64 MB f64 dataset: chunked+deflate 110 -> 69 ms, chunked
72 -> 60 ms, contiguous 56 -> 30 ms.
- read_chunked_data_cached — the path the facade uses — decompressed chunks
one at a time; only the uncached reader was parallel. Cache misses are now
decoded in bounded batches (128), in parallel with the `parallel` feature.
- Every chunk was pushed into the 16 MiB chunk cache, which a larger dataset
just churns (insert, evict moments later). Chunks are cached only when the
whole dataset fits (new ChunkCache::max_bytes).
- Unfiltered chunks went file -> Vec -> aligned cache buffer -> output. They
are copied straight from the file bytes.
- The facade's typed reads convert a contiguous dataset straight from the
borrowed file bytes instead of copying it into a Vec first.
- The native little-endian fast paths allocated vec![0; n] and then overwrote
it; they now fill an uninitialised buffer in one copy (native_le_to_vec).
alloc_output requests zeroed memory from the allocator instead of reserving
and filling.
The unit test that expected unfiltered chunks to land in the decompressed
cache now asserts the new design (index reused, cache not involved).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
With libver='latest', a chunked dataset with two or more unlimited dimensions
indexes its chunks with a v2 B-tree (layout v4, index type 5). Reading one
failed with "unsupported chunked layout version=4, index_type=Some(5)".
read_btree_v2_chunks decodes record types 10 (address + scaled offsets) and 11
(address, stored size, filter mask, scaled offsets). The width of the
stored-size field is taken from the record size the tree header declares
rather than re-deriving the library's formula. Scaled offsets are multiplied
back by the chunk dimensions with overflow checks.
The chunk-index dispatch existed four times (uncached, cached, sweep and
indexed readers). The three copies outside list_chunks now call it, so every
read path — and fill-value handling and partial reads — supports every index
type from one place.
h5py interop test: plain, gzip+shuffle, a 2500-chunk tree with internal nodes,
a sparse dataset with a fill value, and a strided hyperslab.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
read_raw_data_selection computed which chunks a selection intersects, threw
the answer away, decoded the entire dataset and picked elements out of it —
for contiguous layouts too. A 64x64 window of a 64 MB deflate dataset cost
105 ms, about half a full read; every selection cost the same whatever its
size.
New partial_read module: materialise only the selection's bounding box — the
overlapping rows of a contiguous dataset (straight from the file bytes) or the
overlapping chunks (only those are decompressed) — then run the existing
extractor over that buffer with the selection translated to the box origin, so
extraction semantics are exactly the full-read ones. It declines (falling back
to the old path) for All/None, compact/virtual/storage-less layouts, and boxes
covering more than half the dataset. That window now takes 0.39 ms, one row
2.7 ms, one column 5.2 ms.
Selections are validated against the dataset shape first. They were not: a
hyperslab past an edge came back padded with zeros and a point with an
out-of-range column wrapped into the next row, returning the wrong element
with no error. Now FormatError::SelectionOutOfBounds (also rank mismatch and
overlapping blocks); the facade's fill-aware path validates too.
Tests: equivalence against a reference extraction from a full read over 60
random hyperslabs/point lists per layout (contiguous, chunked, deflate) for
ranks 1-3. New read_harness bench binary with before/after in BENCHMARKS.md.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
`parallel` joins the agent's default features, so the HNSW bulk build uses the
thread pool: cold index build at 10K records 1152 -> ~380 ms in a same-moment
A/B (the graph is identical either way). Nothing else on the measured paths
changes — ingest, checkpoint, open and steady-state query times are the same
with the feature on or off. Adds rayon to the default dependency set; opt out
with `--no-default-features --features float16,hnsw`.
Harness: `--e2e-only` runs the end-to-end section without the index
benchmarks. Note for anyone comparing numbers: this machine's absolute timings
drifted ~1.5x over a long session, so only same-moment A/B runs are
comparable.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Profiling the build showed 90% of all distance evaluations are in back-link
pruning (40.8M of 44.9M at 10K): every overflow re-runs the diversity
heuristic pairwise over ~max_conn candidates.
The bulk build now inserts in batches: plan every node's neighbours against
the graph as it stood when the batch began (read-only, so plans are
independent), link, then prune each overflowing list once. A node gaining
several back-links in a batch is pruned once rather than once per link, so
this is faster even single-threaded (10K: 1676 -> 1074 ms). With `parallel`,
planning and pruning use rayon (10K: 388 ms; 100K: ~21 s -> 5.9 s on 16
cores). Batches start at one node and are capped at 1/16 of the linked graph
and 512 nodes; a node that raises the top layer gets a batch to itself. The
result is deterministic and identical with or without the feature (one code
path; test compares two builds byte for byte).
Parallelising within a single insert was tried first: 1.45x on 16 cores, tasks
too small. Incremental insert() stays sequential.
Recall on clustered data is unchanged or slightly better; uniform random data
dips slightly (10K, ef=64: 0.474 -> 0.444).
clawhdf5-agent's `parallel` feature now passes through to the index.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
search() collected ef candidates, then filtered out soft-deleted nodes, then
took k. When the records nearest a query had been deleted, every candidate was
a tombstone and the search returned fewer than k results — 39 of 40 queries in
the new test, which deletes each query's 40 nearest neighbours.
search_layer takes an optional skip mask: a skipped node is still pushed onto
the candidate queue (a tombstone is a valid waypoint) but never into the
result heap, so the ef result slots hold live nodes only. Build and insert
pass no mask. Recall and speed without deletions are unchanged.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Bump all workspace crates, the node package and pyproject to 2.4.0, finalize
the changelog and add upgrade notes for the search behaviour changes.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Fusion min-max normalises over every keyword match, so hybrid_search asked
BM25 for a ranked list of the whole corpus: a hash insert per posting, then a
sort of every match, then the merge sorted every candidate again to keep k.
- BM25Index::scores returns every match unsorted, accumulated in a dense array
(contributions are strictly positive, so zero means untouched). search() is
built on it with the bounded heap.
- merge_vector_keyword partitions out its top k (select_nth) and orders only
those, with the same score-then-id order.
- Both hybrid paths use scores().
Rankings are identical (equivalence tests for both changes). p50 0.24 -> 0.07
ms (1K), 2.1 -> 0.49 ms (10K), 23 -> 4.65 ms (100K).
The harness gains --fusion-study, which measured the alternative — capping the
keyword pool — and found it changes the top-10 for most queries (overlap
0.83-0.92, different #1 for 10-35%) for only a 2x saving. Not adopted.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
- Cosine distance was 1 - dot/(|a||b|), re-deriving both norms on every
evaluation in the innermost loop of build and search. The index now stores
unit vectors (prepared at build, insert, graph load and HDF5 load; the query
once per search) and uses 1 - dot. Zero vectors stay zero, giving distance 1
as before. Returned distances are unchanged.
- search_layer allocated a HashSet of visited nodes per call. It is now an
epoch-stamped u32 array in thread-local scratch, reused across calls, so
search(&self) stays shareable between threads.
- clawhdf5-accel caches the detected SIMD backend in a OnceLock.
Recall is identical. Build 2.75 -> 1.89 s (10K), ~38 -> 21 s (100K); QPS at
ef=64 22.7K -> 39K (10K), 10.4K -> 14K (100K).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
open() marked the HNSW index dirty, so the first search of every session
rebuilt it from scratch — 36 s at 100K records with the (better, slower)
heuristic build. First query after open is now 1.7 / 15 / 159 ms at
1K / 10K / 100K; what remains is the one-off keyword index build.
- clawhdf5-ann: HnswIndex::graph_to_bytes / from_graph_bytes serialize the
graph only (levels, tombstones, adjacency as u32, CRC32). The existing HDF5
serializer embeds a full copy of every vector, which would double a store
that already holds them. Loading validates everything — counts, levels vs
layer count, connection limits, every neighbour id and the layer it must
exist on — so a damaged graph, or a hostile one with a valid checksum, is an
error rather than an out-of-bounds walk during search.
- clawhdf5-agent: each checkpoint writes the graph to <store>.h5.ann (synced,
atomic, before the .h5) and records a fresh generation id in /meta. open()
loads the sidecar only if its generation matches that checkpoint; missing,
stale, damaged or mismatched sidecars are ignored and the index rebuilt.
Records appended through WAL replay join the loaded index incrementally; a
replayed Update or Tombstone invalidates it. snapshot() copies it. Only an
index that exactly mirrors the cache is saved; otherwise a stale sidecar is
removed.
- ensure_hnsw_fresh inserts records appended since the last sync instead of
rebuilding, so save_batch no longer marks the whole index dirty.
- CheckpointMeta { wal_applied, ann_generation } with *_with_meta build/write/
read functions; the *_with_mark ones delegate.
- Harness reports the one-off cold index build separately from the first query
after a reopen.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
hybrid_search rebuilt the BM25 index from scratch (re-tokenising every record)
and rewrote the whole .h5 file on every single query, so a query cost O(store
size) in both CPU and disk I/O. Steady-state p50 per the search harness:
5.5 -> 0.24 ms (1K), 49 -> 2.1 ms (10K), 884 -> 23 ms (100K).
- BM25Index is incremental: add_document / remove_document keep it exactly
equivalent to a fresh build over the same live documents (property test: 60
random op sequences compared against BM25Index::build after every step). IDF
moves to query time since it depends on the live document count. Top-k uses
a bounded heap, ties break by doc id (results were HashMap-ordered), and the
"WAND" code that computed a bound and then discarded it is removed.
- HDF5Memory keeps one index for its lifetime, built lazily. Appends are
picked up by ensure_bm25_fresh whatever path added them; delete and in-place
update report themselves; compaction drops the index. A test drives every
mutation and compares against a fresh build.
- A query no longer calls flush(). Activation boosts are marked dirty and
persisted by the next checkpoint, including a best-effort one on drop so a
search-only session keeps them (approved behaviour change). Activation
weights are capped at 16.0; they previously grew without bound.
The archived mission branch's BM25 cache was reviewed and not used: it was
invalidated by every write, so interleaved save/search still rebuilt per
query, and it changed the default fusion weights.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Neighbours were selected as the plain closest-M, for both a new node's links
and back-link pruning. On clustered data every link of a node inside a tight
cluster then goes to that same cluster, so clusters become islands a search
entering elsewhere can never reach: recall@10 was 0.87 / 0.67 / 0.31 at
1K / 10K / 100K (384-dim) and flat in ef. Uniform random data — all the
existing tests used — does not show it.
Implement the HNSW paper's Algorithm 4 with keepPrunedConnections: accept a
candidate only if it is closer to the node than to every neighbour already
accepted, then fill spare slots with the closest rejected ones. Recall@10 at
ef=64 is now 1.00 / 1.00 / 0.98 and rises with ef; uniform data improves
slightly. Build is ~3.5x slower at 10K (extra distance evaluations), to be
recovered by the distance-kernel work. The needless rayon fan-out over <=33
distances in prune_connections is gone.
Tests: a clustered-data recall test for bulk build and incremental insert
(scores 0.43 with the old selection), and a unit test of the selection rule.
Harness gains --uniform and --ann-only; before/after in BENCHMARKS.md.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
New clawhdf5-bench binary `search_harness`, the measurement baseline for the
search hot-path work. On deterministic clustered 384-dim data it reports HNSW
build time and, per ef, recall@10 against an exact scan, QPS and p50/p99; and
for HDF5Memory: ingest, checkpoint, open, first-query-after-open and
steady-state hybrid_search latency at 1K/10K (and 100K with --full). Optional
JSON output for tracking.
Baseline recorded in BENCHMARKS.md. It shows two problems: HNSW recall@10 does
not respond to ef and falls from 0.87 (1K) to 0.31 (100K) on clustered data,
and end-to-end hybrid_search is ~1000x slower than its vector stage because
every query rebuilds BM25 and rewrites the .h5 file.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Bump all workspace crates, the node package and pyproject to 2.3.0, finalize
the changelog and add upgrade notes for the behaviour changes.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
attrs() silently omitted any attribute whose datatype had no AttrValue variant
— including every Python bool, which h5py stores as an enum — plus complex,
compound and reference attributes, and cast unsigned 64-bit arrays to
I64Array so values above i64::MAX came back negative.
- numpy/h5py-style booleans (an enum of exactly FALSE=0 / TRUE=1 over an
integer base) decode as I64 / I64Array of 0/1.
- AttrValue::U64Array keeps unsigned arrays unsigned. Behaviour change: an
unsigned array attribute no longer arrives as I64Array; the netCDF-4 CF
helpers (_FillValue, valid_range) and the Python bindings handle it.
- AttrValue::Raw { datatype, shape, data } carries any other attribute
verbatim (also used when a value fails to decode as its declared type), so
the attribute list is always complete. Decodable with data_read against the
datatype; Python receives {"dtype", "shape", "data"}.
- Both new variants are writable, so attributes round-trip between files.
h5py interop tests cover reading 13 attribute kinds and h5py reading back a
compound and a u64 attribute written by clawhdf5.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
- tests/wal_properties.rs — deterministic generator, reproducible by seed:
everything appended is read back intact (300 cases), and after ANY damage
to the file (bit flips, truncation, inserted/deleted bytes, duplicated or
rotated regions, overwritten ranges; 1500 cases) reading never panics and
yields an exact prefix of what was written — the guarantee the chained CRC
exists to give. Opening for append then repairs the tail and a new entry
lands right behind the surviving prefix.
- tests/crash_recovery.rs — builds the on-disk images a process crash can
leave and reopens each against a model of what was acknowledged: an image
after every operation (random saves, in-place updates, checkpoints, small
wal_max_entries), the checkpoint window (new .h5 + not-yet-truncated WAL)
over several rounds, and the WAL torn at every byte length, which must
recover the checkpoint plus a prefix of the operations logged since.
- fuzz/fuzz_wal_replay — arbitrary bytes as a WAL: read and open-for-append
must not panic, and open() must not change what is replayable. Based on the
target from the clawmates mission branch (4aee2fa), with the repair
property added. The CI fuzz step now covers both fuzz crates.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The VDS resolver joined the source file name stored in the HDF5 file straight
onto the opened file's directory. That name is untrusted: an absolute path
replaces the base directory outright and `..` components climb out of it, so
a crafted file could make the reader open any path the process can reach.
Only plain relative paths of normal components are accepted now; anything
else resolves to "source not found".
Co-Authored-By: Claude Fable 5.1 <[email protected]>
- Path resolution follows soft links in both old-style (symbol table, cache
type 2) and new-style (compact and dense Link message) groups: absolute and
relative targets, links to groups, links through links, with a depth limit
so a link cycle is NestingDepthExceeded rather than a hang. A dangling link
reports the target it could not find. Previously every soft link was
PathNotFound.
- An external link is FormatError::ExternalLinkUnsupported { filename,
object_path } instead of a misleading PathNotFound.
- Message 0x0007 (External Data Files) is now a known MessageType, and a
dataset carrying it is FormatError::ExternalDataFilesUnsupported. Such a
dataset has no data address in this file, so it would otherwise be read as
"never written" and answered with fill values — wrong data, no error.
- Dense link iteration is shared between hard-link listing and the new
symbolic-link lookup; entry listing behaviour is unchanged.
- h5py interop test for both libver settings.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
HDF5 allocates lazily: a chunk nobody wrote doesn't exist in the file, and a
dataset nobody wrote has no data address. Such regions must read as the
dataset's fill value. There was no Fill Value message parser at all, so:
- a sparse chunked dataset read its holes as zeros — silently wrong whenever
the fill value isn't zero (h5py `fillvalue=-1` came back as 0);
- a dataset that was created but never written failed with NoDataAllocated /
"no address for chunked layout" where h5py returns a filled array.
New clawhdf5_format::fill_value: parses Fill Value messages v1-v3 and the old
0x0004 message (validated against HDF5 2.0 output under default and latest
libver), builds a fully filled dataset when there is no storage, and writes the
fill value into exactly the chunk-grid cells absent from the chunk index —
never mistaking a stored zero for a hole, clipping edge chunks, any rank. It is
skipped entirely for the default (zero) fill value. The chunk index dispatch is
extracted from read_chunked_data into a reusable list_chunks.
The reader, lazy and mmap facades apply it on full reads; selection reads go
through a fill-aware full read when the fill value matters. h5py interop test
compares against h5py's own readback, including a sparse 2-D dataset and a
hyperslab straddling allocated and unallocated chunks.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
A dataset created from a committed (named) datatype stores only a shared-
message reference to it. The facade parsed those reference bytes as the
datatype itself, producing `Time { size: 0 }` and unreadable data, and an
attribute using a committed datatype was silently dropped.
- shared_message::parse_shared_ref had the encoding wrong: it skipped six
reserved bytes for version 2 (only version 1 has them) and had the version 3
types inverted (1 is the SOHM heap, 2 is "committed, in another object
header"). Verified against h5py 3.16 / HDF5 2.0, which writes
`02 02 <address>` under both default and latest libver bounds. Resolution
now dispatches on which field the reference carries.
- New shared_message::message_data resolves a header message through the
indirection; the reader, lazy and mmap facades use it for datatype,
dataspace and filter-pipeline messages.
- AttributeMessage honours the v2/v3 flags (bit 0 datatype shared, bit 1
dataspace shared) via the new parse_in_file, used everywhere file data is
available. Parsing a shared attribute without file access is now
FormatError::UnresolvedSharedMessage instead of a garbage datatype.
- h5py interop test covering both libver settings.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
- Dataset::filter_pipeline() (reader, lazy and mmap variants) swallowed parse
errors with `.ok()`, so a malformed pipeline message silently became "no
filters" and the still-compressed chunk bytes were returned as the data. It
now returns Result<Option<_>>; a present-but-unparseable pipeline is
Error::Format.
- FileBuilder::write used std::fs::write, which truncates the destination
first: a crash mid-write destroyed the existing file. It now writes a
sibling temp file, syncs it, renames it over the target and syncs the
directory, cleaning the temp file up on failure.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Dataspace and chunk dimensions are untrusted 64-bit fields, but the chunked
read paths computed `num_elements() as usize * elem_size` and
`chunk_dims.product() * elem_size` with plain arithmetic and fed the result to
`vec![0u8; n]`. A crafted file could wrap the product (under-sizing the output
buffer that chunks are then copied into) or request an allocation large enough
to abort the process.
- Dataspace::checked_num_elements, checked_byte_len, checked_chunk_byte_len
and alloc_output (try_reserve_exact) replace the plain products and
vec![0; n] at every chunked read site, plus the VDS and hyperslab paths.
Overflow and allocation failure are FormatError::Overflow.
- Dataspace::num_elements saturates instead of wrapping.
- A zero-element dataset returns early, which also keeps the stride products
in range when another dimension is huge.
- parallel_read.rs: the three `c_addr + size > len` bounds checks used a raw
add; they now use checked_add like the rest of the crate.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
test_hebbian_activation_boost failed intermittently. Root causes, all in the
query path:
- normalize_scores mapped a set of identical scores — including the
single-candidate case — to 0.0, so a lone perfect match contributed nothing
to the fused score. Identical positive scores now normalise to 1.0 (all
equally the best match); identical non-positive scores stay 0.0.
- merge_vector_keyword sorted a HashMap's entries by score alone and then
truncated, so which ties survived varied from run to run; hybrid_search had
the same problem in its final sort. Both now break ties by index.
- hybrid_search applied the Hebbian boost to every returned record, including
the zero-score filler that pads the list when fewer than k records match.
With random tie-breaking a filler record could collect as many boosts as the
real hit. Only records with a positive fused score are reinforced now.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
- ProvenanceStore::remap: compaction renumbers cache indices (which are the
provenance record ids) but nothing renumbered the ledger, so after any
compaction — including the automatic one in delete() — every surviving
record's hash was filed under a different record and the next
save_or_update raised a bogus High "integrity mismatch" alert.
- Pending anomaly alerts are capped (newest 1024 kept). Alerts never block a
save, and a session over its write limit alerts on every write, so a caller
that didn't drain them grew the queue without bound.
- WriteAnomalyDetector tracks at most 4096 sessions, forgetting the
least-active half on overflow instead of leaking one entry per session id
for the life of the process.
- snapshot() copies the pending WAL next to the .h5 copy, so a snapshot is
the store as it is now rather than as of the last checkpoint (it used to
silently omit up to wal_max_entries recent saves).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
- HDF5Memory::create/open take an exclusive advisory lock on <store>.h5.lock
(std File::try_lock, no new dependency). The store lives in memory and is
rewritten wholesale at each checkpoint, so two handles on one store used to
silently destroy each other's data; a second writer now gets
MemoryError::Locked. The OS drops the lock with the descriptor, so a crash
never leaves a stale lock. Acquisition retries for ~250 ms to absorb a
previous owner that is mid-teardown; AsyncHDF5Memory::shutdown releases the
lock once its writer task has stopped.
- HDF5Memory::open_read_only: a lock-free, point-in-time view (checkpoint +
current WAL contents, replayed in memory) that never writes — it does not
repair, upgrade or move the WAL, and anything that would persist returns an
error. The CLI's recall/stats/agents-md/export use it, so a store can be
inspected while an agent has it open. Tests that reopened a store purely to
verify on-disk state now use it.
- open() no longer fails on a WAL that cannot possibly be replayed (torn
header, bad magic): it is moved to <store>.h5.wal.corrupt-<ts>, reported via
HDF5Memory::quarantined_wal(), and the healthy .h5 opens from its last
checkpoint. A well-formed header with an unknown version still fails and is
left untouched — most likely a newer build's WAL, which must not be
discarded.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Eight MemoryConfig fields (float16, compression, compression_level,
compact_threshold, hebbian_boost, decay_factor, wal_enabled, wal_max_entries)
were never written to /meta, so reopening a store silently reset them to
defaults — a compressed store was rewritten uncompressed by the first
checkpoint after a reopen, and wal_enabled=false flipped back to true. They
are now stored as /meta attributes; each is optional on load so older files
keep opening with the previous defaults, and non-finite floats are ignored.
Writing the round-trip test exposed that `compression = true` never worked in
a default build: the embeddings dataset called with_zstd() unconditionally but
the agent crate never enabled the zstd feature, so every checkpoint failed
with "unsupported filter: 32015". The default build now compresses with
deflate (always available, pure Rust path); Zstd is opt-in via a new `zstd`
agent feature.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
A save_or_update that hit an existing record was logged as a plain Save, so
replaying the WAL appended a duplicate instead of updating in place. It is now
logged as WalEntryType::Update (0x04) carrying the target index, and replay
applies it with cache.update().
The WAL header version goes 3 -> 4 for the benefit of older binaries: they
don't know record type 0x04, would read it as a torn tail and truncate it and
everything after it. An unknown header version makes them refuse the file
instead. The framing is otherwise identical, so v3 files are read by the same
code and upgraded in place on open (the header is outside the CRC chain).
Also drop the redundant WAL truncate that several callers ran straight after
flush(), which already truncates.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
flush() writes the new .h5 and only then truncates the WAL. A crash in that
window left a .h5 that already contained the pending entries AND a WAL that
still listed them, and open() replayed the WAL unconditionally — every pending
entry came back twice.
A checkpoint now records a WalMark in /meta (wal_applied_len/wal_applied_crc):
the byte length and chained CRC of the WAL prefix it folded in. On open, if
the WAL's v3 CRC chain passes through exactly that position, the entries up to
it are skipped; otherwise (the normal case: the WAL was truncated) everything
is replayed. No WAL format change; files without the attributes behave as
before. WalFile tracks its chain length alongside running_crc and resumes both
on reopen.
Also make the checkpoint and snapshot durable as a unit: sync the temp file
before the rename and the parent directory after it, so a power loss can't
leave an empty or partial .h5 under the final name. This is per-checkpoint
cost only; individual WAL appends remain unsynced by design.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The norms guard was the tautology `n.len() == n.len()`, so a norms dataset
of any length was trusted and corrupted every cosine score; other per-record
datasets were not length-checked at all, so a truncated file loaded and then
panicked on the first index. Mismatches are now MemoryError::Schema, stored
norms are used only when they match the record count, and embedding_dim == 0
with records present is rejected instead of panicking in chunks(0).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Formatting only. cargo fmt --check was already failing on main (accel SIMD
kernels, agent, format, migrate, bench); CI now enforces it.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
- clippy --all-targets plus a clawhdf5-format feature matrix (parallel, lz4,
zstd, pcodec, fast-checksum); fix the accumulated lint backlog in test,
bench and feature-gated code (no behaviour changes).
- Install python3 + h5py/numpy/netCDF4/xarray in the CI container and set
CLAWHDF5_REQUIRE_INTEROP=1, which makes a missing interop dependency a test
failure. Every h5py/netCDF4 interop test used to skip silently in CI. Run
the #[ignore]d writer_h5py_tests suite explicitly.
- cargo bench --no-run so benches can't rot; fix bench.rs and memory_bench.rs,
which no longer compiled against the current strategy/consolidation APIs.
- Optional fuzz smoke run via CLAWHDF5_FUZZ_SECONDS.
- CHANGELOG and docs/known-issues.md updated.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Every test created its own wgpu instance and device (with adapter-maximum
limits) concurrently, which could wedge the driver and hang the suite
indefinitely. Tests now hold a process-wide lock while they own a device, and
GpuAccelerator readback waits are bounded at 30s so a stuck driver surfaces as
GpuError::BufferMap instead of blocking forever.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Compound datasets written with default libver bounds (datatype message
version 1, i.e. plain h5py.File(path, 'w')) could not be read: the v1 member
layout has 28 bytes of legacy array fields after the byte offset
(dimensionality 1, reserved 3, permutation 4, reserved 4, four sizes 16) and
the parser skipped 24, so every following member was read 4 bytes off. v2 was
also wrong: it keeps the 8-byte name padding and has no array fields.
Found by adding a default-libver axis to the h5py-generated-file tests (HDF5
2.0 raised the default low bound to 1.8, so "default" files are a distinct
format path from libver='latest'). Adds byte-level v1/v2 regression tests, a
truncation test, and fuzz corpus seeds for v1 compound and native complex.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Bump all workspace crates, the node package and pyproject to 2.2.0 and
finalize the changelog. The repository URL in every manifest pointed at a
GitHub location that does not resolve; use the real origin.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Class 11 (datatype version 5) properties are a single base floating-point
datatype message, not a compound-style member list. The old parser read the
base type's bytes as member names, yielding a garbage datatype, and failed
with UnexpectedEof when a complex type was nested in a compound.
Parse the base type and surface the type as the equivalent {r, i} compound
(the shape h5py writes for numpy complex dtypes), with a size check against
the base type. Covered by byte-level tests taken from HDF5 2.0 output and an
h5py end-to-end test (writer_h5py_tests is now 27/27 against HDF5 2.0.0).
Found while validating a user report of InvalidDatatypeVersion
{ class: 6, version: 5 } against v2.1.0 (already fixed on main in a13ff51,
never released). Add docs/known-issues.md recording that report, this bug,
the open reference-v4 gap and a gpu_tests parallel-run hang; credit the
reporter in the changelog.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The SIMD migration weakened the near-zero-norm guard in all four
clawhdf5-accel cosine_similarity backends (scalar/avx2/avx512/neon)
from `denom < f32::EPSILON` to `denom == 0.0`. Vectors with a tiny
but nonzero norm (denom in (0, 1.19e-7)) fell through to dot/denom
and scored as identical instead of maximally dissimilar, diverging
from the pre-SIMD scalar loop's documented fallback behavior.
Restores the epsilon threshold in all four backends so
`1.0 - cosine_similarity(...)` in hnsw.rs::compute_distance
reproduces the old fallback exactly. Adds regression tests in
clawhdf5-accel and clawhdf5-ann locking in the near-zero-norm case.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Two missions found largely disjoint ground — v3 in clawhdf5-ann, -io and
-migrate, v6 in -agent and -format — so this merge is mostly additive.
One genuine collision: BOTH runs independently implemented
`Dataset::verify_provenance` in clawhdf5/src/reader.rs, and git kept both,
producing `E0592 duplicate definitions`. They were functionally identical
apart from `self.file.data.as_bytes()` (v3) vs `self.file.as_bytes()` (v6).
Kept v6's — it is the variant that compiles against the current tree and
passed 52 suites, and its doc comment is the more honest one, stating both the
full-read cost and that an unkeyed hash stored beside its data is not a
tamper-evidence guarantee.
Verified present after the merge:
P1 clawhdf5-ann now depends on clawhdf5-accel; compute_distance calls
l2_distance / cosine_similarity instead of a scalar loop
P2 AsyncFileReader caches its handle and length behind a mutex
PR1 clawhdf5-migrate writes SHINES provenance, and validate checks it
v6 WAL torn-tail truncation, char-boundary truncate, agent hardening
52 suites pass, 0 failures.
Co-Authored-By: Claude Opus 5 <[email protected]>
