`MemoryConfig::quantized_index` now defaults to `true`. It holds a
quarter of the index memory and, with the exact re-score, is faster at
equal recall on every configuration measured: 1.63x the queries per
second on x86-64 (AVX2) and 1.18x on a Raspberry Pi 5 (NEON SDOT), with
builds 1.8x and 2.3x faster. The one argument for keeping it off — that
int8 search was slower on ARM — did not survive being measured.
Existing stores do not change. A store written with v2.6.0 or later
keeps its persisted setting. One written before the setting existed has
no stored value, and it loads as `false` rather than as the new default,
so reopening it never changes how its index is held. That case is
guarded by a real store written with the v2.5.0 CLI, committed as
`tests/fixtures/store_v2_5_0.h5` (6.8 KB): the test asserts it reopens
with an f32 index and still searches, and it fails if the load default
is changed to `true`.
The CLI needed more than a new default. `create --quantized-index`
assigned its value straight into the config, so under the new default
every CLI-created store would have been forced back to f32 unless the
caller knew to ask for int8. It is replaced by `--f32-index`, which only
ever switches the default off; `--quantized-index` is still accepted,
hidden, as a no-op, and the two conflict.
The whole agent suite passes under the new default, including the
brute-force recall oracle, now running on int8 plus re-score without
being asked to.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Graph degree and the build- and query-time candidate list sizes were
constants, so a deployment had no way to trade recall against memory or
query speed. They are now `MemoryConfig::hnsw_m`,
`hnsw_ef_construction` and `hnsw_ef_search`, persisted with the store
and defaulting to exactly the previous behaviour (16, 64, and a query
list that scales with `k`).
Two things the straightforward version would have got wrong:
`clawhdf5-ann` asserts a graph degree of at least 2, so a configured 0 —
from a file, or from a caller reading 0 as "use the default" — aborted
the process inside the index builder. The store clamps instead, and a
test covers it: removing the clamp makes that test panic rather than
fail.
`ef_search` and the candidate pool handed to score fusion were the same
number. Tying the pool to the new setting would mean lowering `ef` for
speed also narrows what fusion sees, quietly degrading hybrid results
through a knob that looks like it only costs time. They are now
independent.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
`MemoryConfig::quantized_index` stores the HNSW index's own copy of the
embeddings as i8 rather than f32. At 100k x 384 that takes the index from
266 to 123 MiB and the whole reopened store from 399 to 256 MiB — 2.72x
to 1.74x the raw vectors, the largest remaining item in the footprint.
Quantised distances are approximate and `ef` cannot compensate, because
the loss is in the distances rather than in the graph: recall@10 tops out
at 0.967 against f32's 0.9995 and does not move between ef=128 and
ef=256. The store already holds the exact embeddings, though, so when the
index is quantised the query path re-scores the candidate pool against
them before fusion. That restores recall (0.9940 vs 0.9945 at ef=64) and
costs about 13% of QPS.
Off by default: it trades query speed for memory and which side is worth
more depends on the deployment. The flag is persisted in `/meta`, so a
reopened store does not silently revert to four times the index memory,
and the sidecar graph is rehydrated into the configured storage.
Also on the CLI as `create --quantized-index`.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
MemoryCache held every embedding in two places: a `Vec<Vec<f32>>` and a
flattened copy for the batched kernels, kept in lock-step on every push,
update and compaction. A store loaded from disk therefore carried the corpus
twice, plus one heap allocation per entry.
A new `cache::Embeddings` owns just the flat `[N x dim]` buffer and indexes
into it, so `embeddings[i]` still reads as a `&[f32]` row. The batch kernels
take a `VectorSet` (implemented for both `Embeddings` and `Vec<Vec<f32>>`)
instead of `&[Vec<f32>]`, so their callers and tests are unchanged. Loading no
longer unflattens what it just read.
100k 384-dim entries, reopened from disk: 505 -> 357 MiB, 3.44x -> 2.43x the
raw vectors. Recall (1.0000 at ef=64) and query latency are unchanged.
Rows are now always exactly `dim` long, shorter ones zero-padded. The old
representation allowed ragged rows, which silently misaligned the flattened
copy — every row after a wrong-length embedding — and `update` carried a
comment about falling back to a rebuild to avoid exactly that. It is now
unrepresentable. A record saved without an embedding holds a zero row and is
told apart by its norm, which is what `total_embeddings` now counts.
Measured with a counting allocator rather than RSS: freeing a structure
returns its pages to the allocator's pool, not the OS, so an RSS reading from
inside the process showed the two representations as identical.
Breaking: MemoryCache::embeddings changes type, embeddings_flat is replaced by
flat_embeddings(), rebuild_flat() is a deprecated no-op.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
open() marked the HNSW index dirty, so the first search of every session
rebuilt it from scratch — 36 s at 100K records with the (better, slower)
heuristic build. First query after open is now 1.7 / 15 / 159 ms at
1K / 10K / 100K; what remains is the one-off keyword index build.
- clawhdf5-ann: HnswIndex::graph_to_bytes / from_graph_bytes serialize the
graph only (levels, tombstones, adjacency as u32, CRC32). The existing HDF5
serializer embeds a full copy of every vector, which would double a store
that already holds them. Loading validates everything — counts, levels vs
layer count, connection limits, every neighbour id and the layer it must
exist on — so a damaged graph, or a hostile one with a valid checksum, is an
error rather than an out-of-bounds walk during search.
- clawhdf5-agent: each checkpoint writes the graph to <store>.h5.ann (synced,
atomic, before the .h5) and records a fresh generation id in /meta. open()
loads the sidecar only if its generation matches that checkpoint; missing,
stale, damaged or mismatched sidecars are ignored and the index rebuilt.
Records appended through WAL replay join the loaded index incrementally; a
replayed Update or Tombstone invalidates it. snapshot() copies it. Only an
index that exactly mirrors the cache is saved; otherwise a stale sidecar is
removed.
- ensure_hnsw_fresh inserts records appended since the last sync instead of
rebuilding, so save_batch no longer marks the whole index dirty.
- CheckpointMeta { wal_applied, ann_generation } with *_with_meta build/write/
read functions; the *_with_mark ones delegate.
- Harness reports the one-off cold index build separately from the first query
after a reopen.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Eight MemoryConfig fields (float16, compression, compression_level,
compact_threshold, hebbian_boost, decay_factor, wal_enabled, wal_max_entries)
were never written to /meta, so reopening a store silently reset them to
defaults — a compressed store was rewritten uncompressed by the first
checkpoint after a reopen, and wal_enabled=false flipped back to true. They
are now stored as /meta attributes; each is optional on load so older files
keep opening with the previous defaults, and non-finite floats are ignored.
Writing the round-trip test exposed that `compression = true` never worked in
a default build: the embeddings dataset called with_zstd() unconditionally but
the agent crate never enabled the zstd feature, so every checkpoint failed
with "unsupported filter: 32015". The default build now compresses with
deflate (always available, pure Rust path); Zstd is opt-in via a new `zstd`
agent feature.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
flush() writes the new .h5 and only then truncates the WAL. A crash in that
window left a .h5 that already contained the pending entries AND a WAL that
still listed them, and open() replayed the WAL unconditionally — every pending
entry came back twice.
A checkpoint now records a WalMark in /meta (wal_applied_len/wal_applied_crc):
the byte length and chained CRC of the WAL prefix it folded in. On open, if
the WAL's v3 CRC chain passes through exactly that position, the entries up to
it are skipped; otherwise (the normal case: the WAL was truncated) everything
is replayed. No WAL format change; files without the attributes behave as
before. WalFile tracks its chain length alongside running_crc and resumes both
on reopen.
Also make the checkpoint and snapshot durable as a unit: sync the temp file
before the rename and the parent directory after it, so a power loss can't
leave an empty or partial .h5 under the final name. This is per-checkpoint
cost only; individual WAL appends remain unsynced by design.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The norms guard was the tautology `n.len() == n.len()`, so a norms dataset
of any length was trusted and corrupted every cosine score; other per-record
datasets were not length-checked at all, so a truncated file loaded and then
panicked on the first index. Mismatches are now MemoryError::Schema, stored
norms are used only when they match the record count, and embedding_dim == 0
with records present is rejected instead of panicking in chunks(0).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
blas_cosine_batch and accelerate_cosine_batch_vecs re-flattened the
entire Vec<Vec<f32>> corpus into a fresh Vec<f32> on every single
query before running the batch matmul — an O(N·dim) copy paid per
query when fast-math/accelerate/openblas is enabled, even though a
flat fast-path (blas_cosine_batch_flat / accelerate_cosine_batch)
already existed for pre-flattened input.
Add MemoryCache::embeddings_flat, a contiguous [N × embedding_dim]
buffer maintained incrementally in push/update/compact (O(1) amortized
append, O(dim) in-place overwrite, O(n) rebuild only on compact/bulk
load). schema.rs's direct-push load path calls the new rebuild_flat()
explicitly. flat_embeddings() now just clones the already-maintained
buffer instead of rebuilding it.
Thread the flat buffer through strategy::search_with_metrics as a new
vectors_flat parameter, used only by the Blas/Accelerate arms (now
calling the *_flat variants); other strategies are unaffected. No
current caller wires search_with_metrics into the production query
path yet (only its own tests exercise it) — this fixes the identified
per-query re-flatten and makes the flat buffer available for whenever
that wiring lands.
INT-16
resolve_or_create allocated a fresh lowercased String for every entity
on every call (this runs per extracted mention during entity/relation
extraction) and never short-circuited on an exact dist == 0 match,
scoring every remaining entity regardless. Add Entity::name_lower,
computed once at construction (add_entity, and schema.rs's direct-push
load path), and break out of the scan as soon as an exact match is
found.
INT-12
- Use Zstd level 3 instead of deflate(1) for embedding dataset compression.
Auto-shuffle (already the default since the TDT pre-filter commit) is now
the only shuffle needed — the explicit .with_shuffle() call was redundant.
- Benchmark: save_without_wal_single improves 67 → 61 µs (-9%).
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
clawhdf5-agent: fixed-length string datasets (memory text chunks, session
summaries, ids, tags, entity/relation names) were stored uncompressed behind
a stale "chunked compound not yet supported" comment. Chunked writes work for
fixed-size string/compound datatypes like any other, so write_string_dataset
now chunks + deflates once a dataset's payload reaches 4 KiB — large,
redundant NullPad content compresses well while tiny metadata stays
contiguous (no chunk-overhead bloat). The dead `compress` parameter is
removed in favor of this size heuristic.
clawhdf5-format: enabling string compression exposed a latent bug — the
per-file ChunkCache built its chunk index once and reused it for every
chunked dataset in the file, keyed only by chunk coordinate with no dataset
discrimination. With one chunked dataset per file this never surfaced; with
two of different rank (a 1-D compressed string array and the 2-D embeddings
matrix) the first dataset's rank-1 index was reused for the second, panicking
with an out-of-bounds chunk coordinate. The cache now binds to a dataset by
its chunk-index address and rebinds — dropping the index, chunk-index map,
layout, and decompressed slots — whenever the dataset being read changes,
while still caching repeated/sequential access to the same dataset.
Tests: facade regression reading a 1-D compressed string dataset and a 2-D
compressed f32 dataset through one shared File cache (verified to panic
without the fix); existing agent e2e tests (large text chunks, migration
round-trip) now pass with compression on.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
