perf(agent): store embeddings once, not twice

MemoryCache held every embedding in two places: a `Vec<Vec<f32>>` and a
flattened copy for the batched kernels, kept in lock-step on every push,
update and compaction. A store loaded from disk therefore carried the corpus
twice, plus one heap allocation per entry.

A new `cache::Embeddings` owns just the flat `[N x dim]` buffer and indexes
into it, so `embeddings[i]` still reads as a `&[f32]` row. The batch kernels
take a `VectorSet` (implemented for both `Embeddings` and `Vec<Vec<f32>>`)
instead of `&[Vec<f32>]`, so their callers and tests are unchanged. Loading no
longer unflattens what it just read.

100k 384-dim entries, reopened from disk: 505 -> 357 MiB, 3.44x -> 2.43x the
raw vectors. Recall (1.0000 at ef=64) and query latency are unchanged.

Rows are now always exactly `dim` long, shorter ones zero-padded. The old
representation allowed ragged rows, which silently misaligned the flattened
copy — every row after a wrong-length embedding — and `update` carried a
comment about falling back to a rebuild to avoid exactly that. It is now
unrepresentable. A record saved without an embedding holds a zero row and is
told apart by its norm, which is what `total_embeddings` now counts.

Measured with a counting allocator rather than RSS: freeing a structure
returns its pages to the allocator's pool, not the OS, so an RSS reading from
inside the process showed the two representations as identical.

Breaking: MemoryCache::embeddings changes type, embeddings_flat is replaced by
flat_embeddings(), rebuild_flat() is a deprecated no-op.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-19 20:17:55 -07:00
co-authored by Claude Opus 5
parent dc0113d015
commit 2e7e0456c1
9 changed files with 429 additions and 103 deletions
+5 -3
View File
@@ -466,7 +466,7 @@ impl ClawhdfBackend {
let record = MemoryRecord {
id: i as u64,
chunk: cache.chunks[i].clone(),
embedding: cache.embeddings[i].clone(),
embedding: cache.embeddings[i].to_vec(),
tier: MemoryTier::Working,
importance: cache.activation_weights[i],
access_count: 0,
@@ -717,11 +717,13 @@ impl MemoryBackend for ClawhdfBackend {
let total_records = cache.count_active();
// A record saved without an embedding occupies a zero row, so "has an
// embedding" is "has a non-zero norm" rather than "row is non-empty".
let total_embeddings = cache
.embeddings
.norms
.iter()
.enumerate()
.filter(|(i, emb)| cache.tombstones[*i] == 0 && !emb.is_empty())
.filter(|(i, norm)| cache.tombstones[*i] == 0 && **norm > 0.0)
.count();
let file_size_bytes = std::fs::metadata(&self.hdf5_path)