search() collected ef candidates, then filtered out soft-deleted nodes, then
took k. When the records nearest a query had been deleted, every candidate was
a tombstone and the search returned fewer than k results — 39 of 40 queries in
the new test, which deletes each query's 40 nearest neighbours.
search_layer takes an optional skip mask: a skipped node is still pushed onto
the candidate queue (a tombstone is a valid waypoint) but never into the
result heap, so the ef result slots hold live nodes only. Build and insert
pass no mask. Recall and speed without deletions are unchanged.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Bump all workspace crates, the node package and pyproject to 2.4.0, finalize
the changelog and add upgrade notes for the search behaviour changes.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
- Cosine distance was 1 - dot/(|a||b|), re-deriving both norms on every
evaluation in the innermost loop of build and search. The index now stores
unit vectors (prepared at build, insert, graph load and HDF5 load; the query
once per search) and uses 1 - dot. Zero vectors stay zero, giving distance 1
as before. Returned distances are unchanged.
- search_layer allocated a HashSet of visited nodes per call. It is now an
epoch-stamped u32 array in thread-local scratch, reused across calls, so
search(&self) stays shareable between threads.
- clawhdf5-accel caches the detected SIMD backend in a OnceLock.
Recall is identical. Build 2.75 -> 1.89 s (10K), ~38 -> 21 s (100K); QPS at
ef=64 22.7K -> 39K (10K), 10.4K -> 14K (100K).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
open() marked the HNSW index dirty, so the first search of every session
rebuilt it from scratch — 36 s at 100K records with the (better, slower)
heuristic build. First query after open is now 1.7 / 15 / 159 ms at
1K / 10K / 100K; what remains is the one-off keyword index build.
- clawhdf5-ann: HnswIndex::graph_to_bytes / from_graph_bytes serialize the
graph only (levels, tombstones, adjacency as u32, CRC32). The existing HDF5
serializer embeds a full copy of every vector, which would double a store
that already holds them. Loading validates everything — counts, levels vs
layer count, connection limits, every neighbour id and the layer it must
exist on — so a damaged graph, or a hostile one with a valid checksum, is an
error rather than an out-of-bounds walk during search.
- clawhdf5-agent: each checkpoint writes the graph to <store>.h5.ann (synced,
atomic, before the .h5) and records a fresh generation id in /meta. open()
loads the sidecar only if its generation matches that checkpoint; missing,
stale, damaged or mismatched sidecars are ignored and the index rebuilt.
Records appended through WAL replay join the loaded index incrementally; a
replayed Update or Tombstone invalidates it. snapshot() copies it. Only an
index that exactly mirrors the cache is saved; otherwise a stale sidecar is
removed.
- ensure_hnsw_fresh inserts records appended since the last sync instead of
rebuilding, so save_batch no longer marks the whole index dirty.
- CheckpointMeta { wal_applied, ann_generation } with *_with_meta build/write/
read functions; the *_with_mark ones delegate.
- Harness reports the one-off cold index build separately from the first query
after a reopen.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Neighbours were selected as the plain closest-M, for both a new node's links
and back-link pruning. On clustered data every link of a node inside a tight
cluster then goes to that same cluster, so clusters become islands a search
entering elsewhere can never reach: recall@10 was 0.87 / 0.67 / 0.31 at
1K / 10K / 100K (384-dim) and flat in ef. Uniform random data — all the
existing tests used — does not show it.
Implement the HNSW paper's Algorithm 4 with keepPrunedConnections: accept a
candidate only if it is closer to the node than to every neighbour already
accepted, then fill spare slots with the closest rejected ones. Recall@10 at
ef=64 is now 1.00 / 1.00 / 0.98 and rises with ef; uniform data improves
slightly. Build is ~3.5x slower at 10K (extra distance evaluations), to be
recovered by the distance-kernel work. The needless rayon fan-out over <=33
distances in prune_connections is gone.
Tests: a clustered-data recall test for bulk build and incremental insert
(scores 0.43 with the old selection), and a unit test of the selection rule.
Harness gains --uniform and --ann-only; before/after in BENCHMARKS.md.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Bump all workspace crates, the node package and pyproject to 2.3.0, finalize
the changelog and add upgrade notes for the behaviour changes.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Bump all workspace crates, the node package and pyproject to 2.2.0 and
finalize the changelog. The repository URL in every manifest pointed at a
GitHub location that does not resolve; use the real origin.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The SIMD migration weakened the near-zero-norm guard in all four
clawhdf5-accel cosine_similarity backends (scalar/avx2/avx512/neon)
from `denom < f32::EPSILON` to `denom == 0.0`. Vectors with a tiny
but nonzero norm (denom in (0, 1.19e-7)) fell through to dot/denom
and scored as identical instead of maximally dissimilar, diverging
from the pre-SIMD scalar loop's documented fallback behavior.
Restores the epsilon threshold in all four backends so
`1.0 - cosine_similarity(...)` in hnsw.rs::compute_distance
reproduces the old fallback exactly. Adds regression tests in
clawhdf5-accel and clawhdf5-ann locking in the near-zero-norm case.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Mission: 01a00c41-bac0-7eb3-a8c8-8b7044f3086d
Phase: 01a00c41-bac2-71e3-a58b-c473421200ee
Committed by the ClawMates delivery pipeline from the agents' working tree. Authored by agents, not by the named committer.
Add a default-off `parallel` feature to clawhdf5-ann (rayon optional dep),
matching the convention already used in clawhdf5-format/clawhdf5-agent.
Gate prune_connections' per-neighbor distance computation on it — a pure
read-only map with no shared mutable state, sorted immediately after, so
swapping to rayon's par_iter is low-risk.
Deliberately not touching build_with_metric's outer insert loop per the
original plan: it has genuine cross-iteration data dependencies (graph
mutation, entry-point updates) and needs its own correctness-focused
design pass. The win here is likely small since neighbor lists are
bounded by m/m_max0 (typically small) — this is a low-risk completeness
item, not a headline perf change.
Verified identical results with default features and --features parallel
across the full HNSW test suite (23/23 both ways), including the
build+search end-to-end tests (build_small_index, search_accuracy_cosine,
incremental_insert_matches_batch_recall).
- Add .gitea/workflows/ci.yml running scripts/ci-test.sh (fmt, clippy,
test, no_std check) on push/PR to main.
- Fix stale rustyhdf5-py/rustyhdf5-format package names in
ci-test.sh/check-nostd.sh, which had been silently no-op'ing those
checks (cargo warns but doesn't fail on an unknown --exclude/-p
target).
- With those checks actually running, fix the real issues they surface:
- clippy: useless_conversion in chunked_write.rs, byte_char_slices in
global_heap.rs/object_header.rs.
- cargo fmt: apply formatting across the workspace (whitespace only).
- no_std (thumbv7em-none-eabihf) build errors in clawhdf5-format:
core::sync::atomic::AtomicU64 doesn't exist on that target (no
native 64-bit atomics) — switch profiling.rs's counters to
portable-atomic, which falls back to a CAS-based emulation there
and is a no-op wrapper elsewhere. Add missing alloc imports for
Box (filters.rs), Vec (filters_szip.rs), and format! (dict_encoding.rs)
on no_std paths. Replace f64::powi (std/libm-only) with a small
local exponentiation-by-squaring helper in the scale-offset filter.
Four independent write-path improvements:
1. Cache compressed chunks between Pass 1 and Pass 2 (chunked_write.rs,
file_writer.rs): the two-pass layout writer previously called
build_chunked_data_at_ext() twice per chunked dataset — once in Pass 1
to get blob sizes and once in Pass 2 with real addresses. Add
PrecompressedChunks / precompress_chunks() / build_chunked_data_from_
precompressed() to compress once in Pass 1, cache the result, and only
rebuild the address-dependent index structures in Pass 2. Expected
~2× speedup for chunked+deflate writes (512×512 deflate: 3.33ms → ~1.7ms).
2. SIMD-vectorisable shuffle filter (filters.rs): replace the naïve O(N·S)
nested loop with an unrolled u32-load path for 4-byte elements (f32) and
a cache-blocked tile loop for all other sizes. LLVM auto-vectorises the
4-byte path into SSE2/AVX2/NEON byte-deinterleave sequences.
3. Zstd benchmark variant (h5bench_write.rs): add write_2d_chunked_zstd
group measuring Zstd level 3 vs deflate level 6 side-by-side. Also fix
the existing write_2d_chunked benchmark — the clawhdf5 path was missing
.with_deflate(6), making the comparison apples-to-oranges. Add arXiv-
backed doc recommendation on DatasetBuilder::with_zstd().
4. Zero-copy HNSW save (hnsw.rs, clawhdf5-io/lib.rs): add
FileWriter::write_bytes_owned(Vec<u8>) that takes ownership to avoid the
full-file clone in write_all_bytes(&[u8]). HNSW::save_to_hdf5 uses it.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Sweep of the docs after the v2.0.0 rename and recent changes:
- Per-crate READMEs (13 files): rename leftover rustyhdf5-*/edgehdf5-*
package names and badges to clawhdf5-*, bump usage versions to 2.1.0.
- README: update stale test badge (417 -> 1500+), workspace stats
(15 crates/72K -> 17 crates/84K), agent crate stats (20.7K/32 modules),
and add the missing clawhdf5-napi and clawhdf5-bench crates to the tree.
- CLAUDE.md: correct the CLI subcommand list (inspect/dump/index/search ->
the actual create/save/search/recall/stats/flush-wal/agents-md/export/snapshot).
No code changes. Verified there are zero todo!()/unimplemented!() macros and
no TODO/FIXME comments in the tree.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Bumps [workspace.package] and all 17 crate package versions (and internal
path-dependency requirements) from 2.0.0 to 2.1.0 for a coordinated release.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Resolves two gaps found in a project-state review:
1. Python build was broken: PyO3/numpy 0.23 caps at Python 3.13 but the
environment has 3.14. Bumped to 0.28 and updated the two breaking APIs
(PyObject -> Py<PyAny>, allow_threads -> detach). The extension module now
imports and round-trips under Python 3.14, unblocking cargo build --workspace.
2. The "HNSW vector search over agent memories" headline was unwired:
clawhdf5-ann had zero dependents and the agent used a linear cosine+BM25 scan.
- clawhdf5-ann is now a live index: insert, mark_deleted (soft delete with a
deleted bitset, traversed but never returned), compact, and a format
version tag (v2) with backward-compatible load of v1 files.
- clawhdf5-agent wires HNSW behind the `hnsw` feature (ON by default). The
index mirrors the cache (node id == cache index) and self-heals: it rebuilds
whenever hnsw_synced_len drifts from cache.len(), so unhooked pushes can't
desync it. Non-indexable stores (no/zero-dim/mixed embeddings) and queries
whose dim doesn't match fall back to the exact linear scan.
- hybrid.rs gains merge_vector_keyword, shared by the linear and HNSW paths.
- tests/hnsw_integration.rs validates recall vs a brute-force oracle plus
insert/delete/batch behaviour.
Disable HNSW for exact search with `--no-default-features --features float16`.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
