feat: integrate HNSW into agent search, fix Python 3.14 build

Resolves two gaps found in a project-state review:

1. Python build was broken: PyO3/numpy 0.23 caps at Python 3.13 but the
   environment has 3.14. Bumped to 0.28 and updated the two breaking APIs
   (PyObject -> Py<PyAny>, allow_threads -> detach). The extension module now
   imports and round-trips under Python 3.14, unblocking cargo build --workspace.

2. The "HNSW vector search over agent memories" headline was unwired:
   clawhdf5-ann had zero dependents and the agent used a linear cosine+BM25 scan.
   - clawhdf5-ann is now a live index: insert, mark_deleted (soft delete with a
     deleted bitset, traversed but never returned), compact, and a format
     version tag (v2) with backward-compatible load of v1 files.
   - clawhdf5-agent wires HNSW behind the `hnsw` feature (ON by default). The
     index mirrors the cache (node id == cache index) and self-heals: it rebuilds
     whenever hnsw_synced_len drifts from cache.len(), so unhooked pushes can't
     desync it. Non-indexable stores (no/zero-dim/mixed embeddings) and queries
     whose dim doesn't match fall back to the exact linear scan.
   - hybrid.rs gains merge_vector_keyword, shared by the linear and HNSW paths.
   - tests/hnsw_integration.rs validates recall vs a brute-force oracle plus
     insert/delete/batch behaviour.

Disable HNSW for exact search with `--no-default-features --features float16`.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-06-03 07:10:48 +00:00
co-authored by Claude Opus 4.8
parent dd3e1d0b11
commit 4137846e31
12 changed files with 781 additions and 33 deletions
+8 -1
View File
@@ -14,6 +14,7 @@ clawhdf5-format = { path = "../clawhdf5-format", version = "2.0.0", features = [
clawhdf5 = { path = "../clawhdf5", version = "2.0.0" }
clawhdf5-io = { path = "../clawhdf5-io", version = "2.0.0", features = ["mmap"] }
clawhdf5-accel = { path = "../clawhdf5-accel", version = "2.0.0" }
clawhdf5-ann = { path = "../clawhdf5-ann", version = "2.0.0", optional = true }
clawhdf5-gpu = { path = "../clawhdf5-gpu", version = "2.0.0", optional = true, default-features = false }
serde = { version = "1", features = ["derive"] }
byteorder = "1"
@@ -44,9 +45,15 @@ name = "memory_bench"
harness = false
[features]
default = ["float16"]
default = ["float16", "hnsw"]
float16 = ["half"]
parallel = ["rayon"]
# HNSW approximate-nearest-neighbour acceleration for the vector stage of
# hybrid_search. On by default; the index is rebuilt from the cache on demand
# and stays self-consistent with the persisted memory store. Disable with
# `--no-default-features` (plus re-enabling other defaults) to force the exact
# linear cosine scan.
hnsw = ["clawhdf5-ann"]
agent = []
gpu = ["clawhdf5-gpu/gpu-wgpu"]
fast-math = ["matrixmultiply"]