feat: integrate HNSW into agent search, fix Python 3.14 build
Resolves two gaps found in a project-state review:
1. Python build was broken: PyO3/numpy 0.23 caps at Python 3.13 but the
environment has 3.14. Bumped to 0.28 and updated the two breaking APIs
(PyObject -> Py<PyAny>, allow_threads -> detach). The extension module now
imports and round-trips under Python 3.14, unblocking cargo build --workspace.
2. The "HNSW vector search over agent memories" headline was unwired:
clawhdf5-ann had zero dependents and the agent used a linear cosine+BM25 scan.
- clawhdf5-ann is now a live index: insert, mark_deleted (soft delete with a
deleted bitset, traversed but never returned), compact, and a format
version tag (v2) with backward-compatible load of v1 files.
- clawhdf5-agent wires HNSW behind the `hnsw` feature (ON by default). The
index mirrors the cache (node id == cache index) and self-heals: it rebuilds
whenever hnsw_synced_len drifts from cache.len(), so unhooked pushes can't
desync it. Non-indexable stores (no/zero-dim/mixed embeddings) and queries
whose dim doesn't match fall back to the exact linear scan.
- hybrid.rs gains merge_vector_keyword, shared by the linear and HNSW paths.
- tests/hnsw_integration.rs validates recall vs a brute-force oracle plus
insert/delete/batch behaviour.
Disable HNSW for exact search with `--no-default-features --features float16`.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
This commit is contained in:
@@ -60,6 +60,22 @@ pub fn hybrid_search(
|
||||
};
|
||||
let kw_scores = bm25_index.search(query_text, vectors.len());
|
||||
|
||||
merge_vector_keyword(vec_scores, kw_scores, vector_weight, keyword_weight, k)
|
||||
}
|
||||
|
||||
/// Merge pre-computed vector-similarity and keyword scores into a single ranking.
|
||||
///
|
||||
/// Both score sets are independently min-max normalized to [0, 1] and combined
|
||||
/// with the given weights. This is the shared core of [`hybrid_search`]; it is
|
||||
/// also used by the optional HNSW path, which supplies vector scores from an
|
||||
/// approximate-nearest-neighbour index instead of a full linear scan.
|
||||
pub fn merge_vector_keyword(
|
||||
vec_scores: Vec<(usize, f32)>,
|
||||
kw_scores: Vec<(usize, f32)>,
|
||||
vector_weight: f32,
|
||||
keyword_weight: f32,
|
||||
k: usize,
|
||||
) -> Vec<(usize, f32)> {
|
||||
// Normalize each set to [0, 1].
|
||||
let vec_normalized = normalize_scores(&vec_scores);
|
||||
let kw_normalized = normalize_scores(&kw_scores);
|
||||
|
||||
Reference in New Issue
Block a user