3 Commits

Include renames

Author	SHA1	Message	Date
osobhandClaude Opus 5	c0a9206703	feat(agent): optional int8 vector index, re-scored against exact embeddings ... `MemoryConfig::quantized_index` stores the HNSW index's own copy of the embeddings as i8 rather than f32. At 100k x 384 that takes the index from 266 to 123 MiB and the whole reopened store from 399 to 256 MiB — 2.72x to 1.74x the raw vectors, the largest remaining item in the footprint. Quantised distances are approximate and `ef` cannot compensate, because the loss is in the distances rather than in the graph: recall@10 tops out at 0.967 against f32's 0.9995 and does not move between ef=128 and ef=256. The store already holds the exact embeddings, though, so when the index is quantised the query path re-scores the candidate pool against them before fusion. That restores recall (0.9940 vs 0.9945 at ef=64) and costs about 13% of QPS. Off by default: it trades query speed for memory and which side is worth more depends on the deployment. The flag is persisted in `/meta`, so a reopened store does not silently revert to four times the index memory, and the sidecar graph is rehydrated into the configured storage. Also on the CLI as `create --quantized-index`. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-09-19 20:40:37 -07:00
Omar Sobh	55959b4920	ci: wire up CI, fix no_std build, fix stale package names in scripts ... - Add .gitea/workflows/ci.yml running scripts/ci-test.sh (fmt, clippy, test, no_std check) on push/PR to main. - Fix stale rustyhdf5-py/rustyhdf5-format package names in ci-test.sh/check-nostd.sh, which had been silently no-op'ing those checks (cargo warns but doesn't fail on an unknown --exclude/-p target). - With those checks actually running, fix the real issues they surface: - clippy: useless_conversion in chunked_write.rs, byte_char_slices in global_heap.rs/object_header.rs. - cargo fmt: apply formatting across the workspace (whitespace only). - no_std (thumbv7em-none-eabihf) build errors in clawhdf5-format: core::sync::atomic::AtomicU64 doesn't exist on that target (no native 64-bit atomics) — switch profiling.rs's counters to portable-atomic, which falls back to a CAS-based emulation there and is a no-op wrapper elsewhere. Add missing alloc imports for Box (filters.rs), Vec (filters_szip.rs), and format! (dict_encoding.rs) on no_std paths. Replace f64::powi (std/libm-only) with a small local exponentiation-by-squaring helper in the scale-offset filter.	2026-08-05 10:50:13 -07:00
osobhandClaude Opus 4.8	8f9dbd812c	feat: integrate HNSW into agent search, fix Python 3.14 build ... Resolves two gaps found in a project-state review: 1. Python build was broken: PyO3/numpy 0.23 caps at Python 3.13 but the environment has 3.14. Bumped to 0.28 and updated the two breaking APIs (PyObject -> Py<PyAny>, allow_threads -> detach). The extension module now imports and round-trips under Python 3.14, unblocking cargo build --workspace. 2. The "HNSW vector search over agent memories" headline was unwired: clawhdf5-ann had zero dependents and the agent used a linear cosine+BM25 scan. - clawhdf5-ann is now a live index: insert, mark_deleted (soft delete with a deleted bitset, traversed but never returned), compact, and a format version tag (v2) with backward-compatible load of v1 files. - clawhdf5-agent wires HNSW behind the `hnsw` feature (ON by default). The index mirrors the cache (node id == cache index) and self-heals: it rebuilds whenever hnsw_synced_len drifts from cache.len(), so unhooked pushes can't desync it. Non-indexable stores (no/zero-dim/mixed embeddings) and queries whose dim doesn't match fall back to the exact linear scan. - hybrid.rs gains merge_vector_keyword, shared by the linear and HNSW paths. - tests/hnsw_integration.rs validates recall vs a brute-force oracle plus insert/delete/batch behaviour. Disable HNSW for exact search with `--no-default-features --features float16`. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>	2026-06-03 07:10:48 +00:00