# clawhdf5-ann

An HNSW (Hierarchical Navigable Small World) approximate nearest-neighbour
index in pure Rust, with cosine or L2 distance, optional int8 storage of
the vectors, deletions, and persistence as an HDF5 file. It is the vector
stage of [`clawhdf5-agent`](../clawhdf5-agent/README.md)'s search (the
agent's `hnsw` feature, on by default); distances run on
[`clawhdf5-accel`](../clawhdf5-accel/README.md)'s SIMD kernels.

Neighbours are chosen with the HNSW paper's diversity heuristic, not plain
closest-M (which capped recall on clustered data at 0.31 recall@10 at 100K
vectors).

Not on crates.io yet; depend on it from git:

```toml
[dependencies]
clawhdf5-ann = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" }
```

## Usage

```rust
use clawhdf5_ann::{DistanceMetric, HnswIndex, Storage};

let vectors: Vec<Vec<f32>> = (0..500)
    .map(|i| (0..16).map(|j| ((i * 31 + j * 7) % 97) as f32 / 97.0).collect())
    .collect();

// m = 16 connections per node, ef_construction = 200
let mut index = HnswIndex::build_with(&vectors, 16, 200, DistanceMetric::Cosine, Storage::Int8);
let hits = index.search(&vectors[42], 10, 64); // (id, distance), closest first; ef >= k
assert!(hits[0].1 < 1e-3); // vector 42 itself (or an identical one)

let id = index.insert(vec![0.5; 16]);
index.mark_deleted(id);

// Persist as HDF5 (a self-contained file: graph and vectors) and load it back
let bytes = index.to_hdf5_bytes().unwrap();
let loaded = HnswIndex::load_from_hdf5(&bytes).unwrap();
assert_eq!(loaded.len(), index.len());
```

- `HnswIndex::build` (L2), `build_with_metric`, `build_with` (metric and
  storage); `new`/`new_with` plus `insert` for an index built
  incrementally.
- `Storage::Int8` keeps each vector as `i8`, a quarter of the memory; it
  applies to `Cosine` only (an L2 index keeps `Float32`). Distances are then
  approximate, so a caller that needs exact ranking re-scores the
  candidates, as the agent does.
- `mark_deleted`, `is_deleted`, `deleted_count`, `active_len`, `compact`
  (returns the old-to-new id map).
- `save_to_hdf5(&mut writer)` / `to_hdf5_bytes` / `load_from_hdf5` store
  the whole index; `graph_to_bytes` / `from_graph_bytes` store only the
  graph (with a CRC32) for a caller that keeps the vectors elsewhere — the
  agent's `<store>.h5.ann` sidecar.

## Features

| Feature | Default | What | Builds C |
|---|---|---|---|
| `parallel` | no | build the graph on a rayon pool; the graph is identical with or without it | no |

Recall and speed against exact search, for the index alone and in the
agent: [`BENCHMARKS.md`](../../BENCHMARKS.md), measured with
`cargo run --release -p clawhdf5-bench --bin search_harness`.

## License

MIT