fix(ann): HNSW neighbour-selection heuristic — recall 0.31 -> 0.98 at 100K

Neighbours were selected as the plain closest-M, for both a new node's links
and back-link pruning. On clustered data every link of a node inside a tight
cluster then goes to that same cluster, so clusters become islands a search
entering elsewhere can never reach: recall@10 was 0.87 / 0.67 / 0.31 at
1K / 10K / 100K (384-dim) and flat in ef. Uniform random data — all the
existing tests used — does not show it.

Implement the HNSW paper's Algorithm 4 with keepPrunedConnections: accept a
candidate only if it is closer to the node than to every neighbour already
accepted, then fill spare slots with the closest rejected ones. Recall@10 at
ef=64 is now 1.00 / 1.00 / 0.98 and rises with ef; uniform data improves
slightly. Build is ~3.5x slower at 10K (extra distance evaluations), to be
recovered by the distance-kernel work. The needless rayon fan-out over <=33
distances in prune_connections is gone.

Tests: a clustered-data recall test for bulk build and incremental insert
(scores 0.43 with the old selection), and a unit test of the selection rule.
Harness gains --uniform and --ann-only; before/after in BENCHMARKS.md.

Co-Authored-By: Claude Fable 5.1 <[email protected]>

crates/clawhdf5-bench/src/bin/search_harness.rs

@@ -18,6 +18,7 @@
 //! cargo run --release -p clawhdf5-bench --bin search_harness            # 1K, 10K
 //! cargo run --release -p clawhdf5-bench --bin search_harness -- --full  # + 100K
 //! cargo run --release -p clawhdf5-bench --bin search_harness -- --json out.json
+//! cargo run --release -p clawhdf5-bench --bin search_harness -- --ann-only --uniform
 //! ```
 
 use std::time::{Duration, Instant};
@@ -78,8 +79,26 @@ struct Dataset {
     query_cluster: Vec<usize>,
 }
 
+/// `--uniform`: isotropic random unit vectors instead of clusters. Not a
+/// realistic workload, but a useful second distribution — a recall problem
+/// that appears only on clustered data points at graph connectivity.
+static UNIFORM: std::sync::atomic::AtomicBool = std::sync::atomic::AtomicBool::new(false);
+
 fn make_dataset(n: usize, seed: u64) -> Dataset {
     let mut rng = Rng(seed);
+    if UNIFORM.load(std::sync::atomic::Ordering::Relaxed) {
+        let random_unit = |rng: &mut Rng| {
+            let mut v: Vec<f32> = (0..DIM).map(|_| rng.gauss()).collect();
+            normalize(&mut v);
+            v
+        };
+        return Dataset {
+            vectors: (0..n).map(|_| random_unit(&mut rng)).collect(),
+            queries: (0..N_QUERIES).map(|_| random_unit(&mut rng)).collect(),
+            cluster_of: vec![0; n],
+            query_cluster: vec![0; N_QUERIES],
+        };
+    }
     let n_clusters = (n / 100).clamp(8, 512);
     let centres: Vec<Vec<f32>> = (0..n_clusters)
         .map(|_| {
@@ -344,6 +363,11 @@ fn bench_end_to_end(n: usize, json: &mut Vec<serde_json::Value>) {
 fn main() {
     let args: Vec<String> = std::env::args().skip(1).collect();
     let full = args.iter().any(|a| a == "--full");
+    let ann_only = args.iter().any(|a| a == "--ann-only");
+    if args.iter().any(|a| a == "--uniform") {
+        UNIFORM.store(true, std::sync::atomic::Ordering::Relaxed);
+        println!("(uniform random data)");
+    }
     let json_path = args
         .iter()
         .position(|a| a == "--json")
@@ -365,6 +389,9 @@ fn main() {
         bench_ann(n, &mut json);
     }
 
+    if ann_only {
+        return;
+    }
     println!("\n### End to end: `HDF5Memory::hybrid_search` (k = {K}, weights 0.7 / 0.3)\n");
     println!(
         "| N | ingest ms | checkpoint ms | open ms | first query ms | p50 ms | p99 ms | QPS |"