perf(ann): batched bulk build, parallel with the parallel feature

Profiling the build showed 90% of all distance evaluations are in back-link pruning (40.8M of 44.9M at 10K): every overflow re-runs the diversity heuristic pairwise over ~max_conn candidates. The bulk build now inserts in batches: plan every node's neighbours against the graph as it stood when the batch began (read-only, so plans are independent), link, then prune each overflowing list once. A node gaining several back-links in a batch is pruned once rather than once per link, so this is faster even single-threaded (10K: 1676 -> 1074 ms). With `parallel`, planning and pruning use rayon (10K: 388 ms; 100K: ~21 s -> 5.9 s on 16 cores). Batches start at one node and are capped at 1/16 of the linked graph and 512 nodes; a node that raises the top layer gets a batch to itself. The result is deterministic and identical with or without the feature (one code path; test compares two builds byte for byte). Parallelising within a single insert was tried first: 1.45x on 16 cores, tasks too small. Incremental insert() stays sequential. Recall on clustered data is unchanged or slightly better; uniform random data dips slightly (10K, ef=64: 0.474 -> 0.444). clawhdf5-agent's `parallel` feature now passes through to the index. Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-09-19 13:43:49 -07:00
parent 41db450c92
commit c19199f3eb
4 changed files with 225 additions and 65 deletions
@@ -47,7 +47,9 @@ harness = false
 [features]
 default = ["float16", "hnsw"]
 float16 = ["half"]
-parallel = ["rayon"]
+# Rayon-parallel brute-force search strategies, and a parallel bulk build of
+# the HNSW index (same graph, several times faster on a multi-core machine).
+parallel = ["rayon", "clawhdf5-ann?/parallel"]
 # Compress embeddings with Zstd instead of deflate when
 # `MemoryConfig::compression` is on. Off by default: it links libzstd (C).
 zstd = ["clawhdf5/zstd"]