Files

T

osobhandClaude Opus 5.5 b55b24b7ba docs: crate READMEs describe each crate as it is today

Every crate under crates/ now has a README (android, bench, cli, napi and
wasm had none), each saying what the crate is, its main types and
functions (names checked against the code), its cargo features with
defaults and which ones build C (checked with `cargo tree`), and links to
the top-level docs.

Corrections to the old stubs:
- clawhdf5-derive: the derive is `H5Type`, not `HDF5Type`, and it needs
  clawhdf5-format as a dependency.
- clawhdf5-filters: deflate backends only, and no library crate depends
  on it; the filter pipeline and every other codec are in -format.
- clawhdf5-gpu: vector distance compute, not I/O; not used by
  HDF5Memory::search.
- clawhdf5-io: MpiVol is root-read + broadcast, not collective MPI-IO.
- clawhdf5-ann: from_hdf5/search(q, k) did not exist; load_from_hdf5 and
  search(q, k, ef).
- clawhdf5-accel: checksum::crc32_simd did not exist; the SSE4 and wasm
  backends are reported but run the scalar kernels.
- clawhdf5-gpu: the old example called l2_distances, which does not
  exist (l2_search).
- clawhdf5-agent: it described a "vector store" with "GPU acceleration";
  it now covers HDF5Memory, search options, WAL, signing, the graph.
- crates.io/docs.rs badges removed and `cargo install <crate>` replaced:
  nothing is published; depend on git.
- fuzz: the opt-in CLAWHDF5_FUZZ_SECONDS smoke run in ci-test.sh.
- tools: the FileEditor interop tests that live in this crate.
- remote, py: license, other front ends, limits, File.mode/flush/chunks.

The Rust examples of the facade, format, filters, accel, ann, derive and
agent READMEs were compiled and run as tests (netcdf4, gpu and remote
compiled only) in a scratch crate; the CLI example was run.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

2026-09-28 11:13:30 -05:00

src

fix(format): apply the base address of files with a user block

2026-09-25 22:07:50 -05:00

Cargo.toml

build: declare Rust 1.92 as the MSRV and check it in CI

2026-09-23 11:05:26 -05:00

README.md

docs: crate READMEs describe each crate as it is today

2026-09-28 11:13:30 -05:00

README.md

clawhdf5-ann

An HNSW (Hierarchical Navigable Small World) approximate nearest-neighbour index in pure Rust, with cosine or L2 distance, optional int8 storage of the vectors, deletions, and persistence as an HDF5 file. It is the vector stage of clawhdf5-agent's search (the agent's hnsw feature, on by default); distances run on clawhdf5-accel's SIMD kernels.

Neighbours are chosen with the HNSW paper's diversity heuristic, not plain closest-M (which capped recall on clustered data at 0.31 recall@10 at 100K vectors).

Not on crates.io yet; depend on it from git:

[dependencies]
clawhdf5-ann = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" }

Usage

use clawhdf5_ann::{DistanceMetric, HnswIndex, Storage};

let vectors: Vec<Vec<f32>> = (0..500)
    .map(|i| (0..16).map(|j| ((i * 31 + j * 7) % 97) as f32 / 97.0).collect())
    .collect();

// m = 16 connections per node, ef_construction = 200
let mut index = HnswIndex::build_with(&vectors, 16, 200, DistanceMetric::Cosine, Storage::Int8);
let hits = index.search(&vectors[42], 10, 64); // (id, distance), closest first; ef >= k
assert!(hits[0].1 < 1e-3); // vector 42 itself (or an identical one)

let id = index.insert(vec![0.5; 16]);
index.mark_deleted(id);

// Persist as HDF5 (a self-contained file: graph and vectors) and load it back
let bytes = index.to_hdf5_bytes().unwrap();
let loaded = HnswIndex::load_from_hdf5(&bytes).unwrap();
assert_eq!(loaded.len(), index.len());

HnswIndex::build (L2), build_with_metric, build_with (metric and storage); new/new_with plus insert for an index built incrementally.
Storage::Int8 keeps each vector as i8, a quarter of the memory; it applies to Cosine only (an L2 index keeps Float32). Distances are then approximate, so a caller that needs exact ranking re-scores the candidates, as the agent does.
mark_deleted, is_deleted, deleted_count, active_len, compact (returns the old-to-new id map).
save_to_hdf5(&mut writer) / to_hdf5_bytes / load_from_hdf5 store the whole index; graph_to_bytes / from_graph_bytes store only the graph (with a CRC32) for a caller that keeps the vectors elsewhere — the agent's <store>.h5.ann sidecar.

Features

Feature	Default	What	Builds C
`parallel`	no	build the graph on a rayon pool; the graph is identical with or without it	no

Recall and speed against exact search, for the index alone and in the agent: BENCHMARKS.md, measured with cargo run --release -p clawhdf5-bench --bin search_harness.

License

MIT