osobhandClaude Opus 5.5
b55b24b7ba
Every crate under crates/ now has a README (android, bench, cli, napi and wasm had none), each saying what the crate is, its main types and functions (names checked against the code), its cargo features with defaults and which ones build C (checked with `cargo tree`), and links to the top-level docs. Corrections to the old stubs: - clawhdf5-derive: the derive is `H5Type`, not `HDF5Type`, and it needs clawhdf5-format as a dependency. - clawhdf5-filters: deflate backends only, and no library crate depends on it; the filter pipeline and every other codec are in -format. - clawhdf5-gpu: vector distance compute, not I/O; not used by HDF5Memory::search. - clawhdf5-io: MpiVol is root-read + broadcast, not collective MPI-IO. - clawhdf5-ann: from_hdf5/search(q, k) did not exist; load_from_hdf5 and search(q, k, ef). - clawhdf5-accel: checksum::crc32_simd did not exist; the SSE4 and wasm backends are reported but run the scalar kernels. - clawhdf5-gpu: the old example called l2_distances, which does not exist (l2_search). - clawhdf5-agent: it described a "vector store" with "GPU acceleration"; it now covers HDF5Memory, search options, WAL, signing, the graph. - crates.io/docs.rs badges removed and `cargo install <crate>` replaced: nothing is published; depend on git. - fuzz: the opt-in CLAWHDF5_FUZZ_SECONDS smoke run in ci-test.sh. - tools: the FileEditor interop tests that live in this crate. - remote, py: license, other front ends, limits, File.mode/flush/chunks. The Rust examples of the facade, format, filters, accel, ann, derive and agent READMEs were compiled and run as tests (netcdf4, gpu and remote compiled only) in a scratch crate; the CLI example was run. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
2.1 KiB
clawhdf5-gpu
GPU vector distance computation through wgpu and hand-written WGSL compute shaders: upload a set of vectors once, then run cosine or L2 top-k searches, dot products, distance matrices and norms against them on Vulkan, Metal, DirectX 12 or OpenGL.
This crate does not read or write HDF5: dataset I/O in clawhdf5 is
CPU-only. It is a vector-search accelerator used optionally by
clawhdf5-agent (its gpu feature exposes
gpu_search::GpuSearchBackend and a GPU arm of strategy::search_with_metrics;
HDF5Memory::search itself uses the HNSW index on the CPU).
Not on crates.io yet; depend on it from git:
[dependencies]
clawhdf5-gpu = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" }
Usage
use clawhdf5_gpu::GpuAccelerator;
// Fall back to a CPU path when there is no usable GPU.
let mut gpu = match GpuAccelerator::new() {
Ok(g) => g,
Err(_) => return,
};
let dim = 128;
let vectors = vec![0.5f32; 1000 * dim]; // 1000 vectors, row-major
gpu.upload_vectors(&vectors, dim).unwrap();
let norms = gpu.compute_norms_gpu(&vectors, dim).unwrap();
gpu.upload_norms(&norms).unwrap();
let query = vec![1.0f32; dim];
let top10 = gpu.cosine_search(&query, 10).unwrap(); // (index, similarity), best first
let near10 = gpu.l2_search(&query, 10).unwrap(); // (index, distance), nearest first
GpuAccelerator also has is_available, device_info,
batch_cosine_search, batch_dot_product, distance_matrix,
compute_norms, and f16_to_f32_batch/f32_to_f16_batch. Vector sets
larger than the device's largest storage buffer binding are split into
chunks and the results merged. A GPU→CPU readback waits at most 30 s and
then fails with GpuError::BufferMap instead of hanging.
Features
| Feature | Default | What |
|---|---|---|
gpu-wgpu |
yes | the wgpu implementation. Without it GpuAccelerator::new() returns GpuError::NotCompiled and is_available() is false. |
No C is compiled, but wgpu talks to the system's graphics drivers at run time; the crate is exempt from CI's "no C in the default build" check for that reason.
License
MIT