quantumclaw/clawhdf5
3
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity
82 Commits 11 Branches 2 Tags
4b17bf9101d58fb8cbeb0778f0f3f87986552225
Commit Graph
5 Commits
Author SHA1 Message Date
Omar Sobh a3e1cf8588 perf: Tier 4c — optional rayon parallelism for HNSW prune_connections
CI / test (push) Failing after 3s
Details 
Add a default-off `parallel` feature to clawhdf5-ann (rayon optional dep),
matching the convention already used in clawhdf5-format/clawhdf5-agent.
Gate prune_connections' per-neighbor distance computation on it — a pure
read-only map with no shared mutable state, sorted immediately after, so
swapping to rayon's par_iter is low-risk.

Deliberately not touching build_with_metric's outer insert loop per the
original plan: it has genuine cross-iteration data dependencies (graph
mutation, entry-point updates) and needs its own correctness-focused
design pass. The win here is likely small since neighbor lists are
bounded by m/m_max0 (typically small) — this is a low-risk completeness
item, not a headline perf change.

Verified identical results with default features and --features parallel
across the full HNSW test suite (23/23 both ways), including the
build+search end-to-end tests (build_small_index, search_accuracy_cosine,
incremental_insert_matches_batch_recall).
 2026-08-05 13:15:19 -07:00
Omar Sobh 55959b4920 ci: wire up CI, fix no_std build, fix stale package names in scripts
CI / test (push) Failing after 15s
Details 
- Add .gitea/workflows/ci.yml running scripts/ci-test.sh (fmt, clippy,
  test, no_std check) on push/PR to main.
- Fix stale rustyhdf5-py/rustyhdf5-format package names in
  ci-test.sh/check-nostd.sh, which had been silently no-op'ing those
  checks (cargo warns but doesn't fail on an unknown --exclude/-p
  target).
- With those checks actually running, fix the real issues they surface:
  - clippy: useless_conversion in chunked_write.rs, byte_char_slices in
    global_heap.rs/object_header.rs.
  - cargo fmt: apply formatting across the workspace (whitespace only).
  - no_std (thumbv7em-none-eabihf) build errors in clawhdf5-format:
    core::sync::atomic::AtomicU64 doesn't exist on that target (no
    native 64-bit atomics) — switch profiling.rs's counters to
    portable-atomic, which falls back to a CAS-based emulation there
    and is a no-op wrapper elsewhere. Add missing alloc imports for
    Box (filters.rs), Vec (filters_szip.rs), and format! (dict_encoding.rs)
    on no_std paths. Replace f64::powi (std/libm-only) with a small
    local exponentiation-by-squaring helper in the scale-offset filter.
 2026-08-05 10:50:13 -07:00
Omar SobhandClaude Sonnet 4.6 2ddb22897c perf: eliminate double compression and improve shuffle filter throughput 
Four independent write-path improvements:

1. Cache compressed chunks between Pass 1 and Pass 2 (chunked_write.rs,
   file_writer.rs): the two-pass layout writer previously called
   build_chunked_data_at_ext() twice per chunked dataset — once in Pass 1
   to get blob sizes and once in Pass 2 with real addresses. Add
   PrecompressedChunks / precompress_chunks() / build_chunked_data_from_
   precompressed() to compress once in Pass 1, cache the result, and only
   rebuild the address-dependent index structures in Pass 2. Expected
   ~2× speedup for chunked+deflate writes (512×512 deflate: 3.33ms → ~1.7ms).

2. SIMD-vectorisable shuffle filter (filters.rs): replace the naïve O(N·S)
   nested loop with an unrolled u32-load path for 4-byte elements (f32) and
   a cache-blocked tile loop for all other sizes. LLVM auto-vectorises the
   4-byte path into SSE2/AVX2/NEON byte-deinterleave sequences.

3. Zstd benchmark variant (h5bench_write.rs): add write_2d_chunked_zstd
   group measuring Zstd level 3 vs deflate level 6 side-by-side. Also fix
   the existing write_2d_chunked benchmark — the clawhdf5 path was missing
   .with_deflate(6), making the comparison apples-to-oranges. Add arXiv-
   backed doc recommendation on DatasetBuilder::with_zstd().

4. Zero-copy HNSW save (hnsw.rs, clawhdf5-io/lib.rs): add
   FileWriter::write_bytes_owned(Vec<u8>) that takes ownership to avoid the
   full-file clone in write_all_bytes(&[u8]). HNSW::save_to_hdf5 uses it.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
 2026-06-30 22:36:40 +00:00
osobhandClaude Opus 4.8 8f9dbd812c feat: integrate HNSW into agent search, fix Python 3.14 build 
Resolves two gaps found in a project-state review:

1. Python build was broken: PyO3/numpy 0.23 caps at Python 3.13 but the
   environment has 3.14. Bumped to 0.28 and updated the two breaking APIs
   (PyObject -> Py<PyAny>, allow_threads -> detach). The extension module now
   imports and round-trips under Python 3.14, unblocking cargo build --workspace.

2. The "HNSW vector search over agent memories" headline was unwired:
   clawhdf5-ann had zero dependents and the agent used a linear cosine+BM25 scan.
   - clawhdf5-ann is now a live index: insert, mark_deleted (soft delete with a
     deleted bitset, traversed but never returned), compact, and a format
     version tag (v2) with backward-compatible load of v1 files.
   - clawhdf5-agent wires HNSW behind the `hnsw` feature (ON by default). The
     index mirrors the cache (node id == cache index) and self-heals: it rebuilds
     whenever hnsw_synced_len drifts from cache.len(), so unhooked pushes can't
     desync it. Non-indexable stores (no/zero-dim/mixed embeddings) and queries
     whose dim doesn't match fall back to the exact linear scan.
   - hybrid.rs gains merge_vector_keyword, shared by the linear and HNSW paths.
   - tests/hnsw_integration.rs validates recall vs a brute-force oracle plus
     insert/delete/batch behaviour.

Disable HNSW for exact search with `--no-default-features --features float16`.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
 2026-06-03 07:10:48 +00:00
redclawsystems 3f222f6956 Merge pull request 'docs(clawhdf5): document DType variants, fix unresolved doc links' (#17) from sdlc-docs/clawhdf5-types-20260514-165210 into main 2026-05-14 23:54:48 +00:00