quantumclaw/clawhdf5
3
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity
83 Commits 11 Branches 2 Tags
150afe6f5b56a63de91bb994e908213126338515
Commit Graph
6 Commits
Author SHA1 Message Date
Omar Sobh 534331ffbe chore: Tier 4d — hoist tempfile/criterion/half/serde to workspace.dependencies
CI / test (push) Failing after 13s
Details 
Add [workspace.dependencies] to the root Cargo.toml for the four
duplicated-across-many-crates dependencies flagged by the earlier review:
tempfile (7 crates), criterion (6), half (4 — real version skew, clawhdf5-gpu
pinned 2.7 while others used bare 2), and serde (4). Update every consuming
crate to `dep = { workspace = true }`, preserving crate-local `optional =
true` where it already existed. half now resolves uniformly to 2.7.x
workspace-wide instead of two separate semver ranges.

Also fixed clawhdf5-filters/Cargo.toml's stale "rustyhdf5" description
while touching the file (same class of leftover rename as prior fixes).

Not touching rayon/byteorder/clap (no skew found, lower priority).
 2026-08-05 13:12:17 -07:00
Omar SobhandClaude Sonnet 4.6 2ddb22897c perf: eliminate double compression and improve shuffle filter throughput 
Four independent write-path improvements:

1. Cache compressed chunks between Pass 1 and Pass 2 (chunked_write.rs,
   file_writer.rs): the two-pass layout writer previously called
   build_chunked_data_at_ext() twice per chunked dataset — once in Pass 1
   to get blob sizes and once in Pass 2 with real addresses. Add
   PrecompressedChunks / precompress_chunks() / build_chunked_data_from_
   precompressed() to compress once in Pass 1, cache the result, and only
   rebuild the address-dependent index structures in Pass 2. Expected
   ~2× speedup for chunked+deflate writes (512×512 deflate: 3.33ms → ~1.7ms).

2. SIMD-vectorisable shuffle filter (filters.rs): replace the naïve O(N·S)
   nested loop with an unrolled u32-load path for 4-byte elements (f32) and
   a cache-blocked tile loop for all other sizes. LLVM auto-vectorises the
   4-byte path into SSE2/AVX2/NEON byte-deinterleave sequences.

3. Zstd benchmark variant (h5bench_write.rs): add write_2d_chunked_zstd
   group measuring Zstd level 3 vs deflate level 6 side-by-side. Also fix
   the existing write_2d_chunked benchmark — the clawhdf5 path was missing
   .with_deflate(6), making the comparison apples-to-oranges. Add arXiv-
   backed doc recommendation on DatasetBuilder::with_zstd().

4. Zero-copy HNSW save (hnsw.rs, clawhdf5-io/lib.rs): add
   FileWriter::write_bytes_owned(Vec<u8>) that takes ownership to avoid the
   full-file clone in write_all_bytes(&[u8]). HNSW::save_to_hdf5 uses it.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
 2026-06-30 22:36:40 +00:00
Omar SobhandClaude Sonnet 4.6 d6c4d4f111 feat: implement filter codecs, format write extensions, and MPI-IO VOL 
All three SDD plans fully wired and committed to main:

Filter Codecs (FC):
- FC-1: Implement float E-scale in scaleoffset_decompress (value = minval +
  code * 2^E, negative exponents via cast to i32); add two round-trip tests.
- FC-2: filters_szip.rs — feature-gated SZIP decode via libaec FFI; SZIP
  dispatch arm added to decompress_chunk.
- FC-3: libaec-sys workspace crate with pkg-config probe and aec_buffer_decode
  FFI binding; added to workspace members.

Format Write Extensions (FWE):
- FWE-1: GroupBuilder::add_external_link() API; wired through FinishedGroup
  → GrpFlat → file_writer pass 1/2/3 (OH size, layout cursor, final write);
  external_link_write_roundtrip test.
- FWE-2: data_layout_write.rs — serialize_vds_mappings with length_size param
  and version 0/1 (external vs same-file) selection; declared as pub mod.
- FWE-3: with_virtual_sources empty-mapping guard (Important #9) — empty vec
  is silently ignored; vds_empty_mapping_list test updated to assert non-VDS
  layout results.

MPI-IO VOL Backend (MPI):
- MPI-1/2/3: mpi_vol.rs — MpiVol implementing VirtualObjectLayer; root-read
  + broadcast collective read; gather + root-write collective write; feature-
  gated mpi-io feature; wired into clawhdf5-io lib.rs.
- MPI-4: mpi_io_bench binary (h5bench-equivalent MPI-IO throughput bench).

mpi_vol.rs reviewer fixes:
- Doc-comment updated to accurately describe root-read+broadcast pattern
  (not MPI_File_read_at); MpiVol::expected_capabilities() associated fn
  added so tests can verify capabilities without a live MPI universe;
  rank_and_size_stub_values renamed to no_feature_error_contains_feature_name.

Workspace check: zero warnings, 20 test suites pass.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
 2026-06-30 11:41:41 +00:00
osobhandClaude Opus 4.8 f1762f82a7 docs: fix stale package names, counts, and CLI subcommands 
Sweep of the docs after the v2.0.0 rename and recent changes:
- Per-crate READMEs (13 files): rename leftover rustyhdf5-*/edgehdf5-*
  package names and badges to clawhdf5-*, bump usage versions to 2.1.0.
- README: update stale test badge (417 -> 1500+), workspace stats
  (15 crates/72K -> 17 crates/84K), agent crate stats (20.7K/32 modules),
  and add the missing clawhdf5-napi and clawhdf5-bench crates to the tree.
- CLAUDE.md: correct the CLI subcommand list (inspect/dump/index/search ->
  the actual create/save/search/recall/stats/flush-wal/agents-md/export/snapshot).

No code changes. Verified there are zero todo!()/unimplemented!() macros and
no TODO/FIXME comments in the tree.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
 2026-06-03 11:17:04 +00:00
osobhandClaude Opus 4.8 a16e9ab8cb chore: bump workspace to 2.1.0 
Bumps [workspace.package] and all 17 crate package versions (and internal
path-dependency requirements) from 2.0.0 to 2.1.0 for a coordinated release.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
 2026-06-03 10:36:17 +00:00
redclawsystems 3f222f6956 Merge pull request 'docs(clawhdf5): document DType variants, fix unresolved doc links' (#17) from sdlc-docs/clawhdf5-types-20260514-165210 into main 2026-05-14 23:54:48 +00:00