quantumclaw/clawhdf5
3
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity
80 Commits 11 Branches 2 Tags
339a5bd06aa46851390b46f705da02fa1785ebab
Commit Graph
4 Commits
Author SHA1 Message Date
Omar Sobh 55959b4920 ci: wire up CI, fix no_std build, fix stale package names in scripts
CI / test (push) Failing after 15s
Details 
- Add .gitea/workflows/ci.yml running scripts/ci-test.sh (fmt, clippy,
  test, no_std check) on push/PR to main.
- Fix stale rustyhdf5-py/rustyhdf5-format package names in
  ci-test.sh/check-nostd.sh, which had been silently no-op'ing those
  checks (cargo warns but doesn't fail on an unknown --exclude/-p
  target).
- With those checks actually running, fix the real issues they surface:
  - clippy: useless_conversion in chunked_write.rs, byte_char_slices in
    global_heap.rs/object_header.rs.
  - cargo fmt: apply formatting across the workspace (whitespace only).
  - no_std (thumbv7em-none-eabihf) build errors in clawhdf5-format:
    core::sync::atomic::AtomicU64 doesn't exist on that target (no
    native 64-bit atomics) — switch profiling.rs's counters to
    portable-atomic, which falls back to a CAS-based emulation there
    and is a no-op wrapper elsewhere. Add missing alloc imports for
    Box (filters.rs), Vec (filters_szip.rs), and format! (dict_encoding.rs)
    on no_std paths. Replace f64::powi (std/libm-only) with a small
    local exponentiation-by-squaring helper in the scale-offset filter.
 2026-08-05 10:50:13 -07:00
Omar SobhandClaude Sonnet 4.6 5701e8045d feat: add Pcodec lossless numerical compression filter (arXiv:2502.06112) 
Implements Pcodec (filter ID 32023) via the `pco` 1.0.x crate as a new
optional compression codec. Pcodec achieves 30–94% better compression
ratio than Zstd for f32/f64 columnar data at 1–5 GiB/s decompression
speed, making it ideal for write-once/read-many embedding archives.

Write throughput at 512×512: 591 MiB/s (parity with Zstd-3 at 610 MiB/s).
For smaller chunks Zstd-3 remains faster due to Pcodec's fixed per-chunk
distributional analysis overhead.

- Add FILTER_PCODEC = 32023 constant to filter_pipeline.rs
- Add pcodec_compress/pcodec_decompress using pco::standalone API
- Wire into compress_chunk/decompress_chunk dispatch
- Add ChunkOptions.pcodec field and DatasetBuilder.with_pcodec() method
- Enable pcodec as highest-priority codec in build_pipeline()
- Add pco dep (optional, feature = "pcodec") to clawhdf5-format/clawhdf5
- Add write_2d_chunked_pcodec benchmark comparing pcodec vs zstd-3
- Document results in BENCHMARKS.md

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
 2026-07-01 01:16:08 +00:00
Omar SobhandClaude Sonnet 4.6 2ddb22897c perf: eliminate double compression and improve shuffle filter throughput 
Four independent write-path improvements:

1. Cache compressed chunks between Pass 1 and Pass 2 (chunked_write.rs,
   file_writer.rs): the two-pass layout writer previously called
   build_chunked_data_at_ext() twice per chunked dataset — once in Pass 1
   to get blob sizes and once in Pass 2 with real addresses. Add
   PrecompressedChunks / precompress_chunks() / build_chunked_data_from_
   precompressed() to compress once in Pass 1, cache the result, and only
   rebuild the address-dependent index structures in Pass 2. Expected
   ~2× speedup for chunked+deflate writes (512×512 deflate: 3.33ms → ~1.7ms).

2. SIMD-vectorisable shuffle filter (filters.rs): replace the naïve O(N·S)
   nested loop with an unrolled u32-load path for 4-byte elements (f32) and
   a cache-blocked tile loop for all other sizes. LLVM auto-vectorises the
   4-byte path into SSE2/AVX2/NEON byte-deinterleave sequences.

3. Zstd benchmark variant (h5bench_write.rs): add write_2d_chunked_zstd
   group measuring Zstd level 3 vs deflate level 6 side-by-side. Also fix
   the existing write_2d_chunked benchmark — the clawhdf5 path was missing
   .with_deflate(6), making the comparison apples-to-oranges. Add arXiv-
   backed doc recommendation on DatasetBuilder::with_zstd().

4. Zero-copy HNSW save (hnsw.rs, clawhdf5-io/lib.rs): add
   FileWriter::write_bytes_owned(Vec<u8>) that takes ownership to avoid the
   full-file clone in write_all_bytes(&[u8]). HNSW::save_to_hdf5 uses it.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
 2026-06-30 22:36:40 +00:00
Omar SobhandClaude Sonnet 4.6 90bdd7cd13 feat: add h5bench-equivalent Criterion benchmarks to clawhdf5-bench 
Adds three Criterion benchmark suites mirroring the h5bench HPC I/O
benchmark workloads in pure Rust — no C libhdf5 required for the default
path, with an optional `libhdf5-compare` feature for side-by-side numbers.

  - benches/h5bench_write.rs: write_1d_contiguous, write_2d_chunked,
    write_f64_batch, write_multi_dataset, write_with_attrs
  - benches/h5bench_read.rs: read_sequential, read_f64_sequential,
    read_chunked_2d, read_from_disk, read_hyperslab
  - benches/h5bench_meta.rs: metadata_attrs_write, metadata_attrs_read,
    metadata_groups_create, metadata_groups_traverse, metadata_string_attrs

All benchmarks pass `cargo bench --bench <name> -- --test` and clippy
reports zero warnings.  Run with `cargo bench -p clawhdf5-bench`.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
 2026-06-30 11:41:41 +00:00