Four independent write-path improvements:
1. Cache compressed chunks between Pass 1 and Pass 2 (chunked_write.rs,
file_writer.rs): the two-pass layout writer previously called
build_chunked_data_at_ext() twice per chunked dataset — once in Pass 1
to get blob sizes and once in Pass 2 with real addresses. Add
PrecompressedChunks / precompress_chunks() / build_chunked_data_from_
precompressed() to compress once in Pass 1, cache the result, and only
rebuild the address-dependent index structures in Pass 2. Expected
~2× speedup for chunked+deflate writes (512×512 deflate: 3.33ms → ~1.7ms).
2. SIMD-vectorisable shuffle filter (filters.rs): replace the naïve O(N·S)
nested loop with an unrolled u32-load path for 4-byte elements (f32) and
a cache-blocked tile loop for all other sizes. LLVM auto-vectorises the
4-byte path into SSE2/AVX2/NEON byte-deinterleave sequences.
3. Zstd benchmark variant (h5bench_write.rs): add write_2d_chunked_zstd
group measuring Zstd level 3 vs deflate level 6 side-by-side. Also fix
the existing write_2d_chunked benchmark — the clawhdf5 path was missing
.with_deflate(6), making the comparison apples-to-oranges. Add arXiv-
backed doc recommendation on DatasetBuilder::with_zstd().
4. Zero-copy HNSW save (hnsw.rs, clawhdf5-io/lib.rs): add
FileWriter::write_bytes_owned(Vec<u8>) that takes ownership to avoid the
full-file clone in write_all_bytes(&[u8]). HNSW::save_to_hdf5 uses it.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Adds three Criterion benchmark suites mirroring the h5bench HPC I/O
benchmark workloads in pure Rust — no C libhdf5 required for the default
path, with an optional `libhdf5-compare` feature for side-by-side numbers.
- benches/h5bench_write.rs: write_1d_contiguous, write_2d_chunked,
write_f64_batch, write_multi_dataset, write_with_attrs
- benches/h5bench_read.rs: read_sequential, read_f64_sequential,
read_chunked_2d, read_from_disk, read_hyperslab
- benches/h5bench_meta.rs: metadata_attrs_write, metadata_attrs_read,
metadata_groups_create, metadata_groups_traverse, metadata_string_attrs
All benchmarks pass `cargo bench --bench <name> -- --test` and clippy
reports zero warnings. Run with `cargo bench -p clawhdf5-bench`.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
