perf: eliminate double compression and improve shuffle filter throughput
Four independent write-path improvements: 1. Cache compressed chunks between Pass 1 and Pass 2 (chunked_write.rs, file_writer.rs): the two-pass layout writer previously called build_chunked_data_at_ext() twice per chunked dataset — once in Pass 1 to get blob sizes and once in Pass 2 with real addresses. Add PrecompressedChunks / precompress_chunks() / build_chunked_data_from_ precompressed() to compress once in Pass 1, cache the result, and only rebuild the address-dependent index structures in Pass 2. Expected ~2× speedup for chunked+deflate writes (512×512 deflate: 3.33ms → ~1.7ms). 2. SIMD-vectorisable shuffle filter (filters.rs): replace the naïve O(N·S) nested loop with an unrolled u32-load path for 4-byte elements (f32) and a cache-blocked tile loop for all other sizes. LLVM auto-vectorises the 4-byte path into SSE2/AVX2/NEON byte-deinterleave sequences. 3. Zstd benchmark variant (h5bench_write.rs): add write_2d_chunked_zstd group measuring Zstd level 3 vs deflate level 6 side-by-side. Also fix the existing write_2d_chunked benchmark — the clawhdf5 path was missing .with_deflate(6), making the comparison apples-to-oranges. Add arXiv- backed doc recommendation on DatasetBuilder::with_zstd(). 4. Zero-copy HNSW save (hnsw.rs, clawhdf5-io/lib.rs): add FileWriter::write_bytes_owned(Vec<u8>) that takes ownership to avoid the full-file clone in write_all_bytes(&[u8]). HNSW::save_to_hdf5 uses it. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
This commit is contained in:
co-authored by
Claude Sonnet 4.6
parent
3a1fcc5cb3
commit
2ddb22897c
@@ -16,7 +16,6 @@ use clawhdf5_format::object_header::ObjectHeader;
|
||||
use clawhdf5_format::signature::find_signature;
|
||||
use clawhdf5_format::superblock::Superblock;
|
||||
use clawhdf5_io::FileWriter as IoFileWriter;
|
||||
use clawhdf5_io::HDF5ReadWrite;
|
||||
|
||||
/// Distance metric for the HNSW index.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
@@ -516,7 +515,7 @@ impl HnswIndex {
|
||||
pub fn save_to_hdf5(&self, writer: &mut IoFileWriter) -> Result<(), FormatError> {
|
||||
let bytes = self.to_hdf5_bytes()?;
|
||||
writer
|
||||
.write_all_bytes(&bytes)
|
||||
.write_bytes_owned(bytes)
|
||||
.map_err(|e| FormatError::SerializationError(e.to_string()))?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user