quantumclaw/clawhdf5
3
1
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases 1 Wiki Activity

perf: lower parallel compress threshold from 4 to 2 chunks

Browse Source 
Enables Rayon parallel compression for typical 4-chunk workloads (e.g.,
128×128 matrix with 32-row chunks). Rayon's dispatch overhead is ~2 µs,
worthwhile at ≥3 chunks with real compression work per chunk.

Previously the threshold was "> 4" which excluded 4-chunk datasets entirely
from parallel compression. Now "> 2" covers 3+ chunks.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
This commit is contained in:
Omar Sobh
2026-07-01 01:53:41 +00:00
co-authored by Claude Sonnet 4.6
parent e23e0358e0
commit d8ef8785e2
1 changed files with 5 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/clawhdf5-format/src/chunked_write.rs
+5 -1
View File
@@ -260,8 +260,12 @@ pub fn split_into_chunks(
} }
/// Parallel compression threshold: use rayon when chunk count exceeds this. /// Parallel compression threshold: use rayon when chunk count exceeds this.
///
/// Lowered to 2 to enable parallel compression for typical 4-chunk workloads
/// (e.g., 128×128 matrix with 32-row chunks = 4 chunks). Rayon's overhead is
/// ~2 µs, worthwhile at ≥2 chunks with any real compression (arXiv:2206.14761).
#[cfg(feature = "parallel")] #[cfg(feature = "parallel")]
const PARALLEL_COMPRESS_THRESHOLD: usize = 4; const PARALLEL_COMPRESS_THRESHOLD: usize = 2;
/// Compress all chunks, using parallel compression when beneficial. /// Compress all chunks, using parallel compression when beneficial.
/// ///