quantumclaw/clawhdf5
3
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

docs: add superpowers implementation plans (MPI-IO VOL backend, format write extensions, filter codecs)

Browse Source
This commit is contained in:
Omar Sobh
2026-08-03 02:46:23 +00:00
parent 1347746973
commit b1fc23e975
3 changed files with 2255 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/superpowers/plans/2026-06-29-filter-codecs.md
+654
View File
@@ -0,0 +1,654 @@
# Filter Codecs Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Add SZIP decompression (filter ID 4) and N-Bit E-scale decompression (scale type 1 of filter ID 6) to the clawhdf5-format crate.
**Architecture:** N-Bit E-scale extends the existing `scaleoffset_decompress` function in `filters.rs` with a ~15-line new branch. SZIP is added as an optional `szip` feature using FFI to the system `libaec` C library (same pattern as the existing `system-zlib-decompress` feature), with a `build.rs` that uses `pkg-config` or `cc` to locate/compile it.
**Tech Stack:** Rust (no_std-compatible where possible), `libaec` C library (optional FFI via `cc` crate), `pkg-config` crate for system library discovery.
## Global Constraints
- All code in `crates/clawhdf5-format/` and `crates/clawhdf5-filters/`.
- SZIP must be feature-gated: `szip` feature, disabled by default. When not enabled, `FILTER_SZIP` must return `FormatError::UnsupportedFilter(4)` as it does today.
- N-Bit E-scale requires no new features — it is a fix within the existing `deflate`-free path.
- Tests must not require h5py or Python; use hand-crafted compressed byte sequences verified against the HDF5 reference implementation commentary in the test file.
- Run `cargo test -p clawhdf5-format` after every task.
---
### Task 1: N-Bit E-scale (float scale-offset, scale type 1)
**Files:**
- Modify: `crates/clawhdf5-format/src/filters.rs:96-108` (the `scaleoffset_decompress` dispatch block)
- Test: `crates/clawhdf5-format/src/filters.rs` (new tests in the existing `#[cfg(test)]` block at the bottom)
**Interfaces:**
- Consumes: existing `scaleoffset_decompress(data: &[u8], cd: &[u32], expected_bytes: usize) -> Result<Vec<u8>, FormatError>`.
- Produces: same function, now handling `cd[0] == 1` (H5Z_SO_FLOAT_ESCALE).
**Background:**
- `cd[0]`: scale type — `0` = float D-scale (already done), `1` = float E-scale (this task), `2` = integer (already done).
- E-scale formula: `value = minval + code * 2^E` where `E = cd[1] as i32` (may be negative for sub-unit precision). Compare to D-scale: `value = minval + code / 10^D`.
- The binary layout (minbits, minval, 8 reserved bytes, packed MSB-first codes) is IDENTICAL to D-scale. Only the reconstruction formula differs.
- [ ] **Step 1: Write the failing test**
In `crates/clawhdf5-format/src/filters.rs`, inside the existing `#[cfg(test)] mod tests` block, add:
```rust
#[test]
fn scaleoffset_float_escale_basic() {
// f32 [0.0, 4.0, 8.0, 12.0]: minval=0.0f32, E=2 (scale=4.0 = 2^2),
// stored codes [0, 1, 2, 3] in 2 bits each.
// cd: [scale_type=1, scale_factor=2, nelmts=4, unused=1, elem_size=4,
// signed=0, big_endian=0, fill_defined=1, fill_lo=0, fill_hi=0]
let cd = [1u32, 2, 4, 1, 4, 0, 0, 1, 0, 0];
// Layout: minbits(4 LE) = 2, minval_width(1) = 4, minval(4) = 0.0f32,
// reserved(8), packed codes: 0b_00_01_10_11 = 0x1B in 1 byte
let mut data = Vec::new();
data.extend_from_slice(&2u32.to_le_bytes()); // minbits = 2
data.push(4); // minval_width
data.extend_from_slice(&0.0f32.to_le_bytes()); // minval = 0.0
data.extend_from_slice(&[0u8; 8]); // 8 reserved bytes
data.push(0b0001_1011); // codes: 0,1,2,3 packed MSB-first in 2 bits each
let out = scaleoffset_decompress(&data, &cd, 0).unwrap();
let floats: Vec<f32> = out.chunks_exact(4)
.map(|c| f32::from_le_bytes(c.try_into().unwrap()))
.collect();
assert_eq!(floats.len(), 4);
assert!((floats[0] - 0.0f32).abs() < 1e-5, "got {}", floats[0]);
assert!((floats[1] - 4.0f32).abs() < 1e-5, "got {}", floats[1]);
assert!((floats[2] - 8.0f32).abs() < 1e-5, "got {}", floats[2]);
assert!((floats[3] - 12.0f32).abs() < 1e-5, "got {}", floats[3]);
}
#[test]
fn scaleoffset_float_escale_negative_exponent() {
// f32 [0.0, 0.25, 0.5, 0.75]: minval=0.0, E=-2 (scale=0.25 = 2^-2),
// codes [0,1,2,3]. cd[1] stored as u32; we cast to i32 in decoder.
let e: i32 = -2;
let cd = [1u32, e as u32, 4, 1, 4, 0, 0, 1, 0, 0];
let mut data = Vec::new();
data.extend_from_slice(&2u32.to_le_bytes());
data.push(4);
data.extend_from_slice(&0.0f32.to_le_bytes());
data.extend_from_slice(&[0u8; 8]);
data.push(0b0001_1011);
let out = scaleoffset_decompress(&data, &cd, 0).unwrap();
let floats: Vec<f32> = out.chunks_exact(4)
.map(|c| f32::from_le_bytes(c.try_into().unwrap()))
.collect();
assert!((floats[1] - 0.25f32).abs() < 1e-6, "got {}", floats[1]);
assert!((floats[2] - 0.50f32).abs() < 1e-6, "got {}", floats[2]);
assert!((floats[3] - 0.75f32).abs() < 1e-6, "got {}", floats[3]);
}
```
- [ ] **Step 2: Run tests to verify they fail**
```bash
cargo test -p clawhdf5-format scaleoffset_float_escale 2>&1 | head -30
```
Expected: FAIL — `"UnsupportedFilter(6)"` or similar.
- [ ] **Step 3: Implement E-scale in scaleoffset_decompress**
In `crates/clawhdf5-format/src/filters.rs`, change the dispatch block (around line 96):
```rust
fn scaleoffset_decompress(
data: &[u8],
cd: &[u32],
expected_bytes: usize,
) -> Result<Vec<u8>, FormatError> {
const H5Z_SO_FLOAT_DSCALE: u32 = 0;
const H5Z_SO_FLOAT_ESCALE: u32 = 1;
const H5Z_SO_INT: u32 = 2;
if cd.len() < 8 {
return Err(FormatError::ChunkedReadError(
"scale-offset: missing filter client data".into(),
));
}
let scale_type = cd[0];
let is_float = scale_type == H5Z_SO_FLOAT_DSCALE || scale_type == H5Z_SO_FLOAT_ESCALE;
if scale_type != H5Z_SO_INT && !is_float {
return Err(FormatError::UnsupportedFilter(FILTER_SCALEOFFSET));
}
// ... (rest of the existing parsing logic unchanged until the reconstruction block) ...
```
Then in the float reconstruction block (currently the `if is_float { ... }` branch at line ~192), replace:
```rust
if is_float {
let scale = if scale_type == H5Z_SO_FLOAT_DSCALE {
10f64.powi(cd[1] as i32)
} else {
// E-scale: scale factor is a power of 2; cd[1] interpreted as signed i32
2f64.powi(cd[1] as i32)
};
let minval = read_le_float(minval_bytes, elem_size);
let fill_value = if fill_defined {
let lo = *cd.get(8).unwrap_or(&0) as u64;
let hi = *cd.get(9).unwrap_or(&0) as u64;
bits_to_float(lo | (hi << 32), elem_size)
} else {
0.0
};
let values: Vec<f64> = codes
.iter()
.map(|&code| {
if has_fill_code && code == fill_code {
fill_value
} else if scale_type == H5Z_SO_FLOAT_DSCALE {
minval + code as f64 / scale
} else {
// E-scale: value = minval + code * 2^E
minval + code as f64 * scale
}
})
.collect();
Ok(write_floats(&values, elem_size, big_endian))
} else {
```
- [ ] **Step 4: Run tests to verify they pass**
```bash
cargo test -p clawhdf5-format scaleoffset_float_escale 2>&1
```
Expected: both tests PASS.
- [ ] **Step 5: Run full test suite**
```bash
cargo test -p clawhdf5-format 2>&1 | tail -10
```
Expected: all tests pass, zero failures.
- [ ] **Step 6: Commit**
```bash
git add crates/clawhdf5-format/src/filters.rs
git commit -m "feat: add scale-offset E-scale (float binary-exponent) decompression"
```
---
### Task 2: SZIP feature gate and stub hook
**Files:**
- Modify: `crates/clawhdf5-format/Cargo.toml` (add `szip` feature and `libaec-sys` optional dep)
- Create: `crates/clawhdf5-format/build.rs`
- Modify: `crates/clawhdf5-format/src/filters.rs` (add `szip_decompress` call in `decompress_chunk`)
- Create: `crates/clawhdf5-format/src/filters_szip.rs`
**Interfaces:**
- Produces: `pub(crate) fn szip_decompress(data: &[u8], cd: &[u32], chunk_size: usize) -> Result<Vec<u8>, FormatError>`
- `decompress_chunk` calls it for `FILTER_SZIP` when the `szip` feature is active.
**Background — SZIP parameters from `cd`:**
- `cd[0]` (options mask): bit 2 = NN (nearest-neighbor) preprocessing, bit 4 = EC (entropy coding), bit 5 = LSB order, bit 8 = allow K-13.
- `cd[1]` (pixels per block): 8, 10, 16, or 32.
- `cd[2]` (pixels per scan line): not used for decompression.
- The `libaec` library exposes `aec_decode_init`, `aec_decode`, `aec_decode_end` (struct `aec_stream`).
- [ ] **Step 1: Write the failing test**
In `crates/clawhdf5-format/src/filters_szip.rs` (create the file):
```rust
//! SZIP (libaec Adaptive Entropy Coding) decompression.
//!
//! Gated by the `szip` feature which links against the system libaec library.
use crate::error::FormatError;
use crate::filter_pipeline::FILTER_SZIP;
/// Decompress SZIP-compressed data using libaec.
///
/// `cd` is the HDF5 filter client data:
/// cd[0] = options mask (EC flag = 0x04, NN flag = 0x20, LSB = 0x40, allow_k13 = 0x100)
/// cd[1] = pixels per block (8, 10, 16, or 32)
/// cd[2] = pixels per scan line
/// cd[4] = bits per sample (element bit width)
pub fn szip_decompress(
_data: &[u8],
_cd: &[u32],
_chunk_size: usize,
) -> Result<Vec<u8>, FormatError> {
#[cfg(feature = "szip")]
{
szip_decode_impl(_data, _cd, _chunk_size)
}
#[cfg(not(feature = "szip"))]
{
Err(FormatError::UnsupportedFilter(FILTER_SZIP))
}
}
#[cfg(feature = "szip")]
fn szip_decode_impl(
data: &[u8],
cd: &[u32],
chunk_size: usize,
) -> Result<Vec<u8>, FormatError> {
if cd.len() < 5 {
return Err(FormatError::ChunkedReadError("szip: missing client data".into()));
}
let options = cd[0];
let pixels_per_block = cd[1];
let bits_per_sample = cd[4] as usize;
if bits_per_sample == 0 || bits_per_sample > 32 {
return Err(FormatError::ChunkedReadError("szip: invalid bits per sample".into()));
}
// Map HDF5 options to libaec flags
let flags: u32 = {
let mut f = 0u32;
if options & 0x04 != 0 { f |= AEC_DATA_PREPROCESS; } // NN
if options & 0x40 == 0 { f |= AEC_DATA_MSB; } // MSB (not LSB)
if options & 0x100 != 0 { f |= AEC_ALLOW_K13; }
f
};
let out_len = if chunk_size > 0 { chunk_size } else {
return Err(FormatError::ChunkedReadError("szip: unknown output size".into()));
};
let mut out = vec![0u8; out_len];
let result = unsafe {
libaec_sys::aec_buffer_decode(
data.as_ptr(),
data.len(),
out.as_mut_ptr(),
&mut (out_len as libaec_sys::size_t),
bits_per_sample as u32,
pixels_per_block,
flags,
)
};
if result != 0 {
return Err(FormatError::DecompressionError(format!("szip: libaec error {result}")));
}
Ok(out)
}
// libaec flag constants (from aec.h)
#[cfg(feature = "szip")]
const AEC_DATA_PREPROCESS: u32 = 1;
#[cfg(feature = "szip")]
const AEC_DATA_MSB: u32 = 2;
#[cfg(feature = "szip")]
const AEC_ALLOW_K13: u32 = 8;
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn szip_disabled_returns_unsupported() {
// When szip feature is disabled, must return UnsupportedFilter(4).
#[cfg(not(feature = "szip"))]
{
let result = szip_decompress(&[], &[4, 8, 10, 0, 8], 64);
assert!(
matches!(result, Err(FormatError::UnsupportedFilter(4))),
"expected UnsupportedFilter(4), got {result:?}"
);
}
#[cfg(feature = "szip")]
{
// When szip IS enabled, an empty buffer should error but not panic.
let _ = szip_decompress(&[], &[4, 8, 10, 0, 8], 64);
}
}
}
```
- [ ] **Step 2: Run the new test**
```bash
cargo test -p clawhdf5-format szip_disabled_returns_unsupported 2>&1
```
Expected: the file doesn't compile yet (module not declared). That's the expected failure mode.
- [ ] **Step 3: Add Cargo.toml feature and build.rs**
In `crates/clawhdf5-format/Cargo.toml`, add to `[dependencies]`:
```toml
libaec-sys = { version = "0.1", optional = true }
```
Add to `[features]`:
```toml
szip = ["libaec-sys"]
```
Create `crates/clawhdf5-format/build.rs`:
```rust
fn main() {
#[cfg(feature = "szip")]
{
// Try pkg-config first; fall back to empty link flags (system path).
if std::process::Command::new("pkg-config")
.args(["--exists", "libaec"])
.status()
.map(|s| s.success())
.unwrap_or(false)
{
println!("cargo:rustc-link-lib=aec");
if let Ok(dir) = std::process::Command::new("pkg-config")
.args(["--variable=libdir", "libaec"])
.output()
{
let dir = String::from_utf8_lossy(&dir.stdout).trim().to_string();
if !dir.is_empty() {
println!("cargo:rustc-link-search=native={dir}");
}
}
} else {
// Fallback: assume libaec is in the standard library path.
println!("cargo:rustc-link-lib=aec");
}
}
}
```
Note: `libaec-sys` is a crate that provides raw bindings. If that crate doesn't exist on crates.io with that exact name, use `libaec-sys = { git = "..." }` or add `aec-sys` as a local crate (see Task 3 below for the fallback path).
- [ ] **Step 4: Declare the module in lib.rs**
In `crates/clawhdf5-format/src/lib.rs`, add:
```rust
mod filters_szip;
```
(Place it alongside the other `mod filters;` declaration.)
- [ ] **Step 5: Hook szip_decompress into decompress_chunk**
In `crates/clawhdf5-format/src/filters.rs`, change the dispatch inside `decompress_chunk`:
```rust
// Change this:
other => return Err(FormatError::UnsupportedFilter(other)),
// To this:
FILTER_SZIP => crate::filters_szip::szip_decompress(&data, &filter.client_data, chunk_size)?,
other => return Err(FormatError::UnsupportedFilter(other)),
```
Also add the import at the top of `filters.rs`:
```rust
use crate::filter_pipeline::{
FILTER_DEFLATE, FILTER_FLETCHER32, FILTER_LZ4, FILTER_NBIT, FILTER_SCALEOFFSET,
FILTER_SHUFFLE, FILTER_SZIP, FILTER_ZSTD, FilterPipeline,
};
```
(Add `FILTER_SZIP` to the existing import.)
- [ ] **Step 6: Run tests without szip feature**
```bash
cargo test -p clawhdf5-format 2>&1 | tail -15
```
Expected: all existing tests pass; `szip_disabled_returns_unsupported` passes.
- [ ] **Step 7: Commit**
```bash
git add crates/clawhdf5-format/Cargo.toml \
crates/clawhdf5-format/build.rs \
crates/clawhdf5-format/src/filters_szip.rs \
crates/clawhdf5-format/src/filters.rs \
crates/clawhdf5-format/src/lib.rs
git commit -m "feat: add SZIP filter hook with libaec FFI (feature-gated, disabled by default)"
```
---
### Task 3: libaec-sys bindings crate (if no public crate exists)
> Skip this task if a published `libaec-sys` crate is available on crates.io. Check with `cargo search libaec-sys`.
**Files:**
- Create: `crates/libaec-sys/Cargo.toml`
- Create: `crates/libaec-sys/src/lib.rs`
- Create: `crates/libaec-sys/build.rs`
- Modify: `Cargo.toml` (workspace members)
**Interfaces:**
- Produces: `pub unsafe fn aec_buffer_decode(src: *const u8, src_len: usize, dst: *mut u8, dst_len: *mut usize, bits_per_sample: u32, block_size: u32, flags: u32) -> i32`
- [ ] **Step 1: Create the sys crate**
Create `crates/libaec-sys/Cargo.toml`:
```toml
[package]
name = "libaec-sys"
version = "0.1.0"
edition = "2024"
links = "aec"
[build-dependencies]
pkg-config = "0.3"
```
Create `crates/libaec-sys/build.rs`:
```rust
fn main() {
if pkg_config::Config::new()
.atleast_version("1.0")
.probe("libaec")
.is_ok()
{
return;
}
// If pkg-config fails, try linking directly
println!("cargo:rustc-link-lib=aec");
}
```
Create `crates/libaec-sys/src/lib.rs`:
```rust
//! Raw FFI bindings to libaec (Adaptive Entropy Coding library).
//!
//! Provides the `aec_buffer_decode` convenience function for one-shot decompression.
pub type size_t = usize;
// AEC flag constants matching aec.h
pub const AEC_DATA_PREPROCESS: u32 = 1; // NN preprocessing
pub const AEC_DATA_MSB: u32 = 2; // big-endian sample order
pub const AEC_RESTRICTED: u32 = 4; // restricted coding set
pub const AEC_ALLOW_K13: u32 = 8; // allow k=13 option
extern "C" {
/// One-shot decompression. Returns 0 on success.
///
/// # Safety
/// `src` must be valid for `src_len` bytes; `dst` must be valid for `*dst_len` bytes.
pub fn aec_buffer_decode(
src: *const u8,
src_len: size_t,
dst: *mut u8,
dst_len: *mut size_t,
bits_per_sample: u32,
block_size: u32,
flags: u32,
) -> i32;
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn constants_are_correct() {
assert_eq!(AEC_DATA_PREPROCESS, 1);
assert_eq!(AEC_DATA_MSB, 2);
}
}
```
- [ ] **Step 2: Add to workspace**
In the root `Cargo.toml`, add `"crates/libaec-sys"` to `[workspace] members`.
- [ ] **Step 3: Update clawhdf5-format dependency**
In `crates/clawhdf5-format/Cargo.toml`, change:
```toml
libaec-sys = { version = "0.1", optional = true }
```
to:
```toml
libaec-sys = { path = "../libaec-sys", version = "0.1", optional = true }
```
- [ ] **Step 4: Run tests**
```bash
cargo test -p libaec-sys 2>&1 | tail -10
cargo test -p clawhdf5-format 2>&1 | tail -10
```
Expected: both pass.
- [ ] **Step 5: Commit**
```bash
git add crates/libaec-sys/ Cargo.toml crates/clawhdf5-format/Cargo.toml
git commit -m "feat: add libaec-sys workspace crate for SZIP FFI bindings"
```
---
### Task 4: SZIP integration test with libaec installed
**Files:**
- Modify: `crates/clawhdf5-format/src/filters_szip.rs` (add integration test behind `szip` feature)
**Background:** This test only runs when the `szip` feature is enabled AND libaec is installed. It validates that we can round-trip a known dataset (u8 values 0-63, 8 pixels per block, EC mode).
- [ ] **Step 1: Add integration test**
In `crates/clawhdf5-format/src/filters_szip.rs`, inside `#[cfg(test)] mod tests`, add:
```rust
#[test]
#[cfg(feature = "szip")]
fn szip_ec_roundtrip_u8() {
// Encode 64 values [0..64] with libaec, then decode with our wrapper.
// This tests the full encode→decode cycle.
use crate::filter_pipeline::FilterDescription;
use crate::filters::{compress_chunk, decompress_chunk};
use crate::filter_pipeline::{FilterPipeline, FILTER_SZIP};
// cd: options=EC(0x04)|MSB(0x00), pixels_per_block=8, ppsl=64, unused=0, bits_per_sample=8
let cd = vec![0x04u32, 8, 64, 0, 8];
// Build a test dataset: 64 bytes incrementing
let original: Vec<u8> = (0u8..64).collect();
// Use aec_buffer_encode to generate compressed data for this test
let compressed = unsafe {
let mut out = vec![0u8; original.len() * 4]; // generous buffer
let mut out_len = out.len();
libaec_sys::aec_buffer_encode(
original.as_ptr(),
original.len(),
out.as_mut_ptr(),
&mut out_len,
8, // bits per sample
8, // block size
libaec_sys::AEC_DATA_MSB,
);
out.truncate(out_len);
out
};
let decoded = szip_decompress(&compressed, &cd, original.len()).unwrap();
assert_eq!(decoded, original);
}
```
Also add `aec_buffer_encode` to `crates/libaec-sys/src/lib.rs`:
```rust
extern "C" {
// ... existing aec_buffer_decode ...
/// One-shot compression. Returns 0 on success.
///
/// # Safety
/// `src` must be valid for `src_len` bytes; `dst` must be valid for `*dst_len` bytes.
pub fn aec_buffer_encode(
src: *const u8,
src_len: size_t,
dst: *mut u8,
dst_len: *mut size_t,
bits_per_sample: u32,
block_size: u32,
flags: u32,
) -> i32;
}
```
- [ ] **Step 2: Run the integration test (requires libaec installed)**
```bash
# Install libaec if not present: sudo apt install libaec-dev
cargo test -p clawhdf5-format --features szip szip_ec_roundtrip_u8 2>&1
```
Expected: PASS when libaec is installed.
- [ ] **Step 3: Run full suite without szip feature to verify no regressions**
```bash
cargo test -p clawhdf5-format 2>&1 | tail -10
```
Expected: all tests pass.
- [ ] **Step 4: Commit**
```bash
git add crates/clawhdf5-format/src/filters_szip.rs crates/libaec-sys/src/lib.rs
git commit -m "feat: add SZIP integration test for libaec roundtrip"
```
---
## Verification
```bash
# Full test suite (no szip)
cargo test -p clawhdf5-format 2>&1 | tail -5
# With szip feature (requires libaec installed)
cargo test -p clawhdf5-format --features szip 2>&1 | tail -5
# Specific E-scale tests
cargo test -p clawhdf5-format scaleoffset_float_escale 2>&1
# Confirm SZIP returns UnsupportedFilter without the feature
cargo test -p clawhdf5-format szip_disabled 2>&1
```
docs/superpowers/plans/2026-06-29-format-write-extensions.md
+844
View File
@@ -0,0 +1,844 @@
# Format Write Extensions Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Add three write-side features to clawhdf5-format: (1) external link creation via `GroupBuilder`, (2) external VDS (Virtual Dataset Source) layout writes, and (3) superblock v4 read/write for page-buffering-aware files.
**Architecture:** External links reuse the existing `LinkMessage::serialize()` which already handles `LinkTarget::External` — only the `GroupBuilder` API needs wiring up. External VDS adds `write_vds_layout()` in `file_writer.rs` and `serialize_vds_mappings()` in a new `data_layout_write.rs`. Superblock v4 extends `Superblock::parse` with a new `parse_v4` branch (identical structure to v3 with an extra `page_size` field) and updates `Superblock::serialize` to optionally write v4.
**Tech Stack:** Pure Rust, no new dependencies. All changes in `crates/clawhdf5-format/`.
## Global Constraints
- All code in `crates/clawhdf5-format/`.
- No new Cargo dependencies.
- External links: written as `LinkTarget::External`, readable by h5py (verified in tests).
- VDS: uses data layout version 4, class 3. Global heap at end of file.
- Superblock v4: only adds `page_size: u32` field after the v2/v3 body; checksum placement unchanged.
- Run `cargo test -p clawhdf5-format` after every task.
---
### Task 1: External link write API in GroupBuilder
**Background:** `LinkMessage::serialize()` in `link_message.rs:76–175` already handles `LinkTarget::External { filename, object_path }` (writes link_type byte = 64, then packed filename+path). What's missing is a public API in `file_writer.rs` to create external links from a `GroupBuilder`. Currently `GroupBuilder` only creates datasets and sub-groups via `create_dataset` / `create_group`.
**Files:**
- Modify: `crates/clawhdf5-format/src/file_writer.rs` (add `GroupBuilder::add_external_link`)
- Modify: `crates/clawhdf5-format/src/lib.rs` (re-export `LinkTarget` if not already exported)
- Test: `crates/clawhdf5-format/src/file_writer.rs` (new test in `#[cfg(test)]`)
**Interfaces:**
- Produces: `GroupBuilder::add_external_link(&mut self, name: &str, target_file: &str, target_path: &str) -> &mut Self`
- [ ] **Step 1: Write the failing test**
At the bottom of the `#[cfg(test)]` block in `crates/clawhdf5-format/src/file_writer.rs`, add:
```rust
#[test]
fn external_link_write_roundtrip() {
use crate::group_v2::resolve_path_any;
use crate::link_message::{LinkMessage, LinkTarget};
use crate::message_type::MessageType;
use crate::object_header::ObjectHeader;
use crate::signature::find_signature;
use crate::superblock::Superblock;
let mut fw = FileWriter::new();
let mut grp = fw.create_group("links");
grp.add_external_link("remote_data", "other_file.h5", "/sensors/temp");
fw.add_group(grp.finish());
let bytes = fw.finish().unwrap();
let sig = find_signature(&bytes).unwrap();
let sb = Superblock::parse(&bytes, sig).unwrap();
// Navigate to /links group
let links_addr = resolve_path_any(&bytes, &sb, "links").unwrap();
let links_oh = ObjectHeader::parse(
&bytes, links_addr as usize, sb.offset_size, sb.length_size,
).unwrap();
// Find the link message for "remote_data"
let link_msg = links_oh.messages.iter()
.filter(|m| m.msg_type == MessageType::Link)
.find_map(|m| {
let lm = LinkMessage::parse(&m.data, sb.offset_size).ok()?;
if lm.name == "remote_data" { Some(lm) } else { None }
})
.expect("external link message not found");
assert_eq!(
link_msg.link_target,
LinkTarget::External {
filename: "other_file.h5".into(),
object_path: "/sensors/temp".into(),
}
);
}
```
- [ ] **Step 2: Run test to verify it fails**
```bash
cargo test -p clawhdf5-format external_link_write_roundtrip 2>&1 | head -20
```
Expected: compile error — `add_external_link` not found.
- [ ] **Step 3: Find GroupBuilder in file_writer.rs and add the method**
Locate `GroupBuilder` in `crates/clawhdf5-format/src/file_writer.rs`. It tracks its items as a `Vec` of internal builders. Add a field for external links and the method:
First, locate the `GroupBuilder` struct definition and add a field:
```rust
pub struct GroupBuilder {
name: String,
datasets: Vec<DatasetBuilder>,
groups: Vec<FinishedGroup>,
external_links: Vec<(String, String, String)>, // (name, filename, object_path)
}
```
Update `GroupBuilder::new()` (or equivalent constructor) to initialize `external_links: Vec::new()`.
Add the public method immediately after the existing `create_dataset`/`create_group` methods:
```rust
/// Add a link in this group that points to an object in another HDF5 file.
///
/// `name` is the link name within this group.
/// `target_file` is the relative or absolute path to the target .h5 file.
/// `target_path` is the HDF5 path of the object within the target file.
pub fn add_external_link(
&mut self,
name: &str,
target_file: &str,
target_path: &str,
) -> &mut Self {
self.external_links.push((
name.to_string(),
target_file.to_string(),
target_path.to_string(),
));
self
}
```
- [ ] **Step 4: Wire external links into the group serialization**
Find where the `GroupBuilder` emits `LinkMessage` bytes during `finish()` / `build_group()`. For each external link, emit a `LinkMessage` with `LinkTarget::External`:
```rust
use crate::link_message::{LinkMessage, LinkTarget};
use crate::datatype::CharacterSet;
// Inside the loop/block that serializes links:
for (link_name, filename, object_path) in &self.external_links {
let msg = LinkMessage {
name: link_name.clone(),
link_target: LinkTarget::External {
filename: filename.clone(),
object_path: object_path.clone(),
},
creation_order: None,
charset: CharacterSet::Utf8,
};
let msg_bytes = msg.serialize(offset_size);
// Emit as a Link message (MessageType::Link = 0x0006) into the object header
emit_message(&mut oh_buf, MessageType::Link, &msg_bytes);
}
```
(Follow the exact pattern used for hard links and soft links in the same codebase — find where hard-link `LinkMessage` bytes are pushed and add the external links in the same loop.)
- [ ] **Step 5: Run the failing test**
```bash
cargo test -p clawhdf5-format external_link_write_roundtrip 2>&1
```
Expected: PASS.
- [ ] **Step 6: Run full suite**
```bash
cargo test -p clawhdf5-format 2>&1 | tail -10
```
Expected: all tests pass.
- [ ] **Step 7: Commit**
```bash
git add crates/clawhdf5-format/src/file_writer.rs
git commit -m "feat: add GroupBuilder::add_external_link for writing cross-file HDF5 links"
```
---
### Task 2: VDS mapping serialization helper
**Background:** Reading VDS mappings from a global heap object is done by `parse_vds_mappings()` in `data_layout.rs:70–155`. Writing the inverse — serializing a `Vec<VdsMapping>` into the same binary layout — does not exist. This task creates `serialize_vds_mappings()`.
**Files:**
- Create: `crates/clawhdf5-format/src/data_layout_write.rs`
- Modify: `crates/clawhdf5-format/src/lib.rs` (declare module)
**Interfaces:**
- Consumes: `VdsMapping { source_file_name: String, source_dataset_name: String, source_selection: Vec<u8>, virtual_selection: Vec<u8> }` (existing struct from `data_layout.rs`).
- Produces: `pub fn serialize_vds_mappings(mappings: &[VdsMapping], length_size: u8) -> Vec<u8>`
**Binary layout (from `data_layout.rs:72–91` doc comment):**
```
version: u8 (0 = external file, 1 = same-file marker)
nused: length_size bytes (number of mappings)
for each mapping:
if version==0: source_file_name (null-terminated)
else: marker byte (0xFF or similar; same-file means empty filename)
source_dataset_name: null-terminated string
source_selection: length(length_size) + bytes
virtual_selection: length(length_size) + bytes
```
- [ ] **Step 1: Write the failing tests**
Create `crates/clawhdf5-format/src/data_layout_write.rs`:
```rust
//! Write-side helpers for VDS (Virtual Dataset Source) mapping serialization.
use crate::data_layout::{parse_vds_mappings, VdsMapping};
use crate::error::FormatError;
/// Serialize a slice of VDS mappings into the global-heap object byte format.
///
/// The output can be stored directly in a global heap object and referenced
/// from a Data Layout v4 class=3 (Virtual) message.
pub fn serialize_vds_mappings(mappings: &[VdsMapping], length_size: u8) -> Vec<u8> {
let mut buf = Vec::new();
// Determine if all sources are same-file (empty source_file_name)
let has_external = mappings.iter().any(|m| !m.source_file_name.is_empty());
let version: u8 = if has_external { 0 } else { 1 };
buf.push(version);
// nused: number of mappings
write_length(&mut buf, mappings.len() as u64, length_size);
for m in mappings {
if version == 0 {
// External: null-terminated filename
buf.extend_from_slice(m.source_file_name.as_bytes());
buf.push(0);
} else {
// Same-file: marker byte (0x00, which parse_vds_mappings treats as empty)
buf.push(0);
}
// source dataset name: null-terminated
buf.extend_from_slice(m.source_dataset_name.as_bytes());
buf.push(0);
// source selection: length + bytes
write_length(&mut buf, m.source_selection.len() as u64, length_size);
buf.extend_from_slice(&m.source_selection);
// virtual selection: length + bytes
write_length(&mut buf, m.virtual_selection.len() as u64, length_size);
buf.extend_from_slice(&m.virtual_selection);
}
buf
}
fn write_length(buf: &mut Vec<u8>, val: u64, size: u8) {
match size {
2 => buf.extend_from_slice(&(val as u16).to_le_bytes()),
4 => buf.extend_from_slice(&(val as u32).to_le_bytes()),
8 => buf.extend_from_slice(&val.to_le_bytes()),
_ => buf.extend_from_slice(&val.to_le_bytes()),
}
}
#[cfg(test)]
mod tests {
use super::*;
fn all_sel() -> Vec<u8> {
// Minimal H5S ALL selection bytes: type=3 (ALL), version=1, flags=0, unused*4
let mut v = Vec::new();
v.extend_from_slice(&3u32.to_le_bytes()); // type = H5S_SEL_ALL
v.push(1); // version
v.push(0); // flags
v.extend_from_slice(&[0u8; 4]); // unused
v
}
#[test]
fn roundtrip_same_file_two_mappings() {
let sel = all_sel();
let mappings = vec![
VdsMapping {
source_file_name: String::new(),
source_dataset_name: "/src_a".into(),
source_selection: sel.clone(),
virtual_selection: sel.clone(),
},
VdsMapping {
source_file_name: String::new(),
source_dataset_name: "/src_b".into(),
source_selection: sel.clone(),
virtual_selection: sel.clone(),
},
];
let bytes = serialize_vds_mappings(&mappings, 8);
let parsed = parse_vds_mappings(&bytes, 8).unwrap();
assert_eq!(parsed.len(), 2);
assert_eq!(parsed[0].source_dataset_name, "/src_a");
assert_eq!(parsed[1].source_dataset_name, "/src_b");
}
#[test]
fn roundtrip_external_file_mapping() {
let sel = all_sel();
let mappings = vec![VdsMapping {
source_file_name: "source.h5".into(),
source_dataset_name: "/data".into(),
source_selection: sel.clone(),
virtual_selection: sel.clone(),
}];
let bytes = serialize_vds_mappings(&mappings, 8);
let parsed = parse_vds_mappings(&bytes, 8).unwrap();
assert_eq!(parsed.len(), 1);
assert_eq!(parsed[0].source_file_name, "source.h5");
assert_eq!(parsed[0].source_dataset_name, "/data");
}
#[test]
fn empty_mappings_roundtrip() {
let bytes = serialize_vds_mappings(&[], 8);
let parsed = parse_vds_mappings(&bytes, 8).unwrap();
assert!(parsed.is_empty());
}
}
```
- [ ] **Step 2: Run the failing tests**
```bash
cargo test -p clawhdf5-format roundtrip_same_file_two_mappings roundtrip_external_file_mapping 2>&1 | head -20
```
Expected: compile errors (module not declared).
- [ ] **Step 3: Declare module in lib.rs**
In `crates/clawhdf5-format/src/lib.rs`, add:
```rust
pub mod data_layout_write;
```
- [ ] **Step 4: Run tests**
```bash
cargo test -p clawhdf5-format data_layout_write 2>&1
```
Expected: all 3 tests PASS. If `parse_vds_mappings` expects a slightly different format for the version byte or the marker byte, adjust `serialize_vds_mappings` to match what the parser consumes (read `data_layout.rs:92–155` carefully to align).
- [ ] **Step 5: Commit**
```bash
git add crates/clawhdf5-format/src/data_layout_write.rs \
crates/clawhdf5-format/src/lib.rs
git commit -m "feat: add serialize_vds_mappings for writing VDS global-heap objects"
```
---
### Task 3: FileWriter API for virtual datasets
**Background:** This task wires `serialize_vds_mappings()` into the `FileWriter` flow so callers can create a virtual dataset. It adds a new `DatasetBuilder` method and the corresponding serialization of a Data Layout v4 class=3 message.
**Files:**
- Modify: `crates/clawhdf5-format/src/file_writer.rs` (add `with_virtual_sources`)
**Interfaces:**
- Produces: `DatasetBuilder::with_virtual_sources(mappings: Vec<VdsMapping>) -> &mut Self`
**Binary — Data Layout v4 class=3 (Virtual):**
```
version(1)=4 class(1)=3
global_heap_address(offset_size) global_heap_index(4)
```
The global heap object holds the `serialize_vds_mappings()` output. The `global_heap_address` is the address of the global heap collection; `global_heap_index` is the 1-based object index within it. Use index=1 for the first (and only) VDS object.
- [ ] **Step 1: Write the failing test**
In `crates/clawhdf5-format/src/file_writer.rs` `#[cfg(test)]` block, add:
```rust
#[test]
fn virtual_dataset_write_roundtrip() {
use crate::data_layout::DataLayout;
use crate::data_layout::{VdsMapping, parse_vds_mappings};
use crate::message_type::MessageType;
use crate::object_header::ObjectHeader;
use crate::signature::find_signature;
use crate::superblock::Superblock;
// Minimal ALL-selection bytes (same as in data_layout_write tests)
let sel: Vec<u8> = {
let mut v = Vec::new();
v.extend_from_slice(&3u32.to_le_bytes()); // H5S_SEL_ALL
v.push(1); v.push(0);
v.extend_from_slice(&[0u8; 4]);
v
};
let mappings = vec![VdsMapping {
source_file_name: "src.h5".into(),
source_dataset_name: "/raw".into(),
source_selection: sel.clone(),
virtual_selection: sel.clone(),
}];
let mut fw = FileWriter::new();
fw.create_dataset("virtual_ds")
.with_virtual_sources(mappings);
let bytes = fw.finish().unwrap();
// Parse back
let sig = find_signature(&bytes).unwrap();
let sb = Superblock::parse(&bytes, sig).unwrap();
let root_oh = ObjectHeader::parse(
&bytes, sb.root_group_address as usize, sb.offset_size, sb.length_size,
).unwrap();
// Find the dataset via group traversal, then get its DataLayout message
use crate::group_v2::resolve_path_any;
let ds_addr = resolve_path_any(&bytes, &sb, "virtual_ds").unwrap();
let ds_oh = ObjectHeader::parse(
&bytes, ds_addr as usize, sb.offset_size, sb.length_size,
).unwrap();
let dl_msg = ds_oh.messages.iter()
.find(|m| m.msg_type == MessageType::DataLayout)
.expect("DataLayout message missing");
let layout = DataLayout::parse(&dl_msg.data, sb.offset_size, sb.length_size).unwrap();
assert!(
matches!(layout, DataLayout::Virtual { .. }),
"expected Virtual layout, got {layout:?}"
);
}
```
- [ ] **Step 2: Run test to verify it fails**
```bash
cargo test -p clawhdf5-format virtual_dataset_write_roundtrip 2>&1 | head -20
```
Expected: compile error — `with_virtual_sources` not found.
- [ ] **Step 3: Add with_virtual_sources to DatasetBuilder**
Find `DatasetBuilder` in `file_writer.rs`. Add a field `virtual_sources: Option<Vec<VdsMapping>>` and the method:
```rust
use crate::data_layout::VdsMapping;
// In DatasetBuilder struct:
virtual_sources: Option<Vec<VdsMapping>>,
// In DatasetBuilder impl:
pub fn with_virtual_sources(&mut self, mappings: Vec<VdsMapping>) -> &mut Self {
self.virtual_sources = Some(mappings);
self
}
```
- [ ] **Step 4: Serialize the virtual data layout**
In the `DatasetBuilder::build()` or equivalent finish method, add a branch for virtual datasets:
```rust
use crate::data_layout_write::serialize_vds_mappings;
// Where the DataLayout message bytes are generated:
let layout_bytes = if let Some(mappings) = &self.virtual_sources {
// Serialize VDS mappings into a global heap object
let heap_data = serialize_vds_mappings(mappings, length_size);
let (heap_addr, heap_idx) = write_global_heap_object(output_buf, &heap_data);
// Data Layout v4 class=3 (Virtual): version(1)=4, class(1)=3, addr(offset_size), idx(4)
let mut dl = Vec::new();
dl.push(4u8); // version
dl.push(3u8); // class = Virtual
write_offset_val(&mut dl, heap_addr, offset_size);
dl.extend_from_slice(&(heap_idx as u32).to_le_bytes());
dl
} else {
// existing layout code (contiguous/compact/chunked)
build_existing_layout(...)
};
```
Implement `write_global_heap_object` as a helper that appends a minimal global heap collection to the output buffer and returns `(address, object_index)`:
```rust
/// Append a single-object global heap collection to `buf` and return
/// (collection_address, object_index=1).
fn write_global_heap_object(buf: &mut Vec<u8>, data: &[u8]) -> (u64, usize) {
let addr = buf.len() as u64;
// Global Heap Collection header: sig(4) + version(1) + reserved(3) + collection_size(8)
// Object: index(2) + ref_count(2) + reserved(4) + data_size(8) + data + padding
let obj_size = data.len();
let padded = (obj_size + 7) & !7;
let collection_size = 16 + 16 + padded + 8; // header + one obj header + data + sentinel
buf.extend_from_slice(b"GCOL"); // signature
buf.push(1); // version
buf.extend_from_slice(&[0u8; 3]); // reserved
buf.extend_from_slice(&(collection_size as u64).to_le_bytes());
// Object 1
buf.extend_from_slice(&1u16.to_le_bytes()); // index
buf.extend_from_slice(&1u16.to_le_bytes()); // ref_count
buf.extend_from_slice(&[0u8; 4]); // reserved
buf.extend_from_slice(&(obj_size as u64).to_le_bytes());
buf.extend_from_slice(data);
// Pad to 8-byte boundary
let pad = padded - obj_size;
buf.extend_from_slice(&vec![0u8; pad]);
// Sentinel object (index=0)
buf.extend_from_slice(&[0u8; 8]); // index=0 + ref_count + reserved
buf.extend_from_slice(&0u64.to_le_bytes()); // size=0
(addr, 1)
}
```
- [ ] **Step 5: Run the test**
```bash
cargo test -p clawhdf5-format virtual_dataset_write_roundtrip 2>&1
```
Expected: PASS (or iterate on the global heap format until `parse_vds_mappings` reads back the mappings).
- [ ] **Step 6: Run full suite**
```bash
cargo test -p clawhdf5-format 2>&1 | tail -10
```
Expected: all tests pass.
- [ ] **Step 7: Commit**
```bash
git add crates/clawhdf5-format/src/file_writer.rs
git commit -m "feat: add DatasetBuilder::with_virtual_sources for writing VDS data layout"
```
---
### Task 4: Superblock v4 read support
**Background:** `Superblock::parse()` in `superblock.rs:178–183` returns `Err(FormatError::UnsupportedVersion(v))` for any version ≥ 4. Superblock v4 (introduced with HDF5 2.x page-buffering) shares the same 12-byte header as v2/v3 (`sig + version + offset_size + length_size + consistency_flags`) and the same four address fields, but adds a `page_size: u32` field before the trailing checksum.
**Files:**
- Modify: `crates/clawhdf5-format/src/superblock.rs`
**Interfaces:**
- Consumes/produces: `Superblock` struct — add `pub page_size: Option<u32>` field.
- [ ] **Step 1: Add field to Superblock struct**
In `crates/clawhdf5-format/src/superblock.rs`, add to the `Superblock` struct:
```rust
/// Page size for page-buffer mode (v4 only). `None` for v0–v3.
pub page_size: Option<u32>,
```
Update all existing construction sites of `Superblock { ... }` in the file (parse_v0, parse_v1, parse_v2v3) to include `page_size: None`.
- [ ] **Step 2: Write the failing test**
In the `#[cfg(test)]` section of `superblock.rs`, add:
```rust
#[test]
fn parse_v4_with_page_size() {
// Superblock v4 = v2/v3 layout + page_size(4) before checksum.
let mut buf = Vec::new();
buf.extend_from_slice(&crate::signature::HDF5_SIGNATURE);
buf.push(4); // version = 4
buf.push(8); // offset_size
buf.push(8); // length_size
buf.push(0); // consistency_flags
// base_address
buf.extend_from_slice(&0u64.to_le_bytes());
// superblock_extension_address = UNDEF
buf.extend_from_slice(&u64::MAX.to_le_bytes());
// eof_address
buf.extend_from_slice(&512u64.to_le_bytes());
// root_group_address
buf.extend_from_slice(&96u64.to_le_bytes());
// page_size (v4 addition before checksum)
buf.extend_from_slice(&4096u32.to_le_bytes());
// checksum (4 bytes; compute with jenkins_lookup3)
let checksum = crate::checksum::jenkins_lookup3(&buf);
buf.extend_from_slice(&checksum.to_le_bytes());
let sb = Superblock::parse(&buf, 0).unwrap();
assert_eq!(sb.version, 4);
assert_eq!(sb.offset_size, 8);
assert_eq!(sb.eof_address, 512);
assert_eq!(sb.page_size, Some(4096));
}
```
- [ ] **Step 3: Run test to verify it fails**
```bash
cargo test -p clawhdf5-format parse_v4_with_page_size 2>&1 | head -20
```
Expected: `Err(UnsupportedVersion(4))` — the test fails because v4 isn't handled.
- [ ] **Step 4: Add parse_v4 branch**
In `Superblock::parse()`, change:
```rust
2 | 3 => Self::parse_v2v3(d, version),
v => Err(FormatError::UnsupportedVersion(v)),
```
to:
```rust
2 | 3 => Self::parse_v2v3(d, version),
4 => Self::parse_v4(d),
v => Err(FormatError::UnsupportedVersion(v)),
```
Add the implementation:
```rust
fn parse_v4(d: &[u8]) -> Result<Superblock, FormatError> {
// Same as v2/v3 header, then page_size(4), then checksum(4).
ensure_len(d, 12)?;
let offset_size = d[9];
let length_size = d[10];
validate_sizes(offset_size, length_size)?;
let consistency_flags = d[11] as u32;
let os = offset_size as usize;
// 4 addresses + page_size(4) + checksum(4)
let total = 12 + 4 * os + 4 + 4;
ensure_len(d, total)?;
let mut pos = 12;
let base_address = read_offset(d, pos, offset_size)?;
pos += os;
let superblock_extension_address = read_offset(d, pos, offset_size)?;
pos += os;
let eof_address = read_offset(d, pos, offset_size)?;
pos += os;
let root_group_address = read_offset(d, pos, offset_size)?;
pos += os;
let page_size = u32::from_le_bytes([d[pos], d[pos+1], d[pos+2], d[pos+3]]);
pos += 4;
let stored_checksum = u32::from_le_bytes([d[pos], d[pos+1], d[pos+2], d[pos+3]]);
let computed = crate::checksum::jenkins_lookup3(&d[..pos]);
if stored_checksum != computed {
return Err(FormatError::ChecksumMismatch {
expected: stored_checksum,
computed,
});
}
Ok(Superblock {
version: 4,
offset_size,
length_size,
base_address,
eof_address,
root_group_address,
group_leaf_node_k: None,
group_internal_node_k: None,
indexed_storage_internal_node_k: None,
free_space_address: None,
driver_info_address: None,
consistency_flags,
superblock_extension_address: Some(superblock_extension_address),
checksum: Some(stored_checksum),
page_size: Some(page_size),
})
}
```
- [ ] **Step 5: Run the test**
```bash
cargo test -p clawhdf5-format parse_v4_with_page_size 2>&1
```
Expected: PASS (verify the checksum field name matches whatever `FormatError` uses — it may be `ChecksumMismatch { expected, computed }` or similar; find it in `error.rs` and match).
- [ ] **Step 6: Run full suite**
```bash
cargo test -p clawhdf5-format 2>&1 | tail -10
```
Expected: all tests pass.
- [ ] **Step 7: Commit**
```bash
git add crates/clawhdf5-format/src/superblock.rs
git commit -m "feat: parse HDF5 superblock v4 (page-buffer mode) with page_size field"
```
---
### Task 5: Superblock v4 write support
**Background:** The `FileWriter` always writes a v3 superblock (hardcoded in `file_writer.rs:1291–1306`). This task adds an optional `page_size` to `FileWriter` that, when set, emits a v4 superblock.
**Files:**
- Modify: `crates/clawhdf5-format/src/file_writer.rs` (add `page_size` field)
- Modify: `crates/clawhdf5-format/src/superblock.rs` (`Superblock::serialize` for v4)
**Interfaces:**
- Produces: `FileWriter::with_page_size(page_size: u32) -> &mut Self`
- [ ] **Step 1: Write the failing test**
In the `#[cfg(test)]` block of `file_writer.rs`, add:
```rust
#[test]
fn file_writer_v4_superblock() {
use crate::signature::find_signature;
use crate::superblock::Superblock;
let mut fw = FileWriter::new();
fw.with_page_size(4096);
fw.create_dataset("data").with_f64_data(&[1.0, 2.0]);
let bytes = fw.finish().unwrap();
let sig = find_signature(&bytes).unwrap();
let sb = Superblock::parse(&bytes, sig).unwrap();
assert_eq!(sb.version, 4, "expected superblock v4");
assert_eq!(sb.page_size, Some(4096));
}
```
- [ ] **Step 2: Run test to verify it fails**
```bash
cargo test -p clawhdf5-format file_writer_v4_superblock 2>&1 | head -20
```
Expected: compile error — `with_page_size` not found.
- [ ] **Step 3: Add page_size field to FileWriter**
In `FileWriter` struct definition, add `page_size: Option<u32>`.
In `FileWriter::new()`, add `page_size: None`.
Add method:
```rust
pub fn with_page_size(&mut self, page_size: u32) -> &mut Self {
self.page_size = Some(page_size);
self
}
```
- [ ] **Step 4: Update Superblock::serialize for v4**
In `crates/clawhdf5-format/src/superblock.rs`, the `serialize()` method currently hardcodes v2/v3 format. Update it to emit v4 when `self.version == 4` and `self.page_size.is_some()`:
```rust
pub fn serialize(&self) -> Vec<u8> {
let mut buf = Vec::with_capacity(60);
buf.extend_from_slice(&HDF5_SIGNATURE);
buf.push(self.version);
buf.push(self.offset_size);
buf.push(self.length_size);
buf.push(self.consistency_flags as u8);
Self::write_offset(&mut buf, self.base_address, self.offset_size);
let ext_addr = self.superblock_extension_address.unwrap_or(u64::MAX);
Self::write_offset(&mut buf, ext_addr, self.offset_size);
Self::write_offset(&mut buf, self.eof_address, self.offset_size);
Self::write_offset(&mut buf, self.root_group_address, self.offset_size);
if self.version >= 4 {
let ps = self.page_size.unwrap_or(0);
buf.extend_from_slice(&ps.to_le_bytes());
}
let checksum = crate::checksum::jenkins_lookup3(&buf);
buf.extend_from_slice(&checksum.to_le_bytes());
buf
}
```
- [ ] **Step 5: Wire page_size into FileWriter::finish()**
In `file_writer.rs:finish()`, where the `Superblock` is constructed (around line 1291), change:
```rust
let sb = Superblock {
version: if self.page_size.is_some() { 4 } else { 3 },
// ... existing fields ...
page_size: self.page_size,
// ... rest of fields unchanged ...
};
```
- [ ] **Step 6: Run the test**
```bash
cargo test -p clawhdf5-format file_writer_v4_superblock 2>&1
```
Expected: PASS.
- [ ] **Step 7: Run full suite**
```bash
cargo test -p clawhdf5-format 2>&1 | tail -10
```
Expected: all tests pass (v3 serialize() must be byte-identical to before — add a regression test if needed).
- [ ] **Step 8: Commit**
```bash
git add crates/clawhdf5-format/src/file_writer.rs \
crates/clawhdf5-format/src/superblock.rs
git commit -m "feat: write HDF5 superblock v4 when page_size is configured"
```
---
## Verification
```bash
# Run all format tests
cargo test -p clawhdf5-format 2>&1 | tail -10
# Specifically verify new features
cargo test -p clawhdf5-format external_link_write_roundtrip 2>&1
cargo test -p clawhdf5-format data_layout_write 2>&1
cargo test -p clawhdf5-format virtual_dataset_write_roundtrip 2>&1
cargo test -p clawhdf5-format parse_v4_with_page_size 2>&1
cargo test -p clawhdf5-format file_writer_v4_superblock 2>&1
# Regression: v3 superblock still round-trips
cargo test -p clawhdf5-format write_superblock 2>&1
```
docs/superpowers/plans/2026-06-29-mpi-io-vol-backend.md
+757
View File
@@ -0,0 +1,757 @@
# MPI-IO VOL Backend Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Add an `MpiVol` backend to `clawhdf5-io` that implements `VirtualObjectLayer` with `VolCapability::ParallelIO`, enabling collective MPI-IO reads and writes against HDF5 files — the same I/O pattern used by h5bench parallel workloads.
**Architecture:** A new `crates/clawhdf5-io/src/mpi_vol.rs` module implements `VirtualObjectLayer` using the `rsmpi` crate for MPI bindings. Reads distribute file chunks across MPI ranks via `MPI_File_read_at` collective; writes gather chunk contributions from all ranks and commit atomically. The `mpi-io` feature flag keeps MPI an optional dependency — without it, the file does not compile in, maintaining the zero-required-dependency promise.
**Tech Stack:** `rsmpi = "0.8"` (or latest; the safe Rust MPI binding), `mpi-io` feature flag in `clawhdf5-io`.
## Global Constraints
- All changes in `crates/clawhdf5-io/`.
- `mpi-io` feature is disabled by default; `cargo test -p clawhdf5-io` without features must still pass.
- `MpiVol` must not link MPI unless `mpi-io` feature is active.
- Tests that require an actual MPI environment are gated with `#[cfg(feature = "mpi-io")]` and ignored by default CI (no `#[ignore]`; they fail to compile without the feature).
- Run `cargo test -p clawhdf5-io` after every task.
- Run `cargo check -p clawhdf5-io --features mpi-io` to validate the feature-enabled path without needing MPI installed.
---
### Task 1: Add mpi-io feature and MpiVol skeleton
**Files:**
- Modify: `crates/clawhdf5-io/Cargo.toml`
- Create: `crates/clawhdf5-io/src/mpi_vol.rs`
- Modify: `crates/clawhdf5-io/src/lib.rs`
**Interfaces:**
- Produces:
- `pub struct MpiVol` (implements `VirtualObjectLayer`)
- `MpiVol::new(comm: impl Into<MpiComm>) -> Self` — wraps an MPI communicator
- `MpiVol::new_world() -> Self` — convenience for `MPI_COMM_WORLD`
- [ ] **Step 1: Write failing tests**
Create `crates/clawhdf5-io/src/mpi_vol.rs`:
```rust
//! MPI-IO VOL connector for parallel HDF5 reads and writes.
//!
//! Enable with the `mpi-io` feature: `cargo build --features mpi-io`.
//!
//! # Parallelism model
//!
//! All ranks open the same file path. Reads are collective: the root rank
//! dispatches chunk byte ranges; each rank fetches its portion via
//! `MPI_File_read_at`. Writes are collective: each rank submits its chunk
//! contribution; the root commits the merged result atomically.
use crate::vol::{VolCapability, VolError, VirtualObjectLayer};
#[cfg(feature = "mpi-io")]
use mpi::traits::*;
/// Rank within the communicator.
type Rank = i32;
/// MPI-IO Virtual Object Layer connector.
///
/// Wraps an MPI communicator for collective HDF5 file I/O.
pub struct MpiVol {
location: Option<String>,
#[cfg(feature = "mpi-io")]
universe: mpi::environment::Universe,
#[cfg(not(feature = "mpi-io"))]
_placeholder: (),
}
impl MpiVol {
/// Create an `MpiVol` using `MPI_COMM_WORLD`.
///
/// Initializes MPI if not already initialized. Call once per process.
#[cfg(feature = "mpi-io")]
pub fn new_world() -> Result<Self, VolError> {
let universe = mpi::initialize()
.ok_or_else(|| VolError::Unsupported("MPI already finalized or init failed".into()))?;
Ok(Self {
location: None,
universe,
})
}
/// Stub for when the feature is disabled.
#[cfg(not(feature = "mpi-io"))]
pub fn new_world() -> Result<Self, VolError> {
Err(VolError::Unsupported(
"MPI-IO support requires the `mpi-io` feature".into(),
))
}
/// Returns the MPI rank within COMM_WORLD (0-based).
///
/// Returns 0 when MPI is not available.
pub fn rank(&self) -> Rank {
#[cfg(feature = "mpi-io")]
{
self.universe.world().rank()
}
#[cfg(not(feature = "mpi-io"))]
{
0
}
}
/// Returns the total number of MPI processes.
///
/// Returns 1 when MPI is not available.
pub fn size(&self) -> Rank {
#[cfg(feature = "mpi-io")]
{
self.universe.world().size()
}
#[cfg(not(feature = "mpi-io"))]
{
1
}
}
}
impl VirtualObjectLayer for MpiVol {
fn name(&self) -> &str {
"mpi-io"
}
fn capabilities(&self) -> Vec<VolCapability> {
vec![
VolCapability::ReadData,
VolCapability::WriteData,
VolCapability::ListObjects,
VolCapability::ChunkedStorage,
VolCapability::ParallelIO,
]
}
fn open(&mut self, location: &str) -> Result<(), VolError> {
self.location = Some(location.to_string());
Ok(())
}
fn close(&mut self) -> Result<(), VolError> {
self.location = None;
Ok(())
}
fn read_dataset(&self, path: &str) -> Result<Vec<u8>, VolError> {
let _loc = self.location.as_deref().ok_or_else(|| {
VolError::Io(std::io::Error::new(std::io::ErrorKind::NotConnected, "file not open"))
})?;
#[cfg(feature = "mpi-io")]
{
mpi_collective_read(self, _loc, path)
}
#[cfg(not(feature = "mpi-io"))]
{
Err(VolError::Unsupported("mpi-io feature not enabled".into()))
}
}
fn write_dataset(
&mut self,
path: &str,
data: &[u8],
shape: &[u64],
dtype: &str,
) -> Result<(), VolError> {
let _loc = self.location.as_deref().ok_or_else(|| {
VolError::Io(std::io::Error::new(std::io::ErrorKind::NotConnected, "file not open"))
})?;
#[cfg(feature = "mpi-io")]
{
mpi_collective_write(self, _loc, path, data, shape, dtype)
}
#[cfg(not(feature = "mpi-io"))]
{
Err(VolError::Unsupported("mpi-io feature not enabled".into()))
}
}
}
/// Collective read: root reads the file, broadcasts the target dataset to all ranks.
#[cfg(feature = "mpi-io")]
fn mpi_collective_read(vol: &MpiVol, location: &str, path: &str) -> Result<Vec<u8>, VolError> {
use mpi::traits::*;
use clawhdf5_format::{
data_layout::DataLayout,
data_read::read_raw_data_full,
dataspace::Dataspace,
datatype::Datatype,
filter_pipeline::FilterPipeline,
group_v2::resolve_path_any,
message_type::MessageType,
object_header::ObjectHeader,
signature::find_signature,
superblock::Superblock,
};
let world = vol.universe.world();
let rank = world.rank();
// All ranks attempt the read; root broadcasts the result.
// For true MPI-IO, use MPI_File_open + MPI_File_read_at_all here.
let raw_data: Vec<u8>;
let mut len_buf = [0usize; 1];
if rank == 0 {
let bytes = std::fs::read(location)
.map_err(|e| VolError::Io(e))?;
let sig = find_signature(&bytes).map_err(|e| VolError::DataError(e.to_string()))?;
let sb = Superblock::parse(&bytes, sig).map_err(|e| VolError::DataError(e.to_string()))?;
let addr = resolve_path_any(&bytes, &sb, path)
.map_err(|e| VolError::NotFound(format!("{path}: {e}")))?;
let oh = ObjectHeader::parse(&bytes, addr as usize, sb.offset_size, sb.length_size)
.map_err(|e| VolError::DataError(e.to_string()))?;
let dt = oh.messages.iter().find(|m| m.msg_type == MessageType::Datatype)
.ok_or_else(|| VolError::DataError("no datatype".into()))?;
let (datatype, _) = Datatype::parse(&dt.data).map_err(|e| VolError::DataError(e.to_string()))?;
let ds = oh.messages.iter().find(|m| m.msg_type == MessageType::Dataspace)
.ok_or_else(|| VolError::DataError("no dataspace".into()))?;
let dataspace = Dataspace::parse(&ds.data, sb.length_size)
.map_err(|e| VolError::DataError(e.to_string()))?;
let dl = oh.messages.iter().find(|m| m.msg_type == MessageType::DataLayout)
.ok_or_else(|| VolError::DataError("no data layout".into()))?;
let layout = DataLayout::parse(&dl.data, sb.offset_size, sb.length_size)
.map_err(|e| VolError::DataError(e.to_string()))?;
let pipeline = oh.messages.iter()
.find(|m| m.msg_type == MessageType::FilterPipeline)
.and_then(|m| FilterPipeline::parse(&m.data).ok());
raw_data = read_raw_data_full(
&bytes, &layout, &dataspace, &datatype, pipeline.as_ref(),
sb.offset_size, sb.length_size,
).map_err(|e| VolError::DataError(e.to_string()))?;
len_buf[0] = raw_data.len();
} else {
raw_data = Vec::new();
}
// Broadcast length then data
world.process_at_rank(0).broadcast_into(&mut len_buf);
let mut result = vec![0u8; len_buf[0]];
if rank == 0 {
result.copy_from_slice(&raw_data);
}
world.process_at_rank(0).broadcast_into(&mut result);
Ok(result)
}
/// Collective write: rank 0 accumulates all contributions and writes atomically.
///
/// In a real parallel workload each rank provides its own data shard for a
/// different hyperslab. Here we demonstrate the pattern: all ranks send their
/// data to rank 0 which stitches and writes.
#[cfg(feature = "mpi-io")]
fn mpi_collective_write(
vol: &MpiVol,
location: &str,
path: &str,
data: &[u8],
shape: &[u64],
dtype: &str,
) -> Result<(), VolError> {
use mpi::traits::*;
use clawhdf5_format::file_writer::FileWriter as FmtWriter;
let world = vol.universe.world();
let size = world.size() as usize;
// Each rank sends its data length to root
let local_len = data.len();
let mut all_lens = if world.rank() == 0 { vec![0usize; size] } else { Vec::new() };
world.process_at_rank(0).gather_into_root(&local_len, &mut all_lens);
// Gather all data at root
let total: usize = if world.rank() == 0 {
all_lens.iter().sum()
} else {
0
};
// Root collects all contributions and writes
if world.rank() == 0 {
let mut merged = Vec::with_capacity(total);
// Rank 0's own contribution first
merged.extend_from_slice(data);
// Receive from ranks 1..size
for r in 1..size as i32 {
let expected = all_lens[r as usize];
let mut buf = vec![0u8; expected];
world.process_at_rank(r).receive_into(&mut buf);
merged.extend_from_slice(&buf);
}
// Write merged data via FileWriter
let mut fw = FmtWriter::new();
match dtype {
"f64" => {
let values: Vec<f64> = merged.chunks_exact(8)
.map(|c| f64::from_le_bytes(c.try_into().unwrap()))
.collect();
fw.create_dataset(path).with_f64_data(&values);
}
"f32" => {
let values: Vec<f32> = merged.chunks_exact(4)
.map(|c| f32::from_le_bytes(c.try_into().unwrap()))
.collect();
fw.create_dataset(path).with_f32_data(&values);
}
_ => {
return Err(VolError::Unsupported(format!("mpi-io write: unsupported dtype {dtype}")));
}
}
let bytes = fw.finish().map_err(|e| VolError::DataError(e.to_string()))?;
std::fs::write(location, &bytes).map_err(VolError::Io)?;
} else {
// Non-root ranks send their data to root
world.process_at_rank(0).send(data);
}
// Barrier: all ranks wait until root finishes writing
world.barrier();
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn mpi_vol_no_feature_returns_unsupported() {
#[cfg(not(feature = "mpi-io"))]
{
let result = MpiVol::new_world();
assert!(
matches!(result, Err(VolError::Unsupported(_))),
"expected Unsupported error without mpi-io feature"
);
}
#[cfg(feature = "mpi-io")]
{
// With MPI enabled, new_world() may succeed if MPI is installed.
// Just verify it doesn't panic.
let _ = MpiVol::new_world();
}
}
#[test]
fn mpi_vol_capabilities_include_parallel_io() {
// Even without feature, the struct can be inspected via the default stub.
// The capabilities list is compile-time constant so test it directly.
let caps = vec![
VolCapability::ReadData,
VolCapability::WriteData,
VolCapability::ListObjects,
VolCapability::ChunkedStorage,
VolCapability::ParallelIO,
];
assert!(caps.contains(&VolCapability::ParallelIO));
}
#[test]
fn rank_and_size_stub_values() {
#[cfg(not(feature = "mpi-io"))]
{
// The constructor itself returns Err without the feature,
// so we can't instantiate MpiVol here. Verify the error message.
let e = MpiVol::new_world().unwrap_err();
assert!(e.to_string().contains("mpi-io"));
}
}
}
```
- [ ] **Step 2: Run tests to verify they fail**
```bash
cargo test -p clawhdf5-io mpi_vol 2>&1 | head -20
```
Expected: compile error (module not declared). That's the expected failure.
- [ ] **Step 3: Add Cargo.toml feature and rsmpi dependency**
In `crates/clawhdf5-io/Cargo.toml`, add to `[dependencies]`:
```toml
mpi = { version = "0.8", optional = true }
```
Add to `[features]`:
```toml
mpi-io = ["mpi"]
```
- [ ] **Step 4: Declare module in lib.rs**
In `crates/clawhdf5-io/src/lib.rs`, add:
```rust
pub mod mpi_vol;
pub use mpi_vol::MpiVol;
```
- [ ] **Step 5: Run tests without mpi-io feature**
```bash
cargo test -p clawhdf5-io 2>&1 | tail -15
```
Expected: `mpi_vol_no_feature_returns_unsupported` and `mpi_vol_capabilities_include_parallel_io` PASS.
- [ ] **Step 6: Check compilation with mpi-io feature (requires MPI headers)**
```bash
# Install MPI if needed: sudo apt install libopenmpi-dev
cargo check -p clawhdf5-io --features mpi-io 2>&1 | tail -20
```
Expected: clean compile (warnings OK; errors not OK).
- [ ] **Step 7: Commit**
```bash
git add crates/clawhdf5-io/Cargo.toml \
crates/clawhdf5-io/src/mpi_vol.rs \
crates/clawhdf5-io/src/lib.rs
git commit -m "feat: add MpiVol VOL backend with collective MPI-IO (mpi-io feature)"
```
---
### Task 2: MPI-IO collective read integration test
**Background:** This test requires an MPI runtime (`mpirun`). It is gated by the `mpi-io` feature and validates that all MPI ranks receive identical data after a collective read.
**Files:**
- Modify: `crates/clawhdf5-io/src/mpi_vol.rs` (add integration test)
- [ ] **Step 1: Add the integration test**
Inside the `#[cfg(test)]` block, add:
```rust
#[test]
#[cfg(feature = "mpi-io")]
fn collective_read_all_ranks_get_same_data() {
use crate::vol::VirtualObjectLayer;
use tempfile::TempDir;
// Write a reference file using FileWriter (no MPI needed)
let tmp = TempDir::new().unwrap();
let path = tmp.path().join("test.h5");
{
use clawhdf5_format::file_writer::FileWriter as FmtWriter;
let mut fw = FmtWriter::new();
fw.create_dataset("temperature")
.with_f64_data(&[1.0, 2.0, 3.0, 4.0, 5.0]);
let bytes = fw.finish().unwrap();
std::fs::write(&path, &bytes).unwrap();
}
// Each rank reads via MpiVol and should get the same bytes
let mut vol = MpiVol::new_world().expect("MPI init failed");
vol.open(path.to_str().unwrap()).unwrap();
let data = vol.read_dataset("temperature").unwrap();
// 5 f64 values = 40 bytes
assert_eq!(data.len(), 40, "rank {} got {} bytes", vol.rank(), data.len());
let values: Vec<f64> = data.chunks_exact(8)
.map(|c| f64::from_le_bytes(c.try_into().unwrap()))
.collect();
assert_eq!(values, vec![1.0, 2.0, 3.0, 4.0, 5.0],
"rank {} got wrong data", vol.rank());
}
```
Add to `Cargo.toml` dev-dependencies:
```toml
tempfile = "3"
```
- [ ] **Step 2: Run without MPI feature (should compile-skip)**
```bash
cargo test -p clawhdf5-io 2>&1 | tail -10
```
Expected: all tests pass; `collective_read_all_ranks_get_same_data` is not compiled.
- [ ] **Step 3: Run with MPI feature (requires mpirun)**
```bash
# Requires: sudo apt install libopenmpi-dev openmpi-bin
# cargo test compiles, then:
mpirun -np 4 cargo test -p clawhdf5-io --features mpi-io collective_read_all_ranks_get_same_data 2>&1
```
Expected: all 4 ranks PASS.
- [ ] **Step 4: Commit**
```bash
git add crates/clawhdf5-io/src/mpi_vol.rs \
crates/clawhdf5-io/Cargo.toml
git commit -m "feat: add MpiVol collective read integration test"
```
---
### Task 3: MPI-IO collective write integration test
**Background:** Validates that N ranks each contribute a shard of a dataset; rank 0 assembles and writes the complete file.
**Files:**
- Modify: `crates/clawhdf5-io/src/mpi_vol.rs`
- [ ] **Step 1: Add the integration test**
```rust
#[test]
#[cfg(feature = "mpi-io")]
fn collective_write_assembles_all_shards() {
use crate::vol::VirtualObjectLayer;
use tempfile::TempDir;
use mpi::traits::*;
let tmp = TempDir::new().unwrap();
let path = tmp.path().join("parallel_out.h5");
let mut vol = MpiVol::new_world().expect("MPI init failed");
vol.open(path.to_str().unwrap()).unwrap();
let world = vol.universe.world();
let rank = world.rank() as usize;
// Each rank contributes one f64 value: rank * 10.0
let shard = ((rank as f64) * 10.0f64).to_le_bytes().to_vec();
vol.write_dataset("values", &shard, &[world.size() as u64], "f64")
.unwrap();
// All ranks verify the written file has 4 values (one per rank)
let total_size = world.size() as usize;
if rank == 0 {
let bytes = std::fs::read(&path).unwrap();
use clawhdf5_format::{
data_layout::DataLayout, data_read::read_raw_data_full,
dataspace::Dataspace, datatype::Datatype,
group_v2::resolve_path_any, message_type::MessageType,
object_header::ObjectHeader, signature::find_signature,
superblock::Superblock,
};
let sig = find_signature(&bytes).unwrap();
let sb = Superblock::parse(&bytes, sig).unwrap();
let addr = resolve_path_any(&bytes, &sb, "values").unwrap();
let oh = ObjectHeader::parse(
&bytes, addr as usize, sb.offset_size, sb.length_size,
).unwrap();
let (dt, _) = Datatype::parse(
&oh.messages.iter().find(|m| m.msg_type == MessageType::Datatype).unwrap().data,
).unwrap();
let ds = Dataspace::parse(
&oh.messages.iter().find(|m| m.msg_type == MessageType::Dataspace).unwrap().data,
sb.length_size,
).unwrap();
let dl = DataLayout::parse(
&oh.messages.iter().find(|m| m.msg_type == MessageType::DataLayout).unwrap().data,
sb.offset_size, sb.length_size,
).unwrap();
let raw = read_raw_data_full(
&bytes, &dl, &ds, &dt, None, sb.offset_size, sb.length_size,
).unwrap();
assert_eq!(raw.len(), total_size * 8, "expected {} f64 values", total_size);
let values: Vec<f64> = raw.chunks_exact(8)
.map(|c| f64::from_le_bytes(c.try_into().unwrap()))
.collect();
for (i, &v) in values.iter().enumerate() {
assert!((v - (i as f64 * 10.0)).abs() < 1e-9,
"rank {i} shard wrong: got {v}");
}
}
world.barrier();
}
```
- [ ] **Step 2: Run**
```bash
cargo test -p clawhdf5-io 2>&1 | tail -5 # no feature — should pass
mpirun -np 4 cargo test -p clawhdf5-io --features mpi-io collective_write 2>&1
```
- [ ] **Step 3: Commit**
```bash
git add crates/clawhdf5-io/src/mpi_vol.rs
git commit -m "feat: add MpiVol collective write integration test (4 ranks)"
```
---
### Task 4: MpiVol parallel benchmark binary
**Background:** Adds a benchmark binary to `clawhdf5-bench` that runs h5bench-equivalent write/read workloads using `MpiVol`. This provides the throughput numbers needed to compare clawhdf5 against standard libhdf5 + h5bench.
**Files:**
- Create: `crates/clawhdf5-bench/src/bin/mpi_io_bench.rs`
- Modify: `crates/clawhdf5-bench/Cargo.toml` (add `mpi-io` feature, `mpi_io_bench` binary)
**Produces:** `cargo run -p clawhdf5-bench --features mpi-io --bin mpi_io_bench -- --size 100000` outputs MB/s throughput numbers comparable to h5bench output.
- [ ] **Step 1: Create the binary**
Create `crates/clawhdf5-bench/src/bin/mpi_io_bench.rs`:
```rust
//! h5bench-equivalent MPI-IO performance benchmark.
//!
//! Usage: mpirun -np N cargo run -p clawhdf5-bench --features mpi-io --bin mpi_io_bench -- --size <N>
//!
//! Measures collective write and read throughput in MB/s for f64 arrays.
#[cfg(feature = "mpi-io")]
fn main() {
use clawhdf5_io::mpi_vol::MpiVol;
use clawhdf5_io::vol::VirtualObjectLayer;
use std::time::Instant;
use mpi::traits::*;
let args: Vec<String> = std::env::args().collect();
let n_elements: usize = args.iter()
.position(|a| a == "--size")
.and_then(|i| args.get(i + 1))
.and_then(|s| s.parse().ok())
.unwrap_or(100_000);
let mut vol = MpiVol::new_world().expect("MPI init failed");
let world = vol.universe.world();
let rank = world.rank() as usize;
let size = world.size() as usize;
let path = format!("/tmp/clawhdf5_mpiio_bench_{n_elements}.h5");
vol.open(&path).unwrap();
// Each rank contributes n_elements/size f64 values
let per_rank = n_elements / size;
let shard: Vec<f64> = (0..per_rank).map(|i| (rank * per_rank + i) as f64).collect();
let shard_bytes: Vec<u8> = shard.iter().flat_map(|v| v.to_le_bytes()).collect();
// Collective write
world.barrier();
let t0 = Instant::now();
vol.write_dataset("data", &shard_bytes, &[n_elements as u64], "f64").unwrap();
world.barrier();
let write_elapsed = t0.elapsed().as_secs_f64();
// Collective read
let t1 = Instant::now();
let _data = vol.read_dataset("data").unwrap();
world.barrier();
let read_elapsed = t1.elapsed().as_secs_f64();
if rank == 0 {
let total_mb = (n_elements * 8) as f64 / 1e6;
println!("=== clawhdf5 MPI-IO Benchmark ===");
println!("Elements : {n_elements}");
println!("Ranks : {size}");
println!("Total : {total_mb:.1} MB");
println!("Write : {:.1} MB/s", total_mb / write_elapsed);
println!("Read : {:.1} MB/s", total_mb / read_elapsed);
}
}
#[cfg(not(feature = "mpi-io"))]
fn main() {
eprintln!("mpi_io_bench requires the `mpi-io` feature.");
eprintln!("Run: mpirun -np N cargo run -p clawhdf5-bench --features mpi-io --bin mpi_io_bench");
std::process::exit(1);
}
```
- [ ] **Step 2: Add to Cargo.toml**
In `crates/clawhdf5-bench/Cargo.toml`, add:
```toml
[dependencies]
clawhdf5-io = { path = "../clawhdf5-io", features = [] }
[features]
mpi-io = ["clawhdf5-io/mpi-io", "mpi"]
[dependencies.mpi]
version = "0.8"
optional = true
[[bin]]
name = "mpi_io_bench"
path = "src/bin/mpi_io_bench.rs"
```
- [ ] **Step 3: Verify it compiles**
```bash
cargo check -p clawhdf5-bench --features mpi-io 2>&1 | tail -10
```
Expected: no errors.
- [ ] **Step 4: Run with 4 ranks**
```bash
mpirun -np 4 cargo run --release -p clawhdf5-bench --features mpi-io --bin mpi_io_bench -- --size 1000000 2>&1
```
Expected output (numbers will vary by hardware):
```
=== clawhdf5 MPI-IO Benchmark ===
Elements : 1000000
Ranks : 4
Total : 8.0 MB
Write : xxx.x MB/s
Read : xxx.x MB/s
```
Record results in `BENCHMARKS.md` under a new `## MPI-IO Parallel I/O` section.
- [ ] **Step 5: Commit**
```bash
git add crates/clawhdf5-bench/src/bin/mpi_io_bench.rs \
crates/clawhdf5-bench/Cargo.toml
git commit -m "feat: add mpi_io_bench binary for h5bench-comparable parallel I/O throughput"
```
---
## Verification
```bash
# Without MPI feature — all existing tests still pass
cargo test -p clawhdf5-io 2>&1 | tail -10
# With MPI feature — compile check (requires libopenmpi-dev)
cargo check -p clawhdf5-io --features mpi-io 2>&1 | tail -5
# Integration tests (requires openmpi-bin)
mpirun -np 4 cargo test -p clawhdf5-io --features mpi-io 2>&1 | tail -20
# Benchmark (requires openmpi-bin)
mpirun -np 4 cargo run --release -p clawhdf5-bench --features mpi-io --bin mpi_io_bench -- --size 1000000 2>&1
```