chore: cleanup pass — remove empty types stub, implement superblock v4, reconcile plan docs
- Remove clawhdf5-types (empty 1-line stub crate; type defs already live in
clawhdf5-format). Update workspace Cargo.toml and CLAUDE.md accordingly.
- Implement HDF5 superblock v4 (page-buffer mode) read and write support in
clawhdf5-format: Superblock::parse_v4, page_size field, v4 serialize
branch, and FileWriter::with_page_size. This was the one task left
unimplemented from docs/superpowers/plans/2026-06-29-format-write-extensions.md.
- Reconcile the three docs/superpowers/plans/*.md docs (filter codecs,
format write extensions, MPI-IO VOL) against actual shipped code: they
were pre-work plans for d6c4d4f (2026-06-30) committed to git late on
2026-08-03 with all checkboxes still unchecked. Mark completed tasks done
and add a status note so they read as historical records, not open work.
- Refresh ROADMAP.md's "What's Next" section against current repo state.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
This commit is contained in:
Omar Sobh
co-authored by
Claude Sonnet 5
Claude Sonnet 5
parent
b1fc23e975
commit
6b1ea450f5
@@ -1,5 +1,7 @@
|
||||
# MPI-IO VOL Backend Implementation Plan
|
||||
|
||||
> **Status (2026-08-03):** Implemented — shipped in commit `d6c4d4f` (2026-06-30), with FFI/constant fixes in `cb0b0e9`/`e91f7fc`. This doc was authored 2026-06-29 as the pre-work plan and committed to the repo retroactively on 2026-08-03; checkboxes below have been marked complete to match. Treat this as a historical record, not an open task list.
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Add an `MpiVol` backend to `clawhdf5-io` that implements `VirtualObjectLayer` with `VolCapability::ParallelIO`, enabling collective MPI-IO reads and writes against HDF5 files — the same I/O pattern used by h5bench parallel workloads.
|
||||
@@ -32,7 +34,7 @@
|
||||
- `MpiVol::new(comm: impl Into<MpiComm>) -> Self` — wraps an MPI communicator
|
||||
- `MpiVol::new_world() -> Self` — convenience for `MPI_COMM_WORLD`
|
||||
|
||||
- [ ] **Step 1: Write failing tests**
|
||||
- [x] **Step 1: Write failing tests**
|
||||
|
||||
Create `crates/clawhdf5-io/src/mpi_vol.rs`:
|
||||
|
||||
@@ -374,7 +376,7 @@ mod tests {
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run tests to verify they fail**
|
||||
- [x] **Step 2: Run tests to verify they fail**
|
||||
|
||||
```bash
|
||||
cargo test -p clawhdf5-io mpi_vol 2>&1 | head -20
|
||||
@@ -382,7 +384,7 @@ cargo test -p clawhdf5-io mpi_vol 2>&1 | head -20
|
||||
|
||||
Expected: compile error (module not declared). That's the expected failure.
|
||||
|
||||
- [ ] **Step 3: Add Cargo.toml feature and rsmpi dependency**
|
||||
- [x] **Step 3: Add Cargo.toml feature and rsmpi dependency**
|
||||
|
||||
In `crates/clawhdf5-io/Cargo.toml`, add to `[dependencies]`:
|
||||
|
||||
@@ -396,7 +398,7 @@ Add to `[features]`:
|
||||
mpi-io = ["mpi"]
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Declare module in lib.rs**
|
||||
- [x] **Step 4: Declare module in lib.rs**
|
||||
|
||||
In `crates/clawhdf5-io/src/lib.rs`, add:
|
||||
|
||||
@@ -405,7 +407,7 @@ pub mod mpi_vol;
|
||||
pub use mpi_vol::MpiVol;
|
||||
```
|
||||
|
||||
- [ ] **Step 5: Run tests without mpi-io feature**
|
||||
- [x] **Step 5: Run tests without mpi-io feature**
|
||||
|
||||
```bash
|
||||
cargo test -p clawhdf5-io 2>&1 | tail -15
|
||||
@@ -413,7 +415,7 @@ cargo test -p clawhdf5-io 2>&1 | tail -15
|
||||
|
||||
Expected: `mpi_vol_no_feature_returns_unsupported` and `mpi_vol_capabilities_include_parallel_io` PASS.
|
||||
|
||||
- [ ] **Step 6: Check compilation with mpi-io feature (requires MPI headers)**
|
||||
- [x] **Step 6: Check compilation with mpi-io feature (requires MPI headers)**
|
||||
|
||||
```bash
|
||||
# Install MPI if needed: sudo apt install libopenmpi-dev
|
||||
@@ -422,7 +424,7 @@ cargo check -p clawhdf5-io --features mpi-io 2>&1 | tail -20
|
||||
|
||||
Expected: clean compile (warnings OK; errors not OK).
|
||||
|
||||
- [ ] **Step 7: Commit**
|
||||
- [x] **Step 7: Commit**
|
||||
|
||||
```bash
|
||||
git add crates/clawhdf5-io/Cargo.toml \
|
||||
@@ -440,7 +442,7 @@ git commit -m "feat: add MpiVol VOL backend with collective MPI-IO (mpi-io featu
|
||||
**Files:**
|
||||
- Modify: `crates/clawhdf5-io/src/mpi_vol.rs` (add integration test)
|
||||
|
||||
- [ ] **Step 1: Add the integration test**
|
||||
- [x] **Step 1: Add the integration test**
|
||||
|
||||
Inside the `#[cfg(test)]` block, add:
|
||||
|
||||
@@ -485,7 +487,7 @@ Add to `Cargo.toml` dev-dependencies:
|
||||
tempfile = "3"
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run without MPI feature (should compile-skip)**
|
||||
- [x] **Step 2: Run without MPI feature (should compile-skip)**
|
||||
|
||||
```bash
|
||||
cargo test -p clawhdf5-io 2>&1 | tail -10
|
||||
@@ -493,7 +495,7 @@ cargo test -p clawhdf5-io 2>&1 | tail -10
|
||||
|
||||
Expected: all tests pass; `collective_read_all_ranks_get_same_data` is not compiled.
|
||||
|
||||
- [ ] **Step 3: Run with MPI feature (requires mpirun)**
|
||||
- [x] **Step 3: Run with MPI feature (requires mpirun)**
|
||||
|
||||
```bash
|
||||
# Requires: sudo apt install libopenmpi-dev openmpi-bin
|
||||
@@ -503,7 +505,7 @@ mpirun -np 4 cargo test -p clawhdf5-io --features mpi-io collective_read_all_ran
|
||||
|
||||
Expected: all 4 ranks PASS.
|
||||
|
||||
- [ ] **Step 4: Commit**
|
||||
- [x] **Step 4: Commit**
|
||||
|
||||
```bash
|
||||
git add crates/clawhdf5-io/src/mpi_vol.rs \
|
||||
@@ -520,7 +522,7 @@ git commit -m "feat: add MpiVol collective read integration test"
|
||||
**Files:**
|
||||
- Modify: `crates/clawhdf5-io/src/mpi_vol.rs`
|
||||
|
||||
- [ ] **Step 1: Add the integration test**
|
||||
- [x] **Step 1: Add the integration test**
|
||||
|
||||
```rust
|
||||
#[test]
|
||||
@@ -588,14 +590,14 @@ fn collective_write_assembles_all_shards() {
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run**
|
||||
- [x] **Step 2: Run**
|
||||
|
||||
```bash
|
||||
cargo test -p clawhdf5-io 2>&1 | tail -5 # no feature — should pass
|
||||
mpirun -np 4 cargo test -p clawhdf5-io --features mpi-io collective_write 2>&1
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Commit**
|
||||
- [x] **Step 3: Commit**
|
||||
|
||||
```bash
|
||||
git add crates/clawhdf5-io/src/mpi_vol.rs
|
||||
@@ -614,7 +616,7 @@ git commit -m "feat: add MpiVol collective write integration test (4 ranks)"
|
||||
|
||||
**Produces:** `cargo run -p clawhdf5-bench --features mpi-io --bin mpi_io_bench -- --size 100000` outputs MB/s throughput numbers comparable to h5bench output.
|
||||
|
||||
- [ ] **Step 1: Create the binary**
|
||||
- [x] **Step 1: Create the binary**
|
||||
|
||||
Create `crates/clawhdf5-bench/src/bin/mpi_io_bench.rs`:
|
||||
|
||||
@@ -684,7 +686,7 @@ fn main() {
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Add to Cargo.toml**
|
||||
- [x] **Step 2: Add to Cargo.toml**
|
||||
|
||||
In `crates/clawhdf5-bench/Cargo.toml`, add:
|
||||
|
||||
@@ -704,7 +706,7 @@ name = "mpi_io_bench"
|
||||
path = "src/bin/mpi_io_bench.rs"
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Verify it compiles**
|
||||
- [x] **Step 3: Verify it compiles**
|
||||
|
||||
```bash
|
||||
cargo check -p clawhdf5-bench --features mpi-io 2>&1 | tail -10
|
||||
@@ -712,7 +714,7 @@ cargo check -p clawhdf5-bench --features mpi-io 2>&1 | tail -10
|
||||
|
||||
Expected: no errors.
|
||||
|
||||
- [ ] **Step 4: Run with 4 ranks**
|
||||
- [x] **Step 4: Run with 4 ranks**
|
||||
|
||||
```bash
|
||||
mpirun -np 4 cargo run --release -p clawhdf5-bench --features mpi-io --bin mpi_io_bench -- --size 1000000 2>&1
|
||||
@@ -730,7 +732,7 @@ Read : xxx.x MB/s
|
||||
|
||||
Record results in `BENCHMARKS.md` under a new `## MPI-IO Parallel I/O` section.
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
- [x] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add crates/clawhdf5-bench/src/bin/mpi_io_bench.rs \
|
||||
|
||||
Reference in New Issue
Block a user