docs: archive the improvement log/scan and the June superpowers plans
Moved to docs/archive/ (kept for their history, not deleted), each with a
one-line header saying it is historical and what supersedes it:
- IMPROVEMENT_LOG.md: three automated-loop PRs from April-May 2026, on
the earlier quantumclaw PR numbering, which now collides with this
repo's #12-#15. Superseded by CHANGELOG.md and git log.
- IMPROVEMENT_SCAN.md: one scan's notes (2026-05-04) of changes merged
long ago. Superseded by CHANGELOG.md and git log.
- docs/superpowers/plans/*.md -> docs/archive/plans/: agent pre-work
plans for d6c4d4f (2026-06-30), already marked implemented. The MPI plan
promised collective MPI-IO, which is not what shipped (MpiVol is
root-read + broadcast); its header says so and points at ROADMAP.md,
where collective I/O is an open item.
Nothing links to the old paths (research/*.md mention IMPROVEMENT_LOG.md
and ROADMAP.md in prose only).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
@@ -0,0 +1,848 @@
|
||||
> **Historical (archived 2026-09-28):** a pre-work plan, implemented in `d6c4d4f` (2026-06-30) and on 2026-08-03. Superseded by the code (`crates/clawhdf5-format`), [`CHANGELOG.md`](../../../CHANGELOG.md) and [`ROADMAP.md`](../../../ROADMAP.md); not an open task list.
|
||||
|
||||
# Format Write Extensions Implementation Plan
|
||||
|
||||
> **Status (2026-08-03):** Implemented. Tasks 1–3 (external links, VDS mapping serialization, VDS `FileWriter` API) shipped in commit `d6c4d4f` (2026-06-30). Tasks 4–5 (superblock v4 read/write) were not part of that commit and were completed separately as part of this cleanup pass (2026-08-03) — see `Superblock::parse_v4`/`serialize` and `FileWriter::with_page_size` in `crates/clawhdf5-format`. This doc was authored 2026-06-29 as the pre-work plan and committed to the repo retroactively; checkboxes below have been marked complete to match current state. Treat this as a historical record, not an open task list.
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Add three write-side features to clawhdf5-format: (1) external link creation via `GroupBuilder`, (2) external VDS (Virtual Dataset Source) layout writes, and (3) superblock v4 read/write for page-buffering-aware files.
|
||||
|
||||
**Architecture:** External links reuse the existing `LinkMessage::serialize()` which already handles `LinkTarget::External` — only the `GroupBuilder` API needs wiring up. External VDS adds `write_vds_layout()` in `file_writer.rs` and `serialize_vds_mappings()` in a new `data_layout_write.rs`. Superblock v4 extends `Superblock::parse` with a new `parse_v4` branch (identical structure to v3 with an extra `page_size` field) and updates `Superblock::serialize` to optionally write v4.
|
||||
|
||||
**Tech Stack:** Pure Rust, no new dependencies. All changes in `crates/clawhdf5-format/`.
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- All code in `crates/clawhdf5-format/`.
|
||||
- No new Cargo dependencies.
|
||||
- External links: written as `LinkTarget::External`, readable by h5py (verified in tests).
|
||||
- VDS: uses data layout version 4, class 3. Global heap at end of file.
|
||||
- Superblock v4: only adds `page_size: u32` field after the v2/v3 body; checksum placement unchanged.
|
||||
- Run `cargo test -p clawhdf5-format` after every task.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: External link write API in GroupBuilder
|
||||
|
||||
**Background:** `LinkMessage::serialize()` in `link_message.rs:76–175` already handles `LinkTarget::External { filename, object_path }` (writes link_type byte = 64, then packed filename+path). What's missing is a public API in `file_writer.rs` to create external links from a `GroupBuilder`. Currently `GroupBuilder` only creates datasets and sub-groups via `create_dataset` / `create_group`.
|
||||
|
||||
**Files:**
|
||||
- Modify: `crates/clawhdf5-format/src/file_writer.rs` (add `GroupBuilder::add_external_link`)
|
||||
- Modify: `crates/clawhdf5-format/src/lib.rs` (re-export `LinkTarget` if not already exported)
|
||||
- Test: `crates/clawhdf5-format/src/file_writer.rs` (new test in `#[cfg(test)]`)
|
||||
|
||||
**Interfaces:**
|
||||
- Produces: `GroupBuilder::add_external_link(&mut self, name: &str, target_file: &str, target_path: &str) -> &mut Self`
|
||||
|
||||
- [x] **Step 1: Write the failing test**
|
||||
|
||||
At the bottom of the `#[cfg(test)]` block in `crates/clawhdf5-format/src/file_writer.rs`, add:
|
||||
|
||||
```rust
|
||||
#[test]
|
||||
fn external_link_write_roundtrip() {
|
||||
use crate::group_v2::resolve_path_any;
|
||||
use crate::link_message::{LinkMessage, LinkTarget};
|
||||
use crate::message_type::MessageType;
|
||||
use crate::object_header::ObjectHeader;
|
||||
use crate::signature::find_signature;
|
||||
use crate::superblock::Superblock;
|
||||
|
||||
let mut fw = FileWriter::new();
|
||||
let mut grp = fw.create_group("links");
|
||||
grp.add_external_link("remote_data", "other_file.h5", "/sensors/temp");
|
||||
fw.add_group(grp.finish());
|
||||
let bytes = fw.finish().unwrap();
|
||||
|
||||
let sig = find_signature(&bytes).unwrap();
|
||||
let sb = Superblock::parse(&bytes, sig).unwrap();
|
||||
|
||||
// Navigate to /links group
|
||||
let links_addr = resolve_path_any(&bytes, &sb, "links").unwrap();
|
||||
let links_oh = ObjectHeader::parse(
|
||||
&bytes, links_addr as usize, sb.offset_size, sb.length_size,
|
||||
).unwrap();
|
||||
|
||||
// Find the link message for "remote_data"
|
||||
let link_msg = links_oh.messages.iter()
|
||||
.filter(|m| m.msg_type == MessageType::Link)
|
||||
.find_map(|m| {
|
||||
let lm = LinkMessage::parse(&m.data, sb.offset_size).ok()?;
|
||||
if lm.name == "remote_data" { Some(lm) } else { None }
|
||||
})
|
||||
.expect("external link message not found");
|
||||
|
||||
assert_eq!(
|
||||
link_msg.link_target,
|
||||
LinkTarget::External {
|
||||
filename: "other_file.h5".into(),
|
||||
object_path: "/sensors/temp".into(),
|
||||
}
|
||||
);
|
||||
}
|
||||
```
|
||||
|
||||
- [x] **Step 2: Run test to verify it fails**
|
||||
|
||||
```bash
|
||||
cargo test -p clawhdf5-format external_link_write_roundtrip 2>&1 | head -20
|
||||
```
|
||||
|
||||
Expected: compile error — `add_external_link` not found.
|
||||
|
||||
- [x] **Step 3: Find GroupBuilder in file_writer.rs and add the method**
|
||||
|
||||
Locate `GroupBuilder` in `crates/clawhdf5-format/src/file_writer.rs`. It tracks its items as a `Vec` of internal builders. Add a field for external links and the method:
|
||||
|
||||
First, locate the `GroupBuilder` struct definition and add a field:
|
||||
|
||||
```rust
|
||||
pub struct GroupBuilder {
|
||||
name: String,
|
||||
datasets: Vec<DatasetBuilder>,
|
||||
groups: Vec<FinishedGroup>,
|
||||
external_links: Vec<(String, String, String)>, // (name, filename, object_path)
|
||||
}
|
||||
```
|
||||
|
||||
Update `GroupBuilder::new()` (or equivalent constructor) to initialize `external_links: Vec::new()`.
|
||||
|
||||
Add the public method immediately after the existing `create_dataset`/`create_group` methods:
|
||||
|
||||
```rust
|
||||
/// Add a link in this group that points to an object in another HDF5 file.
|
||||
///
|
||||
/// `name` is the link name within this group.
|
||||
/// `target_file` is the relative or absolute path to the target .h5 file.
|
||||
/// `target_path` is the HDF5 path of the object within the target file.
|
||||
pub fn add_external_link(
|
||||
&mut self,
|
||||
name: &str,
|
||||
target_file: &str,
|
||||
target_path: &str,
|
||||
) -> &mut Self {
|
||||
self.external_links.push((
|
||||
name.to_string(),
|
||||
target_file.to_string(),
|
||||
target_path.to_string(),
|
||||
));
|
||||
self
|
||||
}
|
||||
```
|
||||
|
||||
- [x] **Step 4: Wire external links into the group serialization**
|
||||
|
||||
Find where the `GroupBuilder` emits `LinkMessage` bytes during `finish()` / `build_group()`. For each external link, emit a `LinkMessage` with `LinkTarget::External`:
|
||||
|
||||
```rust
|
||||
use crate::link_message::{LinkMessage, LinkTarget};
|
||||
use crate::datatype::CharacterSet;
|
||||
|
||||
// Inside the loop/block that serializes links:
|
||||
for (link_name, filename, object_path) in &self.external_links {
|
||||
let msg = LinkMessage {
|
||||
name: link_name.clone(),
|
||||
link_target: LinkTarget::External {
|
||||
filename: filename.clone(),
|
||||
object_path: object_path.clone(),
|
||||
},
|
||||
creation_order: None,
|
||||
charset: CharacterSet::Utf8,
|
||||
};
|
||||
let msg_bytes = msg.serialize(offset_size);
|
||||
// Emit as a Link message (MessageType::Link = 0x0006) into the object header
|
||||
emit_message(&mut oh_buf, MessageType::Link, &msg_bytes);
|
||||
}
|
||||
```
|
||||
|
||||
(Follow the exact pattern used for hard links and soft links in the same codebase — find where hard-link `LinkMessage` bytes are pushed and add the external links in the same loop.)
|
||||
|
||||
- [x] **Step 5: Run the failing test**
|
||||
|
||||
```bash
|
||||
cargo test -p clawhdf5-format external_link_write_roundtrip 2>&1
|
||||
```
|
||||
|
||||
Expected: PASS.
|
||||
|
||||
- [x] **Step 6: Run full suite**
|
||||
|
||||
```bash
|
||||
cargo test -p clawhdf5-format 2>&1 | tail -10
|
||||
```
|
||||
|
||||
Expected: all tests pass.
|
||||
|
||||
- [x] **Step 7: Commit**
|
||||
|
||||
```bash
|
||||
git add crates/clawhdf5-format/src/file_writer.rs
|
||||
git commit -m "feat: add GroupBuilder::add_external_link for writing cross-file HDF5 links"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 2: VDS mapping serialization helper
|
||||
|
||||
**Background:** Reading VDS mappings from a global heap object is done by `parse_vds_mappings()` in `data_layout.rs:70–155`. Writing the inverse — serializing a `Vec<VdsMapping>` into the same binary layout — does not exist. This task creates `serialize_vds_mappings()`.
|
||||
|
||||
**Files:**
|
||||
- Create: `crates/clawhdf5-format/src/data_layout_write.rs`
|
||||
- Modify: `crates/clawhdf5-format/src/lib.rs` (declare module)
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `VdsMapping { source_file_name: String, source_dataset_name: String, source_selection: Vec<u8>, virtual_selection: Vec<u8> }` (existing struct from `data_layout.rs`).
|
||||
- Produces: `pub fn serialize_vds_mappings(mappings: &[VdsMapping], length_size: u8) -> Vec<u8>`
|
||||
|
||||
**Binary layout (from `data_layout.rs:72–91` doc comment):**
|
||||
```
|
||||
version: u8 (0 = external file, 1 = same-file marker)
|
||||
nused: length_size bytes (number of mappings)
|
||||
for each mapping:
|
||||
if version==0: source_file_name (null-terminated)
|
||||
else: marker byte (0xFF or similar; same-file means empty filename)
|
||||
source_dataset_name: null-terminated string
|
||||
source_selection: length(length_size) + bytes
|
||||
virtual_selection: length(length_size) + bytes
|
||||
```
|
||||
|
||||
- [x] **Step 1: Write the failing tests**
|
||||
|
||||
Create `crates/clawhdf5-format/src/data_layout_write.rs`:
|
||||
|
||||
```rust
|
||||
//! Write-side helpers for VDS (Virtual Dataset Source) mapping serialization.
|
||||
|
||||
use crate::data_layout::{parse_vds_mappings, VdsMapping};
|
||||
use crate::error::FormatError;
|
||||
|
||||
/// Serialize a slice of VDS mappings into the global-heap object byte format.
|
||||
///
|
||||
/// The output can be stored directly in a global heap object and referenced
|
||||
/// from a Data Layout v4 class=3 (Virtual) message.
|
||||
pub fn serialize_vds_mappings(mappings: &[VdsMapping], length_size: u8) -> Vec<u8> {
|
||||
let mut buf = Vec::new();
|
||||
|
||||
// Determine if all sources are same-file (empty source_file_name)
|
||||
let has_external = mappings.iter().any(|m| !m.source_file_name.is_empty());
|
||||
let version: u8 = if has_external { 0 } else { 1 };
|
||||
buf.push(version);
|
||||
|
||||
// nused: number of mappings
|
||||
write_length(&mut buf, mappings.len() as u64, length_size);
|
||||
|
||||
for m in mappings {
|
||||
if version == 0 {
|
||||
// External: null-terminated filename
|
||||
buf.extend_from_slice(m.source_file_name.as_bytes());
|
||||
buf.push(0);
|
||||
} else {
|
||||
// Same-file: marker byte (0x00, which parse_vds_mappings treats as empty)
|
||||
buf.push(0);
|
||||
}
|
||||
// source dataset name: null-terminated
|
||||
buf.extend_from_slice(m.source_dataset_name.as_bytes());
|
||||
buf.push(0);
|
||||
// source selection: length + bytes
|
||||
write_length(&mut buf, m.source_selection.len() as u64, length_size);
|
||||
buf.extend_from_slice(&m.source_selection);
|
||||
// virtual selection: length + bytes
|
||||
write_length(&mut buf, m.virtual_selection.len() as u64, length_size);
|
||||
buf.extend_from_slice(&m.virtual_selection);
|
||||
}
|
||||
|
||||
buf
|
||||
}
|
||||
|
||||
fn write_length(buf: &mut Vec<u8>, val: u64, size: u8) {
|
||||
match size {
|
||||
2 => buf.extend_from_slice(&(val as u16).to_le_bytes()),
|
||||
4 => buf.extend_from_slice(&(val as u32).to_le_bytes()),
|
||||
8 => buf.extend_from_slice(&val.to_le_bytes()),
|
||||
_ => buf.extend_from_slice(&val.to_le_bytes()),
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
fn all_sel() -> Vec<u8> {
|
||||
// Minimal H5S ALL selection bytes: type=3 (ALL), version=1, flags=0, unused*4
|
||||
let mut v = Vec::new();
|
||||
v.extend_from_slice(&3u32.to_le_bytes()); // type = H5S_SEL_ALL
|
||||
v.push(1); // version
|
||||
v.push(0); // flags
|
||||
v.extend_from_slice(&[0u8; 4]); // unused
|
||||
v
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn roundtrip_same_file_two_mappings() {
|
||||
let sel = all_sel();
|
||||
let mappings = vec![
|
||||
VdsMapping {
|
||||
source_file_name: String::new(),
|
||||
source_dataset_name: "/src_a".into(),
|
||||
source_selection: sel.clone(),
|
||||
virtual_selection: sel.clone(),
|
||||
},
|
||||
VdsMapping {
|
||||
source_file_name: String::new(),
|
||||
source_dataset_name: "/src_b".into(),
|
||||
source_selection: sel.clone(),
|
||||
virtual_selection: sel.clone(),
|
||||
},
|
||||
];
|
||||
let bytes = serialize_vds_mappings(&mappings, 8);
|
||||
let parsed = parse_vds_mappings(&bytes, 8).unwrap();
|
||||
assert_eq!(parsed.len(), 2);
|
||||
assert_eq!(parsed[0].source_dataset_name, "/src_a");
|
||||
assert_eq!(parsed[1].source_dataset_name, "/src_b");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn roundtrip_external_file_mapping() {
|
||||
let sel = all_sel();
|
||||
let mappings = vec![VdsMapping {
|
||||
source_file_name: "source.h5".into(),
|
||||
source_dataset_name: "/data".into(),
|
||||
source_selection: sel.clone(),
|
||||
virtual_selection: sel.clone(),
|
||||
}];
|
||||
let bytes = serialize_vds_mappings(&mappings, 8);
|
||||
let parsed = parse_vds_mappings(&bytes, 8).unwrap();
|
||||
assert_eq!(parsed.len(), 1);
|
||||
assert_eq!(parsed[0].source_file_name, "source.h5");
|
||||
assert_eq!(parsed[0].source_dataset_name, "/data");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn empty_mappings_roundtrip() {
|
||||
let bytes = serialize_vds_mappings(&[], 8);
|
||||
let parsed = parse_vds_mappings(&bytes, 8).unwrap();
|
||||
assert!(parsed.is_empty());
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [x] **Step 2: Run the failing tests**
|
||||
|
||||
```bash
|
||||
cargo test -p clawhdf5-format roundtrip_same_file_two_mappings roundtrip_external_file_mapping 2>&1 | head -20
|
||||
```
|
||||
|
||||
Expected: compile errors (module not declared).
|
||||
|
||||
- [x] **Step 3: Declare module in lib.rs**
|
||||
|
||||
In `crates/clawhdf5-format/src/lib.rs`, add:
|
||||
|
||||
```rust
|
||||
pub mod data_layout_write;
|
||||
```
|
||||
|
||||
- [x] **Step 4: Run tests**
|
||||
|
||||
```bash
|
||||
cargo test -p clawhdf5-format data_layout_write 2>&1
|
||||
```
|
||||
|
||||
Expected: all 3 tests PASS. If `parse_vds_mappings` expects a slightly different format for the version byte or the marker byte, adjust `serialize_vds_mappings` to match what the parser consumes (read `data_layout.rs:92–155` carefully to align).
|
||||
|
||||
- [x] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add crates/clawhdf5-format/src/data_layout_write.rs \
|
||||
crates/clawhdf5-format/src/lib.rs
|
||||
git commit -m "feat: add serialize_vds_mappings for writing VDS global-heap objects"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 3: FileWriter API for virtual datasets
|
||||
|
||||
**Background:** This task wires `serialize_vds_mappings()` into the `FileWriter` flow so callers can create a virtual dataset. It adds a new `DatasetBuilder` method and the corresponding serialization of a Data Layout v4 class=3 message.
|
||||
|
||||
**Files:**
|
||||
- Modify: `crates/clawhdf5-format/src/file_writer.rs` (add `with_virtual_sources`)
|
||||
|
||||
**Interfaces:**
|
||||
- Produces: `DatasetBuilder::with_virtual_sources(mappings: Vec<VdsMapping>) -> &mut Self`
|
||||
|
||||
**Binary — Data Layout v4 class=3 (Virtual):**
|
||||
```
|
||||
version(1)=4 class(1)=3
|
||||
global_heap_address(offset_size) global_heap_index(4)
|
||||
```
|
||||
The global heap object holds the `serialize_vds_mappings()` output. The `global_heap_address` is the address of the global heap collection; `global_heap_index` is the 1-based object index within it. Use index=1 for the first (and only) VDS object.
|
||||
|
||||
- [x] **Step 1: Write the failing test**
|
||||
|
||||
In `crates/clawhdf5-format/src/file_writer.rs` `#[cfg(test)]` block, add:
|
||||
|
||||
```rust
|
||||
#[test]
|
||||
fn virtual_dataset_write_roundtrip() {
|
||||
use crate::data_layout::DataLayout;
|
||||
use crate::data_layout::{VdsMapping, parse_vds_mappings};
|
||||
use crate::message_type::MessageType;
|
||||
use crate::object_header::ObjectHeader;
|
||||
use crate::signature::find_signature;
|
||||
use crate::superblock::Superblock;
|
||||
|
||||
// Minimal ALL-selection bytes (same as in data_layout_write tests)
|
||||
let sel: Vec<u8> = {
|
||||
let mut v = Vec::new();
|
||||
v.extend_from_slice(&3u32.to_le_bytes()); // H5S_SEL_ALL
|
||||
v.push(1); v.push(0);
|
||||
v.extend_from_slice(&[0u8; 4]);
|
||||
v
|
||||
};
|
||||
|
||||
let mappings = vec![VdsMapping {
|
||||
source_file_name: "src.h5".into(),
|
||||
source_dataset_name: "/raw".into(),
|
||||
source_selection: sel.clone(),
|
||||
virtual_selection: sel.clone(),
|
||||
}];
|
||||
|
||||
let mut fw = FileWriter::new();
|
||||
fw.create_dataset("virtual_ds")
|
||||
.with_virtual_sources(mappings);
|
||||
let bytes = fw.finish().unwrap();
|
||||
|
||||
// Parse back
|
||||
let sig = find_signature(&bytes).unwrap();
|
||||
let sb = Superblock::parse(&bytes, sig).unwrap();
|
||||
let root_oh = ObjectHeader::parse(
|
||||
&bytes, sb.root_group_address as usize, sb.offset_size, sb.length_size,
|
||||
).unwrap();
|
||||
|
||||
// Find the dataset via group traversal, then get its DataLayout message
|
||||
use crate::group_v2::resolve_path_any;
|
||||
let ds_addr = resolve_path_any(&bytes, &sb, "virtual_ds").unwrap();
|
||||
let ds_oh = ObjectHeader::parse(
|
||||
&bytes, ds_addr as usize, sb.offset_size, sb.length_size,
|
||||
).unwrap();
|
||||
let dl_msg = ds_oh.messages.iter()
|
||||
.find(|m| m.msg_type == MessageType::DataLayout)
|
||||
.expect("DataLayout message missing");
|
||||
|
||||
let layout = DataLayout::parse(&dl_msg.data, sb.offset_size, sb.length_size).unwrap();
|
||||
assert!(
|
||||
matches!(layout, DataLayout::Virtual { .. }),
|
||||
"expected Virtual layout, got {layout:?}"
|
||||
);
|
||||
}
|
||||
```
|
||||
|
||||
- [x] **Step 2: Run test to verify it fails**
|
||||
|
||||
```bash
|
||||
cargo test -p clawhdf5-format virtual_dataset_write_roundtrip 2>&1 | head -20
|
||||
```
|
||||
|
||||
Expected: compile error — `with_virtual_sources` not found.
|
||||
|
||||
- [x] **Step 3: Add with_virtual_sources to DatasetBuilder**
|
||||
|
||||
Find `DatasetBuilder` in `file_writer.rs`. Add a field `virtual_sources: Option<Vec<VdsMapping>>` and the method:
|
||||
|
||||
```rust
|
||||
use crate::data_layout::VdsMapping;
|
||||
|
||||
// In DatasetBuilder struct:
|
||||
virtual_sources: Option<Vec<VdsMapping>>,
|
||||
|
||||
// In DatasetBuilder impl:
|
||||
pub fn with_virtual_sources(&mut self, mappings: Vec<VdsMapping>) -> &mut Self {
|
||||
self.virtual_sources = Some(mappings);
|
||||
self
|
||||
}
|
||||
```
|
||||
|
||||
- [x] **Step 4: Serialize the virtual data layout**
|
||||
|
||||
In the `DatasetBuilder::build()` or equivalent finish method, add a branch for virtual datasets:
|
||||
|
||||
```rust
|
||||
use crate::data_layout_write::serialize_vds_mappings;
|
||||
|
||||
// Where the DataLayout message bytes are generated:
|
||||
let layout_bytes = if let Some(mappings) = &self.virtual_sources {
|
||||
// Serialize VDS mappings into a global heap object
|
||||
let heap_data = serialize_vds_mappings(mappings, length_size);
|
||||
let (heap_addr, heap_idx) = write_global_heap_object(output_buf, &heap_data);
|
||||
// Data Layout v4 class=3 (Virtual): version(1)=4, class(1)=3, addr(offset_size), idx(4)
|
||||
let mut dl = Vec::new();
|
||||
dl.push(4u8); // version
|
||||
dl.push(3u8); // class = Virtual
|
||||
write_offset_val(&mut dl, heap_addr, offset_size);
|
||||
dl.extend_from_slice(&(heap_idx as u32).to_le_bytes());
|
||||
dl
|
||||
} else {
|
||||
// existing layout code (contiguous/compact/chunked)
|
||||
build_existing_layout(...)
|
||||
};
|
||||
```
|
||||
|
||||
Implement `write_global_heap_object` as a helper that appends a minimal global heap collection to the output buffer and returns `(address, object_index)`:
|
||||
|
||||
```rust
|
||||
/// Append a single-object global heap collection to `buf` and return
|
||||
/// (collection_address, object_index=1).
|
||||
fn write_global_heap_object(buf: &mut Vec<u8>, data: &[u8]) -> (u64, usize) {
|
||||
let addr = buf.len() as u64;
|
||||
// Global Heap Collection header: sig(4) + version(1) + reserved(3) + collection_size(8)
|
||||
// Object: index(2) + ref_count(2) + reserved(4) + data_size(8) + data + padding
|
||||
let obj_size = data.len();
|
||||
let padded = (obj_size + 7) & !7;
|
||||
let collection_size = 16 + 16 + padded + 8; // header + one obj header + data + sentinel
|
||||
buf.extend_from_slice(b"GCOL"); // signature
|
||||
buf.push(1); // version
|
||||
buf.extend_from_slice(&[0u8; 3]); // reserved
|
||||
buf.extend_from_slice(&(collection_size as u64).to_le_bytes());
|
||||
// Object 1
|
||||
buf.extend_from_slice(&1u16.to_le_bytes()); // index
|
||||
buf.extend_from_slice(&1u16.to_le_bytes()); // ref_count
|
||||
buf.extend_from_slice(&[0u8; 4]); // reserved
|
||||
buf.extend_from_slice(&(obj_size as u64).to_le_bytes());
|
||||
buf.extend_from_slice(data);
|
||||
// Pad to 8-byte boundary
|
||||
let pad = padded - obj_size;
|
||||
buf.extend_from_slice(&vec![0u8; pad]);
|
||||
// Sentinel object (index=0)
|
||||
buf.extend_from_slice(&[0u8; 8]); // index=0 + ref_count + reserved
|
||||
buf.extend_from_slice(&0u64.to_le_bytes()); // size=0
|
||||
(addr, 1)
|
||||
}
|
||||
```
|
||||
|
||||
- [x] **Step 5: Run the test**
|
||||
|
||||
```bash
|
||||
cargo test -p clawhdf5-format virtual_dataset_write_roundtrip 2>&1
|
||||
```
|
||||
|
||||
Expected: PASS (or iterate on the global heap format until `parse_vds_mappings` reads back the mappings).
|
||||
|
||||
- [x] **Step 6: Run full suite**
|
||||
|
||||
```bash
|
||||
cargo test -p clawhdf5-format 2>&1 | tail -10
|
||||
```
|
||||
|
||||
Expected: all tests pass.
|
||||
|
||||
- [x] **Step 7: Commit**
|
||||
|
||||
```bash
|
||||
git add crates/clawhdf5-format/src/file_writer.rs
|
||||
git commit -m "feat: add DatasetBuilder::with_virtual_sources for writing VDS data layout"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 4: Superblock v4 read support
|
||||
|
||||
**Background:** `Superblock::parse()` in `superblock.rs:178–183` returns `Err(FormatError::UnsupportedVersion(v))` for any version ≥ 4. Superblock v4 (introduced with HDF5 2.x page-buffering) shares the same 12-byte header as v2/v3 (`sig + version + offset_size + length_size + consistency_flags`) and the same four address fields, but adds a `page_size: u32` field before the trailing checksum.
|
||||
|
||||
**Files:**
|
||||
- Modify: `crates/clawhdf5-format/src/superblock.rs`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes/produces: `Superblock` struct — add `pub page_size: Option<u32>` field.
|
||||
|
||||
- [x] **Step 1: Add field to Superblock struct**
|
||||
|
||||
In `crates/clawhdf5-format/src/superblock.rs`, add to the `Superblock` struct:
|
||||
|
||||
```rust
|
||||
/// Page size for page-buffer mode (v4 only). `None` for v0–v3.
|
||||
pub page_size: Option<u32>,
|
||||
```
|
||||
|
||||
Update all existing construction sites of `Superblock { ... }` in the file (parse_v0, parse_v1, parse_v2v3) to include `page_size: None`.
|
||||
|
||||
- [x] **Step 2: Write the failing test**
|
||||
|
||||
In the `#[cfg(test)]` section of `superblock.rs`, add:
|
||||
|
||||
```rust
|
||||
#[test]
|
||||
fn parse_v4_with_page_size() {
|
||||
// Superblock v4 = v2/v3 layout + page_size(4) before checksum.
|
||||
let mut buf = Vec::new();
|
||||
buf.extend_from_slice(&crate::signature::HDF5_SIGNATURE);
|
||||
buf.push(4); // version = 4
|
||||
buf.push(8); // offset_size
|
||||
buf.push(8); // length_size
|
||||
buf.push(0); // consistency_flags
|
||||
// base_address
|
||||
buf.extend_from_slice(&0u64.to_le_bytes());
|
||||
// superblock_extension_address = UNDEF
|
||||
buf.extend_from_slice(&u64::MAX.to_le_bytes());
|
||||
// eof_address
|
||||
buf.extend_from_slice(&512u64.to_le_bytes());
|
||||
// root_group_address
|
||||
buf.extend_from_slice(&96u64.to_le_bytes());
|
||||
// page_size (v4 addition before checksum)
|
||||
buf.extend_from_slice(&4096u32.to_le_bytes());
|
||||
// checksum (4 bytes; compute with jenkins_lookup3)
|
||||
let checksum = crate::checksum::jenkins_lookup3(&buf);
|
||||
buf.extend_from_slice(&checksum.to_le_bytes());
|
||||
|
||||
let sb = Superblock::parse(&buf, 0).unwrap();
|
||||
assert_eq!(sb.version, 4);
|
||||
assert_eq!(sb.offset_size, 8);
|
||||
assert_eq!(sb.eof_address, 512);
|
||||
assert_eq!(sb.page_size, Some(4096));
|
||||
}
|
||||
```
|
||||
|
||||
- [x] **Step 3: Run test to verify it fails**
|
||||
|
||||
```bash
|
||||
cargo test -p clawhdf5-format parse_v4_with_page_size 2>&1 | head -20
|
||||
```
|
||||
|
||||
Expected: `Err(UnsupportedVersion(4))` — the test fails because v4 isn't handled.
|
||||
|
||||
- [x] **Step 4: Add parse_v4 branch**
|
||||
|
||||
In `Superblock::parse()`, change:
|
||||
|
||||
```rust
|
||||
2 | 3 => Self::parse_v2v3(d, version),
|
||||
v => Err(FormatError::UnsupportedVersion(v)),
|
||||
```
|
||||
|
||||
to:
|
||||
|
||||
```rust
|
||||
2 | 3 => Self::parse_v2v3(d, version),
|
||||
4 => Self::parse_v4(d),
|
||||
v => Err(FormatError::UnsupportedVersion(v)),
|
||||
```
|
||||
|
||||
Add the implementation:
|
||||
|
||||
```rust
|
||||
fn parse_v4(d: &[u8]) -> Result<Superblock, FormatError> {
|
||||
// Same as v2/v3 header, then page_size(4), then checksum(4).
|
||||
ensure_len(d, 12)?;
|
||||
let offset_size = d[9];
|
||||
let length_size = d[10];
|
||||
validate_sizes(offset_size, length_size)?;
|
||||
let consistency_flags = d[11] as u32;
|
||||
|
||||
let os = offset_size as usize;
|
||||
// 4 addresses + page_size(4) + checksum(4)
|
||||
let total = 12 + 4 * os + 4 + 4;
|
||||
ensure_len(d, total)?;
|
||||
|
||||
let mut pos = 12;
|
||||
let base_address = read_offset(d, pos, offset_size)?;
|
||||
pos += os;
|
||||
let superblock_extension_address = read_offset(d, pos, offset_size)?;
|
||||
pos += os;
|
||||
let eof_address = read_offset(d, pos, offset_size)?;
|
||||
pos += os;
|
||||
let root_group_address = read_offset(d, pos, offset_size)?;
|
||||
pos += os;
|
||||
|
||||
let page_size = u32::from_le_bytes([d[pos], d[pos+1], d[pos+2], d[pos+3]]);
|
||||
pos += 4;
|
||||
|
||||
let stored_checksum = u32::from_le_bytes([d[pos], d[pos+1], d[pos+2], d[pos+3]]);
|
||||
let computed = crate::checksum::jenkins_lookup3(&d[..pos]);
|
||||
if stored_checksum != computed {
|
||||
return Err(FormatError::ChecksumMismatch {
|
||||
expected: stored_checksum,
|
||||
computed,
|
||||
});
|
||||
}
|
||||
|
||||
Ok(Superblock {
|
||||
version: 4,
|
||||
offset_size,
|
||||
length_size,
|
||||
base_address,
|
||||
eof_address,
|
||||
root_group_address,
|
||||
group_leaf_node_k: None,
|
||||
group_internal_node_k: None,
|
||||
indexed_storage_internal_node_k: None,
|
||||
free_space_address: None,
|
||||
driver_info_address: None,
|
||||
consistency_flags,
|
||||
superblock_extension_address: Some(superblock_extension_address),
|
||||
checksum: Some(stored_checksum),
|
||||
page_size: Some(page_size),
|
||||
})
|
||||
}
|
||||
```
|
||||
|
||||
- [x] **Step 5: Run the test**
|
||||
|
||||
```bash
|
||||
cargo test -p clawhdf5-format parse_v4_with_page_size 2>&1
|
||||
```
|
||||
|
||||
Expected: PASS (verify the checksum field name matches whatever `FormatError` uses — it may be `ChecksumMismatch { expected, computed }` or similar; find it in `error.rs` and match).
|
||||
|
||||
- [x] **Step 6: Run full suite**
|
||||
|
||||
```bash
|
||||
cargo test -p clawhdf5-format 2>&1 | tail -10
|
||||
```
|
||||
|
||||
Expected: all tests pass.
|
||||
|
||||
- [x] **Step 7: Commit**
|
||||
|
||||
```bash
|
||||
git add crates/clawhdf5-format/src/superblock.rs
|
||||
git commit -m "feat: parse HDF5 superblock v4 (page-buffer mode) with page_size field"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 5: Superblock v4 write support
|
||||
|
||||
**Background:** The `FileWriter` always writes a v3 superblock (hardcoded in `file_writer.rs:1291–1306`). This task adds an optional `page_size` to `FileWriter` that, when set, emits a v4 superblock.
|
||||
|
||||
**Files:**
|
||||
- Modify: `crates/clawhdf5-format/src/file_writer.rs` (add `page_size` field)
|
||||
- Modify: `crates/clawhdf5-format/src/superblock.rs` (`Superblock::serialize` for v4)
|
||||
|
||||
**Interfaces:**
|
||||
- Produces: `FileWriter::with_page_size(page_size: u32) -> &mut Self`
|
||||
|
||||
- [x] **Step 1: Write the failing test**
|
||||
|
||||
In the `#[cfg(test)]` block of `file_writer.rs`, add:
|
||||
|
||||
```rust
|
||||
#[test]
|
||||
fn file_writer_v4_superblock() {
|
||||
use crate::signature::find_signature;
|
||||
use crate::superblock::Superblock;
|
||||
|
||||
let mut fw = FileWriter::new();
|
||||
fw.with_page_size(4096);
|
||||
fw.create_dataset("data").with_f64_data(&[1.0, 2.0]);
|
||||
let bytes = fw.finish().unwrap();
|
||||
|
||||
let sig = find_signature(&bytes).unwrap();
|
||||
let sb = Superblock::parse(&bytes, sig).unwrap();
|
||||
assert_eq!(sb.version, 4, "expected superblock v4");
|
||||
assert_eq!(sb.page_size, Some(4096));
|
||||
}
|
||||
```
|
||||
|
||||
- [x] **Step 2: Run test to verify it fails**
|
||||
|
||||
```bash
|
||||
cargo test -p clawhdf5-format file_writer_v4_superblock 2>&1 | head -20
|
||||
```
|
||||
|
||||
Expected: compile error — `with_page_size` not found.
|
||||
|
||||
- [x] **Step 3: Add page_size field to FileWriter**
|
||||
|
||||
In `FileWriter` struct definition, add `page_size: Option<u32>`.
|
||||
In `FileWriter::new()`, add `page_size: None`.
|
||||
Add method:
|
||||
|
||||
```rust
|
||||
pub fn with_page_size(&mut self, page_size: u32) -> &mut Self {
|
||||
self.page_size = Some(page_size);
|
||||
self
|
||||
}
|
||||
```
|
||||
|
||||
- [x] **Step 4: Update Superblock::serialize for v4**
|
||||
|
||||
In `crates/clawhdf5-format/src/superblock.rs`, the `serialize()` method currently hardcodes v2/v3 format. Update it to emit v4 when `self.version == 4` and `self.page_size.is_some()`:
|
||||
|
||||
```rust
|
||||
pub fn serialize(&self) -> Vec<u8> {
|
||||
let mut buf = Vec::with_capacity(60);
|
||||
buf.extend_from_slice(&HDF5_SIGNATURE);
|
||||
buf.push(self.version);
|
||||
buf.push(self.offset_size);
|
||||
buf.push(self.length_size);
|
||||
buf.push(self.consistency_flags as u8);
|
||||
Self::write_offset(&mut buf, self.base_address, self.offset_size);
|
||||
let ext_addr = self.superblock_extension_address.unwrap_or(u64::MAX);
|
||||
Self::write_offset(&mut buf, ext_addr, self.offset_size);
|
||||
Self::write_offset(&mut buf, self.eof_address, self.offset_size);
|
||||
Self::write_offset(&mut buf, self.root_group_address, self.offset_size);
|
||||
if self.version >= 4 {
|
||||
let ps = self.page_size.unwrap_or(0);
|
||||
buf.extend_from_slice(&ps.to_le_bytes());
|
||||
}
|
||||
let checksum = crate::checksum::jenkins_lookup3(&buf);
|
||||
buf.extend_from_slice(&checksum.to_le_bytes());
|
||||
buf
|
||||
}
|
||||
```
|
||||
|
||||
- [x] **Step 5: Wire page_size into FileWriter::finish()**
|
||||
|
||||
In `file_writer.rs:finish()`, where the `Superblock` is constructed (around line 1291), change:
|
||||
|
||||
```rust
|
||||
let sb = Superblock {
|
||||
version: if self.page_size.is_some() { 4 } else { 3 },
|
||||
// ... existing fields ...
|
||||
page_size: self.page_size,
|
||||
// ... rest of fields unchanged ...
|
||||
};
|
||||
```
|
||||
|
||||
- [x] **Step 6: Run the test**
|
||||
|
||||
```bash
|
||||
cargo test -p clawhdf5-format file_writer_v4_superblock 2>&1
|
||||
```
|
||||
|
||||
Expected: PASS.
|
||||
|
||||
- [x] **Step 7: Run full suite**
|
||||
|
||||
```bash
|
||||
cargo test -p clawhdf5-format 2>&1 | tail -10
|
||||
```
|
||||
|
||||
Expected: all tests pass (v3 serialize() must be byte-identical to before — add a regression test if needed).
|
||||
|
||||
- [x] **Step 8: Commit**
|
||||
|
||||
```bash
|
||||
git add crates/clawhdf5-format/src/file_writer.rs \
|
||||
crates/clawhdf5-format/src/superblock.rs
|
||||
git commit -m "feat: write HDF5 superblock v4 when page_size is configured"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Verification
|
||||
|
||||
```bash
|
||||
# Run all format tests
|
||||
cargo test -p clawhdf5-format 2>&1 | tail -10
|
||||
|
||||
# Specifically verify new features
|
||||
cargo test -p clawhdf5-format external_link_write_roundtrip 2>&1
|
||||
cargo test -p clawhdf5-format data_layout_write 2>&1
|
||||
cargo test -p clawhdf5-format virtual_dataset_write_roundtrip 2>&1
|
||||
cargo test -p clawhdf5-format parse_v4_with_page_size 2>&1
|
||||
cargo test -p clawhdf5-format file_writer_v4_superblock 2>&1
|
||||
|
||||
# Regression: v3 superblock still round-trips
|
||||
cargo test -p clawhdf5-format write_superblock 2>&1
|
||||
```
|
||||
Reference in New Issue
Block a user