clawhdf5/docs/superpowers/plans/2026-06-29-format-write-extensions.md

Format Write Extensions Implementation Plan

Status (2026-08-03): Implemented. Tasks 1–3 (external links, VDS mapping serialization, VDS FileWriter API) shipped in commit d6c4d4f (2026-06-30). Tasks 4–5 (superblock v4 read/write) were not part of that commit and were completed separately as part of this cleanup pass (2026-08-03) — see Superblock::parse_v4/serialize and FileWriter::with_page_size in crates/clawhdf5-format. This doc was authored 2026-06-29 as the pre-work plan and committed to the repo retroactively; checkboxes below have been marked complete to match current state. Treat this as a historical record, not an open task list.

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Add three write-side features to clawhdf5-format: (1) external link creation via GroupBuilder, (2) external VDS (Virtual Dataset Source) layout writes, and (3) superblock v4 read/write for page-buffering-aware files.

Architecture: External links reuse the existing LinkMessage::serialize() which already handles LinkTarget::External — only the GroupBuilder API needs wiring up. External VDS adds write_vds_layout() in file_writer.rs and serialize_vds_mappings() in a new data_layout_write.rs. Superblock v4 extends Superblock::parse with a new parse_v4 branch (identical structure to v3 with an extra page_size field) and updates Superblock::serialize to optionally write v4.

Tech Stack: Pure Rust, no new dependencies. All changes in crates/clawhdf5-format/.

Global Constraints

• All code in crates/clawhdf5-format/.
• No new Cargo dependencies.
• External links: written as LinkTarget::External, readable by h5py (verified in tests).
• VDS: uses data layout version 4, class 3. Global heap at end of file.
• Superblock v4: only adds page_size: u32 field after the v2/v3 body; checksum placement unchanged.
• Run cargo test -p clawhdf5-format after every task.

---

Task 1: External link write API in GroupBuilder

Background: LinkMessage::serialize() in link_message.rs:76–175 already handles LinkTarget::External { filename, object_path } (writes link_type byte = 64, then packed filename+path). What's missing is a public API in file_writer.rs to create external links from a GroupBuilder. Currently GroupBuilder only creates datasets and sub-groups via create_dataset / create_group.

Files:

• Modify: crates/clawhdf5-format/src/file_writer.rs (add GroupBuilder::add_external_link)
• Modify: crates/clawhdf5-format/src/lib.rs (re-export LinkTarget if not already exported)
• Test: crates/clawhdf5-format/src/file_writer.rs (new test in #[cfg(test)])

Interfaces:

• Produces: GroupBuilder::add_external_link(&mut self, name: &str, target_file: &str, target_path: &str) -> &mut Self

• Step 1: Write the failing test

At the bottom of the #[cfg(test)] block in crates/clawhdf5-format/src/file_writer.rs, add:

#[test]
fn external_link_write_roundtrip() {
    use crate::group_v2::resolve_path_any;
    use crate::link_message::{LinkMessage, LinkTarget};
    use crate::message_type::MessageType;
    use crate::object_header::ObjectHeader;
    use crate::signature::find_signature;
    use crate::superblock::Superblock;

    let mut fw = FileWriter::new();
    let mut grp = fw.create_group("links");
    grp.add_external_link("remote_data", "other_file.h5", "/sensors/temp");
    fw.add_group(grp.finish());
    let bytes = fw.finish().unwrap();

    let sig = find_signature(&bytes).unwrap();
    let sb = Superblock::parse(&bytes, sig).unwrap();

    // Navigate to /links group
    let links_addr = resolve_path_any(&bytes, &sb, "links").unwrap();
    let links_oh = ObjectHeader::parse(
        &bytes, links_addr as usize, sb.offset_size, sb.length_size,
    ).unwrap();

    // Find the link message for "remote_data"
    let link_msg = links_oh.messages.iter()
        .filter(|m| m.msg_type == MessageType::Link)
        .find_map(|m| {
            let lm = LinkMessage::parse(&m.data, sb.offset_size).ok()?;
            if lm.name == "remote_data" { Some(lm) } else { None }
        })
        .expect("external link message not found");

    assert_eq!(
        link_msg.link_target,
        LinkTarget::External {
            filename: "other_file.h5".into(),
            object_path: "/sensors/temp".into(),
        }
    );
}

• Step 2: Run test to verify it fails

cargo test -p clawhdf5-format external_link_write_roundtrip 2>&1 | head -20

Expected: compile error — add_external_link not found.

• Step 3: Find GroupBuilder in file_writer.rs and add the method

Locate GroupBuilder in crates/clawhdf5-format/src/file_writer.rs. It tracks its items as a Vec of internal builders. Add a field for external links and the method:

First, locate the GroupBuilder struct definition and add a field:

pub struct GroupBuilder {
    name: String,
    datasets: Vec<DatasetBuilder>,
    groups: Vec<FinishedGroup>,
    external_links: Vec<(String, String, String)>, // (name, filename, object_path)
}

Update GroupBuilder::new() (or equivalent constructor) to initialize external_links: Vec::new().

Add the public method immediately after the existing create_dataset/create_group methods:

/// Add a link in this group that points to an object in another HDF5 file.
///
/// `name` is the link name within this group.
/// `target_file` is the relative or absolute path to the target .h5 file.
/// `target_path` is the HDF5 path of the object within the target file.
pub fn add_external_link(
    &mut self,
    name: &str,
    target_file: &str,
    target_path: &str,
) -> &mut Self {
    self.external_links.push((
        name.to_string(),
        target_file.to_string(),
        target_path.to_string(),
    ));
    self
}

• Step 4: Wire external links into the group serialization

Find where the GroupBuilder emits LinkMessage bytes during finish() / build_group(). For each external link, emit a LinkMessage with LinkTarget::External:

use crate::link_message::{LinkMessage, LinkTarget};
use crate::datatype::CharacterSet;

// Inside the loop/block that serializes links:
for (link_name, filename, object_path) in &self.external_links {
    let msg = LinkMessage {
        name: link_name.clone(),
        link_target: LinkTarget::External {
            filename: filename.clone(),
            object_path: object_path.clone(),
        },
        creation_order: None,
        charset: CharacterSet::Utf8,
    };
    let msg_bytes = msg.serialize(offset_size);
    // Emit as a Link message (MessageType::Link = 0x0006) into the object header
    emit_message(&mut oh_buf, MessageType::Link, &msg_bytes);
}

(Follow the exact pattern used for hard links and soft links in the same codebase — find where hard-link LinkMessage bytes are pushed and add the external links in the same loop.)

• Step 5: Run the failing test

cargo test -p clawhdf5-format external_link_write_roundtrip 2>&1

Expected: PASS.

• Step 6: Run full suite

cargo test -p clawhdf5-format 2>&1 | tail -10

Expected: all tests pass.

• Step 7: Commit

git add crates/clawhdf5-format/src/file_writer.rs
git commit -m "feat: add GroupBuilder::add_external_link for writing cross-file HDF5 links"

---

Task 2: VDS mapping serialization helper

Background: Reading VDS mappings from a global heap object is done by parse_vds_mappings() in data_layout.rs:70–155. Writing the inverse — serializing a Vec<VdsMapping> into the same binary layout — does not exist. This task creates serialize_vds_mappings().

Files:

• Create: crates/clawhdf5-format/src/data_layout_write.rs
• Modify: crates/clawhdf5-format/src/lib.rs (declare module)

Interfaces:

• Consumes: VdsMapping { source_file_name: String, source_dataset_name: String, source_selection: Vec<u8>, virtual_selection: Vec<u8> } (existing struct from data_layout.rs).
• Produces: pub fn serialize_vds_mappings(mappings: &[VdsMapping], length_size: u8) -> Vec<u8>

Binary layout (from data_layout.rs:72–91 doc comment):

version: u8 (0 = external file, 1 = same-file marker)
nused: length_size bytes (number of mappings)
for each mapping:
  if version==0: source_file_name (null-terminated)
  else: marker byte (0xFF or similar; same-file means empty filename)
  source_dataset_name: null-terminated string
  source_selection: length(length_size) + bytes
  virtual_selection: length(length_size) + bytes

• Step 1: Write the failing tests

Create crates/clawhdf5-format/src/data_layout_write.rs:

//! Write-side helpers for VDS (Virtual Dataset Source) mapping serialization.

use crate::data_layout::{parse_vds_mappings, VdsMapping};
use crate::error::FormatError;

/// Serialize a slice of VDS mappings into the global-heap object byte format.
///
/// The output can be stored directly in a global heap object and referenced
/// from a Data Layout v4 class=3 (Virtual) message.
pub fn serialize_vds_mappings(mappings: &[VdsMapping], length_size: u8) -> Vec<u8> {
    let mut buf = Vec::new();

    // Determine if all sources are same-file (empty source_file_name)
    let has_external = mappings.iter().any(|m| !m.source_file_name.is_empty());
    let version: u8 = if has_external { 0 } else { 1 };
    buf.push(version);

    // nused: number of mappings
    write_length(&mut buf, mappings.len() as u64, length_size);

    for m in mappings {
        if version == 0 {
            // External: null-terminated filename
            buf.extend_from_slice(m.source_file_name.as_bytes());
            buf.push(0);
        } else {
            // Same-file: marker byte (0x00, which parse_vds_mappings treats as empty)
            buf.push(0);
        }
        // source dataset name: null-terminated
        buf.extend_from_slice(m.source_dataset_name.as_bytes());
        buf.push(0);
        // source selection: length + bytes
        write_length(&mut buf, m.source_selection.len() as u64, length_size);
        buf.extend_from_slice(&m.source_selection);
        // virtual selection: length + bytes
        write_length(&mut buf, m.virtual_selection.len() as u64, length_size);
        buf.extend_from_slice(&m.virtual_selection);
    }

    buf
}

fn write_length(buf: &mut Vec<u8>, val: u64, size: u8) {
    match size {
        2 => buf.extend_from_slice(&(val as u16).to_le_bytes()),
        4 => buf.extend_from_slice(&(val as u32).to_le_bytes()),
        8 => buf.extend_from_slice(&val.to_le_bytes()),
        _ => buf.extend_from_slice(&val.to_le_bytes()),
    }
}

#[cfg(test)]
mod tests {
    use super::*;

    fn all_sel() -> Vec<u8> {
        // Minimal H5S ALL selection bytes: type=3 (ALL), version=1, flags=0, unused*4
        let mut v = Vec::new();
        v.extend_from_slice(&3u32.to_le_bytes()); // type = H5S_SEL_ALL
        v.push(1); // version
        v.push(0); // flags
        v.extend_from_slice(&[0u8; 4]); // unused
        v
    }

    #[test]
    fn roundtrip_same_file_two_mappings() {
        let sel = all_sel();
        let mappings = vec![
            VdsMapping {
                source_file_name: String::new(),
                source_dataset_name: "/src_a".into(),
                source_selection: sel.clone(),
                virtual_selection: sel.clone(),
            },
            VdsMapping {
                source_file_name: String::new(),
                source_dataset_name: "/src_b".into(),
                source_selection: sel.clone(),
                virtual_selection: sel.clone(),
            },
        ];
        let bytes = serialize_vds_mappings(&mappings, 8);
        let parsed = parse_vds_mappings(&bytes, 8).unwrap();
        assert_eq!(parsed.len(), 2);
        assert_eq!(parsed[0].source_dataset_name, "/src_a");
        assert_eq!(parsed[1].source_dataset_name, "/src_b");
    }

    #[test]
    fn roundtrip_external_file_mapping() {
        let sel = all_sel();
        let mappings = vec![VdsMapping {
            source_file_name: "source.h5".into(),
            source_dataset_name: "/data".into(),
            source_selection: sel.clone(),
            virtual_selection: sel.clone(),
        }];
        let bytes = serialize_vds_mappings(&mappings, 8);
        let parsed = parse_vds_mappings(&bytes, 8).unwrap();
        assert_eq!(parsed.len(), 1);
        assert_eq!(parsed[0].source_file_name, "source.h5");
        assert_eq!(parsed[0].source_dataset_name, "/data");
    }

    #[test]
    fn empty_mappings_roundtrip() {
        let bytes = serialize_vds_mappings(&[], 8);
        let parsed = parse_vds_mappings(&bytes, 8).unwrap();
        assert!(parsed.is_empty());
    }
}

• Step 2: Run the failing tests

cargo test -p clawhdf5-format roundtrip_same_file_two_mappings roundtrip_external_file_mapping 2>&1 | head -20

Expected: compile errors (module not declared).

• Step 3: Declare module in lib.rs

In crates/clawhdf5-format/src/lib.rs, add:

pub mod data_layout_write;

• Step 4: Run tests

cargo test -p clawhdf5-format data_layout_write 2>&1

Expected: all 3 tests PASS. If parse_vds_mappings expects a slightly different format for the version byte or the marker byte, adjust serialize_vds_mappings to match what the parser consumes (read data_layout.rs:92–155 carefully to align).

• Step 5: Commit

git add crates/clawhdf5-format/src/data_layout_write.rs \
        crates/clawhdf5-format/src/lib.rs
git commit -m "feat: add serialize_vds_mappings for writing VDS global-heap objects"

---

Task 3: FileWriter API for virtual datasets

Background: This task wires serialize_vds_mappings() into the FileWriter flow so callers can create a virtual dataset. It adds a new DatasetBuilder method and the corresponding serialization of a Data Layout v4 class=3 message.

Files:

• Modify: crates/clawhdf5-format/src/file_writer.rs (add with_virtual_sources)

Interfaces:

• Produces: DatasetBuilder::with_virtual_sources(mappings: Vec<VdsMapping>) -> &mut Self

Binary — Data Layout v4 class=3 (Virtual):

version(1)=4  class(1)=3
global_heap_address(offset_size)  global_heap_index(4)

The global heap object holds the serialize_vds_mappings() output. The global_heap_address is the address of the global heap collection; global_heap_index is the 1-based object index within it. Use index=1 for the first (and only) VDS object.

• Step 1: Write the failing test

In crates/clawhdf5-format/src/file_writer.rs #[cfg(test)] block, add:

#[test]
fn virtual_dataset_write_roundtrip() {
    use crate::data_layout::DataLayout;
    use crate::data_layout::{VdsMapping, parse_vds_mappings};
    use crate::message_type::MessageType;
    use crate::object_header::ObjectHeader;
    use crate::signature::find_signature;
    use crate::superblock::Superblock;

    // Minimal ALL-selection bytes (same as in data_layout_write tests)
    let sel: Vec<u8> = {
        let mut v = Vec::new();
        v.extend_from_slice(&3u32.to_le_bytes()); // H5S_SEL_ALL
        v.push(1); v.push(0);
        v.extend_from_slice(&[0u8; 4]);
        v
    };

    let mappings = vec![VdsMapping {
        source_file_name: "src.h5".into(),
        source_dataset_name: "/raw".into(),
        source_selection: sel.clone(),
        virtual_selection: sel.clone(),
    }];

    let mut fw = FileWriter::new();
    fw.create_dataset("virtual_ds")
      .with_virtual_sources(mappings);
    let bytes = fw.finish().unwrap();

    // Parse back
    let sig = find_signature(&bytes).unwrap();
    let sb = Superblock::parse(&bytes, sig).unwrap();
    let root_oh = ObjectHeader::parse(
        &bytes, sb.root_group_address as usize, sb.offset_size, sb.length_size,
    ).unwrap();

    // Find the dataset via group traversal, then get its DataLayout message
    use crate::group_v2::resolve_path_any;
    let ds_addr = resolve_path_any(&bytes, &sb, "virtual_ds").unwrap();
    let ds_oh = ObjectHeader::parse(
        &bytes, ds_addr as usize, sb.offset_size, sb.length_size,
    ).unwrap();
    let dl_msg = ds_oh.messages.iter()
        .find(|m| m.msg_type == MessageType::DataLayout)
        .expect("DataLayout message missing");

    let layout = DataLayout::parse(&dl_msg.data, sb.offset_size, sb.length_size).unwrap();
    assert!(
        matches!(layout, DataLayout::Virtual { .. }),
        "expected Virtual layout, got {layout:?}"
    );
}

• Step 2: Run test to verify it fails

cargo test -p clawhdf5-format virtual_dataset_write_roundtrip 2>&1 | head -20

Expected: compile error — with_virtual_sources not found.

• Step 3: Add with_virtual_sources to DatasetBuilder

Find DatasetBuilder in file_writer.rs. Add a field virtual_sources: Option<Vec<VdsMapping>> and the method:

use crate::data_layout::VdsMapping;

// In DatasetBuilder struct:
virtual_sources: Option<Vec<VdsMapping>>,

// In DatasetBuilder impl:
pub fn with_virtual_sources(&mut self, mappings: Vec<VdsMapping>) -> &mut Self {
    self.virtual_sources = Some(mappings);
    self
}

• Step 4: Serialize the virtual data layout

In the DatasetBuilder::build() or equivalent finish method, add a branch for virtual datasets:

use crate::data_layout_write::serialize_vds_mappings;

// Where the DataLayout message bytes are generated:
let layout_bytes = if let Some(mappings) = &self.virtual_sources {
    // Serialize VDS mappings into a global heap object
    let heap_data = serialize_vds_mappings(mappings, length_size);
    let (heap_addr, heap_idx) = write_global_heap_object(output_buf, &heap_data);
    // Data Layout v4 class=3 (Virtual): version(1)=4, class(1)=3, addr(offset_size), idx(4)
    let mut dl = Vec::new();
    dl.push(4u8); // version
    dl.push(3u8); // class = Virtual
    write_offset_val(&mut dl, heap_addr, offset_size);
    dl.extend_from_slice(&(heap_idx as u32).to_le_bytes());
    dl
} else {
    // existing layout code (contiguous/compact/chunked)
    build_existing_layout(...)
};

Implement write_global_heap_object as a helper that appends a minimal global heap collection to the output buffer and returns (address, object_index):

/// Append a single-object global heap collection to `buf` and return
/// (collection_address, object_index=1).
fn write_global_heap_object(buf: &mut Vec<u8>, data: &[u8]) -> (u64, usize) {
    let addr = buf.len() as u64;
    // Global Heap Collection header: sig(4) + version(1) + reserved(3) + collection_size(8)
    // Object: index(2) + ref_count(2) + reserved(4) + data_size(8) + data + padding
    let obj_size = data.len();
    let padded = (obj_size + 7) & !7;
    let collection_size = 16 + 16 + padded + 8; // header + one obj header + data + sentinel
    buf.extend_from_slice(b"GCOL"); // signature
    buf.push(1); // version
    buf.extend_from_slice(&[0u8; 3]); // reserved
    buf.extend_from_slice(&(collection_size as u64).to_le_bytes());
    // Object 1
    buf.extend_from_slice(&1u16.to_le_bytes()); // index
    buf.extend_from_slice(&1u16.to_le_bytes()); // ref_count
    buf.extend_from_slice(&[0u8; 4]); // reserved
    buf.extend_from_slice(&(obj_size as u64).to_le_bytes());
    buf.extend_from_slice(data);
    // Pad to 8-byte boundary
    let pad = padded - obj_size;
    buf.extend_from_slice(&vec![0u8; pad]);
    // Sentinel object (index=0)
    buf.extend_from_slice(&[0u8; 8]); // index=0 + ref_count + reserved
    buf.extend_from_slice(&0u64.to_le_bytes()); // size=0
    (addr, 1)
}

• Step 5: Run the test

cargo test -p clawhdf5-format virtual_dataset_write_roundtrip 2>&1

Expected: PASS (or iterate on the global heap format until parse_vds_mappings reads back the mappings).

• Step 6: Run full suite

cargo test -p clawhdf5-format 2>&1 | tail -10

Expected: all tests pass.

• Step 7: Commit

git add crates/clawhdf5-format/src/file_writer.rs
git commit -m "feat: add DatasetBuilder::with_virtual_sources for writing VDS data layout"

---

Task 4: Superblock v4 read support

Background: Superblock::parse() in superblock.rs:178–183 returns Err(FormatError::UnsupportedVersion(v)) for any version ≥ 4. Superblock v4 (introduced with HDF5 2.x page-buffering) shares the same 12-byte header as v2/v3 (sig + version + offset_size + length_size + consistency_flags) and the same four address fields, but adds a page_size: u32 field before the trailing checksum.

Files:

• Modify: crates/clawhdf5-format/src/superblock.rs

Interfaces:

• Consumes/produces: Superblock struct — add pub page_size: Option<u32> field.

• Step 1: Add field to Superblock struct

In crates/clawhdf5-format/src/superblock.rs, add to the Superblock struct:

/// Page size for page-buffer mode (v4 only). `None` for v0–v3.
pub page_size: Option<u32>,

Update all existing construction sites of Superblock { ... } in the file (parse_v0, parse_v1, parse_v2v3) to include page_size: None.

• Step 2: Write the failing test

In the #[cfg(test)] section of superblock.rs, add:

#[test]
fn parse_v4_with_page_size() {
    // Superblock v4 = v2/v3 layout + page_size(4) before checksum.
    let mut buf = Vec::new();
    buf.extend_from_slice(&crate::signature::HDF5_SIGNATURE);
    buf.push(4);  // version = 4
    buf.push(8);  // offset_size
    buf.push(8);  // length_size
    buf.push(0);  // consistency_flags
    // base_address
    buf.extend_from_slice(&0u64.to_le_bytes());
    // superblock_extension_address = UNDEF
    buf.extend_from_slice(&u64::MAX.to_le_bytes());
    // eof_address
    buf.extend_from_slice(&512u64.to_le_bytes());
    // root_group_address
    buf.extend_from_slice(&96u64.to_le_bytes());
    // page_size (v4 addition before checksum)
    buf.extend_from_slice(&4096u32.to_le_bytes());
    // checksum (4 bytes; compute with jenkins_lookup3)
    let checksum = crate::checksum::jenkins_lookup3(&buf);
    buf.extend_from_slice(&checksum.to_le_bytes());

    let sb = Superblock::parse(&buf, 0).unwrap();
    assert_eq!(sb.version, 4);
    assert_eq!(sb.offset_size, 8);
    assert_eq!(sb.eof_address, 512);
    assert_eq!(sb.page_size, Some(4096));
}

• Step 3: Run test to verify it fails

cargo test -p clawhdf5-format parse_v4_with_page_size 2>&1 | head -20

Expected: Err(UnsupportedVersion(4)) — the test fails because v4 isn't handled.

• Step 4: Add parse_v4 branch

In Superblock::parse(), change:

2 | 3 => Self::parse_v2v3(d, version),
v => Err(FormatError::UnsupportedVersion(v)),

to:

2 | 3 => Self::parse_v2v3(d, version),
4 => Self::parse_v4(d),
v => Err(FormatError::UnsupportedVersion(v)),

Add the implementation:

fn parse_v4(d: &[u8]) -> Result<Superblock, FormatError> {
    // Same as v2/v3 header, then page_size(4), then checksum(4).
    ensure_len(d, 12)?;
    let offset_size = d[9];
    let length_size = d[10];
    validate_sizes(offset_size, length_size)?;
    let consistency_flags = d[11] as u32;

    let os = offset_size as usize;
    // 4 addresses + page_size(4) + checksum(4)
    let total = 12 + 4 * os + 4 + 4;
    ensure_len(d, total)?;

    let mut pos = 12;
    let base_address = read_offset(d, pos, offset_size)?;
    pos += os;
    let superblock_extension_address = read_offset(d, pos, offset_size)?;
    pos += os;
    let eof_address = read_offset(d, pos, offset_size)?;
    pos += os;
    let root_group_address = read_offset(d, pos, offset_size)?;
    pos += os;

    let page_size = u32::from_le_bytes([d[pos], d[pos+1], d[pos+2], d[pos+3]]);
    pos += 4;

    let stored_checksum = u32::from_le_bytes([d[pos], d[pos+1], d[pos+2], d[pos+3]]);
    let computed = crate::checksum::jenkins_lookup3(&d[..pos]);
    if stored_checksum != computed {
        return Err(FormatError::ChecksumMismatch {
            expected: stored_checksum,
            computed,
        });
    }

    Ok(Superblock {
        version: 4,
        offset_size,
        length_size,
        base_address,
        eof_address,
        root_group_address,
        group_leaf_node_k: None,
        group_internal_node_k: None,
        indexed_storage_internal_node_k: None,
        free_space_address: None,
        driver_info_address: None,
        consistency_flags,
        superblock_extension_address: Some(superblock_extension_address),
        checksum: Some(stored_checksum),
        page_size: Some(page_size),
    })
}

• Step 5: Run the test

cargo test -p clawhdf5-format parse_v4_with_page_size 2>&1

Expected: PASS (verify the checksum field name matches whatever FormatError uses — it may be ChecksumMismatch { expected, computed } or similar; find it in error.rs and match).

• Step 6: Run full suite

cargo test -p clawhdf5-format 2>&1 | tail -10

Expected: all tests pass.

• Step 7: Commit

git add crates/clawhdf5-format/src/superblock.rs
git commit -m "feat: parse HDF5 superblock v4 (page-buffer mode) with page_size field"

---

Task 5: Superblock v4 write support

Background: The FileWriter always writes a v3 superblock (hardcoded in file_writer.rs:1291–1306). This task adds an optional page_size to FileWriter that, when set, emits a v4 superblock.

Files:

• Modify: crates/clawhdf5-format/src/file_writer.rs (add page_size field)
• Modify: crates/clawhdf5-format/src/superblock.rs (Superblock::serialize for v4)

Interfaces:

• Produces: FileWriter::with_page_size(page_size: u32) -> &mut Self

• Step 1: Write the failing test

In the #[cfg(test)] block of file_writer.rs, add:

#[test]
fn file_writer_v4_superblock() {
    use crate::signature::find_signature;
    use crate::superblock::Superblock;

    let mut fw = FileWriter::new();
    fw.with_page_size(4096);
    fw.create_dataset("data").with_f64_data(&[1.0, 2.0]);
    let bytes = fw.finish().unwrap();

    let sig = find_signature(&bytes).unwrap();
    let sb = Superblock::parse(&bytes, sig).unwrap();
    assert_eq!(sb.version, 4, "expected superblock v4");
    assert_eq!(sb.page_size, Some(4096));
}

• Step 2: Run test to verify it fails

cargo test -p clawhdf5-format file_writer_v4_superblock 2>&1 | head -20

Expected: compile error — with_page_size not found.

• Step 3: Add page_size field to FileWriter

In FileWriter struct definition, add page_size: Option<u32>. In FileWriter::new(), add page_size: None. Add method:

pub fn with_page_size(&mut self, page_size: u32) -> &mut Self {
    self.page_size = Some(page_size);
    self
}

• Step 4: Update Superblock::serialize for v4

In crates/clawhdf5-format/src/superblock.rs, the serialize() method currently hardcodes v2/v3 format. Update it to emit v4 when self.version == 4 and self.page_size.is_some():

pub fn serialize(&self) -> Vec<u8> {
    let mut buf = Vec::with_capacity(60);
    buf.extend_from_slice(&HDF5_SIGNATURE);
    buf.push(self.version);
    buf.push(self.offset_size);
    buf.push(self.length_size);
    buf.push(self.consistency_flags as u8);
    Self::write_offset(&mut buf, self.base_address, self.offset_size);
    let ext_addr = self.superblock_extension_address.unwrap_or(u64::MAX);
    Self::write_offset(&mut buf, ext_addr, self.offset_size);
    Self::write_offset(&mut buf, self.eof_address, self.offset_size);
    Self::write_offset(&mut buf, self.root_group_address, self.offset_size);
    if self.version >= 4 {
        let ps = self.page_size.unwrap_or(0);
        buf.extend_from_slice(&ps.to_le_bytes());
    }
    let checksum = crate::checksum::jenkins_lookup3(&buf);
    buf.extend_from_slice(&checksum.to_le_bytes());
    buf
}

• Step 5: Wire page_size into FileWriter::finish()

In file_writer.rs:finish(), where the Superblock is constructed (around line 1291), change:

let sb = Superblock {
    version: if self.page_size.is_some() { 4 } else { 3 },
    // ... existing fields ...
    page_size: self.page_size,
    // ... rest of fields unchanged ...
};

• Step 6: Run the test

cargo test -p clawhdf5-format file_writer_v4_superblock 2>&1

Expected: PASS.

• Step 7: Run full suite

cargo test -p clawhdf5-format 2>&1 | tail -10

Expected: all tests pass (v3 serialize() must be byte-identical to before — add a regression test if needed).

• Step 8: Commit

git add crates/clawhdf5-format/src/file_writer.rs \
        crates/clawhdf5-format/src/superblock.rs
git commit -m "feat: write HDF5 superblock v4 when page_size is configured"

---

Verification

# Run all format tests
cargo test -p clawhdf5-format 2>&1 | tail -10

# Specifically verify new features
cargo test -p clawhdf5-format external_link_write_roundtrip 2>&1
cargo test -p clawhdf5-format data_layout_write 2>&1
cargo test -p clawhdf5-format virtual_dataset_write_roundtrip 2>&1
cargo test -p clawhdf5-format parse_v4_with_page_size 2>&1
cargo test -p clawhdf5-format file_writer_v4_superblock 2>&1

# Regression: v3 superblock still round-trips
cargo test -p clawhdf5-format write_superblock 2>&1