feat: write multi-block fractal heaps (root indirect block)

Dense attribute and dense link storage capped at a single fractal-heap direct
block (~64 KiB of heap data — a few thousand objects); beyond that the writer
produced an invalid oversized block. Lift the cap with a root indirect block.

When the serialized objects don't fit in one direct block, build a root
indirect block (FHIB) over multiple direct blocks sized by the doubling table
(start 512, width 4, doubling per row up to 64 KiB). Objects are packed
row-major across blocks, each block carries its logical block offset, and heap
IDs encode each object's heap offset (block offset + position). The FRHP points
root -> FHIB with the row count; unused slots in the current rows are undefined.

The fractal-heap builder is unified: FractalHeapBlock now carries the full heap
blob, and a shared write_frhp helper serializes the header for both the
single-block and multi-block paths. The single-block path is unchanged
(byte-identical), so existing dense attrs/links stay valid.

Validated end-to-end: a 2,500-attribute object and a 2,500-link group round-trip
through our reader and are read correctly by h5py. Objects still may not span a
block (no huge-object path).

Tests: facade round-trips for multi-block dense attrs and dense links, plus an
h5py-gated interop test (verified against the real h5py environment).

Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
osobh
2026-06-04 01:37:42 +00:00
co-authored by Claude Opus 4.8
parent 0aab49f2f0
commit 0754afb7f2
4 changed files with 375 additions and 19 deletions
@@ -519,3 +519,39 @@ print("OK")
let out = run_python_output(&script);
assert_eq!(out, "OK");
}
// ---------------------------------------------------------------------------
// A_multiblock. Write a multi-direct-block fractal heap -> h5py reads
// ---------------------------------------------------------------------------
#[test]
fn clawhdf5_writes_multiblock_heap_h5py_reads() {
skip_if_no_python!();
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("multiblock.h5");
let path_str = path.display().to_string();
// ~1600 dense attributes overflow a single 64KiB fractal-heap direct block.
let mut b = FileBuilder::new();
let mut g = b.create_group("g");
for i in 0..1600i64 {
g.set_attr(&format!("attribute_number_{i:05}"), AttrValue::I64(i * 2));
}
g.create_dataset("d").with_i32_data(&[1]);
b.add_group(g.finish());
b.write(&path).unwrap();
let script = format!(
r#"
import h5py
with h5py.File("{path_str}", "r") as f:
a = f["g"].attrs
assert len(a) == 1600, f"expected 1600 attrs, got {{len(a)}}"
for i in (0, 1, 999, 1599):
v = int(a[f"attribute_number_{{i:05}}"])
assert v == i*2, f"attr {{i}} = {{v}}"
print("OK")
"#
);
assert_eq!(run_python_output(&script), "OK");
}