feat: write multi-block fractal heaps (root indirect block)
Dense attribute and dense link storage capped at a single fractal-heap direct block (~64 KiB of heap data — a few thousand objects); beyond that the writer produced an invalid oversized block. Lift the cap with a root indirect block. When the serialized objects don't fit in one direct block, build a root indirect block (FHIB) over multiple direct blocks sized by the doubling table (start 512, width 4, doubling per row up to 64 KiB). Objects are packed row-major across blocks, each block carries its logical block offset, and heap IDs encode each object's heap offset (block offset + position). The FRHP points root -> FHIB with the row count; unused slots in the current rows are undefined. The fractal-heap builder is unified: FractalHeapBlock now carries the full heap blob, and a shared write_frhp helper serializes the header for both the single-block and multi-block paths. The single-block path is unchanged (byte-identical), so existing dense attrs/links stay valid. Validated end-to-end: a 2,500-attribute object and a 2,500-link group round-trip through our reader and are read correctly by h5py. Objects still may not span a block (no huge-object path). Tests: facade round-trips for multi-block dense attrs and dense links, plus an h5py-gated interop test (verified against the real h5py environment). Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
@@ -519,3 +519,39 @@ print("OK")
|
||||
let out = run_python_output(&script);
|
||||
assert_eq!(out, "OK");
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// A_multiblock. Write a multi-direct-block fractal heap -> h5py reads
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
#[test]
|
||||
fn clawhdf5_writes_multiblock_heap_h5py_reads() {
|
||||
skip_if_no_python!();
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let path = dir.path().join("multiblock.h5");
|
||||
let path_str = path.display().to_string();
|
||||
|
||||
// ~1600 dense attributes overflow a single 64KiB fractal-heap direct block.
|
||||
let mut b = FileBuilder::new();
|
||||
let mut g = b.create_group("g");
|
||||
for i in 0..1600i64 {
|
||||
g.set_attr(&format!("attribute_number_{i:05}"), AttrValue::I64(i * 2));
|
||||
}
|
||||
g.create_dataset("d").with_i32_data(&[1]);
|
||||
b.add_group(g.finish());
|
||||
b.write(&path).unwrap();
|
||||
|
||||
let script = format!(
|
||||
r#"
|
||||
import h5py
|
||||
with h5py.File("{path_str}", "r") as f:
|
||||
a = f["g"].attrs
|
||||
assert len(a) == 1600, f"expected 1600 attrs, got {{len(a)}}"
|
||||
for i in (0, 1, 999, 1599):
|
||||
v = int(a[f"attribute_number_{{i:05}}"])
|
||||
assert v == i*2, f"attr {{i}} = {{v}}"
|
||||
print("OK")
|
||||
"#
|
||||
);
|
||||
assert_eq!(run_python_output(&script), "OK");
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user