feat: write dense group link storage (fractal heap + v2 B-tree)
A group with more than 8 links (libhdf5's compact max_compact default) is now written densely instead of as inline Link messages: the links live in a single-direct-block fractal heap indexed by a v2 B-tree of type 5 (link-name index), referenced from the group's LinkInfo message. This matches libhdf5's compact->dense switchover and keeps large groups out of the object header. - Extract the byte-identical fractal-heap builder shared by dense attributes and dense links, parameterized by heap_id_length / max_heap_size. Attributes keep 8 / 40; links use 7 / 32 to match libhdf5 (reverse-engineered: an h5py-written dense group uses heap_id_length 7, max_heap_size 32, type-5 record = hash(4) + heap_id(7) = 11 bytes). This was the cause of an initial "object overruns end of direct block" error from h5py. - build_group_oh gained an optional dense LinkInfo (omitting inline Link messages); the two-pass file assembly allocates each group's link blob after its object header and rebuilds it with real target addresses in the final pass (link-message size is address-independent, so layout is stable). Validated end-to-end: our reader round-trips dense groups, h5py reads the groups we write, and dense attributes remain byte-identical (still h5py-valid). The agent's 9-dataset memory group now writes densely and round-trips. Single direct block only (~a couple thousand links); indirect blocks remain a TODO. Tests: facade round-trip (20-link dense group + compact sibling) and an h5py-gated interop test confirming libhdf5 reads our dense groups. Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
@@ -481,3 +481,41 @@ fn serde_json_minimal_parse(s: &str) -> Vec<f64> {
|
||||
.map(|v| v.trim().parse::<f64>().unwrap())
|
||||
.collect()
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// A_dense. Write a group with many links (dense storage) -> h5py reads all
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
#[test]
|
||||
fn clawhdf5_writes_dense_group_h5py_reads() {
|
||||
skip_if_no_python!();
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let path = dir.path().join("dense_group.h5");
|
||||
let path_str = path.display().to_string();
|
||||
|
||||
// 20 links exceeds the compact threshold (8) -> dense fractal-heap storage.
|
||||
let mut b = FileBuilder::new();
|
||||
let mut g = b.create_group("big");
|
||||
for i in 0..20 {
|
||||
g.create_dataset(&format!("dataset_{i:03}"))
|
||||
.with_i32_data(&[i, i * 10]);
|
||||
}
|
||||
b.add_group(g.finish());
|
||||
b.write(&path).unwrap();
|
||||
|
||||
let script = format!(
|
||||
r#"
|
||||
import h5py
|
||||
with h5py.File("{path_str}", "r") as f:
|
||||
big = f["big"]
|
||||
names = sorted(big.keys())
|
||||
assert len(names) == 20, f"expected 20 links, got {{len(names)}}"
|
||||
for i in range(20):
|
||||
v = big[f"dataset_{{i:03}}"][()].tolist()
|
||||
assert v == [i, i*10], f"link {{i}} = {{v}}"
|
||||
print("OK")
|
||||
"#
|
||||
);
|
||||
let out = run_python_output(&script);
|
||||
assert_eq!(out, "OK");
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user