feat: write dense group link storage (fractal heap + v2 B-tree)

A group with more than 8 links (libhdf5's compact max_compact default) is now
written densely instead of as inline Link messages: the links live in a
single-direct-block fractal heap indexed by a v2 B-tree of type 5 (link-name
index), referenced from the group's LinkInfo message. This matches libhdf5's
compact->dense switchover and keeps large groups out of the object header.

- Extract the byte-identical fractal-heap builder shared by dense attributes
  and dense links, parameterized by heap_id_length / max_heap_size. Attributes
  keep 8 / 40; links use 7 / 32 to match libhdf5 (reverse-engineered: an
  h5py-written dense group uses heap_id_length 7, max_heap_size 32, type-5
  record = hash(4) + heap_id(7) = 11 bytes). This was the cause of an initial
  "object overruns end of direct block" error from h5py.
- build_group_oh gained an optional dense LinkInfo (omitting inline Link
  messages); the two-pass file assembly allocates each group's link blob after
  its object header and rebuilds it with real target addresses in the final
  pass (link-message size is address-independent, so layout is stable).

Validated end-to-end: our reader round-trips dense groups, h5py reads the
groups we write, and dense attributes remain byte-identical (still h5py-valid).
The agent's 9-dataset memory group now writes densely and round-trips. Single
direct block only (~a couple thousand links); indirect blocks remain a TODO.

Tests: facade round-trip (20-link dense group + compact sibling) and an
h5py-gated interop test confirming libhdf5 reads our dense groups.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
osobh
2026-06-04 01:12:23 +00:00
co-authored by Claude Opus 4.8
parent e0189cd5c4
commit 20ad16ab69
4 changed files with 337 additions and 40 deletions
@@ -481,3 +481,41 @@ fn serde_json_minimal_parse(s: &str) -> Vec<f64> {
.map(|v| v.trim().parse::<f64>().unwrap())
.collect()
}
// ---------------------------------------------------------------------------
// A_dense. Write a group with many links (dense storage) -> h5py reads all
// ---------------------------------------------------------------------------
#[test]
fn clawhdf5_writes_dense_group_h5py_reads() {
skip_if_no_python!();
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("dense_group.h5");
let path_str = path.display().to_string();
// 20 links exceeds the compact threshold (8) -> dense fractal-heap storage.
let mut b = FileBuilder::new();
let mut g = b.create_group("big");
for i in 0..20 {
g.create_dataset(&format!("dataset_{i:03}"))
.with_i32_data(&[i, i * 10]);
}
b.add_group(g.finish());
b.write(&path).unwrap();
let script = format!(
r#"
import h5py
with h5py.File("{path_str}", "r") as f:
big = f["big"]
names = sorted(big.keys())
assert len(names) == 20, f"expected 20 links, got {{len(names)}}"
for i in range(20):
v = big[f"dataset_{{i:03}}"][()].tolist()
assert v == [i, i*10], f"link {{i}} = {{v}}"
print("OK")
"#
);
let out = run_python_output(&script);
assert_eq!(out, "OK");
}