fix(format): write child indirect blocks in big fractal heaps
Dense link and attribute storage keeps its messages in a fractal heap. Its
root indirect block holds direct blocks up to 64 KiB, 512 KiB in all; rows
past that are child indirect blocks. The writer kept adding rows of direct
blocks instead, and libhdf5 and h5rs read them as indirect blocks: a group
with 20 000 links of 20-byte names was written without error and could not
be listed ("incorrect metadata checksum"), and 150 dense attributes of up
to 56 KB could not be opened. The heap writer now follows the doubling
table: rows past the direct ones hold child indirect blocks, each with its
own rows, nested as deep as the heap needs.
Two more heap bugs are fixed on the way. An object bigger than the next
block's free space was written into it anyway and cut off; the block is
now left unallocated and the object goes in the first block big enough, as
libhdf5 skips blocks. And the header's next-block offset was 0, so libhdf5
adding a link to such a group overwrote the heap's first block ("bad
version number for message"); it is now the offset after the last block.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
@@ -858,3 +858,34 @@ fn check_and_dump_files_with_nested_groups_and_links() {
|
||||
assert_eq!(stdout(&ours), r, "{name}");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn check_files_with_big_dense_storage() {
|
||||
// Dense links and attributes past the 512 KiB the root indirect block's
|
||||
// direct blocks hold: the heap then needs child indirect blocks, which
|
||||
// the writer used to write as direct blocks ("fractal heap indirect
|
||||
// block: bad signature").
|
||||
use clawhdf5::{AttrValue, FileBuilder};
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let mut b = FileBuilder::new();
|
||||
let x = b.create_dataset("x");
|
||||
x.with_i32_data(&[7]);
|
||||
for i in 0..150usize {
|
||||
let len = if i % 3 == 0 { 7_000 } else { 1 + i };
|
||||
x.set_attr(
|
||||
&format!("a{i:03}"),
|
||||
AttrValue::F64Array(vec![i as f64; len]),
|
||||
);
|
||||
}
|
||||
let mut g = b.create_group("g");
|
||||
for i in 0..40_000 {
|
||||
g.add_hard_link(&format!("link_{i:06}_{}", "x".repeat(88)), "/x");
|
||||
}
|
||||
b.add_group(g.finish());
|
||||
let p = dir.path().join("big.h5").to_string_lossy().into_owned();
|
||||
b.write(&p).unwrap();
|
||||
// Structure only: `--data` looks every link up by a linear scan.
|
||||
let o = h5rs(&["check", &p]);
|
||||
assert_eq!(code(&o), 0, "{p}:\n{}", stdout(&o));
|
||||
assert!(stdout(&o).contains("no problems found"), "{}", stdout(&o));
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user