writer: v2 B-trees with internal nodes (no 65 535-record limit)

Dense link and attribute indexes and the chunk index for several
unlimited dimensions were single leaves, capping them at 65 535
records. btree_v2_write builds trees of any depth, with node capacities
and pointer widths from libhdf5's H5B2__hdr_init arithmetic (now shared
with the reader as btree_v2::node_info) and libhdf5's node sizes (512
dense, 2048 chunks). Indexes that fit the old one-leaf layout are
written byte for byte as before (compared for 10..65 535 links, attrs
and chunks, tracked and filtered).

Tests: 100 000 links (short names; long names with creation order),
70 000 attributes, 200 000 chunks (and 80 000 deflated), read by h5py,
h5dump and clawhdf5 and edited by h5py r+; h5rs check on the same
shapes, asserting depths 2-3.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-26 10:12:07 -05:00
co-authored by Claude Opus 5.5
parent 7acfb79584
commit d63c76e7ab
11 changed files with 1005 additions and 185 deletions
@@ -559,20 +559,6 @@ fn we_write_btree_v2_for_several_unlimited_dims() {
check_we_write(&cases);
}
/// A single-leaf B-tree has a 16-bit record count; beyond it the writer
/// refuses rather than writing a tree libhdf5 would misread.
#[test]
fn btree_v2_index_past_one_leaf_is_refused() {
let mut b = FileBuilder::new();
b.create_dataset("d")
.with_i32_data(&vec![0i32; 70_000])
.with_shape(&[70_000, 1])
.with_chunks(&[1, 1])
.with_maxshape(&[u64::MAX, u64::MAX]);
let dir = tempfile::tempdir().unwrap();
assert!(b.write(dir.path().join("too_many.h5")).is_err());
}
/// A maxshape equal to the shape cannot grow, so it needs no chunks: the
/// dataset stays contiguous (as h5py makes it) unless chunks are requested.
#[test]