quantumclaw/clawhdf5
3
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

writer: v2 B-trees with internal nodes (no 65 535-record limit)

Browse Source 
Dense link and attribute indexes and the chunk index for several
unlimited dimensions were single leaves, capping them at 65 535
records. btree_v2_write builds trees of any depth, with node capacities
and pointer widths from libhdf5's H5B2__hdr_init arithmetic (now shared
with the reader as btree_v2::node_info) and libhdf5's node sizes (512
dense, 2048 chunks). Indexes that fit the old one-leaf layout are
written byte for byte as before (compared for 10..65 535 links, attrs
and chunks, tracked and filtered).

Tests: 100 000 links (short names; long names with creation order),
70 000 attributes, 200 000 chunks (and 80 000 deflated), read by h5py,
h5dump and clawhdf5 and edited by h5py r+; h5rs check on the same
shapes, asserting depths 2-3.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-26 10:12:07 -05:00
co-authored by Claude Opus 5.5
parent 7acfb79584
commit d63c76e7ab
11 changed files with 1005 additions and 185 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/known-issues.md
+22 -6
View File
@@ -264,10 +264,21 @@ fill-value item that did is fixed).
external links at any depth and optional creation-order tracking;
h5py, h5dump and `h5rs check --data` read them
(`crates/clawhdf5/tests/writer_groups_interop.rs`,
`crates/clawhdf5-tools/tests/h5rs_interop.rs`). Still missing: a group
with more than 65 535 links, or an object with more than 65 535 dense
attributes, is an error (the index is one B-tree leaf), and attribute
creation order is not tracked.
`crates/clawhdf5-tools/tests/h5rs_interop.rs`). Still missing:
attribute creation order is not tracked.
- ~~A group with more than 65 535 links, or an object with more than
65 535 dense attributes, is an error (the index is one B-tree leaf).~~
**Fixed 2026-09-26:** the dense indexes are v2 B-trees of any depth
(libhdf5's 512-byte nodes once the records outgrow the one-leaf layout,
which smaller indexes keep byte for byte). Tested on tank with 100 000
links in one group (short names, and 111-byte names with creation order
tracked) and 70 000 attributes on one object: h5py lists them in order
and reads the values, h5dump reads spot checks, `h5rs check` reads every
record, and h5py in "r+" mode adds and deletes thousands of links and
attributes in those trees (`cargo test -p clawhdf5 --test
deep_btree_interop`; `cargo test -p clawhdf5-tools --test h5rs_interop
check_files_with_deep_btrees`). The name indexes are ordered by hash and
then name, as libhdf5 needs for names whose hashes collide.
- ~~Dense link or attribute storage past 512 KiB of messages was written
unreadable (child indirect blocks of the fractal heap written as direct
blocks).~~ **Fixed 2026-09-26** (it affected 2.7.0 too): tested with
@@ -280,8 +291,13 @@ fill-value item that did is fixed).
an object, or more than 8 links in a group) one attribute or link
message over 65 515 bytes is an error.
- Output that HDF5 1.8 can read.
- A B-tree v2 chunk index larger than one leaf, so datasets with several
unlimited dimensions are limited to 65 535 chunks.
- ~~A B-tree v2 chunk index larger than one leaf, so datasets with
several unlimited dimensions are limited to 65 535 chunks.~~ **Fixed
2026-09-26:** more chunks get libhdf5's 2048-byte nodes with internal
nodes above the leaves. Tested on tank with 200 000 chunks (and 80 000
deflated): h5py and clawhdf5 read every value, and h5py resizes the
dataset and writes 4 510 new chunks into the tree (same commands as
above).
---