quantumclaw/clawhdf5
3
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

writer: v2 B-trees with internal nodes (no 65 535-record limit)

Browse Source 
Dense link and attribute indexes and the chunk index for several
unlimited dimensions were single leaves, capping them at 65 535
records. btree_v2_write builds trees of any depth, with node capacities
and pointer widths from libhdf5's H5B2__hdr_init arithmetic (now shared
with the reader as btree_v2::node_info) and libhdf5's node sizes (512
dense, 2048 chunks). Indexes that fit the old one-leaf layout are
written byte for byte as before (compared for 10..65 535 links, attrs
and chunks, tracked and filtered).

Tests: 100 000 links (short names; long names with creation order),
70 000 attributes, 200 000 chunks (and 80 000 deflated), read by h5py,
h5dump and clawhdf5 and edited by h5py r+; h5rs check on the same
shapes, asserting depths 2-3.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-26 10:12:07 -05:00
co-authored by Claude Opus 5.5
parent 7acfb79584
commit d63c76e7ab
11 changed files with 1005 additions and 185 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+14
View File
@@ -3,6 +3,20 @@
## Unreleased
### Writer: large dense indexes (2026-09-26)
- **No more 65 535-record limit on the writer's v2 B-trees.** Dense link
storage (name index and creation-order index), dense attribute storage
and the chunk index of datasets with more than one unlimited dimension
were written as a single leaf, so a group with more than 65 535 links, an
object with more than 65 535 dense attributes, or such a dataset with more
than 65 535 chunks was an error. The writer now builds internal nodes to
any depth (`clawhdf5_format::btree_v2_write`), with node capacities and
child-pointer widths from the same arithmetic as libhdf5's
`H5B2__hdr_init` (shared with the reader, `btree_v2::node_info`) and
libhdf5's node sizes (512 bytes for dense storage, 2048 for chunks).
Indexes that fit the old one-leaf layout are written byte for byte as
before. New tests `crates/clawhdf5/tests/deep_btree_interop.rs` (100 000
links, 70 000 attributes, 200 000 chunks; h5py, h5dump, clawhdf5, and
h5py "r+" edits) and `check_files_with_deep_btrees` (`h5rs check`).
- **Links and attributes whose name hashes collide are found by name.** The
dense name indexes (a group's links: B-tree v2 type 5; an object's
attributes: type 8) are ordered by the name's lookup3 hash and, when two