Merge branch 'feat/p2b-writer-btree-internal-nodes' into feat/p2b-scale

# Conflicts:
#	CHANGELOG.md
This commit is contained in:
osobh
2026-09-26 11:57:11 -05:00
15 changed files with 1587 additions and 292 deletions
+30 -6
View File
@@ -275,10 +275,29 @@ fill-value item that did is fixed).
external links at any depth and optional creation-order tracking;
h5py, h5dump and `h5rs check --data` read them
(`crates/clawhdf5/tests/writer_groups_interop.rs`,
`crates/clawhdf5-tools/tests/h5rs_interop.rs`). Still missing: a group
with more than 65 535 links, or an object with more than 65 535 dense
attributes, is an error (the index is one B-tree leaf), and attribute
creation order is not tracked.
`crates/clawhdf5-tools/tests/h5rs_interop.rs`). ~~Still missing:
attribute creation order is not tracked.~~ **Fixed 2026-09-26:**
`track_order` (file default, `GroupBuilder`, and the new
`DatasetBuilder::track_order`) tracks and indexes attribute creation
order as h5py's `track_order=True` does; h5py lists the attributes in
the order they were set, inline and dense (20 000 on one dataset), and
keeps numbering them in "r+" mode (tank,
`cargo test -p clawhdf5 --test writer_groups_interop
track_order_lists_attributes_in_creation_order`). libhdf5 numbers at
most 65 535 attributes on such an object, so more is an error.
- ~~A group with more than 65 535 links, or an object with more than
65 535 dense attributes, is an error (the index is one B-tree leaf).~~
**Fixed 2026-09-26:** the dense indexes are v2 B-trees of any depth
(libhdf5's 512-byte nodes once the records outgrow the one-leaf layout,
which smaller indexes keep byte for byte). Tested on tank with 100 000
links in one group (short names, and 111-byte names with creation order
tracked) and 70 000 attributes on one object: h5py lists them in order
and reads the values, h5dump reads spot checks, `h5rs check` reads every
record, and h5py in "r+" mode adds and deletes thousands of links and
attributes in those trees (`cargo test -p clawhdf5 --test
deep_btree_interop`; `cargo test -p clawhdf5-tools --test h5rs_interop
check_files_with_deep_btrees`). The name indexes are ordered by hash and
then name, as libhdf5 needs for names whose hashes collide.
- ~~Dense link or attribute storage past 512 KiB of messages was written
unreadable (child indirect blocks of the fractal heap written as direct
blocks).~~ **Fixed 2026-09-26** (it affected 2.7.0 too): tested with
@@ -291,8 +310,13 @@ fill-value item that did is fixed).
an object, or more than 8 links in a group) one attribute or link
message over 65 515 bytes is an error.
- Output that HDF5 1.8 can read.
- A B-tree v2 chunk index larger than one leaf, so datasets with several
unlimited dimensions are limited to 65 535 chunks.
- ~~A B-tree v2 chunk index larger than one leaf, so datasets with
several unlimited dimensions are limited to 65 535 chunks.~~ **Fixed
2026-09-26:** more chunks get libhdf5's 2048-byte nodes with internal
nodes above the leaves. Tested on tank with 200 000 chunks (and 80 000
deflated): h5py and clawhdf5 read every value, and h5py resizes the
dataset and writes 4 510 new chunks into the tree (same commands as
above).
---