Merge branch 'feat/p2b-writer-btree-internal-nodes' into feat/p2b-scale

# Conflicts:
#	CHANGELOG.md
This commit is contained in:
osobh
2026-09-26 11:57:11 -05:00
15 changed files with 1587 additions and 292 deletions
+41
View File
@@ -48,6 +48,47 @@
with partial edge chunks, sparse datasets and fill values, and datasets
larger than the chunk cache.
### Writer: large dense indexes (2026-09-26)
- **`track_order` orders attributes too, as h5py's `track_order=True`
does.** It tracked link creation order only, so h5py listed a tracked
object's attributes by name. A tracking object's header now has the
attribute creation order tracked and indexed flags, an Attribute Info
message with the next creation order (also for inline attributes), and a
creation order on each inline attribute message; dense attribute storage
gets a creation-order index (B-tree type 9). `FileWriter::track_order` /
`FileBuilder::track_order` now apply to datasets' attributes as well, and
`DatasetBuilder::track_order` sets it per dataset. Groups and datasets
that track order are written differently from before; others are
unchanged. More than 65 535 attributes on a tracking object is an error
(libhdf5's creation order counter is 2 bytes). The reader
(`attribute::extract_attributes*`) lists a tracking object's attributes
in creation order. Test `track_order_lists_attributes_in_creation_order`
(h5py lists, reads and extends them in "r+" mode, including libhdf5's
move from inline to dense storage).
- **No more 65 535-record limit on the writer's v2 B-trees.** Dense link
storage (name index and creation-order index), dense attribute storage
and the chunk index of datasets with more than one unlimited dimension
were written as a single leaf, so a group with more than 65 535 links, an
object with more than 65 535 dense attributes, or such a dataset with more
than 65 535 chunks was an error. The writer now builds internal nodes to
any depth (`clawhdf5_format::btree_v2_write`), with node capacities and
child-pointer widths from the same arithmetic as libhdf5's
`H5B2__hdr_init` (shared with the reader, `btree_v2::node_info`) and
libhdf5's node sizes (512 bytes for dense storage, 2048 for chunks).
Indexes that fit the old one-leaf layout are written byte for byte as
before. New tests `crates/clawhdf5/tests/deep_btree_interop.rs` (100 000
links, 70 000 attributes, 200 000 chunks; h5py, h5dump, clawhdf5, and
h5py "r+" edits) and `check_files_with_deep_btrees` (`h5rs check`).
- **Links and attributes whose name hashes collide are found by name.** The
dense name indexes (a group's links: B-tree v2 type 5; an object's
attributes: type 8) are ordered by the name's lookup3 hash and, when two
hashes are equal, by the name itself, as libhdf5 compares them. The writer
broke ties by insertion order, so libhdf5 could not open one of two
colliding names (`"k69209"` and `"k155448"` share hash `0x3a0b13e6`;
collisions are likely from about 77 000 names). Regression test
`names_whose_hashes_collide_are_found_by_name` in
`crates/clawhdf5/tests/writer_groups_interop.rs`.
### Concurrent reads (2026-09-26)
- **Full reads of chunked datasets scale with threads again when rayon's
pool has one thread.** Each full read handed its chunks to rayon to