Merge branch 'feat/p2b-writer-btree-internal-nodes' into feat/p2b-scale
# Conflicts: # CHANGELOG.md
This commit is contained in:
@@ -48,6 +48,47 @@
|
||||
with partial edge chunks, sparse datasets and fill values, and datasets
|
||||
larger than the chunk cache.
|
||||
|
||||
### Writer: large dense indexes (2026-09-26)
|
||||
- **`track_order` orders attributes too, as h5py's `track_order=True`
|
||||
does.** It tracked link creation order only, so h5py listed a tracked
|
||||
object's attributes by name. A tracking object's header now has the
|
||||
attribute creation order tracked and indexed flags, an Attribute Info
|
||||
message with the next creation order (also for inline attributes), and a
|
||||
creation order on each inline attribute message; dense attribute storage
|
||||
gets a creation-order index (B-tree type 9). `FileWriter::track_order` /
|
||||
`FileBuilder::track_order` now apply to datasets' attributes as well, and
|
||||
`DatasetBuilder::track_order` sets it per dataset. Groups and datasets
|
||||
that track order are written differently from before; others are
|
||||
unchanged. More than 65 535 attributes on a tracking object is an error
|
||||
(libhdf5's creation order counter is 2 bytes). The reader
|
||||
(`attribute::extract_attributes*`) lists a tracking object's attributes
|
||||
in creation order. Test `track_order_lists_attributes_in_creation_order`
|
||||
(h5py lists, reads and extends them in "r+" mode, including libhdf5's
|
||||
move from inline to dense storage).
|
||||
- **No more 65 535-record limit on the writer's v2 B-trees.** Dense link
|
||||
storage (name index and creation-order index), dense attribute storage
|
||||
and the chunk index of datasets with more than one unlimited dimension
|
||||
were written as a single leaf, so a group with more than 65 535 links, an
|
||||
object with more than 65 535 dense attributes, or such a dataset with more
|
||||
than 65 535 chunks was an error. The writer now builds internal nodes to
|
||||
any depth (`clawhdf5_format::btree_v2_write`), with node capacities and
|
||||
child-pointer widths from the same arithmetic as libhdf5's
|
||||
`H5B2__hdr_init` (shared with the reader, `btree_v2::node_info`) and
|
||||
libhdf5's node sizes (512 bytes for dense storage, 2048 for chunks).
|
||||
Indexes that fit the old one-leaf layout are written byte for byte as
|
||||
before. New tests `crates/clawhdf5/tests/deep_btree_interop.rs` (100 000
|
||||
links, 70 000 attributes, 200 000 chunks; h5py, h5dump, clawhdf5, and
|
||||
h5py "r+" edits) and `check_files_with_deep_btrees` (`h5rs check`).
|
||||
- **Links and attributes whose name hashes collide are found by name.** The
|
||||
dense name indexes (a group's links: B-tree v2 type 5; an object's
|
||||
attributes: type 8) are ordered by the name's lookup3 hash and, when two
|
||||
hashes are equal, by the name itself, as libhdf5 compares them. The writer
|
||||
broke ties by insertion order, so libhdf5 could not open one of two
|
||||
colliding names (`"k69209"` and `"k155448"` share hash `0x3a0b13e6`;
|
||||
collisions are likely from about 77 000 names). Regression test
|
||||
`names_whose_hashes_collide_are_found_by_name` in
|
||||
`crates/clawhdf5/tests/writer_groups_interop.rs`.
|
||||
|
||||
### Concurrent reads (2026-09-26)
|
||||
- **Full reads of chunked datasets scale with threads again when rayon's
|
||||
pool has one thread.** Each full read handed its chunks to rayon to
|
||||
|
||||
Reference in New Issue
Block a user