fix(format): read Extensible Array chunk indexes correctly

A dataset with exactly one unlimited dimension — the ordinary
append-only case — is indexed by an Extensible Array. Only its first
few chunk entries (4 by default) sit inline in the index block, and
everything past them was read with the wrong layout. In the default
shape the 37th chunk onward came back from the wrong place: a
400-chunk dataset returned 364 wrong values while reporting success,
and beyond about a thousand chunks the read failed outright. Silently
wrong data is the worse half of that.

It survived because the only Extensible Array fixture in the suite had
three chunks — inside the inline limit — so no test ever reached a data
block.

Four layout errors, each confirmed against files written by HDF5 2.0 and
against the library source rather than inferred:

- super block `u` owns 2^(u/2) data blocks, not 2^u;
- each holds 2^((u+1)/2) * data_blk_min_elmts elements — the two
  quantities double every *other* level, a half step apart;
- a super block carries a block-offset field before its data block
  addresses, which was not skipped;
- the page-init bitmap belongs to the super block, one bit per page
  packed across all of its data blocks and read MSB-first, rather than
  living inside the data block; a paged data block also ends its prefix
  with a checksum before the first page.

Where the spec left room for doubt the file settled it: decoding a
paged block's elements and reading the chunk values they address
identifies the mapping exactly, and the bitmap's 68 set bits matched
the 34 data blocks x 2 pages that 200 000 elements need, which only
holds MSB-first.

New interop tests cross every boundary — 4, 37, 400, 5 000 and 200 000
chunks, the last with paged data blocks — plus sparse (uninitialised
pages taking fill values), gzip-filtered elements and a 2-D dataset.
All three fail against the old traversal.

Writing is untouched; this was a read-path bug.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-20 17:18:59 -07:00
co-authored by Claude Opus 5
parent 0901fb1499
commit 367faad7f7
4 changed files with 445 additions and 257 deletions
+34
View File
@@ -193,3 +193,37 @@ B-tree v2 backs dense attribute storage, v2 groups, shared object header
messages and chunk indexes, so opening an object that uses any of them is
enough. Both are now errors: depth is capped at 64, and traversal stops once it
has produced more records than the file could physically hold.
---
## Extensible Array chunk indexes read back wrong data past the inline elements
**Status:** fixed on `main` (2026-09-20), after v2.6.0. **Every release up to
and including v2.6.0 is affected.**
A dataset created with exactly one unlimited dimension (`maxshape=(None, ...)`,
the usual append-only/resizable case) is indexed by an Extensible Array. Its
index block holds the first `idx_blk_elmts` chunk entries inline — 4 by
default — and everything after that lives in data blocks and super blocks whose
layout `clawhdf5-format` computed incorrectly.
Consequences, by dataset size (1 chunk per element):
| chunks | result before the fix |
|---|---|
| <= 36 | correct (inline, plus two data blocks that happened to line up) |
| 37 | 1 element wrong |
| 400 | 364 elements wrong |
| >= ~1000 | `invalid Extensible Array data block signature` |
The dangerous case is the middle one: values were returned from the wrong
chunks rather than an error being raised. Any reader that accepted the data at
face value saw plausible but incorrect numbers.
The root causes were the super block sizing formulas (`ndblks` and
`dblk_nelmts` each double every *other* level, a half-step apart), a missing
block-offset field in the super block, and a page-init bitmap read from the
wrong structure. All four are fixed and covered by interop tests against
HDF5 2.0 at sizes that cross each boundary, including paged data blocks.
Files written by this crate are unaffected — this was purely a read-path bug.