quantumclaw/clawhdf5
3
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

feat(format): read chunked datasets indexed by a version-2 B-tree

Browse Source 
With libver='latest', a chunked dataset with two or more unlimited dimensions
indexes its chunks with a v2 B-tree (layout v4, index type 5). Reading one
failed with "unsupported chunked layout version=4, index_type=Some(5)".

read_btree_v2_chunks decodes record types 10 (address + scaled offsets) and 11
(address, stored size, filter mask, scaled offsets). The width of the
stored-size field is taken from the record size the tree header declares
rather than re-deriving the library's formula. Scaled offsets are multiplied
back by the chunk dimensions with overflow checks.

The chunk-index dispatch existed four times (uncached, cached, sweep and
indexed readers). The three copies outside list_chunks now call it, so every
read path — and fill-value handling and partial reads — supports every index
type from one place.

h5py interop test: plain, gzip+shuffle, a 2500-chunk tree with internal nodes,
a sparse dataset with a fill value, and a strided hyperslab.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
This commit is contained in:
osobh
2026-09-19 13:59:18 -07:00
co-authored by Claude Fable 5.1
parent c6a7bbfc67
commit d668e45ab5
4 changed files with 229 additions and 256 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/known-issues.md
+8 -7
View File
@@ -116,16 +116,17 @@ not reported).
## B-tree v2 chunk index (layout v4, index type 5) is not supported
**Status:** open.
**Status:** fixed 2026-09-19.
**Summary:** a chunked dataset with **two or more unlimited dimensions** written
with `libver='latest'` indexes its chunks with a version-2 B-tree. Reading it
fails with `ChunkedReadError("unsupported chunked layout version=4,
index_type=Some(5)")`. Single-chunk, implicit, fixed-array and
extensible-array indexes (and the v3 B-tree v1) are supported.
with `libver='latest'` indexes its chunks with a version-2 B-tree, and reading it
failed with `unsupported chunked layout version=4, index_type=Some(5)`.
**Repro:** `f.create_dataset("d", shape=(5, 7), chunks=(2, 3), maxshape=(None, None))`
with `h5py.File(..., libver='latest')`.
**Fix:** record types 10 (unfiltered) and 11 (filtered) are decoded — address,
stored size, filter mask, scaled offsets — through the shared chunk-listing
function, so full reads, cached reads, partial reads and fill-value handling
all work. Covered by an h5py interop test (plain, gzip+shuffle, a 2500-chunk
tree with internal nodes, a sparse dataset with a fill value, a hyperslab).
## External links and external raw data are not followed