The Blosc2 fuzz tests checked only for panics and the size of the output,
so the offsets-chunk amplification passed all 40,000 iterations.
tests/blosc2_alloc_bounds.rs now measures peak allocation (a counting
global allocator) and asserts it stays within 6x the HDF5 chunk size plus
twice the input plus 2 MiB (ruzstd's fixed state) for:
- 20,000 mutated fixture frames, decoded with their real chunk size as
the limit, with edits aimed at the frame's and chunks' size fields;
- 20,000 mutated first chunks on their own;
- 5,000 frames built from random header sizes, offsets chunks and B2ND
shapes (padding chunk and block shapes, chunks larger than the array,
NaN, zero and repeated-value chunks).
Against the code before this series every test in the file fails (the
fuzz tests at frame iteration 3913, a zstd window, and random frame 289,
the offsets chunk); now the worst frame peaks at 0.48 of the bound.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
ruzstd reserves a frame's declared window (up to its 100 MiB default)
when a decoder is reset for a new frame, before decoding anything. The
Blosc, Blosc2 and bitshuffle decoders reuse one decoder per chunk, so a
Blosc2 chunk of two 16-byte streams, each declaring a 96 MiB window,
allocated 128 MiB. zstd_decode_into now sets the decoder's maximum window
to twice the stream's output (at least 128 KiB): c-blosc, c-blosc2 and
bitshuffle compress each block in one call with its size known, so
libzstd's window never exceeds the block.
Found by tracking peak allocation in the Blosc2 fuzz test.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
B2ND chunks were decoded whole, padding included, with up to 16x the HDF5
chunk size as their limit, so a crafted frame made each chunk allocate and
fill up to 16x the output (4 GiB for a 256 MiB HDF5 chunk). The padding is
the real bound (prod(ceil(c/b)*b) per chunk), but that can be 2^ndim times
the array, so it is no longer held at all:
- A Blosc2 chunk is now decoded block by block (decode_blocks), each block
handed to a sink as it is ready, with at most three blocks of scratch.
blosc2_decompress_chunk and plain frames still collect every block.
- reassemble places each B2ND block straight into the output and skips
blocks that are all padding (they are not decoded unless the delta
filter needs the first block). A frame's NaN chunks are handed over one
B2ND block at a time and its zero chunks cost nothing.
- A B2ND chunk must decode to exactly its padded size, its Blosc2 blocks
must be whole B2ND blocks no larger than the output, and a chunk may not
be larger than the array (hdf5-blosc2's chunk is the array), so a block
is never larger than the output.
Peak allocation for a 10-D array padded to 13x (NaN, repeated-value and
stored-block chunks) and for a 16x chunk was 4.5 MB and 17.8 MB for
315 KB and 1 MiB outputs before, and is now within the tests' bound.
Blosc2 files written by hdf5plugin in 9-D and 12-D, an 8 MiB single chunk,
1x1x1 and edge-chunk shapes still read exactly.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A chunk header with nbytes 0 and no special type kept its declared block
size (up to 512 MiB): the block size was clamped to nbytes only when nbytes
was positive, and the scratch blocks were allocated before the (empty)
block loop, so a 20-byte chunk allocated about 1 GiB. The block size is now
clamped to nbytes always, and an empty chunk returns before any scratch is
allocated.
A frame chunk must also decode to the size the frame header gives it
(chunksize, or the remainder for the last chunk), and is decoded with that
as its limit, so an empty chunk in a frame for a non-empty HDF5 chunk is an
error rather than an empty result.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
parse_frame sized the offsets chunk from the frame header's own nbytes and
chunksize, so a 173-byte frame declaring 32 Mi chunks, with a 40-byte
repeated-value offsets chunk, built 256 MiB (up to 2 GiB) of offsets for a
1 MiB HDF5 chunk and then returned 4 bytes. The offsets chunk is now capped
at the output limit (at least 128 bytes); a frame whose nbytes/chunksize
imply more chunks than that is refused before anything is allocated.
tests/blosc2_alloc_bounds.rs measures peak allocation with a counting
global allocator; the reviewer's frame failed it (decoded Ok(4)) before.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Review follow-ups. With run_with_helpers a one-thread rayon pool gave each
read a second core (the caller plus the worker), so --decode-threads 1 no
longer matched h5py's one core per call; such a pool now adds no helper.
Per-thread decode scratch is kept up to 1 MiB per buffer (was 4 MiB),
bounding what never-exiting pool workers hold.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Placing a chunk cast its u64 offsets to usize; on a 32-bit target an
offset past the address space wrapped into the output (and could then
overlap another chunk's region when chunks are placed concurrently).
Such an offset is past the dataset, so it now saturates and the chunk
writes nothing, as the concurrent-placement check already assumed. No
change on 64-bit targets, where the cast cannot wrap (so no test here).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
CHANGELOG entry for the chunked read changes, and the known-issues entry
on concurrent chunked reads updated: both causes it names (per-read page
faults, readers waiting on a small pool) are fixed; the 16-thread
comparison with h5py stays open until re-measured on an idle machine.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Record the `blosc2` feature in the changelog, the README's feature table
and the crate table, and mark the Blosc2 half of the known "Filters"
issue fixed (dated, with the conformance run that shows h5ex_d_blosc2
reading). What stays open: ZFP, writing Blosc2, and the Blosc2 features
hdf5plugin never writes (dictionaries, lazy chunks, variable-length
blocks, user-defined codecs and registered filters), which are errors.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
hdf5plugin's Blosc2 was a clear "not implemented" error. It stores each
HDF5 chunk as a Blosc2 contiguous frame; for chunks of 2+ dimensions the
frame holds a B2ND array whose data are cut into (padded) blocks stored
one after another.
filters_blosc2 (feature `blosc2`, in `plugin-filters`; the facade
forwards both) decodes, following c-blosc2's decoder and hdf5-blosc2's
blosc2_filter.c:
- the frame: the msgpack header's fixed fields, the metalayer index, the
offsets chunk (with the special zero/NaN/uninitialised offsets) and
chunk lookup;
- Blosc2 chunks: 16- and 32-byte headers, special chunks (zeros, NaN,
uninitialised, one repeated value), split and unsplit streams, zero and
run-length streams, and the filter pipeline run backwards (shuffle,
shuffle with a byte-group size, bit shuffle including the version-2 and
later handling of a partial group of 8, delta against the first block,
truncated precision);
- the codecs, shared with Blosc 1: BloscLZ, LZ4/LZ4HC, Zlib, Zstandard;
- B2ND arrays: blocks gathered into C order, padding dropped, several
chunks per array, and the array shape checked against the chunk shape
in cd_values as the HDF5 filter does.
Dictionaries, lazy chunks, variable-length blocks, user-defined codecs
and registered filters (e.g. bytedelta) are errors. Uninitialised chunks
read as zeros. No encoder.
Tests: h5py + hdf5plugin write every codec x filter (none, shuffle,
bitshuffle, delta) and levels 0-9 over the plugin-filter cases, then
i1..u8/f4/f8 in 1-D to 5-D chunks with partial edge chunks, datasets of
zeros, one value and NaN, and Fletcher32 before Blosc2 (plain frames for
n-D chunks); clawhdf5 reads each exactly as its unfiltered twin, and
truncated precision exactly as h5py reads it. Fixture frames from
python-blosc2 (tests/fixtures/blosc2/generate.py) cover what hdf5plugin
never writes: special chunks, delta over many blocks and odd type sizes,
odd bit-shuffle blocks, forced splitting, multi-chunk B2ND arrays with a
zero chunk, and the refused features. The decoder is fuzzed (random and
mutated frames and chunks: no panic, output within the limit).
Conformance: h5ex_d_blosc2.h5 now reads (576 of 697 ok, baseline 575).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A selection read (a hyperslab or points covering at most half the
dataset) decoded each chunk it overlaps into fresh buffers, one per
filter stage; it now uses the thread's chunk-decoding scratch like the
full readers. Covered by tests/chunked_read_paths_interop.rs (small
hyperslabs and points over every filter and type) and the partial-read
equivalence tests.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
read_f32/read_f64/read_i32/read_i64/read_u64 of a chunked dataset that
stores exactly that type in native byte order now decode every chunk
straight into the Vec<T> they return (data_read::read_chunked_native),
on File (through its chunk cache), MmapFile and LazyFile. Before, the
chunks went into a byte buffer that read_as_* then copied into a second,
typed one: two dataset-sized allocations and a full extra copy per read.
The output is zeroed pages from the allocator, backed by transparent
huge pages when large, like the byte reader's. Other types and byte
orders, and datasets with no storage or external data, keep converting
through the byte readers; unallocated chunks read as the fill value as
before.
tests/chunked_read_paths_interop.rs checks every chunked read path
(File twice, so cached; from_bytes; MmapFile; LazyFile; small, strided
and point selections; with and without the parallel feature) against
h5py for 1-8 byte integers and 2-8 byte floats in both byte orders,
through deflate, shuffle, Fletcher32, LZF, SZIP and Blosc, with partial
edge chunks, sparse datasets with default and non-default fill values,
and datasets larger than the chunk cache. A filter this build lacks must
be an error (or, when an optional filter declined every chunk, the right
data).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
h5py's track_order=True orders attributes as well as links; the writer
tracked links only. A tracking object's header now sets the attribute
creation order tracked/indexed flags and carries per-message creation
orders, an Attribute Info message holds the next order (inline too),
and dense storage gets a type-9 creation-order index. The file default
applies to datasets, with DatasetBuilder::track_order per dataset; more
than 65 535 attributes on a tracking object is an error (libhdf5's
counter is 2 bytes). The reader lists such attributes in creation
order.
h5py lists them in order (inline, dense, 20 000 on one dataset) and
keeps numbering in r+ mode, including its inline-to-dense move.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Full reads of a chunked dataset (the cached reader behind the facade's
read_* and the uncached one behind mmap/lazy files and verify_provenance)
now decode each chunk into this thread's reusable scratch buffers and copy
it straight to its place in the output. Before, the cached reader decoded
batches of 128 chunks into fresh Vecs and the uncached one decoded every
chunk of the dataset into its own buffer before assembling any: a new
256 KiB allocation (and its page faults) per chunk and per filter stage.
Chunks are still inserted into the file's chunk cache when the whole
dataset fits in it.
With the parallel feature the calling thread now decodes too, sharing the
chunks with whichever rayon workers are free (run_with_helpers): a helper
the busy pool only starts after the read is done returns at once. Before,
the caller handed every chunk to the pool and slept, so readers outside a
small pool (2-4 threads) queued behind its workers; with a one-thread pool
the reads went sequential. Chunks are placed concurrently only when the
index puts them on the chunk grid at distinct places (a corrupt index is
read one chunk at a time), and the error returned is still the first
failing chunk's.
Fix: a chunk stored unfiltered in a filtered dataset (every filter-mask
bit set) that is shorter than a chunk read as zeros where its data was
missing through the cached reader (the facade's read_*); it is now an
error naming the chunk, as the uncached reader already made it.
Regression tests, both failing before this change:
tests/busy_decode_pool.rs (both workers of a two-thread pool busy, four
readers) and short_unfiltered_chunk_of_a_filtered_dataset_is_an_error.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Dense link and attribute indexes and the chunk index for several
unlimited dimensions were single leaves, capping them at 65 535
records. btree_v2_write builds trees of any depth, with node capacities
and pointer widths from libhdf5's H5B2__hdr_init arithmetic (now shared
with the reader as btree_v2::node_info) and libhdf5's node sizes (512
dense, 2048 chunks). Indexes that fit the old one-leaf layout are
written byte for byte as before (compared for 10..65 535 links, attrs
and chunks, tracked and filtered).
Tests: 100 000 links (short names; long names with creation order),
70 000 attributes, 200 000 chunks (and 80 000 deflated), read by h5py,
h5dump and clawhdf5 and edited by h5py r+; h5rs check on the same
shapes, asserting depths 2-3.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
decompress_chunk_exact_with decodes a chunk into a DecodeScratch the
caller keeps between chunks, instead of a new Vec per chunk and per filter
stage. Deflate inflates into a kept buffer with a reset (not rebuilt)
inflater, shuffle interleaves into the other buffer, and Fletcher32 checks
and drops its checksum in place (on the stored bytes when it is the first
filter undone). Other filters go through the registry as before. Output
and errors are those of decompress_chunk_exact; a unit test checks that
for every pipeline shape and filter mask with one reused scratch.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
libhdf5 compares the name when two hashes are equal; the writer broke
ties by insertion order, and libhdf5 could not find one of two names
whose lookup3 hashes collide (k69209 / k155448). Test fails before the
fix with h5py's KeyError.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Idle tank at 408f69e, h5py re-run in the same session. Contiguous reads
went from 0.25x to 1.44x h5py (full) and 0.12x to 6.3x (256x256
hyperslabs) on one thread; deflate full reads at 8 threads 887 -> 2943
MB/s (h5py processes 3042). Full chunked reads at 16 threads are still
0.69x-0.76x h5py processes; the issue stays open.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
libhdf5 fails to read a VL element whose global heap address is
undefined (all 0xff), even at length 0 ("addr undefined"); we returned
"" (or an empty sequence) in every reader. Checked with h5py first:
libhdf5 writes a null element with address 0, which still reads as
empty, and h5py writes "" as a zero-size heap object at a real address,
so no file they write relies on the old behaviour. read_vl_bytes now
treats address 0 as null whatever the length, as VlResolver does.
Tests, each failing before: vl_data unit test (8- and 4-byte offsets,
lengths 0 and 1); clawhdf5 vl_data_interop
a_vl_element_at_the_undefined_heap_address_fails_like_h5py (also checks
where h5py writes ""); h5rs dump --json and check --data on the patched
`undef` dataset; clawhdf5-wasm vl_strings.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The READMEs said ds[...] reads only the selected elements, and the
facade's read_selection docs that only intersecting chunks are
decompressed. The bounding-box path runs only when the box covers at
most half the dataset; larger boxes (any strided slice across the
dataset), compact, virtual and unwritten datasets and chunked ones with
a non-default fill value decode the whole dataset. The READMEs, the
facade and format docs, the bindings' docstrings and known-issues now
say so, and how index lists are read.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
h5rs (dump, ls, diff, check --data) kept its own lenient VL decoder:
a heap object longer than its element was cut to the element's length
(libhdf5 and h5py refuse it), a null string printed "" where h5dump
prints NULL, the stored element size was trusted, and every heap
collection was kept as an owned copy for the whole run. It now resolves
each element with VlResolver::element / string_element (new: one element
in place, borrowing from the file), and refuses a VL type whose stored
element size is not 4 + offset size + 4, as File does. H5::heap_object
and its cache are gone. h5diff compares a null VL string equal to an
empty one; so does h5rs diff.
clawhdf5-wasm already resolved VL strings with read_vl_strings; it now
uses VlResolver and checks the stored element size before reading, as
File::read_string does.
Tests (h5py writes the files, patched for "a\0b", a null element and
mis-sized heap objects, with 8- and 4-byte offsets):
- h5rs_interop dump_prints_vl_data_like_h5dump: byte-identical to h5dump;
- dump_json_vl_values_match_h5py: h5py's values, errors where h5py fails;
- check_data_flags_mis_sized_vl_heap_objects;
- clawhdf5-wasm tests/vl_strings.rs: wasm, File and h5py agree.
All four fail before. check --data over the 150 cve_hdf5 CVE and fuzzer
files now passes 15 (h5dump rejects 8 of them), was 16 and 9: the
stored-size check flags cve-2024-32608. h5rs-check-ok-files.sh --data:
0 of 422 flagged; h5rs-fuzz.sh: clean on 180 files.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
test_threads_read_the_same_file passed with the GIL held. The new
test_reads_release_the_gil measures the longest stall of a spinning
Python thread while another reads: with py.detach removed from the read
it stalled 0.062 s of a 0.064 s read and failed; with it, about 3 ms.
test_errors_match_h5py now compares the result whenever h5py reads the
key, instead of only checking that we raise when h5py raises, over a
longer key list.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The changelog, known issues and README said a group holds up to 65 535
links while a group of about 17 000 was already unreadable. Record the
fixes (child indirect blocks, the next-block offset, the index leaf cap,
refusing oversized dense messages, hard-link memoisation, dataset
attribute overwrite) and the limits that remain true: 65 535 links or
dense attributes per object, and 65 515 bytes per dense message.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
ds[np.array(1)] went down the index-list path, where tolist() returns a
scalar and extracting a list of indices raised a confusing TypeError.
h5py treats it as an integer index; so do we now. The h5py comparison
keys include 0-d arrays (signed and unsigned) on each axis; they failed
before.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
b0a1e4f fixed this for group and root attributes only. Setting a dataset
attribute twice still wrote two attribute messages with one name, and h5py
read back the first value: set_attr("a", 1) then set_attr("a", 2) read as
1, and list(attrs) was ["a", "a"]. DatasetBuilder::set_attr now replaces
the earlier value, compact or dense. Likewise, a hand-set attribute named
like a provenance attribute (_provenance_sha256, ...) is replaced by the
computed one instead of being written next to it and read first.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Every ds[...] and g[k] resolved the path from the root again, two or
three times per open, and resolving a name in a large group scans its
links: visiting a group was O(n^2). 4000 scalar datasets in one group
took 39 s (v1 group) and 131 s (dense) to list, read and re-read; now
0.3 s each. A Dataset keeps its object address, a Group (and the file's
root) its address and, after the first lookup, its link table.
New facade API File::dataset_at(address), tested in integration_tests.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A hard link's target may go through other hard links, and each was
resolved again every time a path went through it. With each link's
target naming the previous link twice (g/s{i} -> /g/s{i-1}/s{i-1}) the
work doubled per link: finish() took 46 s for 26 links in a debug build,
and 60 would never finish. Resolved links are now remembered, so the work
is linear in the links, and a hard link met again while it is being
resolved is reported as a cycle by name. The depth limit (64) still bounds
the recursion through links not yet resolved.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The link and attribute name indexes are one v2 B-tree leaf, sized to the
next power of two. libhdf5 takes a leaf's capacity from that node size,
but a leaf's record count is a 2-byte field. From about 47 700 links the
node had room for more than 65 535 records, so adding a link in h5py
overflowed the count: a group of 65 535 links crashed h5py, or could no
longer be listed ("unknown link class"). The node is now capped at a full
leaf of 65 535 records, so libhdf5 splits it instead.
Dense attributes now go through the same index builder. Their record
count was written modulo 65 536, without error; more than 65 535
attributes on one object are now refused, like links.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A message in dense storage is a fractal heap object, and an object must
fit one direct block: 65 515 bytes here, since the writer has no
huge-object path. A bigger one (a soft link with a 80 000-byte target in
a group of more than 8 links) was written without error, cut off at the
end of its block, and libhdf5 could not list the group ("object overruns
end of direct block"). finish() now fails with an error that names the
limit, for links and for dense attributes; a 65 001-byte soft link target
still works and h5py reads it back. The heap packer also skips a child
indirect block whose blocks are all too small for the next object instead
of walking it.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Each run of consecutive indices was its own uncached hyperslab read, so
a list over a compressed chunked dataset decoded the same chunk once per
run (d[range(0, 200000, 40)] over 20 gzip chunks: 8 s, h5py 0.014 s).
Plan::reads now groups the indices — a group ends only where a whole
chunk holds no selected index, or, unchunked, at a gap over 64 KiB — and
the selected rows are gathered from each group's block in Rust. Now
3.8 ms (h5py 4.1 ms, release, tank). The new test (1-D, 2-D and
contiguous, compared with h5py, 2 s bound) took 5.8 s before.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Dense link and attribute storage keeps its messages in a fractal heap. Its
root indirect block holds direct blocks up to 64 KiB, 512 KiB in all; rows
past that are child indirect blocks. The writer kept adding rows of direct
blocks instead, and libhdf5 and h5rs read them as indirect blocks: a group
with 20 000 links of 20-byte names was written without error and could not
be listed ("incorrect metadata checksum"), and 150 dense attributes of up
to 56 KB could not be opened. The heap writer now follows the doubling
table: rows past the direct ones hold child indirect blocks, each with its
own rows, nested as deep as the heap needs.
Two more heap bugs are fixed on the way. An object bigger than the next
block's free space was written into it anyway and cut off; the block is
now left unallocated and the object goes in the first block big enough, as
libhdf5 skips blocks. And the header's next-block offset was 0, so libhdf5
adding a link to such a group overwrote the heap's first block ("bad
version number for message"); it is now the offset after the last block.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
VlResolver kept an owned copy of every object of every heap collection
it parsed, for the whole read. Collections nested inside each other's
object data, 32 bytes apart with each element pointing at a different
one, made retained memory O(elements x file size): 1.58 GB for a 744 KB
file (read_vl_strings did the same before VlResolver). Chaining every
collection's objects into one shared run of tiny objects made parse
time O(elements x objects) as well. libhdf5 refuses these files.
- The cache records where each object lies (GlobalHeapCollection::
parse_index, new) instead of copying it, and is dropped past a 32 MiB
budget.
- A collection overlapping one already read is an error: libhdf5 gives
every collection its own block, so only a crafted file has them.
- parse and parse_index refuse a collection that runs past the end of
the file and an object that runs past the end of its collection.
tests/vl_heap_bounds.rs measures peak heap use with a counting
allocator: 129 MB and 350 MB live before on its two crafted files (64 KB
and 176 KB), 97 KB and 0.9 MB now. Conformance unchanged at 575 of 697.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
np.concatenate copies structured dtypes field by field into np.empty, so
the padding of ds[[0, 3, 6]] held process memory. The runs' bytes are
joined in Rust, whole elements at a time, before anything becomes numpy:
the padding is the file's bytes (h5py's) and the result is still a view
of the Rust buffer. The h5py comparisons now compare every byte of
structured values; the new test failed on the padding before.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
PanicException derives from BaseException, so `except Exception` let a
library bug through. Every call from the bindings into the library now
runs under catch_unwind and a panic becomes InternalError (RuntimeError)
naming the object. Tests: a hidden hook panics inside the guard; and the
v4 chunk indexes are compared with h5py from Python — with the library
fix reverted, ds[0:30] of the implicit-index dataset now raises
InternalError instead of aborting the test run.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
read_raw_data_selection's chunked fallback (taken when partial_read
declines, e.g. a bounding box over half the dataset) handed the layout's
chunk dimensions, element-size dimension included, to
generate_implicit_chunks, which indexed past the dataset rank. It then
decoded the whole dataset regardless, so the enumeration is gone: the
arm decodes and extracts for every chunk index.
The new test reads small and large hyperslabs of all five v4 indexes
written by h5py and compares with h5py's values; it panicked before.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The review measured the default pool unchanged (about 2900 MB/s at 16
threads before and after) and still short of 16 h5py processes; small
pools still make outside readers wait. Say so instead of marking the
scaling issue fixed.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Full reads of chunked datasets handed their chunks to rayon. With a
one-thread pool (concurrent_read --decode-threads 1, RAYON_NUM_THREADS=1)
every thread reading through a File queued behind that single worker, so
16 readers decoded on one core: per-thread CPU time showed one thread
doing all the decoding and the readers almost none, and full reads
stopped at about 2x one thread. The cached full-read path and the
uncached reader behind verify_provenance now decode inline when the pool
cannot parallelise (parallel_read::pool_can_parallelise).
The File's chunk cache was the suspect but not the cause: datasets over
its budget were already read without inserting, and skipping its lookups
gained only a few percent at 16 threads.
The regression test keeps a one-thread global pool's worker busy and
requires a full read and verify_provenance to finish anyway; before the
fix both waited for the worker (timed out).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
osobh
osobh