known-issues records what the editor refuses, that freed space is never
reused (append-workload file sizes measured 2026-09-26 on tank with the
ignored measure_append_waste test; sizes are deterministic), and that there
is no journal.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
New clawhdf5::FileEditor opens an HDF5 file (h5py-written at any libver,
HDF5 2.0 format included, or clawhdf5-written) under an exclusive flock
and changes only what an edit touches:
- write_selection/write_all/write_values: compact, contiguous (also
late-allocated) and chunked datasets, any selection. Chunks are decoded,
updated and re-encoded; a filtered chunk that no longer fits moves to the
end of the file unless it is the file's last structure, which grows in
place. New chunks go into v1 B-tree, Extensible Array (paged data blocks
included), Fixed Array and single-chunk indexes, created on first use.
- resize: grow chunked datasets up to maxshape.
- set_attr: add/replace compact attributes, in a NIL slot or a new
continuation chunk.
Each edit is planned in an in-memory image and refused whole
(Error::Unsupported) when any part is unsupported (v2 B-tree / implicit
new chunks, shrinking, vlen/reference data, dense or order-tracked
attributes, cache images, paged/persistent free space). Commit writes and
syncs new space before patching existing bytes. Layout v5 (HDF5 2.0)
array indexes use 8-byte filtered chunk sizes, as libhdf5 does.
Error gains Unsupported/InvalidArgument/Locked and is #[non_exhaustive];
the Python bindings map them. build_attr_message is public.
Tests (h5py, h5dump, h5rs check --data after every round; h5py r+
afterwards): appends crossing EA super/data blocks and B-tree splits, the
same B-tree node counts and EA statistics as libhdf5 for the same writes
(in order, reversed and shuffled; paged blocks), every layout and chunk
index overwritten under random selections, attributes to continuation
chunks, random operations against a model, refused edits leave the file
byte-identical, locking.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Idle-start run on tank at c5334b1 (noisier than the last: compare ratios
within the run). Full reads of deflate data at 16 threads: 4944 MB/s vs
3135 for 16 h5py processes (1.58x; 0.69x-0.76x before). One thread with
the default pool: 6143 MB/s, 15x one h5py call. The concurrent-read
known issue is closed.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Regenerated on tank: ok 575 -> 599, our-error 10 -> 4, mismatch 20 -> 2,
no panics, hangs, crashes or OOM. Newly ok: metadata-cache-image files,
the CVE open-time checks, v1 chunk-key lookup as libhdf5 does it, the
scale-offset fixes, Blosc2, and harness corrections. Baseline raised.
h5rs check --data flags none of the 435 fully-read ok files.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
cve-2025-44905 now reads as h5py reads it (the v1 chunk B-tree lookup),
leaving 5 our-errors: cve-2025-2308, cve-2025-44904 and
bad_nbit_parms_walk (corrupt data HDF5 2.0 reads through a bug), and
the Blosc2 and ZFP filters. The five unloadable-cache-image files stay
ok, now with the library behaving as the probe reports.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
clippy (with the async feature) flags chunks_exact with a constant size
in the AsyncHDF5File cache-image test added in d493d47.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The scale-offset fix (d110b1d) was covered only by unit vectors from
CVE chunks, and documented as three corner cases. The review found it
is much bigger: of 1480 scale-offset datasets h5py writes (every
integer type i1..u8, f4 and f8, both byte orders, with and without a
fill value, scaleoffset 0..full width), v2.7.0's decoder read 332
differently from h5py: 151 returned wrong values with no error (82
integer datasets with scaleoffset=0 and a wide range, 51 full-width
i4/u4/i8/u8, 18 f4 D-scale datasets with a large range) and 181 failed
to read. The cause in every case is a chunk libhdf5 stores at full
width, whose elements were decoded as offsets from minval.
tests/scaleoffset_interop.rs generates that matrix with h5py at test
time, stores h5py's decoded values uncompressed next to it, and
compares every dataset's bytes. It passes on this branch; with the
filters.rs before d110b1d it reports "332 of 1480 scale-offset datasets
differ from h5py".
CHANGELOG: a Correctness entry stating this was silent wrong data in
every release that decoded scale-offset (v2.2.0 to v2.7.0), replacing
the corner-case wording. docs/known-issues.md: a fixed entry with the
affected cases.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
File, MmapFile and LazyFile decoded the superblock extension and laid a
metadata cache image over the file's metadata; the other readers did
not, so the same file read differently by entry point: NativeVol,
AsyncHDF5File and MpiVol (clawhdf5-io) and the external source files of
a virtual dataset (clawhdf5-format vds.rs) read a file with an image
from its own bytes, which libhdf5 does not (they may be stale, or zeros:
h5clear_mdc_image.h5 failed with InvalidObjectHeaderVersion(0)), and
skipped the extension checks File::open makes (cve-2020-10810/10812).
Each of them owns its buffer, so each now calls the shared
superblock_ext::apply_cache_image_in_place, which checks the extension
and writes the image's entries in place (only the image block is
copied). These readers read whole datasets and cannot open a file and
fail each object, so an image libhdf5 cannot load is refused with the
image's error, never read around. clawhdf5-io's vol::load_hdf5 wraps it
for NativeVol (at open; for from_bytes the error is reported on read,
as a truncated file already was) and MpiVol. The MpiVol edit is minimal
and was not compiled: the mpi-io feature needs an MPI installation this
machine does not have (mpi-sys's build script panics).
Tests: NativeVol (open_path and from_bytes), AsyncHDF5File and a VDS
whose source file is h5clear_mdc_image.h5 (vds_interop.rs, against
h5py) read the fixture's values; the corrupted-image variants are
refused. Each fails without its fix.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
For a metadata cache image libhdf5 cannot load, libhdf5 opens the file
and fails the first metadata read (the image loads on the first
H5C_protect after open); h5py reports the error on the root group. The
conformance probe reported it that way, but File::open refused the file,
so the gate counted cve-2025-6269-1..4 and cve-2025-6516 as agreeing
with h5py for behaviour the library did not have.
The library now behaves as the probe reports: File (mmap, buffered and
from_bytes) and MmapFile open the file and every object lookup (dataset,
dataset_at, group, group listings and attributes, VL decoding) fails with
the image's error; LazyFile reads the root group's header at open, so
its open is that first read and fails. Probe and library take the
three-way decision (refuse at open / image loads / image cannot load)
from the same clawhdf5_format::superblock_ext::cache_image_state.
One deliberate difference from libhdf5 remains, documented: after the
failed first read libhdf5 reads the file's own metadata, which the image
was meant to replace and may be stale; here every lookup keeps failing.
File::cache_image_error / MmapFile::cache_image_error expose the error
to code that parses as_bytes() itself; h5rs checks it before reading any
object header (h5rs ls on cve-2025-6269-1 said "invalid object header
version: 0" from the stale bytes).
Test: metadata_cache_image.rs an_image_libhdf5_cannot_load_fails_every_object
(the fixture with its image signature broken; h5py opens that file and
fails the first read with "Bad metadata cache image header signature").
It fails on the previous commit, where File::open refuses the file.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
apply_cache_image returned a copy of the whole file with the image's
entries written in, and File (mmap by default), MmapFile and LazyFile
used that copy for every read: opening a 1 GiB sparse file with an image
needed 2 GB of memory, and an 8 GiB one aborted the process, where
de2a53f (which ignored the image) opened them in a few MB.
The metadata parsers read one contiguous slice, so the image still has
to be laid over the file's bytes; it is now laid over a private copy
that costs only the pages it touches:
- clawhdf5_format::superblock_ext::CacheImage decodes the image into an
entry list (address, offset in the block, length) and applies it to
any destination; cache_image_state tells an opener whether the file
has no image, a loadable one, or one libhdf5 cannot load;
apply_cache_image_in_place is for readers that own their buffer.
apply_cache_image and metadata_view (which copied) are gone.
- clawhdf5_io::HDF5Read::private_copy returns a writable private copy
of a reader's bytes: MmapReader gives a MAP_PRIVATE copy-on-write
mapping (memmap2 map_copy), so only the pages the entries land on are
copied; the default copies the bytes (in-memory readers).
- File, MmapFile and LazyFile write the image into that mapping
(crate::cache_image). File::from_bytes / open_buffered patch their own
buffer in place, copying only the image block, as libhdf5 does. A
file without an image is read straight from the mapping, unchanged.
An image entry that runs past the end of file is now refused: libhdf5
checks only that it starts inside the file, and the images libhdf5
writes never do this, but those bytes have nowhere to go in a view of
the file.
Tests: tests/cache_image_memory.rs has libhdf5 (through ctypes) add an
image to a 1 GiB sparse file and bounds resident-memory growth for all
three openers at 256 MiB; it fails on the previous commit (File::open
grew 2,148,720,640 bytes). reader.rs zero_copy_tests check that a file
without an image is read from the mapping itself and that an image goes
into a copy-on-write mapping, not a heap copy; clawhdf5-io checks that
private_copy writes never reach the reader or the file.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The review suggested libhdf5 loads every image entry and resolves
flush-dependency parents afterwards. It does not:
H5C__reconstruct_cache_contents (HDF5 1.14.6 and 2.0.0, and develop)
inserts each entry and then searches the cache index for its parents in
the same loop, failing with "fd parent not in cache?!?" when one is
missing. So a parent must be an earlier image entry, as before, or
metadata cached before the image loads: the superblock (address 0) and
the superblock extension's object header, which libhdf5 reads to find
the image. Those two were refused as parents; they are now accepted.
A parent listed after its child is still refused, as libhdf5 refuses
it, and so is an entry that is its own parent ("Child entry flush
dependency parent can't be itself").
apply_cache_image takes the superblock to know the extension address.
Test: superblock_ext::tests::flush_dependency_parents_must_already_be_cached
(parent-first loads, child-first refused, extension header accepted,
self-parent refused); the extension-header case fails without the fix.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
libhdf5 does not walk the chunk B-tree to read a dataset: it looks each
chunk up (H5B_find with H5D__btree_cmp3 and H5D__btree_found), asking for
the element-size coordinate as 0. collect_chunk_info_checked now parses
the tree with its keys and returns each stored chunk only when that
lookup, replayed over the scaled keys, finds it.
A key with a non-zero element-size coordinate is therefore found in a
1-D dataset (cmp3 compares only the first coordinate there, and found
compares with <=) and missed in a dataset of rank 2 or more, which reads
fill values. The previous commit refused every such key, which refused
1-D files libhdf5 reads correctly; before that, the rank-2 case read the
chunk's data where h5py reads fill values (cve-2025-44905
/Shuffle_float_data_le, now identical to h5py, so it leaves the
conformance report's list of libhdf5 bugs).
Test: chunk_keys_with_an_element_offset_read_as_libhdf5_reads_them
compares 1-D and 2-D files against h5py's values. It fails on the
previous commit (the 1-D file is refused) and with the refusal removed
(the 2-D file reads 0..23 where h5py reads fill values).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The Blosc2 fuzz tests checked only for panics and the size of the output,
so the offsets-chunk amplification passed all 40,000 iterations.
tests/blosc2_alloc_bounds.rs now measures peak allocation (a counting
global allocator) and asserts it stays within 6x the HDF5 chunk size plus
twice the input plus 2 MiB (ruzstd's fixed state) for:
- 20,000 mutated fixture frames, decoded with their real chunk size as
the limit, with edits aimed at the frame's and chunks' size fields;
- 20,000 mutated first chunks on their own;
- 5,000 frames built from random header sizes, offsets chunks and B2ND
shapes (padding chunk and block shapes, chunks larger than the array,
NaN, zero and repeated-value chunks).
Against the code before this series every test in the file fails (the
fuzz tests at frame iteration 3913, a zstd window, and random frame 289,
the offsets chunk); now the worst frame peaks at 0.48 of the bound.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
ruzstd reserves a frame's declared window (up to its 100 MiB default)
when a decoder is reset for a new frame, before decoding anything. The
Blosc, Blosc2 and bitshuffle decoders reuse one decoder per chunk, so a
Blosc2 chunk of two 16-byte streams, each declaring a 96 MiB window,
allocated 128 MiB. zstd_decode_into now sets the decoder's maximum window
to twice the stream's output (at least 128 KiB): c-blosc, c-blosc2 and
bitshuffle compress each block in one call with its size known, so
libzstd's window never exceeds the block.
Found by tracking peak allocation in the Blosc2 fuzz test.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
B2ND chunks were decoded whole, padding included, with up to 16x the HDF5
chunk size as their limit, so a crafted frame made each chunk allocate and
fill up to 16x the output (4 GiB for a 256 MiB HDF5 chunk). The padding is
the real bound (prod(ceil(c/b)*b) per chunk), but that can be 2^ndim times
the array, so it is no longer held at all:
- A Blosc2 chunk is now decoded block by block (decode_blocks), each block
handed to a sink as it is ready, with at most three blocks of scratch.
blosc2_decompress_chunk and plain frames still collect every block.
- reassemble places each B2ND block straight into the output and skips
blocks that are all padding (they are not decoded unless the delta
filter needs the first block). A frame's NaN chunks are handed over one
B2ND block at a time and its zero chunks cost nothing.
- A B2ND chunk must decode to exactly its padded size, its Blosc2 blocks
must be whole B2ND blocks no larger than the output, and a chunk may not
be larger than the array (hdf5-blosc2's chunk is the array), so a block
is never larger than the output.
Peak allocation for a 10-D array padded to 13x (NaN, repeated-value and
stored-block chunks) and for a 16x chunk was 4.5 MB and 17.8 MB for
315 KB and 1 MiB outputs before, and is now within the tests' bound.
Blosc2 files written by hdf5plugin in 9-D and 12-D, an 8 MiB single chunk,
1x1x1 and edge-chunk shapes still read exactly.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A chunk header with nbytes 0 and no special type kept its declared block
size (up to 512 MiB): the block size was clamped to nbytes only when nbytes
was positive, and the scratch blocks were allocated before the (empty)
block loop, so a 20-byte chunk allocated about 1 GiB. The block size is now
clamped to nbytes always, and an empty chunk returns before any scratch is
allocated.
A frame chunk must also decode to the size the frame header gives it
(chunksize, or the remainder for the last chunk), and is decoded with that
as its limit, so an empty chunk in a frame for a non-empty HDF5 chunk is an
error rather than an empty result.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
parse_frame sized the offsets chunk from the frame header's own nbytes and
chunksize, so a 173-byte frame declaring 32 Mi chunks, with a 40-byte
repeated-value offsets chunk, built 256 MiB (up to 2 GiB) of offsets for a
1 MiB HDF5 chunk and then returned 4 bytes. The offsets chunk is now capped
at the output limit (at least 128 bytes); a frame whose nbytes/chunksize
imply more chunks than that is refused before anything is allocated.
tests/blosc2_alloc_bounds.rs measures peak allocation with a counting
global allocator; the reviewer's frame failed it (decoded Ok(4)) before.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Review follow-ups. With run_with_helpers a one-thread rayon pool gave each
read a second core (the caller plus the worker), so --decode-threads 1 no
longer matched h5py's one core per call; such a pool now adds no helper.
Per-thread decode scratch is kept up to 1 MiB per buffer (was 4 MiB),
bounding what never-exiting pool workers hold.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Conformance on tank, conformance/run.sh --no-fetch (2026-09-26): 597 of
697 ok, 6 our-errors (4 corrupt objects HDF5 2.0 reads through a bug, the
Blosc2 and ZFP filters), 2 mismatches (the known h5py big-endian VL bug).
Closes the known-issues entries for metadata cache images,
cve-2024-32624, cve-2020-10810/10812, and unfiltered chunks of the wrong
size; the N-Bit / 64-bit scale-offset entry is recorded as not our bug.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Four of the remaining our-errors are objects the reference (h5py 3.16 /
HDF5 2.0) reads only through a libhdf5 bug, and clawhdf5 refuses:
cve-2025-2308 (scale-offset codes past the end of the chunk),
cve-2025-44904 (short unfiltered chunks), bad_nbit_parms_walk.h5 (an N-Bit
parameter list one value short; libhdf5's own test_filter_bad_params now
requires the read to fail) and cve-2025-44905 /Shuffle_float_data_le (a
chunk key libhdf5's lookup misses, reading fill values). report.py lists
them under Known not-our-bug and counts them in the summary; they stay
our-errors in the class counts.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
libhdf5 loads a metadata cache image when it first reads metadata (the
root group), not at open, so for cve-2025-6269-1..4 and cve-2025-6516 (all
corrupt images) h5py opens the file and fails on "/". The probe reported
the image's error as an open error, which made those files our-errors;
it now records it on the root object, where h5py reports it. File::open
still refuses such a file outright: nothing in it can be read.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Placing a chunk cast its u64 offsets to usize; on a 32-bit target an
offset past the address space wrapped into the output (and could then
overlap another chunk's region when chunks are placed concurrently).
Such an offset is past the dataset, so it now saturates and the chunk
writes nothing, as the concurrent-placement check already assumed. No
change on 64-bit targets, where the cast cannot wrap (so no test here).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
CHANGELOG entry for the chunked read changes, and the known-issues entry
on concurrent chunked reads updated: both causes it names (per-read page
faults, readers waiting on a small pool) are fixed; the 16-thread
comparison with h5py stays open until re-measured on an idle machine.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
- A dataset without filters stores every chunk at the chunk's full size.
A chunk its index records at another size was read at that size, with
the rest of the chunk left as zeros (cve-2025-44904's
Scale_offset_float_data_le: 38- and 37-byte chunks for 48-byte chunks,
where HDF5 2.0 fills the rest with whatever its buffer held). It is now
refused, as later libhdf5 releases refuse it ("incorrect chunk size
returned from index for unfiltered chunk"):
chunked_read::list_chunks_for_read, used by every read path.
- A v1 B-tree chunk key carries 0 in the element-size dimension. libhdf5
compares that coordinate when it looks a chunk up, so whether it finds a
chunk keyed otherwise depends on where the key falls (in cve-2025-44905
/Shuffle_float_data_le, offset 4096, it does not, and h5py reads fill
values); we read the chunk. Such a key is now refused.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Scale-offset (filter 6) now follows H5Z__filter_scaleoffset:
- the packed codes start at byte 21 whatever size the chunk records for
minval (libhdf5 reads min(8, size) bytes of minval and always starts the
codes at buf_offset 21). We started them after minval plus 8 bytes, so a
chunk recording a size of 0 (cve-2025-44905 /Scale_offset_short_data_be)
decoded differently from h5py;
- with a fill value defined, a code equal to the all-ones code of minbits
bits is the fill value, including minbits 0 (code 0): a chunk of nothing
but fill values read as minval;
- minbits of the full width stores the elements as they are (no minval
added), and an integer scale factor of the full width means the chunk was
left untouched; minbits or a scale factor wider than the type is an
error;
- the class parameter (integer or float) decides the decode, a scale type
that does not match it is refused, and E-scale is refused, as in libhdf5
(no library writes it; it was decoded here unchecked);
- minval is the stored bytes zero-extended, as libhdf5 reads it.
Codes past the end of the chunk stay an error, as in libhdf5 releases
after 2.0 ("Buffer too short"; 2.0 reads past the buffer, cve-2025-2308).
Shuffle (filter 2) uses its own parameter as the element size, as libhdf5
does, instead of the dataset's element size; a parameter larger than the
chunk leaves the chunk as it is (cve-2025-44905 /Shuffle_float_data_be),
and a parameter of 0 is an error.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
libhdf5 decodes the messages of a v2/v3 superblock's extension when it
opens a file (H5F__super_read) and refuses the file when one does not
decode. We never looked at them, so we opened cve-2020-10810 (a File
Space Info message too short for the free-space manager addresses it
announces) and cve-2020-10812 (a metadata cache image past the end of the
file), both of which libhdf5 refuses.
A file written with a metadata cache image keeps its metadata cache
entries in an image block the extension points at; libhdf5 loads them
over the file's own bytes before it reads any metadata
(H5C__load_cache_image, H5C__reconstruct_cache_contents). In
h5clear_mdc_image.h5 the root group's header exists only in the image, so
every reader failed with InvalidObjectHeaderVersion(0).
The new clawhdf5_format::superblock_ext module:
- read_superblock_extension decodes the v1 B-tree K, File Space Info and
Metadata Cache Image messages with libhdf5's checks (versions, page size
512 B .. 1 GiB, the addresses a persisting message lists, the image
inside the file), with the new FormatError::InvalidSuperblockExtension;
- apply_cache_image checks an image block as libhdf5 does (signature,
version, recorded length, entry types, rings, ages, addresses inside
the file and not repeated, flush-dependency parents) and returns the
file's bytes with every entry written at its address
(FormatError::InvalidCacheImage);
- metadata_view does both.
File, MmapFile and LazyFile (and so h5rs) call metadata_view at open and
read an image file through the patched copy; the conformance probe does
the same. The image's trailing checksum is not verified, as libhdf5 does
not verify it. tests/fixtures/h5clear_mdc_image.h5 is libhdf5's own test
file.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Record the `blosc2` feature in the changelog, the README's feature table
and the crate table, and mark the Blosc2 half of the known "Filters"
issue fixed (dated, with the conformance run that shows h5ex_d_blosc2
reading). What stays open: ZFP, writing Blosc2, and the Blosc2 features
hdf5plugin never writes (dictionaries, lazy chunks, variable-length
blocks, user-defined codecs and registered filters), which are errors.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
hdf5plugin's Blosc2 was a clear "not implemented" error. It stores each
HDF5 chunk as a Blosc2 contiguous frame; for chunks of 2+ dimensions the
frame holds a B2ND array whose data are cut into (padded) blocks stored
one after another.
filters_blosc2 (feature `blosc2`, in `plugin-filters`; the facade
forwards both) decodes, following c-blosc2's decoder and hdf5-blosc2's
blosc2_filter.c:
- the frame: the msgpack header's fixed fields, the metalayer index, the
offsets chunk (with the special zero/NaN/uninitialised offsets) and
chunk lookup;
- Blosc2 chunks: 16- and 32-byte headers, special chunks (zeros, NaN,
uninitialised, one repeated value), split and unsplit streams, zero and
run-length streams, and the filter pipeline run backwards (shuffle,
shuffle with a byte-group size, bit shuffle including the version-2 and
later handling of a partial group of 8, delta against the first block,
truncated precision);
- the codecs, shared with Blosc 1: BloscLZ, LZ4/LZ4HC, Zlib, Zstandard;
- B2ND arrays: blocks gathered into C order, padding dropped, several
chunks per array, and the array shape checked against the chunk shape
in cd_values as the HDF5 filter does.
Dictionaries, lazy chunks, variable-length blocks, user-defined codecs
and registered filters (e.g. bytedelta) are errors. Uninitialised chunks
read as zeros. No encoder.
Tests: h5py + hdf5plugin write every codec x filter (none, shuffle,
bitshuffle, delta) and levels 0-9 over the plugin-filter cases, then
i1..u8/f4/f8 in 1-D to 5-D chunks with partial edge chunks, datasets of
zeros, one value and NaN, and Fletcher32 before Blosc2 (plain frames for
n-D chunks); clawhdf5 reads each exactly as its unfiltered twin, and
truncated precision exactly as h5py reads it. Fixture frames from
python-blosc2 (tests/fixtures/blosc2/generate.py) cover what hdf5plugin
never writes: special chunks, delta over many blocks and odd type sizes,
odd bit-shuffle blocks, forced splitting, multi-chunk B2ND arrays with a
zero chunk, and the refused features. The decoder is fuzzed (random and
mutated frames and chunks: no panic, output within the limit).
Conformance: h5ex_d_blosc2.h5 now reads (576 of 697 ok, baseline 575).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
libhdf5 checks a dataset's storage when it opens the dataset
(H5D__contig_check, H5D__compact_init): the element count times the
element size must not overflow, contiguous storage must end inside the
file, compact data must be the dataset's size. File::dataset opened
cve-2024-32624's /Dset_OBJREF (2^62 + 2 references of 8 bytes) and
reported its shape; only reading failed.
data_read::check_dataset_storage makes those checks (new
FormatError::InvalidDatasetStorage), and File, MmapFile and LazyFile run
it whenever they open a dataset (by path, by address, from a group), as
does the conformance probe. As before, a datatype, dataspace or layout
that does not decode is left for the read to report, so such a dataset
still opens and its attributes still read. An empty contiguous dataset at
a defined address, which libhdf5 refuses, is still accepted: clawhdf5 up
to v2.7.0 wrote them.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A selection read (a hyperslab or points covering at most half the
dataset) decoded each chunk it overlaps into fresh buffers, one per
filter stage; it now uses the thread's chunk-decoding scratch like the
full readers. Covered by tests/chunked_read_paths_interop.rs (small
hyperslabs and points over every filter and type) and the partial-read
equivalence tests.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
read_f32/read_f64/read_i32/read_i64/read_u64 of a chunked dataset that
stores exactly that type in native byte order now decode every chunk
straight into the Vec<T> they return (data_read::read_chunked_native),
on File (through its chunk cache), MmapFile and LazyFile. Before, the
chunks went into a byte buffer that read_as_* then copied into a second,
typed one: two dataset-sized allocations and a full extra copy per read.
The output is zeroed pages from the allocator, backed by transparent
huge pages when large, like the byte reader's. Other types and byte
orders, and datasets with no storage or external data, keep converting
through the byte readers; unallocated chunks read as the fill value as
before.
tests/chunked_read_paths_interop.rs checks every chunked read path
(File twice, so cached; from_bytes; MmapFile; LazyFile; small, strided
and point selections; with and without the parallel feature) against
h5py for 1-8 byte integers and 2-8 byte floats in both byte orders,
through deflate, shuffle, Fletcher32, LZF, SZIP and Blosc, with partial
edge chunks, sparse datasets with default and non-default fill values,
and datasets larger than the chunk cache. A filter this build lacks must
be an error (or, when an optional filter declined every chunk, the right
data).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
h5py's track_order=True orders attributes as well as links; the writer
tracked links only. A tracking object's header now sets the attribute
creation order tracked/indexed flags and carries per-message creation
orders, an Attribute Info message holds the next order (inline too),
and dense storage gets a type-9 creation-order index. The file default
applies to datasets, with DatasetBuilder::track_order per dataset; more
than 65 535 attributes on a tracking object is an error (libhdf5's
counter is 2 bytes). The reader lists such attributes in creation
order.
h5py lists them in order (inline, dense, 20 000 on one dataset) and
keeps numbering in r+ mode, including its inline-to-dense move.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
- A simple dataspace of rank 0 holds one element in libhdf5 (the product
of no dimensions; h5py reads it as shape ()). num_elements() said 0, so
cve-2020-18494's /dset1 failed with DataSizeMismatch { expected: 0 }.
- A contiguous dataset whose storage is larger than its elements reads:
libhdf5 reads the elements from the start of the storage and ignores the
rest (H5D__contig_check checks only that they fit in the file). We
required the sizes to be equal, so the scalar /Dset1 of cve-2024-32623
and cve-2025-2309 (240 bytes of storage for one int) failed. Storage too
small for the elements is still an error. data_read::contiguous_read_len
is the rule, used by every contiguous read path.
- Dataspace::parse refuses what H5O__sdspace_decode refuses: more than 32
dimensions, a rank on a scalar or null dataspace, a dimension larger
than its maximum (new FormatError::InvalidDataspace).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Inventory of the 109 whole-file parser functions, a measurement of the
metadata ranges clawhdf5 touches on three corpus files against libhdf5,
options (storage trait, virtual slice, metadata prefetch, userfaultfd)
with how ros3, h5py+fsspec, pyfive, h5wasm and object_store do it, and
an incremental plan: M0 name-index lookups, M1 metadata over a Storage
trait, M2 raw data with batched ranges, M3 object_store backend,
M4 wasm fetch-driven lazy loading.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
inventory.py counts the functions and call sites that take the whole
file as &[u8]; range-trace (standalone crate, x86-64 Linux) records every
load clawhdf5 makes from a file by mprotect + single-step, unchanged
library code; libhdf5_reads.py counts libhdf5's reads through h5py's
fileobj driver and prints a dataset's chunk extents.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
libhdf5 refuses to list a symbol-table group that has an entry with an
empty name (H5G__ent_to_link: "invalid link name"), so h5py cannot list
cve-2021-46244's /BAG_root. We listed it, with an object at "/BAG_root/"
(the empty name, pointing at address 0). resolve_v1_group_entries — the
listing — now fails with the new FormatError::InvalidLinkName; path
lookups still find the group's other names, as libhdf5's by-name lookup
does.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
libhdf5 decides what an object header is in a fixed order
(H5O__obj_class_real): a group if it has a Symbol Table or Link Info
message, a dataset if it has a Datatype *and* a Dataspace message, a named
datatype if it has a Datatype message. The conformance probe called any
header with a Data Layout message a dataset, so cve-2024-33874's /Dset1 (a
datatype and a layout, no dataspace), which h5py opens as a named
datatype, was reported as a dataset we failed to read
(MissingMessage(Dataspace)). The probe now classifies with
object_class().
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Full reads of a chunked dataset (the cached reader behind the facade's
read_* and the uncached one behind mmap/lazy files and verify_provenance)
now decode each chunk into this thread's reusable scratch buffers and copy
it straight to its place in the output. Before, the cached reader decoded
batches of 128 chunks into fresh Vecs and the uncached one decoded every
chunk of the dataset into its own buffer before assembling any: a new
256 KiB allocation (and its page faults) per chunk and per filter stage.
Chunks are still inserted into the file's chunk cache when the whole
dataset fits in it.
With the parallel feature the calling thread now decodes too, sharing the
chunks with whichever rayon workers are free (run_with_helpers): a helper
the busy pool only starts after the read is done returns at once. Before,
the caller handed every chunk to the pool and slept, so readers outside a
small pool (2-4 threads) queued behind its workers; with a one-thread pool
the reads went sequential. Chunks are placed concurrently only when the
index puts them on the chunk grid at distinct places (a corrupt index is
read one chunk at a time), and the error returned is still the first
failing chunk's.
Fix: a chunk stored unfiltered in a filtered dataset (every filter-mask
bit set) that is shorter than a chunk read as zeros where its data was
missing through the cached reader (the facade's read_*); it is now an
error naming the chunk, as the uncached reader already made it.
Regression tests, both failing before this change:
tests/busy_decode_pool.rs (both workers of a two-thread pool busy, four
readers) and short_unfiltered_chunk_of_a_filtered_dataset_is_an_error.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
ref.py and compare.py reported 13 files as mismatches or our-errors that
were artefacts of the harness, not differences between the readers:
- User-defined links (tall.h5, tudlink.h5, twithub*.h5, tmany.h5, ...):
h5py's `get(name, getlink=True)` reports a user-defined link as a
HardLink, so ref.py listed it as an object. Read the link type from
H5Lget_info instead.
- Objects h5py cannot open (cve-2019-8397/8398, cve-2021-46243,
cve-2024-32618): the probe deduplicates by header address, ref.py by
ObjectID, which an unopenable object does not have, so each extra hard
link to it was listed again. Deduplicate those by link address.
- Nested array types (tarray3.h5): h5py expands them into trailing dims;
hash_values stripped one level and numpy broadcast every element into a
whole subarray. Strip every level.
compare.py no longer compares the attributes or links of an object h5py
could not open at all (cve-2018-17438/17439, cve-2019-9151): h5py read
none, so ours are neither extra nor errors against it.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Dense link and attribute indexes and the chunk index for several
unlimited dimensions were single leaves, capping them at 65 535
records. btree_v2_write builds trees of any depth, with node capacities
and pointer widths from libhdf5's H5B2__hdr_init arithmetic (now shared
with the reader as btree_v2::node_info) and libhdf5's node sizes (512
dense, 2048 chunks). Indexes that fit the old one-leaf layout are
written byte for byte as before (compared for 10..65 535 links, attrs
and chunks, tracked and filtered).
Tests: 100 000 links (short names; long names with creation order),
70 000 attributes, 200 000 chunks (and 80 000 deflated), read by h5py,
h5dump and clawhdf5 and edited by h5py r+; h5rs check on the same
shapes, asserting depths 2-3.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
decompress_chunk_exact_with decodes a chunk into a DecodeScratch the
caller keeps between chunks, instead of a new Vec per chunk and per filter
stage. Deflate inflates into a kept buffer with a reset (not rebuilt)
inflater, shuffle interleaves into the other buffer, and Fletcher32 checks
and drops its checksum in place (on the stored bytes when it is the first
filter undone). Other filters go through the registry as before. Output
and errors are those of decompress_chunk_exact; a unit test checks that
for every pipeline shape and filter mask with one reused scratch.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
libhdf5 compares the name when two hashes are equal; the writer broke
ties by insertion order, and libhdf5 could not find one of two names
whose lookup3 hashes collide (k69209 / k155448). Test fails before the
fix with h5py's KeyError.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
