Regenerated on tank: ok 599 -> 600 (h5ex_d_zfp.h5), our-error 4 -> 3,
mismatch 2, no panics, hangs, crashes or OOM. Baseline raised.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
FileBuilder stored every chunk of an LZF or Blosc dataset through the
filter with filter mask 0. libhdf5 counts LZF and Blosc output no smaller
than the chunk as a failure of the optional filter and stores the chunk raw
with the filter's mask bit set. For an LZF chunk whose stream was exactly
the chunk's size, the first libhdf5 rewrite stored raw data at the same
size and kept our stale mask 0 in the index, and h5py could no longer read
the dataset.
precompress_chunks now runs chunks through compress_chunk_masked (as
FileEditor does since f7e2ab1), sequentially and on the parallel path, and
build_chunked_data_from_precompressed records each chunk's real mask in
every index the writer builds: single chunk (layout field), Fixed Array and
Extensible Array filtered elements, and version-2 B-tree type 11 records
(create_datasets_parallel goes through the same path). The writer builds
no version-1 B-tree or implicit index. PrecompressedChunks::chunks gains
the mask. Files whose chunks all compress are byte-identical.
Latent only in the unreleased LZF/Blosc writer (added 2026-09-26); no
tagged release writes either filter.
Tests:
- plugin_filters_interop skipped_optional_filters_are_masked_as_libhdf5_masks_them:
LZF, shuffle+LZF+fletcher32 and Blosc over random, compressible and
alternating chunks in every index; masks equal an h5py-written twin's;
h5py r+ rewrites and extends them; h5py, h5dump and our reader read
every value. Before: 20 of 24 datasets had masks other than h5py's, and
with that check disabled h5py failed to read the rewritten datasets
("filter returned failure during read").
- plugin_filters_interop files_whose_chunks_all_compress_are_unchanged:
pins the pre-fix bytes of five all-compressing files.
- chunked_write skipped_lzf_chunks_are_masked_in_every_index (fails before:
mask 0, want 2).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
- Per-file probe output is identical for 696 of 697 files, not all:
cve-2025-2310.h5's error string depends on which parallel chunk decode
fails first, at f2ff2c4 as on this branch.
- The parser cores are generic (S: Storage + ?Sized); provisional A/B
numbers against f2ff2c4, including the one bench that still shows
ObjectHeader::parse slower when old and new are separate binaries.
- Reads sized by untrusted fields are bounded; the harness only accepts
the known whole-file fallbacks.
- range-reads.md records why M1 went generic rather than &dyn.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
On a backend without the file in memory, a structure read whose length
comes from untrusted header fields was clamped only by the end of the
file, so a crafted size made one read (and copy) of up to the rest of
the file. Each such read now covers what the parser actually uses:
- local heap names: read in growing pieces (64 bytes first, then 4x)
up to the end of the data segment, instead of the rest of the segment
per name (quadratic for a big symbol-table group);
- fractal heap indirect blocks: the doubling-table geometry locates the
entry covering the object, and the first read ends at that entry; only
if it is unallocated does the walk read the rest of the block (it
visits every entry then). One walk implementation serves both;
- paged fixed/extensible array data blocks over 1 MiB: the prefix and
page bitmap, then each page in use on its own (smaller blocks are
still one read);
- blocks under one checksum (non-paged array data blocks, extensible
array index and super blocks): the bounds check that comes first (the
checksum's; the page bitmap's for a super block) is made against the
file length before reading (Window::check_extent), so a block claimed
past the end of the file costs no read. With the checksum feature off
the parser has no such first check and the old read stands.
Other windows were already bounded (the superblock and object header
prefixes, the fractal heap header by a u16, SOHM tables by u8/u16
counts) or are exact reads checked against the file length first.
In memory nothing changes: the pieces are borrowed slices.
Tests: CountingStorage over a crafted heap (16 MiB file, width and rows
0xFFFF: under 1 KiB read, 16.7 MB before), a heap segment claiming 64 MiB
(one 64-byte read per short name), long names at every piece boundary,
a fixed array block claimed past the end of a 16 MiB file (under 64
bytes read), and in the equivalence harness an h5py file with a 2.4 MB
fixed array block and a >1 MiB extensible array block, whole and cut at
97 points: every chunk index agrees with the slice read and the largest
takes 205 KB (2.4 MB and 1.2 MB when read whole).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The harness counted ContiguousStorageRequired from any parser as an
allowed difference, so a converted module that wrongly fell back to the
whole file would still pass. It now accepts the error only from the
three sites that are not converted yet (dense attribute storage, a SOHM
B-tree index, huge fractal-heap objects, all found through a v2 B-tree)
and only in the checks that can reach them; anything else fails with
the check and the site named.
Checked by making LocalHeap::parse_in return the error first: the
fixture and h5py runs fail ("local heap fell back to the whole file").
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Every `*_in` core and the read helpers take `file: &S` with
`S: Storage + ?Sized` instead of `&dyn Storage`, and the `&[u8]`
wrappers pass the slice itself, so they compile to a `[u8]` instance:
`as_contiguous()` inlines to `Some(self)` and each structure read is the
slice code's bounds check again, with no indirect call. `&dyn Storage`
still works (`S = dyn Storage`); there is one parser implementation.
Also, so the structure reads cost no more than the slice checks did:
- ObjectHeader::parse_in reads the prefix once (signature included)
instead of the signature and then the prefix: two reads for a
one-chunk header instead of three on a range backend;
- the symbol-table node and group B-tree (v1) loops walk their entries
with chunks_exact over the bytes read, and the node's redundant second
bounds check is gone (the entries' read is the check, same error);
- a version-1 header's message list is sized from its (capped) count.
Same results and errors; the unit and equivalence tests are unchanged.
New Criterion bench `clawhdf5/benches/local_metadata_bench.rs` over a
400-group version-1 file written by h5py (new fixture
`v1_groups_400.h5`): ObjectHeader::parse, symbol-table nodes, the group
B-tree walk and a facade listing, using only APIs that exist at f2ff2c4
so it builds there for an A/B.
Provisional A/B against f2ff2c4 (busy machine, not for docs): both
builds linked into one binary and timed in alternation, 200 rounds;
median ratio new/old: facade listing -0.5% to -3.5% (was +14%),
ObjectHeader::parse +1% to +2% (was +25%), symbol-table nodes -18%,
group B-tree walk -18%, local-heap names and resolve_group_children
within +-1.5%. An old-vs-old-copy run shows +-2% from code layout alone.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Each edit planned over the reader's memory map of the file and committed
while that File, and the Image's &[u8] over the mapping, were still alive,
writing the same file through the editor's descriptor. Nothing read the
mapping during the writes, but a shared slice whose memory changes
underneath it is undefined behaviour under Rust's aliasing rules.
Image::into_plan now detaches the edit's writes (patches, end of
allocation) into a Plan that owns all of its bytes and borrows nothing;
edit() takes the user-block size, drops the File — unmapping the file —
and only then commits the Plan. The invariant is documented in the image
module and the editor's module docs.
Test: edit::tests::file_is_not_mapped_while_an_edit_writes_it checks
/proc/self/maps at the moment each commit starts (write, resize, set_attr):
never mapped. With the commit moved back before the reader is dropped (the
previous order) it reports all three commits with the file mapped.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A dataset whose filter this build cannot encode (scale-offset, N-Bit, SZIP;
a plugin filter the build lacks) failed with Error::Format("unsupported
filter: 6"), although the editor documents every refused edit as
Error::Unsupported, and the Python bindings raised ValueError rather than
NotImplementedError. Every edit now maps FormatError::UnsupportedFilter to
Error::Unsupported; the file is left untouched as before.
Test: edit_interop unencodable_filters_are_unsupported — h5py scale-offset
datasets (integer with chunks, integer never written, float D-scale):
Error::Unsupported naming the filter, and the file byte for byte unchanged.
Fails on the previous editor (Format(UnsupportedFilter(6))).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
libhdf5 counts a version-2 object header's attributes through its Attribute
Info message (0x15) and reports none when the header has none. set_attr gave
v110/latest groups, the root group and datasets without attributes an
attribute message only, so h5py listed the attribute but len(obj.attrs) and
H5Oget_info's num_attrs said 0, and stayed wrong after h5py r+ added more.
Like H5O__attr_create, the edit now adds the message when a version-2 header
lacks it, in the same planned edit: version 0, the header's creation-order
track/index flags, maximum creation index 0, undefined fractal heap and
B-tree addresses, message flag DONTSHARE — byte for byte what libhdf5
writes. It goes before the attribute (libhdf5's order) when free space
holds both, else after it, so a continuation chunk made for the attribute
also takes it.
Test: edit_interop attribute_count_in_version_2_headers — v110 and latest
files, attributes set on the root group, groups and datasets with and
without existing attributes: h5py's len/num_attrs/list/values, h5dump -A
and our reader agree, also after h5py r+ adds attributes up to and past the
compact limit. Fails on the previous editor (h5py len 0).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A valid group has one link per name, but a damaged or hand-made one can
have two. resolve_child followed the first soft link of the name, the
listing skipped a dangling one and listed the name via a later link, and
path resolution followed the last symbolic link: three answers. All now
take the first link of the name (header message order in a compact group,
name index order in a dense one) and ignore the rest, even if the first
dangles. That is libhdf5's rule for compact groups (H5G__compact_lookup
stops at the first Link message); h5py opens nothing for a dangling first
link although a later one resolves. For a dense group libhdf5
binary-searches the index and may land on another of several exact
duplicates; documented on first_link_named. find_symbolic_link's v2 branch
was dead (only v1 groups reach it) and is now v1-only.
Test: an h5py compact group with soft links dup_A (dangling, or to /d) and
dup_B (the other), dup_B renamed to dup_A in the header and re-checksummed.
Lookup, path and listing through all three readers match h5py for both
orders. With the old group_v2.rs the path lookup returned 42 where h5py
opens nothing.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
addr::to_usize's error branch only ran where usize is narrower than u64,
and no such target runs tests in CI, so on x86_64 the test checked only
that every u64 fits. to_usize and saturating_usize are now the usize
instances of generic to_index/saturating_index; the test runs the same
code with u32 standing in for a 32-bit usize: values past u32::MAX
(including one an `as` cast would wrap to 0x1234) are Overflow, and the
saturating form clamps. A mutant that truncates instead fails the test;
the old addr.rs does not provide the helper the test needs.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The editor stored every chunk through the whole pipeline with filter mask 0.
For LZF that did not shrink a chunk, h5py instead stores it raw with the
filter's mask bit set. A chunk the editor stored LZF-encoded at exactly the
raw size was then rewritten raw by libhdf5 at the same size; libhdf5 does not
touch the index entry when the size is unchanged, so the stale mask 0 stayed
and h5py (and h5dump) could no longer read the dataset.
clawhdf5_format::filters::compress_chunk_masked runs the pipeline as
H5Z_pipeline does: an optional filter (H5Z_FLAG_OPTIONAL) that fails is
skipped and its bit set, a mandatory one fails the write, and LZF/Blosc
output no smaller than the input counts as failure, as in the reference
filters (their output buffer is the input's size). Deflate, LZ4, Zstd,
bitshuffle and bzip2 never fail on size in libhdf5 and are kept as before.
Test: edit_interop optional_filters_that_fail_are_skipped — the reviewer's
repro at every libver: the editor stores the chunk exactly as h5py does
(mask 1, size 5; shuffle+LZF+fletcher32 mask 2), h5py r+ rewrites and
extends the datasets, and h5py, h5dump and our reader read every value.
Fails on the previous editor (mask 0; h5dump cannot read /u8).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Three `chunk_info.address as usize` casts behind the `parallel` feature
survived the conversion, because check-32bit-casts.sh linted only default
features plus plugin-filters. On a 32-bit target with rayon a chunk address
past 4 GiB still wrapped onto another part of the file. They go through
addr::to_usize now, and the lane index (h % n, always < n) through
saturating_usize.
The script now lints no default features, default features, and every
optional feature but szip (wasm32; the set with zstd, which does not build
for wasm32, on the host, where the lint reports the same casts). With the
old parallel_read.rs/lane_partition.rs it fails listing the four casts; the
old script passed them. CHANGELOG and the design note give the exact count
(119) and what is not covered.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Only leaves and the header were checked. Harmless while every lookup read
the whole tree, but the indexed lookup prunes children by the keys stored
in internal nodes, so one corrupted byte there could route a name to the
wrong child and report it missing with no error. A BTIN whose lookup3
checksum does not match is now ChecksumMismatch on every read (lookups and
full traversals), as in libhdf5.
Test: one byte of the root BTIN of the 35 001-link h5py group's name index
changed -> lookups, paths and listings through File, MmapFile and LazyFile
all fail with ChecksumMismatch, and h5py refuses both. Before, lookups
returned Ok.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
read_exact_at and read_upto (and so Window::read) ask as_contiguous()
first and slice the file directly when the backend holds it in memory:
one dynamic call per structure read instead of two or three (len,
read_at, then len again for errors). Same results and errors.
Provisional (busy machine, not for docs): a listing that walks 400
symbol-table groups through the facade went from about 18% to about 14%
slower than before the Storage conversion; the extra cost is a few tens
of nanoseconds per structure read, which the facade's per-lookup
re-listing (range-reads.md M0) multiplies.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
known-issues records what the editor refuses, that freed space is never
reused (append-workload file sizes measured 2026-09-26 on tank with the
ignored measure_append_waste test; sizes are deterministic), and that there
is no journal.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
New clawhdf5::FileEditor opens an HDF5 file (h5py-written at any libver,
HDF5 2.0 format included, or clawhdf5-written) under an exclusive flock
and changes only what an edit touches:
- write_selection/write_all/write_values: compact, contiguous (also
late-allocated) and chunked datasets, any selection. Chunks are decoded,
updated and re-encoded; a filtered chunk that no longer fits moves to the
end of the file unless it is the file's last structure, which grows in
place. New chunks go into v1 B-tree, Extensible Array (paged data blocks
included), Fixed Array and single-chunk indexes, created on first use.
- resize: grow chunked datasets up to maxshape.
- set_attr: add/replace compact attributes, in a NIL slot or a new
continuation chunk.
Each edit is planned in an in-memory image and refused whole
(Error::Unsupported) when any part is unsupported (v2 B-tree / implicit
new chunks, shrinking, vlen/reference data, dense or order-tracked
attributes, cache images, paged/persistent free space). Commit writes and
syncs new space before patching existing bytes. Layout v5 (HDF5 2.0)
array indexes use 8-byte filtered chunk sizes, as libhdf5 does.
Error gains Unsupported/InvalidArgument/Locked and is #[non_exhaustive];
the Python bindings map them. build_attr_message is public.
Tests (h5py, h5dump, h5rs check --data after every round; h5py r+
afterwards): appends crossing EA super/data blocks and B-tree splits, the
same B-tree node counts and EA statistics as libhdf5 for the same writes
(in order, reversed and shuffled; paged blocks), every layout and chunk
index overwritten under random selections, attributes to continuation
chunks, random operations against a model, refused edits leave the file
byte-identical, locking.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Every `u64 as usize` cast in clawhdf5-format (115 on wasm32) now goes
through addr::to_usize for values read from the file — addresses, lengths,
counts, dimensions: FormatError::Overflow where the value does not fit
instead of wrapping onto another part of the file on a 32-bit target — or
addr::saturating_usize for counts bounded by something in memory (codec
progress counters, writer sizes), which fail a bounds check or allocation
rather than wrap. A chunk whose offset does not fit lies outside the
dataset and is skipped; partial reads treat such an offset as out of the
buffers. On 64-bit targets nothing changes.
scripts/check-32bit-casts.sh (run by ci-test.sh) lints the wasm32 build
with clippy's cast_possible_truncation and fails on any u64 -> usize
finding; before this commit it listed 115.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Finding one link or attribute by name read every entry: Group::dataset and
Group::group (File, MmapFile, LazyFile) listed the whole group per call, and
path resolution scanned each group's links. Opening every child of a
35 001-link group by name decoded ~1.2e9 links.
Now a dense group's v2 B-tree name index (type 5, lookup3 hash of the
name) is descended to the records with the name's hash
(btree_v2::find_btree_v2_records reads only the nodes whose key interval
overlaps), and only those links are read and compared; all hash-equal
records are compared, so libhdf5's tie order does not matter. Dense
attributes the same through their type 8 index
(attribute::find_attribute_in_file, facade attr(name)); huge heap objects
through their ID-ordered index. group_v2::resolve_child returns what the
listing has under a name (soft links followed, dangling/external ones not
found). Group::entries and File::group_at hand out a listing's addresses.
The lookup-stats feature counts heap objects read. Tests: one lookup in
an h5py-written 35 001-link group with colliding hashes reads at most two
links (before: 35 001, failing), attribute lookups likewise (before: 3 000,
failing), every child opens through all three readers and matches h5py,
every link kind resolves as h5py resolves it in dense and compact groups,
300 huge attributes are found, and a range search matches a full scan at
every tree depth.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Lists the parsers now reading through Storage, what still needs the
whole file (v2 B-tree-indexed structures: a clean error; raw data: M2),
the equivalence harness, and the evidence that nothing changed: existing
tests, a byte-identical conformance results.json and per-file probe
output against f2ff2c4, and identical slice-API transcripts over 748
files between the two builds.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
With Storage in scope, `data.len()` on a `&&[u8]` resolves to
Storage::len (already a u64), so `as u64` was a no-op cast; name the
slice method explicitly.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Making dataset_fill_value_in generic over Storage broke callers that pass
an array (`include_bytes!`): a generic parameter does not unsize-coerce
`&[u8; N]`. It takes &[u8] again, as before this branch, and wraps the
new dataset_fill_value_from_storage(&dyn Storage, ..).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
tests/storage_equivalence.rs walks real files and runs every metadata
parser converted to Storage twice per object — over the file as a slice
and over a read_at-only CountingStorage — and requires identical results,
values and errors alike; the only allowed difference is the clean
ContiguousStorageRequired from the structures still indexed by a v2
B-tree (dense attributes, a SOHM B-tree index), which is counted. It
covers superblock and extension, cache image, SOHM table/list/B-tree,
object headers, attributes, fill values, shared messages, symbol-table
groups (local heap, B-tree, nodes, names), fractal heaps and their
objects, VDS mappings, and fixed/extensible array chunk indexes.
Inputs: every fixture; files h5py writes for what the fixtures lack
(extensible arrays with super blocks and paged data blocks, paged fixed
arrays, a 400-group v1 file with a user block, a 300-link dense group,
dense, shared and committed-type attributes, SOHM list and B-tree
indexes; honours CLAWHDF5_PYTHON / CLAWHDF5_REQUIRE_INTEROP); and, with
CLAWHDF5_STORAGE_CORPUS set, a corpus such as conformance/.cache/corpus.
Milestones M2/M3 add their parsers to check_object.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
ZFP (32013) was the one plugin filter still listed as UnsupportedFilter.
Conformance on tank, `conformance/run.sh --no-fetch` (2026-09-26): 600 of
697 files ok (baseline 599); h5ex_d_zfp.h5 is newly ok.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
DataLayout::resolve_vds_mappings_in reads the global heap collection
holding a virtual dataset's mappings through GlobalHeapCollection::
parse_in; resolve_vds_mappings is a wrapper. The rest of the data layout
module parses message payloads and does not read the file.
New test: the virtual dataset of vds_same_file.h5 resolves to the same
mappings through a read_at-only CountingStorage, in two reads.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
dataset_fill_value_in now takes the file as any Storage (generic, so
every &[u8] caller compiles unchanged) and follows a shared fill value
message through message_data_with_sohm_in. The raw-data helpers in the
module (read_full_with_fill, apply_to_unallocated_chunks) walk chunk
indexes and stay on &[u8] until milestone M2.
New test: the four datasets of shared_fill_value.h5, two with their
fill value in the SOHM heap, resolve identically through a read_at-only
CountingStorage.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
AttributeMessage::parse_in_storage, extract_attributes_full_in and
extract_attributes_tolerant_in take the file as &dyn Storage: shared
datatypes, dataspaces and attributes are resolved through the Storage
shared-message path, and dense attributes' fractal heap through
FractalHeapHeader::parse_in / read_managed_object_in. The dense-storage
name index is a v2 B-tree, which is not read over Storage yet: over a
backend without the whole file in memory it is a clean
ContiguousStorageRequired error, never a partial list. The &[u8]
functions are wrappers.
New test: every object in five h5py-written fixtures (compact, shared
and dense attributes) reads identically through a slice as Storage, and
through a read_at-only CountingStorage except the dense ones, which give
the clean error.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Every shared-message entry point has an *_in(&dyn Storage, ..) core:
message_data(_with_sohm), resolve_shared_message(_with_sohm),
resolve_sohm_message, load_sohm_table, and the SMTB/SMLI parsers, which
read the signature and then all entries in one bounded read (the list's
open-ended &file_data[pos..] slice is gone). Object headers and the SOHM
fractal heap are read through their Storage parsers; a SOHM B-tree index
still needs the v2 B-tree over a slice, so over a backend without one it
is a clean ContiguousStorageRequired error. New signature::find_signature_in
probes the candidate offsets with 8-byte reads. The &[u8] functions are
wrappers.
New tests: SOHM tables and lists with 4- and 8-byte offsets, at two
offsets, cut at every length and with a bad signature, parse identically
through a read_at-only CountingStorage in at most two reads; the
signature search matches the slice search.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
ExtensibleArrayHeader::parse_in reads the header in one read, and
read_extensible_array_chunks_in reads each index block, super block and
data block as a prefix read and then one window of the whole structure
(paged data blocks included); checksums and elements are checked in the
window with bounds errors reported as the whole-file checks did. The
&[u8] functions are wrappers. New test: an array with inline elements
and a data block, cut at every length and damaged in each structure,
reads identically through a read_at-only CountingStorage.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
FixedArrayHeader::parse_in reads the header in one read, and
read_fixed_array_chunks_in reads the data block's prefix and then the
whole block, paged or not, as one window; checksums and elements are
checked in it, with bounds errors reported as the whole-file checks did
(also in builds without the checksum feature, where the per-element
checks are the only ones). The open-ended &file_data[offset..] slices are
gone. The &[u8] functions are wrappers. New test: non-paged and paged,
filtered and unfiltered arrays, cut through the data block and with
damaged bytes, read identically through a read_at-only CountingStorage.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The facade forwards `zfp` and adds it to `plugin-filters`.
tests/zfp_interop.rs: h5py + hdf5plugin (H5Z-ZFP 1.1.1, zfp 1.0.1)
write 2205 datasets over 16 modes (rate, precision, accuracy,
reversible, expert settings at their edges) x int32/int64/float/double
x 1-4-D shapes with partial edge chunks, partial blocks and unit chunk
dimensions x smooth/noisy/wide-range/zero/inf-NaN data; clawhdf5 must
read each byte for byte as h5py does (read back after closing the file:
h5py returns a chunk still in libhdf5's cache without decoding it). A
second test swaps a file's header words to what a big-endian writer
stores and checks the byte-swapped values match h5py's.
The left-out-filter test now expects ZFP only in builds without it; CI
lints `zfp` alone and runs the new test with the plugin-filter step.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A port of the zfp 1.0.1 decoder and of H5Z-ZFP 1.1.1's decompression
path behind a new `zfp` feature (in `plugin-filters`): every mode
(fixed rate, precision, accuracy, reversible, expert), int32, int64,
float and double, 1-4 dimensional fields with partial blocks, and
headers written big-endian (values byte-swapped as H5Z-ZFP does).
H5Z-ZFP keeps the zfp header in cd_values (version word, then the
magic/metadata/mode bit stream); each chunk is the bare stream. The
decoder reproduces libzfp bit for bit: integer arithmetic wraps as
libzfp's, block exponents scale by exact powers of two, and the mode
goes through zfp_stream_mode as H5Z-ZFP hands it to zfp. A stream that
ends early is an error (libzfp reads past its buffer), as is a field
whose size is not the chunk's; the output is allocated only once the
stream holds a bit per block.
Tests: header/mode unit tests, and a counting-allocator fuzz of random
headers (expert parameters at their edges) and streams.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
FractalHeapHeader::parse_in reads the header as one window (a second,
longer one when it holds an I/O filter pipeline); read_managed_object_in
reads direct blocks, indirect blocks (one window up to the last child
entry) and huge objects with bounded reads. The &[u8] methods are
wrappers. A huge object indexed by the huge-object v2 B-tree, which is
not converted yet, is a clean ContiguousStorageRequired error on a
backend without the whole file in memory (after the "no index" check,
so the error order is unchanged).
storage::Window (crate-internal) reads a window of a structure and
reports bounds failures exactly as the whole-file ensure_len did, and a
short read inside the file is now a Storage error rather than an EOF.
New tests: headers (with and without a filter pipeline) cut at every
length, and managed objects in a direct root and through an indirect
root, huge objects with direct IDs and tiny objects, give identical
results through a read_at-only CountingStorage.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
BTreeV1Node::parse_in reads a node's header and then its keys and
children, two bounded reads; collect_symbol_table_nodes_in walks the tree
over any Storage. The &[u8] functions are wrappers. New test: nodes with
siblings and 4- and 8-byte offsets cut at every length, and a two-level
tree with truncated leaves, give identical results through a read_at-only
CountingStorage (six reads for the three nodes).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
SymbolTableNode::parse_in reads the node's 8-byte header, checks the
entries against the end of the file as before, and reads them in one
bounded read. parse is a wrapper. New test: nodes with 4- and 8-byte
offsets, valid and with a bad version, at two offsets and cut at every
length, parse identically through a read_at-only CountingStorage.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
GlobalHeapCollection::parse_in / parse_index_in read the header, check
the collection against the end of the file, and read the collection in
one bounded read; objects are indexed in it with file offsets, as before.
The &[u8] functions are wrappers. New test: collections with 4- and
8-byte lengths, one whose size runs past the file and one whose object
overruns it, at two offsets and cut at every length, give identical
results through a read_at-only CountingStorage.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
LocalHeap::parse_in reads the header in one bounded read,
validate_free_list_in reads each free block's two lengths, and
read_string_in reads from the string to the end of the data segment
once and looks for the terminator there. The &[u8] methods are wrappers.
New test: a heap without free space, with a valid free block and with a
free block overrunning the segment, cut at every length, parse, validate
and read strings identically through a read_at-only CountingStorage.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
read_superblock_extension_in, cache_image_state_in, CacheImage::decode_in
and CacheImage::block_in take &dyn Storage (whose length is the end of
file); the image block is one bounded read. The &[u8] functions are
wrappers; applying an image in place still needs the bytes in memory.
New test: extension messages, a cache image and a corrupt one decode to
the same results through a read_at-only CountingStorage.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
ObjectHeader::parse_in(&dyn Storage, u64, ..) reads the signature, the
prefix (a window of at most 34 bytes for version 2) and then each chunk,
continuation chunks included, as one bounded read; the message loops run
unchanged on the chunk with chunk-relative positions. parse keeps its
&[u8] signature as a wrapper. Bounds errors are reported as before, with
absolute positions and the file's length. New test: headers of both
versions, with times, phase-change values, creation order and a
continuation chunk, and every truncation of each, parse identically
through a read_at-only CountingStorage.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Superblock::parse_in(&dyn Storage, offset) reads one bounded window of
128 bytes (the largest superblock is 100) and runs the existing version
parsers on it; parse and refresh_eof keep their &[u8] signatures as
wrappers. No behaviour change: on a file longer than the window no bounds
check can fail, and on a shorter one the window is the whole file. New
test: every version and truncation parses to the same result through a
read_at-only CountingStorage, in one read.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Range-read milestone M1, first step (docs/design/range-reads.md §3(a)):
a synchronous, no_std read interface with u64 offsets, read_at returning
Cow<[u8]>, read_ranges, len and an as_contiguous fast path. Implemented
for [u8], Vec<u8>, &T, Box<T> and Arc<T>; slices serve borrowed bytes.
read_exact_at reproduces the parsers' UnexpectedEof bounds error exactly,
so converted modules keep their error values.
FormatError gains Storage(String) and ContiguousStorageRequired; it and
the facade Error are now #[non_exhaustive] (breaking for exhaustive
matches, noted in the changelog; the Python bindings' match gets a
wildcard arm).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
addr::to_usize turns a 64-bit file address or length into a slice index,
failing with FormatError::Overflow where it does not fit usize (32-bit
targets such as wasm32) instead of truncating like an `as usize` cast.
Callers are converted in the following commits.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Idle-start run on tank at c5334b1 (noisier than the last: compare ratios
within the run). Full reads of deflate data at 16 threads: 4944 MB/s vs
3135 for 16 h5py processes (1.58x; 0.69x-0.76x before). One thread with
the default pool: 6143 MB/s, 15x one h5py call. The concurrent-read
known issue is closed.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Regenerated on tank: ok 575 -> 599, our-error 10 -> 4, mismatch 20 -> 2,
no panics, hangs, crashes or OOM. Newly ok: metadata-cache-image files,
the CVE open-time checks, v1 chunk-key lookup as libhdf5 does it, the
scale-offset fixes, Blosc2, and harness corrections. Baseline raised.
h5rs check --data flags none of the 435 fully-read ok files.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
cve-2025-44905 now reads as h5py reads it (the v1 chunk B-tree lookup),
leaving 5 our-errors: cve-2025-2308, cve-2025-44904 and
bad_nbit_parms_walk (corrupt data HDF5 2.0 reads through a bug), and
the Blosc2 and ZFP filters. The five unloadable-cache-image files stay
ok, now with the library behaving as the probe reports.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
clippy (with the async feature) flags chunks_exact with a constant size
in the AsyncHDF5File cache-image test added in d493d47.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The scale-offset fix (d110b1d) was covered only by unit vectors from
CVE chunks, and documented as three corner cases. The review found it
is much bigger: of 1480 scale-offset datasets h5py writes (every
integer type i1..u8, f4 and f8, both byte orders, with and without a
fill value, scaleoffset 0..full width), v2.7.0's decoder read 332
differently from h5py: 151 returned wrong values with no error (82
integer datasets with scaleoffset=0 and a wide range, 51 full-width
i4/u4/i8/u8, 18 f4 D-scale datasets with a large range) and 181 failed
to read. The cause in every case is a chunk libhdf5 stores at full
width, whose elements were decoded as offsets from minval.
tests/scaleoffset_interop.rs generates that matrix with h5py at test
time, stores h5py's decoded values uncompressed next to it, and
compares every dataset's bytes. It passes on this branch; with the
filters.rs before d110b1d it reports "332 of 1480 scale-offset datasets
differ from h5py".
CHANGELOG: a Correctness entry stating this was silent wrong data in
every release that decoded scale-offset (v2.2.0 to v2.7.0), replacing
the corner-case wording. docs/known-issues.md: a fixed entry with the
affected cases.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
File, MmapFile and LazyFile decoded the superblock extension and laid a
metadata cache image over the file's metadata; the other readers did
not, so the same file read differently by entry point: NativeVol,
AsyncHDF5File and MpiVol (clawhdf5-io) and the external source files of
a virtual dataset (clawhdf5-format vds.rs) read a file with an image
from its own bytes, which libhdf5 does not (they may be stale, or zeros:
h5clear_mdc_image.h5 failed with InvalidObjectHeaderVersion(0)), and
skipped the extension checks File::open makes (cve-2020-10810/10812).
Each of them owns its buffer, so each now calls the shared
superblock_ext::apply_cache_image_in_place, which checks the extension
and writes the image's entries in place (only the image block is
copied). These readers read whole datasets and cannot open a file and
fail each object, so an image libhdf5 cannot load is refused with the
image's error, never read around. clawhdf5-io's vol::load_hdf5 wraps it
for NativeVol (at open; for from_bytes the error is reported on read,
as a truncated file already was) and MpiVol. The MpiVol edit is minimal
and was not compiled: the mpi-io feature needs an MPI installation this
machine does not have (mpi-sys's build script panics).
Tests: NativeVol (open_path and from_bytes), AsyncHDF5File and a VDS
whose source file is h5clear_mdc_image.h5 (vds_interop.rs, against
h5py) read the fixture's values; the corrupted-image variants are
refused. Each fails without its fix.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
For a metadata cache image libhdf5 cannot load, libhdf5 opens the file
and fails the first metadata read (the image loads on the first
H5C_protect after open); h5py reports the error on the root group. The
conformance probe reported it that way, but File::open refused the file,
so the gate counted cve-2025-6269-1..4 and cve-2025-6516 as agreeing
with h5py for behaviour the library did not have.
The library now behaves as the probe reports: File (mmap, buffered and
from_bytes) and MmapFile open the file and every object lookup (dataset,
dataset_at, group, group listings and attributes, VL decoding) fails with
the image's error; LazyFile reads the root group's header at open, so
its open is that first read and fails. Probe and library take the
three-way decision (refuse at open / image loads / image cannot load)
from the same clawhdf5_format::superblock_ext::cache_image_state.
One deliberate difference from libhdf5 remains, documented: after the
failed first read libhdf5 reads the file's own metadata, which the image
was meant to replace and may be stale; here every lookup keeps failing.
File::cache_image_error / MmapFile::cache_image_error expose the error
to code that parses as_bytes() itself; h5rs checks it before reading any
object header (h5rs ls on cve-2025-6269-1 said "invalid object header
version: 0" from the stale bytes).
Test: metadata_cache_image.rs an_image_libhdf5_cannot_load_fails_every_object
(the fixture with its image signature broken; h5py opens that file and
fails the first read with "Bad metadata cache image header signature").
It fails on the previous commit, where File::open refuses the file.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
apply_cache_image returned a copy of the whole file with the image's
entries written in, and File (mmap by default), MmapFile and LazyFile
used that copy for every read: opening a 1 GiB sparse file with an image
needed 2 GB of memory, and an 8 GiB one aborted the process, where
de2a53f (which ignored the image) opened them in a few MB.
The metadata parsers read one contiguous slice, so the image still has
to be laid over the file's bytes; it is now laid over a private copy
that costs only the pages it touches:
- clawhdf5_format::superblock_ext::CacheImage decodes the image into an
entry list (address, offset in the block, length) and applies it to
any destination; cache_image_state tells an opener whether the file
has no image, a loadable one, or one libhdf5 cannot load;
apply_cache_image_in_place is for readers that own their buffer.
apply_cache_image and metadata_view (which copied) are gone.
- clawhdf5_io::HDF5Read::private_copy returns a writable private copy
of a reader's bytes: MmapReader gives a MAP_PRIVATE copy-on-write
mapping (memmap2 map_copy), so only the pages the entries land on are
copied; the default copies the bytes (in-memory readers).
- File, MmapFile and LazyFile write the image into that mapping
(crate::cache_image). File::from_bytes / open_buffered patch their own
buffer in place, copying only the image block, as libhdf5 does. A
file without an image is read straight from the mapping, unchanged.
An image entry that runs past the end of file is now refused: libhdf5
checks only that it starts inside the file, and the images libhdf5
writes never do this, but those bytes have nowhere to go in a view of
the file.
Tests: tests/cache_image_memory.rs has libhdf5 (through ctypes) add an
image to a 1 GiB sparse file and bounds resident-memory growth for all
three openers at 256 MiB; it fails on the previous commit (File::open
grew 2,148,720,640 bytes). reader.rs zero_copy_tests check that a file
without an image is read from the mapping itself and that an image goes
into a copy-on-write mapping, not a heap copy; clawhdf5-io checks that
private_copy writes never reach the reader or the file.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The review suggested libhdf5 loads every image entry and resolves
flush-dependency parents afterwards. It does not:
H5C__reconstruct_cache_contents (HDF5 1.14.6 and 2.0.0, and develop)
inserts each entry and then searches the cache index for its parents in
the same loop, failing with "fd parent not in cache?!?" when one is
missing. So a parent must be an earlier image entry, as before, or
metadata cached before the image loads: the superblock (address 0) and
the superblock extension's object header, which libhdf5 reads to find
the image. Those two were refused as parents; they are now accepted.
A parent listed after its child is still refused, as libhdf5 refuses
it, and so is an entry that is its own parent ("Child entry flush
dependency parent can't be itself").
apply_cache_image takes the superblock to know the extension address.
Test: superblock_ext::tests::flush_dependency_parents_must_already_be_cached
(parent-first loads, child-first refused, extension header accepted,
self-parent refused); the extension-header case fails without the fix.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
libhdf5 does not walk the chunk B-tree to read a dataset: it looks each
chunk up (H5B_find with H5D__btree_cmp3 and H5D__btree_found), asking for
the element-size coordinate as 0. collect_chunk_info_checked now parses
the tree with its keys and returns each stored chunk only when that
lookup, replayed over the scaled keys, finds it.
A key with a non-zero element-size coordinate is therefore found in a
1-D dataset (cmp3 compares only the first coordinate there, and found
compares with <=) and missed in a dataset of rank 2 or more, which reads
fill values. The previous commit refused every such key, which refused
1-D files libhdf5 reads correctly; before that, the rank-2 case read the
chunk's data where h5py reads fill values (cve-2025-44905
/Shuffle_float_data_le, now identical to h5py, so it leaves the
conformance report's list of libhdf5 bugs).
Test: chunk_keys_with_an_element_offset_read_as_libhdf5_reads_them
compares 1-D and 2-D files against h5py's values. It fails on the
previous commit (the 1-D file is refused) and with the refusal removed
(the 2-D file reads 0..23 where h5py reads fill values).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The Blosc2 fuzz tests checked only for panics and the size of the output,
so the offsets-chunk amplification passed all 40,000 iterations.
tests/blosc2_alloc_bounds.rs now measures peak allocation (a counting
global allocator) and asserts it stays within 6x the HDF5 chunk size plus
twice the input plus 2 MiB (ruzstd's fixed state) for:
- 20,000 mutated fixture frames, decoded with their real chunk size as
the limit, with edits aimed at the frame's and chunks' size fields;
- 20,000 mutated first chunks on their own;
- 5,000 frames built from random header sizes, offsets chunks and B2ND
shapes (padding chunk and block shapes, chunks larger than the array,
NaN, zero and repeated-value chunks).
Against the code before this series every test in the file fails (the
fuzz tests at frame iteration 3913, a zstd window, and random frame 289,
the offsets chunk); now the worst frame peaks at 0.48 of the bound.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
ruzstd reserves a frame's declared window (up to its 100 MiB default)
when a decoder is reset for a new frame, before decoding anything. The
Blosc, Blosc2 and bitshuffle decoders reuse one decoder per chunk, so a
Blosc2 chunk of two 16-byte streams, each declaring a 96 MiB window,
allocated 128 MiB. zstd_decode_into now sets the decoder's maximum window
to twice the stream's output (at least 128 KiB): c-blosc, c-blosc2 and
bitshuffle compress each block in one call with its size known, so
libzstd's window never exceeds the block.
Found by tracking peak allocation in the Blosc2 fuzz test.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
B2ND chunks were decoded whole, padding included, with up to 16x the HDF5
chunk size as their limit, so a crafted frame made each chunk allocate and
fill up to 16x the output (4 GiB for a 256 MiB HDF5 chunk). The padding is
the real bound (prod(ceil(c/b)*b) per chunk), but that can be 2^ndim times
the array, so it is no longer held at all:
- A Blosc2 chunk is now decoded block by block (decode_blocks), each block
handed to a sink as it is ready, with at most three blocks of scratch.
blosc2_decompress_chunk and plain frames still collect every block.
- reassemble places each B2ND block straight into the output and skips
blocks that are all padding (they are not decoded unless the delta
filter needs the first block). A frame's NaN chunks are handed over one
B2ND block at a time and its zero chunks cost nothing.
- A B2ND chunk must decode to exactly its padded size, its Blosc2 blocks
must be whole B2ND blocks no larger than the output, and a chunk may not
be larger than the array (hdf5-blosc2's chunk is the array), so a block
is never larger than the output.
Peak allocation for a 10-D array padded to 13x (NaN, repeated-value and
stored-block chunks) and for a 16x chunk was 4.5 MB and 17.8 MB for
315 KB and 1 MiB outputs before, and is now within the tests' bound.
Blosc2 files written by hdf5plugin in 9-D and 12-D, an 8 MiB single chunk,
1x1x1 and edge-chunk shapes still read exactly.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A chunk header with nbytes 0 and no special type kept its declared block
size (up to 512 MiB): the block size was clamped to nbytes only when nbytes
was positive, and the scratch blocks were allocated before the (empty)
block loop, so a 20-byte chunk allocated about 1 GiB. The block size is now
clamped to nbytes always, and an empty chunk returns before any scratch is
allocated.
A frame chunk must also decode to the size the frame header gives it
(chunksize, or the remainder for the last chunk), and is decoded with that
as its limit, so an empty chunk in a frame for a non-empty HDF5 chunk is an
error rather than an empty result.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
parse_frame sized the offsets chunk from the frame header's own nbytes and
chunksize, so a 173-byte frame declaring 32 Mi chunks, with a 40-byte
repeated-value offsets chunk, built 256 MiB (up to 2 GiB) of offsets for a
1 MiB HDF5 chunk and then returned 4 bytes. The offsets chunk is now capped
at the output limit (at least 128 bytes); a frame whose nbytes/chunksize
imply more chunks than that is refused before anything is allocated.
tests/blosc2_alloc_bounds.rs measures peak allocation with a counting
global allocator; the reviewer's frame failed it (decoded Ok(4)) before.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Review follow-ups. With run_with_helpers a one-thread rayon pool gave each
read a second core (the caller plus the worker), so --decode-threads 1 no
longer matched h5py's one core per call; such a pool now adds no helper.
Per-thread decode scratch is kept up to 1 MiB per buffer (was 4 MiB),
bounding what never-exiting pool workers hold.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Conformance on tank, conformance/run.sh --no-fetch (2026-09-26): 597 of
697 ok, 6 our-errors (4 corrupt objects HDF5 2.0 reads through a bug, the
Blosc2 and ZFP filters), 2 mismatches (the known h5py big-endian VL bug).
Closes the known-issues entries for metadata cache images,
cve-2024-32624, cve-2020-10810/10812, and unfiltered chunks of the wrong
size; the N-Bit / 64-bit scale-offset entry is recorded as not our bug.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Four of the remaining our-errors are objects the reference (h5py 3.16 /
HDF5 2.0) reads only through a libhdf5 bug, and clawhdf5 refuses:
cve-2025-2308 (scale-offset codes past the end of the chunk),
cve-2025-44904 (short unfiltered chunks), bad_nbit_parms_walk.h5 (an N-Bit
parameter list one value short; libhdf5's own test_filter_bad_params now
requires the read to fail) and cve-2025-44905 /Shuffle_float_data_le (a
chunk key libhdf5's lookup misses, reading fill values). report.py lists
them under Known not-our-bug and counts them in the summary; they stay
our-errors in the class counts.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
libhdf5 loads a metadata cache image when it first reads metadata (the
root group), not at open, so for cve-2025-6269-1..4 and cve-2025-6516 (all
corrupt images) h5py opens the file and fails on "/". The probe reported
the image's error as an open error, which made those files our-errors;
it now records it on the root object, where h5py reports it. File::open
still refuses such a file outright: nothing in it can be read.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Placing a chunk cast its u64 offsets to usize; on a 32-bit target an
offset past the address space wrapped into the output (and could then
overlap another chunk's region when chunks are placed concurrently).
Such an offset is past the dataset, so it now saturates and the chunk
writes nothing, as the concurrent-placement check already assumed. No
change on 64-bit targets, where the cast cannot wrap (so no test here).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
CHANGELOG entry for the chunked read changes, and the known-issues entry
on concurrent chunked reads updated: both causes it names (per-read page
faults, readers waiting on a small pool) are fixed; the 16-thread
comparison with h5py stays open until re-measured on an idle machine.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
- A dataset without filters stores every chunk at the chunk's full size.
A chunk its index records at another size was read at that size, with
the rest of the chunk left as zeros (cve-2025-44904's
Scale_offset_float_data_le: 38- and 37-byte chunks for 48-byte chunks,
where HDF5 2.0 fills the rest with whatever its buffer held). It is now
refused, as later libhdf5 releases refuse it ("incorrect chunk size
returned from index for unfiltered chunk"):
chunked_read::list_chunks_for_read, used by every read path.
- A v1 B-tree chunk key carries 0 in the element-size dimension. libhdf5
compares that coordinate when it looks a chunk up, so whether it finds a
chunk keyed otherwise depends on where the key falls (in cve-2025-44905
/Shuffle_float_data_le, offset 4096, it does not, and h5py reads fill
values); we read the chunk. Such a key is now refused.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Scale-offset (filter 6) now follows H5Z__filter_scaleoffset:
- the packed codes start at byte 21 whatever size the chunk records for
minval (libhdf5 reads min(8, size) bytes of minval and always starts the
codes at buf_offset 21). We started them after minval plus 8 bytes, so a
chunk recording a size of 0 (cve-2025-44905 /Scale_offset_short_data_be)
decoded differently from h5py;
- with a fill value defined, a code equal to the all-ones code of minbits
bits is the fill value, including minbits 0 (code 0): a chunk of nothing
but fill values read as minval;
- minbits of the full width stores the elements as they are (no minval
added), and an integer scale factor of the full width means the chunk was
left untouched; minbits or a scale factor wider than the type is an
error;
- the class parameter (integer or float) decides the decode, a scale type
that does not match it is refused, and E-scale is refused, as in libhdf5
(no library writes it; it was decoded here unchecked);
- minval is the stored bytes zero-extended, as libhdf5 reads it.
Codes past the end of the chunk stay an error, as in libhdf5 releases
after 2.0 ("Buffer too short"; 2.0 reads past the buffer, cve-2025-2308).
Shuffle (filter 2) uses its own parameter as the element size, as libhdf5
does, instead of the dataset's element size; a parameter larger than the
chunk leaves the chunk as it is (cve-2025-44905 /Shuffle_float_data_be),
and a parameter of 0 is an error.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
libhdf5 decodes the messages of a v2/v3 superblock's extension when it
opens a file (H5F__super_read) and refuses the file when one does not
decode. We never looked at them, so we opened cve-2020-10810 (a File
Space Info message too short for the free-space manager addresses it
announces) and cve-2020-10812 (a metadata cache image past the end of the
file), both of which libhdf5 refuses.
A file written with a metadata cache image keeps its metadata cache
entries in an image block the extension points at; libhdf5 loads them
over the file's own bytes before it reads any metadata
(H5C__load_cache_image, H5C__reconstruct_cache_contents). In
h5clear_mdc_image.h5 the root group's header exists only in the image, so
every reader failed with InvalidObjectHeaderVersion(0).
The new clawhdf5_format::superblock_ext module:
- read_superblock_extension decodes the v1 B-tree K, File Space Info and
Metadata Cache Image messages with libhdf5's checks (versions, page size
512 B .. 1 GiB, the addresses a persisting message lists, the image
inside the file), with the new FormatError::InvalidSuperblockExtension;
- apply_cache_image checks an image block as libhdf5 does (signature,
version, recorded length, entry types, rings, ages, addresses inside
the file and not repeated, flush-dependency parents) and returns the
file's bytes with every entry written at its address
(FormatError::InvalidCacheImage);
- metadata_view does both.
File, MmapFile and LazyFile (and so h5rs) call metadata_view at open and
read an image file through the patched copy; the conformance probe does
the same. The image's trailing checksum is not verified, as libhdf5 does
not verify it. tests/fixtures/h5clear_mdc_image.h5 is libhdf5's own test
file.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Record the `blosc2` feature in the changelog, the README's feature table
and the crate table, and mark the Blosc2 half of the known "Filters"
issue fixed (dated, with the conformance run that shows h5ex_d_blosc2
reading). What stays open: ZFP, writing Blosc2, and the Blosc2 features
hdf5plugin never writes (dictionaries, lazy chunks, variable-length
blocks, user-defined codecs and registered filters), which are errors.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
hdf5plugin's Blosc2 was a clear "not implemented" error. It stores each
HDF5 chunk as a Blosc2 contiguous frame; for chunks of 2+ dimensions the
frame holds a B2ND array whose data are cut into (padded) blocks stored
one after another.
filters_blosc2 (feature `blosc2`, in `plugin-filters`; the facade
forwards both) decodes, following c-blosc2's decoder and hdf5-blosc2's
blosc2_filter.c:
- the frame: the msgpack header's fixed fields, the metalayer index, the
offsets chunk (with the special zero/NaN/uninitialised offsets) and
chunk lookup;
- Blosc2 chunks: 16- and 32-byte headers, special chunks (zeros, NaN,
uninitialised, one repeated value), split and unsplit streams, zero and
run-length streams, and the filter pipeline run backwards (shuffle,
shuffle with a byte-group size, bit shuffle including the version-2 and
later handling of a partial group of 8, delta against the first block,
truncated precision);
- the codecs, shared with Blosc 1: BloscLZ, LZ4/LZ4HC, Zlib, Zstandard;
- B2ND arrays: blocks gathered into C order, padding dropped, several
chunks per array, and the array shape checked against the chunk shape
in cd_values as the HDF5 filter does.
Dictionaries, lazy chunks, variable-length blocks, user-defined codecs
and registered filters (e.g. bytedelta) are errors. Uninitialised chunks
read as zeros. No encoder.
Tests: h5py + hdf5plugin write every codec x filter (none, shuffle,
bitshuffle, delta) and levels 0-9 over the plugin-filter cases, then
i1..u8/f4/f8 in 1-D to 5-D chunks with partial edge chunks, datasets of
zeros, one value and NaN, and Fletcher32 before Blosc2 (plain frames for
n-D chunks); clawhdf5 reads each exactly as its unfiltered twin, and
truncated precision exactly as h5py reads it. Fixture frames from
python-blosc2 (tests/fixtures/blosc2/generate.py) cover what hdf5plugin
never writes: special chunks, delta over many blocks and odd type sizes,
odd bit-shuffle blocks, forced splitting, multi-chunk B2ND arrays with a
zero chunk, and the refused features. The decoder is fuzzed (random and
mutated frames and chunks: no panic, output within the limit).
Conformance: h5ex_d_blosc2.h5 now reads (576 of 697 ok, baseline 575).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
libhdf5 checks a dataset's storage when it opens the dataset
(H5D__contig_check, H5D__compact_init): the element count times the
element size must not overflow, contiguous storage must end inside the
file, compact data must be the dataset's size. File::dataset opened
cve-2024-32624's /Dset_OBJREF (2^62 + 2 references of 8 bytes) and
reported its shape; only reading failed.
data_read::check_dataset_storage makes those checks (new
FormatError::InvalidDatasetStorage), and File, MmapFile and LazyFile run
it whenever they open a dataset (by path, by address, from a group), as
does the conformance probe. As before, a datatype, dataspace or layout
that does not decode is left for the read to report, so such a dataset
still opens and its attributes still read. An empty contiguous dataset at
a defined address, which libhdf5 refuses, is still accepted: clawhdf5 up
to v2.7.0 wrote them.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A selection read (a hyperslab or points covering at most half the
dataset) decoded each chunk it overlaps into fresh buffers, one per
filter stage; it now uses the thread's chunk-decoding scratch like the
full readers. Covered by tests/chunked_read_paths_interop.rs (small
hyperslabs and points over every filter and type) and the partial-read
equivalence tests.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
read_f32/read_f64/read_i32/read_i64/read_u64 of a chunked dataset that
stores exactly that type in native byte order now decode every chunk
straight into the Vec<T> they return (data_read::read_chunked_native),
on File (through its chunk cache), MmapFile and LazyFile. Before, the
chunks went into a byte buffer that read_as_* then copied into a second,
typed one: two dataset-sized allocations and a full extra copy per read.
The output is zeroed pages from the allocator, backed by transparent
huge pages when large, like the byte reader's. Other types and byte
orders, and datasets with no storage or external data, keep converting
through the byte readers; unallocated chunks read as the fill value as
before.
tests/chunked_read_paths_interop.rs checks every chunked read path
(File twice, so cached; from_bytes; MmapFile; LazyFile; small, strided
and point selections; with and without the parallel feature) against
h5py for 1-8 byte integers and 2-8 byte floats in both byte orders,
through deflate, shuffle, Fletcher32, LZF, SZIP and Blosc, with partial
edge chunks, sparse datasets with default and non-default fill values,
and datasets larger than the chunk cache. A filter this build lacks must
be an error (or, when an optional filter declined every chunk, the right
data).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
h5py's track_order=True orders attributes as well as links; the writer
tracked links only. A tracking object's header now sets the attribute
creation order tracked/indexed flags and carries per-message creation
orders, an Attribute Info message holds the next order (inline too),
and dense storage gets a type-9 creation-order index. The file default
applies to datasets, with DatasetBuilder::track_order per dataset; more
than 65 535 attributes on a tracking object is an error (libhdf5's
counter is 2 bytes). The reader lists such attributes in creation
order.
h5py lists them in order (inline, dense, 20 000 on one dataset) and
keeps numbering in r+ mode, including its inline-to-dense move.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
- A simple dataspace of rank 0 holds one element in libhdf5 (the product
of no dimensions; h5py reads it as shape ()). num_elements() said 0, so
cve-2020-18494's /dset1 failed with DataSizeMismatch { expected: 0 }.
- A contiguous dataset whose storage is larger than its elements reads:
libhdf5 reads the elements from the start of the storage and ignores the
rest (H5D__contig_check checks only that they fit in the file). We
required the sizes to be equal, so the scalar /Dset1 of cve-2024-32623
and cve-2025-2309 (240 bytes of storage for one int) failed. Storage too
small for the elements is still an error. data_read::contiguous_read_len
is the rule, used by every contiguous read path.
- Dataspace::parse refuses what H5O__sdspace_decode refuses: more than 32
dimensions, a rank on a scalar or null dataspace, a dimension larger
than its maximum (new FormatError::InvalidDataspace).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Inventory of the 109 whole-file parser functions, a measurement of the
metadata ranges clawhdf5 touches on three corpus files against libhdf5,
options (storage trait, virtual slice, metadata prefetch, userfaultfd)
with how ros3, h5py+fsspec, pyfive, h5wasm and object_store do it, and
an incremental plan: M0 name-index lookups, M1 metadata over a Storage
trait, M2 raw data with batched ranges, M3 object_store backend,
M4 wasm fetch-driven lazy loading.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
inventory.py counts the functions and call sites that take the whole
file as &[u8]; range-trace (standalone crate, x86-64 Linux) records every
load clawhdf5 makes from a file by mprotect + single-step, unchanged
library code; libhdf5_reads.py counts libhdf5's reads through h5py's
fileobj driver and prints a dataset's chunk extents.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
libhdf5 refuses to list a symbol-table group that has an entry with an
empty name (H5G__ent_to_link: "invalid link name"), so h5py cannot list
cve-2021-46244's /BAG_root. We listed it, with an object at "/BAG_root/"
(the empty name, pointing at address 0). resolve_v1_group_entries — the
listing — now fails with the new FormatError::InvalidLinkName; path
lookups still find the group's other names, as libhdf5's by-name lookup
does.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
libhdf5 decides what an object header is in a fixed order
(H5O__obj_class_real): a group if it has a Symbol Table or Link Info
message, a dataset if it has a Datatype *and* a Dataspace message, a named
datatype if it has a Datatype message. The conformance probe called any
header with a Data Layout message a dataset, so cve-2024-33874's /Dset1 (a
datatype and a layout, no dataspace), which h5py opens as a named
datatype, was reported as a dataset we failed to read
(MissingMessage(Dataspace)). The probe now classifies with
object_class().
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Full reads of a chunked dataset (the cached reader behind the facade's
read_* and the uncached one behind mmap/lazy files and verify_provenance)
now decode each chunk into this thread's reusable scratch buffers and copy
it straight to its place in the output. Before, the cached reader decoded
batches of 128 chunks into fresh Vecs and the uncached one decoded every
chunk of the dataset into its own buffer before assembling any: a new
256 KiB allocation (and its page faults) per chunk and per filter stage.
Chunks are still inserted into the file's chunk cache when the whole
dataset fits in it.
With the parallel feature the calling thread now decodes too, sharing the
chunks with whichever rayon workers are free (run_with_helpers): a helper
the busy pool only starts after the read is done returns at once. Before,
the caller handed every chunk to the pool and slept, so readers outside a
small pool (2-4 threads) queued behind its workers; with a one-thread pool
the reads went sequential. Chunks are placed concurrently only when the
index puts them on the chunk grid at distinct places (a corrupt index is
read one chunk at a time), and the error returned is still the first
failing chunk's.
Fix: a chunk stored unfiltered in a filtered dataset (every filter-mask
bit set) that is shorter than a chunk read as zeros where its data was
missing through the cached reader (the facade's read_*); it is now an
error naming the chunk, as the uncached reader already made it.
Regression tests, both failing before this change:
tests/busy_decode_pool.rs (both workers of a two-thread pool busy, four
readers) and short_unfiltered_chunk_of_a_filtered_dataset_is_an_error.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
ref.py and compare.py reported 13 files as mismatches or our-errors that
were artefacts of the harness, not differences between the readers:
- User-defined links (tall.h5, tudlink.h5, twithub*.h5, tmany.h5, ...):
h5py's `get(name, getlink=True)` reports a user-defined link as a
HardLink, so ref.py listed it as an object. Read the link type from
H5Lget_info instead.
- Objects h5py cannot open (cve-2019-8397/8398, cve-2021-46243,
cve-2024-32618): the probe deduplicates by header address, ref.py by
ObjectID, which an unopenable object does not have, so each extra hard
link to it was listed again. Deduplicate those by link address.
- Nested array types (tarray3.h5): h5py expands them into trailing dims;
hash_values stripped one level and numpy broadcast every element into a
whole subarray. Strip every level.
compare.py no longer compares the attributes or links of an object h5py
could not open at all (cve-2018-17438/17439, cve-2019-9151): h5py read
none, so ours are neither extra nor errors against it.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Dense link and attribute indexes and the chunk index for several
unlimited dimensions were single leaves, capping them at 65 535
records. btree_v2_write builds trees of any depth, with node capacities
and pointer widths from libhdf5's H5B2__hdr_init arithmetic (now shared
with the reader as btree_v2::node_info) and libhdf5's node sizes (512
dense, 2048 chunks). Indexes that fit the old one-leaf layout are
written byte for byte as before (compared for 10..65 535 links, attrs
and chunks, tracked and filtered).
Tests: 100 000 links (short names; long names with creation order),
70 000 attributes, 200 000 chunks (and 80 000 deflated), read by h5py,
h5dump and clawhdf5 and edited by h5py r+; h5rs check on the same
shapes, asserting depths 2-3.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
decompress_chunk_exact_with decodes a chunk into a DecodeScratch the
caller keeps between chunks, instead of a new Vec per chunk and per filter
stage. Deflate inflates into a kept buffer with a reset (not rebuilt)
inflater, shuffle interleaves into the other buffer, and Fletcher32 checks
and drops its checksum in place (on the stored bytes when it is the first
filter undone). Other filters go through the registry as before. Output
and errors are those of decompress_chunk_exact; a unit test checks that
for every pipeline shape and filter mask with one reused scratch.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
libhdf5 compares the name when two hashes are equal; the writer broke
ties by insertion order, and libhdf5 could not find one of two names
whose lookup3 hashes collide (k69209 / k155448). Test fails before the
fix with h5py's KeyError.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Idle tank at 408f69e, h5py re-run in the same session. Contiguous reads
went from 0.25x to 1.44x h5py (full) and 0.12x to 6.3x (256x256
hyperslabs) on one thread; deflate full reads at 8 threads 887 -> 2943
MB/s (h5py processes 3042). Full chunked reads at 16 threads are still
0.69x-0.76x h5py processes; the issue stays open.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
