Regenerated on tank: ok 599 -> 600 (h5ex_d_zfp.h5), our-error 4 -> 3,
mismatch 2, no panics, hangs, crashes or OOM. Baseline raised.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
FileBuilder stored every chunk of an LZF or Blosc dataset through the
filter with filter mask 0. libhdf5 counts LZF and Blosc output no smaller
than the chunk as a failure of the optional filter and stores the chunk raw
with the filter's mask bit set. For an LZF chunk whose stream was exactly
the chunk's size, the first libhdf5 rewrite stored raw data at the same
size and kept our stale mask 0 in the index, and h5py could no longer read
the dataset.
precompress_chunks now runs chunks through compress_chunk_masked (as
FileEditor does since f7e2ab1), sequentially and on the parallel path, and
build_chunked_data_from_precompressed records each chunk's real mask in
every index the writer builds: single chunk (layout field), Fixed Array and
Extensible Array filtered elements, and version-2 B-tree type 11 records
(create_datasets_parallel goes through the same path). The writer builds
no version-1 B-tree or implicit index. PrecompressedChunks::chunks gains
the mask. Files whose chunks all compress are byte-identical.
Latent only in the unreleased LZF/Blosc writer (added 2026-09-26); no
tagged release writes either filter.
Tests:
- plugin_filters_interop skipped_optional_filters_are_masked_as_libhdf5_masks_them:
LZF, shuffle+LZF+fletcher32 and Blosc over random, compressible and
alternating chunks in every index; masks equal an h5py-written twin's;
h5py r+ rewrites and extends them; h5py, h5dump and our reader read
every value. Before: 20 of 24 datasets had masks other than h5py's, and
with that check disabled h5py failed to read the rewritten datasets
("filter returned failure during read").
- plugin_filters_interop files_whose_chunks_all_compress_are_unchanged:
pins the pre-fix bytes of five all-compressing files.
- chunked_write skipped_lzf_chunks_are_masked_in_every_index (fails before:
mask 0, want 2).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
- Per-file probe output is identical for 696 of 697 files, not all:
cve-2025-2310.h5's error string depends on which parallel chunk decode
fails first, at f2ff2c4 as on this branch.
- The parser cores are generic (S: Storage + ?Sized); provisional A/B
numbers against f2ff2c4, including the one bench that still shows
ObjectHeader::parse slower when old and new are separate binaries.
- Reads sized by untrusted fields are bounded; the harness only accepts
the known whole-file fallbacks.
- range-reads.md records why M1 went generic rather than &dyn.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
On a backend without the file in memory, a structure read whose length
comes from untrusted header fields was clamped only by the end of the
file, so a crafted size made one read (and copy) of up to the rest of
the file. Each such read now covers what the parser actually uses:
- local heap names: read in growing pieces (64 bytes first, then 4x)
up to the end of the data segment, instead of the rest of the segment
per name (quadratic for a big symbol-table group);
- fractal heap indirect blocks: the doubling-table geometry locates the
entry covering the object, and the first read ends at that entry; only
if it is unallocated does the walk read the rest of the block (it
visits every entry then). One walk implementation serves both;
- paged fixed/extensible array data blocks over 1 MiB: the prefix and
page bitmap, then each page in use on its own (smaller blocks are
still one read);
- blocks under one checksum (non-paged array data blocks, extensible
array index and super blocks): the bounds check that comes first (the
checksum's; the page bitmap's for a super block) is made against the
file length before reading (Window::check_extent), so a block claimed
past the end of the file costs no read. With the checksum feature off
the parser has no such first check and the old read stands.
Other windows were already bounded (the superblock and object header
prefixes, the fractal heap header by a u16, SOHM tables by u8/u16
counts) or are exact reads checked against the file length first.
In memory nothing changes: the pieces are borrowed slices.
Tests: CountingStorage over a crafted heap (16 MiB file, width and rows
0xFFFF: under 1 KiB read, 16.7 MB before), a heap segment claiming 64 MiB
(one 64-byte read per short name), long names at every piece boundary,
a fixed array block claimed past the end of a 16 MiB file (under 64
bytes read), and in the equivalence harness an h5py file with a 2.4 MB
fixed array block and a >1 MiB extensible array block, whole and cut at
97 points: every chunk index agrees with the slice read and the largest
takes 205 KB (2.4 MB and 1.2 MB when read whole).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The harness counted ContiguousStorageRequired from any parser as an
allowed difference, so a converted module that wrongly fell back to the
whole file would still pass. It now accepts the error only from the
three sites that are not converted yet (dense attribute storage, a SOHM
B-tree index, huge fractal-heap objects, all found through a v2 B-tree)
and only in the checks that can reach them; anything else fails with
the check and the site named.
Checked by making LocalHeap::parse_in return the error first: the
fixture and h5py runs fail ("local heap fell back to the whole file").
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Every `*_in` core and the read helpers take `file: &S` with
`S: Storage + ?Sized` instead of `&dyn Storage`, and the `&[u8]`
wrappers pass the slice itself, so they compile to a `[u8]` instance:
`as_contiguous()` inlines to `Some(self)` and each structure read is the
slice code's bounds check again, with no indirect call. `&dyn Storage`
still works (`S = dyn Storage`); there is one parser implementation.
Also, so the structure reads cost no more than the slice checks did:
- ObjectHeader::parse_in reads the prefix once (signature included)
instead of the signature and then the prefix: two reads for a
one-chunk header instead of three on a range backend;
- the symbol-table node and group B-tree (v1) loops walk their entries
with chunks_exact over the bytes read, and the node's redundant second
bounds check is gone (the entries' read is the check, same error);
- a version-1 header's message list is sized from its (capped) count.
Same results and errors; the unit and equivalence tests are unchanged.
New Criterion bench `clawhdf5/benches/local_metadata_bench.rs` over a
400-group version-1 file written by h5py (new fixture
`v1_groups_400.h5`): ObjectHeader::parse, symbol-table nodes, the group
B-tree walk and a facade listing, using only APIs that exist at f2ff2c4
so it builds there for an A/B.
Provisional A/B against f2ff2c4 (busy machine, not for docs): both
builds linked into one binary and timed in alternation, 200 rounds;
median ratio new/old: facade listing -0.5% to -3.5% (was +14%),
ObjectHeader::parse +1% to +2% (was +25%), symbol-table nodes -18%,
group B-tree walk -18%, local-heap names and resolve_group_children
within +-1.5%. An old-vs-old-copy run shows +-2% from code layout alone.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Each edit planned over the reader's memory map of the file and committed
while that File, and the Image's &[u8] over the mapping, were still alive,
writing the same file through the editor's descriptor. Nothing read the
mapping during the writes, but a shared slice whose memory changes
underneath it is undefined behaviour under Rust's aliasing rules.
Image::into_plan now detaches the edit's writes (patches, end of
allocation) into a Plan that owns all of its bytes and borrows nothing;
edit() takes the user-block size, drops the File — unmapping the file —
and only then commits the Plan. The invariant is documented in the image
module and the editor's module docs.
Test: edit::tests::file_is_not_mapped_while_an_edit_writes_it checks
/proc/self/maps at the moment each commit starts (write, resize, set_attr):
never mapped. With the commit moved back before the reader is dropped (the
previous order) it reports all three commits with the file mapped.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A dataset whose filter this build cannot encode (scale-offset, N-Bit, SZIP;
a plugin filter the build lacks) failed with Error::Format("unsupported
filter: 6"), although the editor documents every refused edit as
Error::Unsupported, and the Python bindings raised ValueError rather than
NotImplementedError. Every edit now maps FormatError::UnsupportedFilter to
Error::Unsupported; the file is left untouched as before.
Test: edit_interop unencodable_filters_are_unsupported — h5py scale-offset
datasets (integer with chunks, integer never written, float D-scale):
Error::Unsupported naming the filter, and the file byte for byte unchanged.
Fails on the previous editor (Format(UnsupportedFilter(6))).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
libhdf5 counts a version-2 object header's attributes through its Attribute
Info message (0x15) and reports none when the header has none. set_attr gave
v110/latest groups, the root group and datasets without attributes an
attribute message only, so h5py listed the attribute but len(obj.attrs) and
H5Oget_info's num_attrs said 0, and stayed wrong after h5py r+ added more.
Like H5O__attr_create, the edit now adds the message when a version-2 header
lacks it, in the same planned edit: version 0, the header's creation-order
track/index flags, maximum creation index 0, undefined fractal heap and
B-tree addresses, message flag DONTSHARE — byte for byte what libhdf5
writes. It goes before the attribute (libhdf5's order) when free space
holds both, else after it, so a continuation chunk made for the attribute
also takes it.
Test: edit_interop attribute_count_in_version_2_headers — v110 and latest
files, attributes set on the root group, groups and datasets with and
without existing attributes: h5py's len/num_attrs/list/values, h5dump -A
and our reader agree, also after h5py r+ adds attributes up to and past the
compact limit. Fails on the previous editor (h5py len 0).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A valid group has one link per name, but a damaged or hand-made one can
have two. resolve_child followed the first soft link of the name, the
listing skipped a dangling one and listed the name via a later link, and
path resolution followed the last symbolic link: three answers. All now
take the first link of the name (header message order in a compact group,
name index order in a dense one) and ignore the rest, even if the first
dangles. That is libhdf5's rule for compact groups (H5G__compact_lookup
stops at the first Link message); h5py opens nothing for a dangling first
link although a later one resolves. For a dense group libhdf5
binary-searches the index and may land on another of several exact
duplicates; documented on first_link_named. find_symbolic_link's v2 branch
was dead (only v1 groups reach it) and is now v1-only.
Test: an h5py compact group with soft links dup_A (dangling, or to /d) and
dup_B (the other), dup_B renamed to dup_A in the header and re-checksummed.
Lookup, path and listing through all three readers match h5py for both
orders. With the old group_v2.rs the path lookup returned 42 where h5py
opens nothing.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
addr::to_usize's error branch only ran where usize is narrower than u64,
and no such target runs tests in CI, so on x86_64 the test checked only
that every u64 fits. to_usize and saturating_usize are now the usize
instances of generic to_index/saturating_index; the test runs the same
code with u32 standing in for a 32-bit usize: values past u32::MAX
(including one an `as` cast would wrap to 0x1234) are Overflow, and the
saturating form clamps. A mutant that truncates instead fails the test;
the old addr.rs does not provide the helper the test needs.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The editor stored every chunk through the whole pipeline with filter mask 0.
For LZF that did not shrink a chunk, h5py instead stores it raw with the
filter's mask bit set. A chunk the editor stored LZF-encoded at exactly the
raw size was then rewritten raw by libhdf5 at the same size; libhdf5 does not
touch the index entry when the size is unchanged, so the stale mask 0 stayed
and h5py (and h5dump) could no longer read the dataset.
clawhdf5_format::filters::compress_chunk_masked runs the pipeline as
H5Z_pipeline does: an optional filter (H5Z_FLAG_OPTIONAL) that fails is
skipped and its bit set, a mandatory one fails the write, and LZF/Blosc
output no smaller than the input counts as failure, as in the reference
filters (their output buffer is the input's size). Deflate, LZ4, Zstd,
bitshuffle and bzip2 never fail on size in libhdf5 and are kept as before.
Test: edit_interop optional_filters_that_fail_are_skipped — the reviewer's
repro at every libver: the editor stores the chunk exactly as h5py does
(mask 1, size 5; shuffle+LZF+fletcher32 mask 2), h5py r+ rewrites and
extends the datasets, and h5py, h5dump and our reader read every value.
Fails on the previous editor (mask 0; h5dump cannot read /u8).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Three `chunk_info.address as usize` casts behind the `parallel` feature
survived the conversion, because check-32bit-casts.sh linted only default
features plus plugin-filters. On a 32-bit target with rayon a chunk address
past 4 GiB still wrapped onto another part of the file. They go through
addr::to_usize now, and the lane index (h % n, always < n) through
saturating_usize.
The script now lints no default features, default features, and every
optional feature but szip (wasm32; the set with zstd, which does not build
for wasm32, on the host, where the lint reports the same casts). With the
old parallel_read.rs/lane_partition.rs it fails listing the four casts; the
old script passed them. CHANGELOG and the design note give the exact count
(119) and what is not covered.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Only leaves and the header were checked. Harmless while every lookup read
the whole tree, but the indexed lookup prunes children by the keys stored
in internal nodes, so one corrupted byte there could route a name to the
wrong child and report it missing with no error. A BTIN whose lookup3
checksum does not match is now ChecksumMismatch on every read (lookups and
full traversals), as in libhdf5.
Test: one byte of the root BTIN of the 35 001-link h5py group's name index
changed -> lookups, paths and listings through File, MmapFile and LazyFile
all fail with ChecksumMismatch, and h5py refuses both. Before, lookups
returned Ok.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
read_exact_at and read_upto (and so Window::read) ask as_contiguous()
first and slice the file directly when the backend holds it in memory:
one dynamic call per structure read instead of two or three (len,
read_at, then len again for errors). Same results and errors.
Provisional (busy machine, not for docs): a listing that walks 400
symbol-table groups through the facade went from about 18% to about 14%
slower than before the Storage conversion; the extra cost is a few tens
of nanoseconds per structure read, which the facade's per-lookup
re-listing (range-reads.md M0) multiplies.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
known-issues records what the editor refuses, that freed space is never
reused (append-workload file sizes measured 2026-09-26 on tank with the
ignored measure_append_waste test; sizes are deterministic), and that there
is no journal.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
New clawhdf5::FileEditor opens an HDF5 file (h5py-written at any libver,
HDF5 2.0 format included, or clawhdf5-written) under an exclusive flock
and changes only what an edit touches:
- write_selection/write_all/write_values: compact, contiguous (also
late-allocated) and chunked datasets, any selection. Chunks are decoded,
updated and re-encoded; a filtered chunk that no longer fits moves to the
end of the file unless it is the file's last structure, which grows in
place. New chunks go into v1 B-tree, Extensible Array (paged data blocks
included), Fixed Array and single-chunk indexes, created on first use.
- resize: grow chunked datasets up to maxshape.
- set_attr: add/replace compact attributes, in a NIL slot or a new
continuation chunk.
Each edit is planned in an in-memory image and refused whole
(Error::Unsupported) when any part is unsupported (v2 B-tree / implicit
new chunks, shrinking, vlen/reference data, dense or order-tracked
attributes, cache images, paged/persistent free space). Commit writes and
syncs new space before patching existing bytes. Layout v5 (HDF5 2.0)
array indexes use 8-byte filtered chunk sizes, as libhdf5 does.
Error gains Unsupported/InvalidArgument/Locked and is #[non_exhaustive];
the Python bindings map them. build_attr_message is public.
Tests (h5py, h5dump, h5rs check --data after every round; h5py r+
afterwards): appends crossing EA super/data blocks and B-tree splits, the
same B-tree node counts and EA statistics as libhdf5 for the same writes
(in order, reversed and shuffled; paged blocks), every layout and chunk
index overwritten under random selections, attributes to continuation
chunks, random operations against a model, refused edits leave the file
byte-identical, locking.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Every `u64 as usize` cast in clawhdf5-format (115 on wasm32) now goes
through addr::to_usize for values read from the file — addresses, lengths,
counts, dimensions: FormatError::Overflow where the value does not fit
instead of wrapping onto another part of the file on a 32-bit target — or
addr::saturating_usize for counts bounded by something in memory (codec
progress counters, writer sizes), which fail a bounds check or allocation
rather than wrap. A chunk whose offset does not fit lies outside the
dataset and is skipped; partial reads treat such an offset as out of the
buffers. On 64-bit targets nothing changes.
scripts/check-32bit-casts.sh (run by ci-test.sh) lints the wasm32 build
with clippy's cast_possible_truncation and fails on any u64 -> usize
finding; before this commit it listed 115.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Finding one link or attribute by name read every entry: Group::dataset and
Group::group (File, MmapFile, LazyFile) listed the whole group per call, and
path resolution scanned each group's links. Opening every child of a
35 001-link group by name decoded ~1.2e9 links.
Now a dense group's v2 B-tree name index (type 5, lookup3 hash of the
name) is descended to the records with the name's hash
(btree_v2::find_btree_v2_records reads only the nodes whose key interval
overlaps), and only those links are read and compared; all hash-equal
records are compared, so libhdf5's tie order does not matter. Dense
attributes the same through their type 8 index
(attribute::find_attribute_in_file, facade attr(name)); huge heap objects
through their ID-ordered index. group_v2::resolve_child returns what the
listing has under a name (soft links followed, dangling/external ones not
found). Group::entries and File::group_at hand out a listing's addresses.
The lookup-stats feature counts heap objects read. Tests: one lookup in
an h5py-written 35 001-link group with colliding hashes reads at most two
links (before: 35 001, failing), attribute lookups likewise (before: 3 000,
failing), every child opens through all three readers and matches h5py,
every link kind resolves as h5py resolves it in dense and compact groups,
300 huge attributes are found, and a range search matches a full scan at
every tree depth.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Lists the parsers now reading through Storage, what still needs the
whole file (v2 B-tree-indexed structures: a clean error; raw data: M2),
the equivalence harness, and the evidence that nothing changed: existing
tests, a byte-identical conformance results.json and per-file probe
output against f2ff2c4, and identical slice-API transcripts over 748
files between the two builds.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
With Storage in scope, `data.len()` on a `&&[u8]` resolves to
Storage::len (already a u64), so `as u64` was a no-op cast; name the
slice method explicitly.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Making dataset_fill_value_in generic over Storage broke callers that pass
an array (`include_bytes!`): a generic parameter does not unsize-coerce
`&[u8; N]`. It takes &[u8] again, as before this branch, and wraps the
new dataset_fill_value_from_storage(&dyn Storage, ..).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
tests/storage_equivalence.rs walks real files and runs every metadata
parser converted to Storage twice per object — over the file as a slice
and over a read_at-only CountingStorage — and requires identical results,
values and errors alike; the only allowed difference is the clean
ContiguousStorageRequired from the structures still indexed by a v2
B-tree (dense attributes, a SOHM B-tree index), which is counted. It
covers superblock and extension, cache image, SOHM table/list/B-tree,
object headers, attributes, fill values, shared messages, symbol-table
groups (local heap, B-tree, nodes, names), fractal heaps and their
objects, VDS mappings, and fixed/extensible array chunk indexes.
Inputs: every fixture; files h5py writes for what the fixtures lack
(extensible arrays with super blocks and paged data blocks, paged fixed
arrays, a 400-group v1 file with a user block, a 300-link dense group,
dense, shared and committed-type attributes, SOHM list and B-tree
indexes; honours CLAWHDF5_PYTHON / CLAWHDF5_REQUIRE_INTEROP); and, with
CLAWHDF5_STORAGE_CORPUS set, a corpus such as conformance/.cache/corpus.
Milestones M2/M3 add their parsers to check_object.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
ZFP (32013) was the one plugin filter still listed as UnsupportedFilter.
Conformance on tank, `conformance/run.sh --no-fetch` (2026-09-26): 600 of
697 files ok (baseline 599); h5ex_d_zfp.h5 is newly ok.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
DataLayout::resolve_vds_mappings_in reads the global heap collection
holding a virtual dataset's mappings through GlobalHeapCollection::
parse_in; resolve_vds_mappings is a wrapper. The rest of the data layout
module parses message payloads and does not read the file.
New test: the virtual dataset of vds_same_file.h5 resolves to the same
mappings through a read_at-only CountingStorage, in two reads.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
dataset_fill_value_in now takes the file as any Storage (generic, so
every &[u8] caller compiles unchanged) and follows a shared fill value
message through message_data_with_sohm_in. The raw-data helpers in the
module (read_full_with_fill, apply_to_unallocated_chunks) walk chunk
indexes and stay on &[u8] until milestone M2.
New test: the four datasets of shared_fill_value.h5, two with their
fill value in the SOHM heap, resolve identically through a read_at-only
CountingStorage.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
AttributeMessage::parse_in_storage, extract_attributes_full_in and
extract_attributes_tolerant_in take the file as &dyn Storage: shared
datatypes, dataspaces and attributes are resolved through the Storage
shared-message path, and dense attributes' fractal heap through
FractalHeapHeader::parse_in / read_managed_object_in. The dense-storage
name index is a v2 B-tree, which is not read over Storage yet: over a
backend without the whole file in memory it is a clean
ContiguousStorageRequired error, never a partial list. The &[u8]
functions are wrappers.
New test: every object in five h5py-written fixtures (compact, shared
and dense attributes) reads identically through a slice as Storage, and
through a read_at-only CountingStorage except the dense ones, which give
the clean error.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Every shared-message entry point has an *_in(&dyn Storage, ..) core:
message_data(_with_sohm), resolve_shared_message(_with_sohm),
resolve_sohm_message, load_sohm_table, and the SMTB/SMLI parsers, which
read the signature and then all entries in one bounded read (the list's
open-ended &file_data[pos..] slice is gone). Object headers and the SOHM
fractal heap are read through their Storage parsers; a SOHM B-tree index
still needs the v2 B-tree over a slice, so over a backend without one it
is a clean ContiguousStorageRequired error. New signature::find_signature_in
probes the candidate offsets with 8-byte reads. The &[u8] functions are
wrappers.
New tests: SOHM tables and lists with 4- and 8-byte offsets, at two
offsets, cut at every length and with a bad signature, parse identically
through a read_at-only CountingStorage in at most two reads; the
signature search matches the slice search.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
ExtensibleArrayHeader::parse_in reads the header in one read, and
read_extensible_array_chunks_in reads each index block, super block and
data block as a prefix read and then one window of the whole structure
(paged data blocks included); checksums and elements are checked in the
window with bounds errors reported as the whole-file checks did. The
&[u8] functions are wrappers. New test: an array with inline elements
and a data block, cut at every length and damaged in each structure,
reads identically through a read_at-only CountingStorage.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
FixedArrayHeader::parse_in reads the header in one read, and
read_fixed_array_chunks_in reads the data block's prefix and then the
whole block, paged or not, as one window; checksums and elements are
checked in it, with bounds errors reported as the whole-file checks did
(also in builds without the checksum feature, where the per-element
checks are the only ones). The open-ended &file_data[offset..] slices are
gone. The &[u8] functions are wrappers. New test: non-paged and paged,
filtered and unfiltered arrays, cut through the data block and with
damaged bytes, read identically through a read_at-only CountingStorage.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The facade forwards `zfp` and adds it to `plugin-filters`.
tests/zfp_interop.rs: h5py + hdf5plugin (H5Z-ZFP 1.1.1, zfp 1.0.1)
write 2205 datasets over 16 modes (rate, precision, accuracy,
reversible, expert settings at their edges) x int32/int64/float/double
x 1-4-D shapes with partial edge chunks, partial blocks and unit chunk
dimensions x smooth/noisy/wide-range/zero/inf-NaN data; clawhdf5 must
read each byte for byte as h5py does (read back after closing the file:
h5py returns a chunk still in libhdf5's cache without decoding it). A
second test swaps a file's header words to what a big-endian writer
stores and checks the byte-swapped values match h5py's.
The left-out-filter test now expects ZFP only in builds without it; CI
lints `zfp` alone and runs the new test with the plugin-filter step.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A port of the zfp 1.0.1 decoder and of H5Z-ZFP 1.1.1's decompression
path behind a new `zfp` feature (in `plugin-filters`): every mode
(fixed rate, precision, accuracy, reversible, expert), int32, int64,
float and double, 1-4 dimensional fields with partial blocks, and
headers written big-endian (values byte-swapped as H5Z-ZFP does).
H5Z-ZFP keeps the zfp header in cd_values (version word, then the
magic/metadata/mode bit stream); each chunk is the bare stream. The
decoder reproduces libzfp bit for bit: integer arithmetic wraps as
libzfp's, block exponents scale by exact powers of two, and the mode
goes through zfp_stream_mode as H5Z-ZFP hands it to zfp. A stream that
ends early is an error (libzfp reads past its buffer), as is a field
whose size is not the chunk's; the output is allocated only once the
stream holds a bit per block.
Tests: header/mode unit tests, and a counting-allocator fuzz of random
headers (expert parameters at their edges) and streams.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
FractalHeapHeader::parse_in reads the header as one window (a second,
longer one when it holds an I/O filter pipeline); read_managed_object_in
reads direct blocks, indirect blocks (one window up to the last child
entry) and huge objects with bounded reads. The &[u8] methods are
wrappers. A huge object indexed by the huge-object v2 B-tree, which is
not converted yet, is a clean ContiguousStorageRequired error on a
backend without the whole file in memory (after the "no index" check,
so the error order is unchanged).
storage::Window (crate-internal) reads a window of a structure and
reports bounds failures exactly as the whole-file ensure_len did, and a
short read inside the file is now a Storage error rather than an EOF.
New tests: headers (with and without a filter pipeline) cut at every
length, and managed objects in a direct root and through an indirect
root, huge objects with direct IDs and tiny objects, give identical
results through a read_at-only CountingStorage.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
BTreeV1Node::parse_in reads a node's header and then its keys and
children, two bounded reads; collect_symbol_table_nodes_in walks the tree
over any Storage. The &[u8] functions are wrappers. New test: nodes with
siblings and 4- and 8-byte offsets cut at every length, and a two-level
tree with truncated leaves, give identical results through a read_at-only
CountingStorage (six reads for the three nodes).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
SymbolTableNode::parse_in reads the node's 8-byte header, checks the
entries against the end of the file as before, and reads them in one
bounded read. parse is a wrapper. New test: nodes with 4- and 8-byte
offsets, valid and with a bad version, at two offsets and cut at every
length, parse identically through a read_at-only CountingStorage.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
GlobalHeapCollection::parse_in / parse_index_in read the header, check
the collection against the end of the file, and read the collection in
one bounded read; objects are indexed in it with file offsets, as before.
The &[u8] functions are wrappers. New test: collections with 4- and
8-byte lengths, one whose size runs past the file and one whose object
overruns it, at two offsets and cut at every length, give identical
results through a read_at-only CountingStorage.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
LocalHeap::parse_in reads the header in one bounded read,
validate_free_list_in reads each free block's two lengths, and
read_string_in reads from the string to the end of the data segment
once and looks for the terminator there. The &[u8] methods are wrappers.
New test: a heap without free space, with a valid free block and with a
free block overrunning the segment, cut at every length, parse, validate
and read strings identically through a read_at-only CountingStorage.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
read_superblock_extension_in, cache_image_state_in, CacheImage::decode_in
and CacheImage::block_in take &dyn Storage (whose length is the end of
file); the image block is one bounded read. The &[u8] functions are
wrappers; applying an image in place still needs the bytes in memory.
New test: extension messages, a cache image and a corrupt one decode to
the same results through a read_at-only CountingStorage.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
ObjectHeader::parse_in(&dyn Storage, u64, ..) reads the signature, the
prefix (a window of at most 34 bytes for version 2) and then each chunk,
continuation chunks included, as one bounded read; the message loops run
unchanged on the chunk with chunk-relative positions. parse keeps its
&[u8] signature as a wrapper. Bounds errors are reported as before, with
absolute positions and the file's length. New test: headers of both
versions, with times, phase-change values, creation order and a
continuation chunk, and every truncation of each, parse identically
through a read_at-only CountingStorage.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Superblock::parse_in(&dyn Storage, offset) reads one bounded window of
128 bytes (the largest superblock is 100) and runs the existing version
parsers on it; parse and refresh_eof keep their &[u8] signatures as
wrappers. No behaviour change: on a file longer than the window no bounds
check can fail, and on a shorter one the window is the whole file. New
test: every version and truncation parses to the same result through a
read_at-only CountingStorage, in one read.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Range-read milestone M1, first step (docs/design/range-reads.md §3(a)):
a synchronous, no_std read interface with u64 offsets, read_at returning
Cow<[u8]>, read_ranges, len and an as_contiguous fast path. Implemented
for [u8], Vec<u8>, &T, Box<T> and Arc<T>; slices serve borrowed bytes.
read_exact_at reproduces the parsers' UnexpectedEof bounds error exactly,
so converted modules keep their error values.
FormatError gains Storage(String) and ContiguousStorageRequired; it and
the facade Error are now #[non_exhaustive] (breaking for exhaustive
matches, noted in the changelog; the Python bindings' match gets a
wildcard arm).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
addr::to_usize turns a 64-bit file address or length into a slice index,
failing with FormatError::Overflow where it does not fit usize (32-bit
targets such as wasm32) instead of truncating like an `as usize` cast.
Callers are converted in the following commits.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
