libhdf5 fails to read a VL element whose global heap address is
undefined (all 0xff), even at length 0 ("addr undefined"); we returned
"" (or an empty sequence) in every reader. Checked with h5py first:
libhdf5 writes a null element with address 0, which still reads as
empty, and h5py writes "" as a zero-size heap object at a real address,
so no file they write relies on the old behaviour. read_vl_bytes now
treats address 0 as null whatever the length, as VlResolver does.
Tests, each failing before: vl_data unit test (8- and 4-byte offsets,
lengths 0 and 1); clawhdf5 vl_data_interop
a_vl_element_at_the_undefined_heap_address_fails_like_h5py (also checks
where h5py writes ""); h5rs dump --json and check --data on the patched
`undef` dataset; clawhdf5-wasm vl_strings.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The READMEs said ds[...] reads only the selected elements, and the
facade's read_selection docs that only intersecting chunks are
decompressed. The bounding-box path runs only when the box covers at
most half the dataset; larger boxes (any strided slice across the
dataset), compact, virtual and unwritten datasets and chunked ones with
a non-default fill value decode the whole dataset. The READMEs, the
facade and format docs, the bindings' docstrings and known-issues now
say so, and how index lists are read.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
h5rs (dump, ls, diff, check --data) kept its own lenient VL decoder:
a heap object longer than its element was cut to the element's length
(libhdf5 and h5py refuse it), a null string printed "" where h5dump
prints NULL, the stored element size was trusted, and every heap
collection was kept as an owned copy for the whole run. It now resolves
each element with VlResolver::element / string_element (new: one element
in place, borrowing from the file), and refuses a VL type whose stored
element size is not 4 + offset size + 4, as File does. H5::heap_object
and its cache are gone. h5diff compares a null VL string equal to an
empty one; so does h5rs diff.
clawhdf5-wasm already resolved VL strings with read_vl_strings; it now
uses VlResolver and checks the stored element size before reading, as
File::read_string does.
Tests (h5py writes the files, patched for "a\0b", a null element and
mis-sized heap objects, with 8- and 4-byte offsets):
- h5rs_interop dump_prints_vl_data_like_h5dump: byte-identical to h5dump;
- dump_json_vl_values_match_h5py: h5py's values, errors where h5py fails;
- check_data_flags_mis_sized_vl_heap_objects;
- clawhdf5-wasm tests/vl_strings.rs: wasm, File and h5py agree.
All four fail before. check --data over the 150 cve_hdf5 CVE and fuzzer
files now passes 15 (h5dump rejects 8 of them), was 16 and 9: the
stored-size check flags cve-2024-32608. h5rs-check-ok-files.sh --data:
0 of 422 flagged; h5rs-fuzz.sh: clean on 180 files.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
test_threads_read_the_same_file passed with the GIL held. The new
test_reads_release_the_gil measures the longest stall of a spinning
Python thread while another reads: with py.detach removed from the read
it stalled 0.062 s of a 0.064 s read and failed; with it, about 3 ms.
test_errors_match_h5py now compares the result whenever h5py reads the
key, instead of only checking that we raise when h5py raises, over a
longer key list.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The changelog, known issues and README said a group holds up to 65 535
links while a group of about 17 000 was already unreadable. Record the
fixes (child indirect blocks, the next-block offset, the index leaf cap,
refusing oversized dense messages, hard-link memoisation, dataset
attribute overwrite) and the limits that remain true: 65 535 links or
dense attributes per object, and 65 515 bytes per dense message.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
ds[np.array(1)] went down the index-list path, where tolist() returns a
scalar and extracting a list of indices raised a confusing TypeError.
h5py treats it as an integer index; so do we now. The h5py comparison
keys include 0-d arrays (signed and unsigned) on each axis; they failed
before.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
b0a1e4f fixed this for group and root attributes only. Setting a dataset
attribute twice still wrote two attribute messages with one name, and h5py
read back the first value: set_attr("a", 1) then set_attr("a", 2) read as
1, and list(attrs) was ["a", "a"]. DatasetBuilder::set_attr now replaces
the earlier value, compact or dense. Likewise, a hand-set attribute named
like a provenance attribute (_provenance_sha256, ...) is replaced by the
computed one instead of being written next to it and read first.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Every ds[...] and g[k] resolved the path from the root again, two or
three times per open, and resolving a name in a large group scans its
links: visiting a group was O(n^2). 4000 scalar datasets in one group
took 39 s (v1 group) and 131 s (dense) to list, read and re-read; now
0.3 s each. A Dataset keeps its object address, a Group (and the file's
root) its address and, after the first lookup, its link table.
New facade API File::dataset_at(address), tested in integration_tests.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A hard link's target may go through other hard links, and each was
resolved again every time a path went through it. With each link's
target naming the previous link twice (g/s{i} -> /g/s{i-1}/s{i-1}) the
work doubled per link: finish() took 46 s for 26 links in a debug build,
and 60 would never finish. Resolved links are now remembered, so the work
is linear in the links, and a hard link met again while it is being
resolved is reported as a cycle by name. The depth limit (64) still bounds
the recursion through links not yet resolved.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The link and attribute name indexes are one v2 B-tree leaf, sized to the
next power of two. libhdf5 takes a leaf's capacity from that node size,
but a leaf's record count is a 2-byte field. From about 47 700 links the
node had room for more than 65 535 records, so adding a link in h5py
overflowed the count: a group of 65 535 links crashed h5py, or could no
longer be listed ("unknown link class"). The node is now capped at a full
leaf of 65 535 records, so libhdf5 splits it instead.
Dense attributes now go through the same index builder. Their record
count was written modulo 65 536, without error; more than 65 535
attributes on one object are now refused, like links.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A message in dense storage is a fractal heap object, and an object must
fit one direct block: 65 515 bytes here, since the writer has no
huge-object path. A bigger one (a soft link with a 80 000-byte target in
a group of more than 8 links) was written without error, cut off at the
end of its block, and libhdf5 could not list the group ("object overruns
end of direct block"). finish() now fails with an error that names the
limit, for links and for dense attributes; a 65 001-byte soft link target
still works and h5py reads it back. The heap packer also skips a child
indirect block whose blocks are all too small for the next object instead
of walking it.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Each run of consecutive indices was its own uncached hyperslab read, so
a list over a compressed chunked dataset decoded the same chunk once per
run (d[range(0, 200000, 40)] over 20 gzip chunks: 8 s, h5py 0.014 s).
Plan::reads now groups the indices — a group ends only where a whole
chunk holds no selected index, or, unchunked, at a gap over 64 KiB — and
the selected rows are gathered from each group's block in Rust. Now
3.8 ms (h5py 4.1 ms, release, tank). The new test (1-D, 2-D and
contiguous, compared with h5py, 2 s bound) took 5.8 s before.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Dense link and attribute storage keeps its messages in a fractal heap. Its
root indirect block holds direct blocks up to 64 KiB, 512 KiB in all; rows
past that are child indirect blocks. The writer kept adding rows of direct
blocks instead, and libhdf5 and h5rs read them as indirect blocks: a group
with 20 000 links of 20-byte names was written without error and could not
be listed ("incorrect metadata checksum"), and 150 dense attributes of up
to 56 KB could not be opened. The heap writer now follows the doubling
table: rows past the direct ones hold child indirect blocks, each with its
own rows, nested as deep as the heap needs.
Two more heap bugs are fixed on the way. An object bigger than the next
block's free space was written into it anyway and cut off; the block is
now left unallocated and the object goes in the first block big enough, as
libhdf5 skips blocks. And the header's next-block offset was 0, so libhdf5
adding a link to such a group overwrote the heap's first block ("bad
version number for message"); it is now the offset after the last block.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
VlResolver kept an owned copy of every object of every heap collection
it parsed, for the whole read. Collections nested inside each other's
object data, 32 bytes apart with each element pointing at a different
one, made retained memory O(elements x file size): 1.58 GB for a 744 KB
file (read_vl_strings did the same before VlResolver). Chaining every
collection's objects into one shared run of tiny objects made parse
time O(elements x objects) as well. libhdf5 refuses these files.
- The cache records where each object lies (GlobalHeapCollection::
parse_index, new) instead of copying it, and is dropped past a 32 MiB
budget.
- A collection overlapping one already read is an error: libhdf5 gives
every collection its own block, so only a crafted file has them.
- parse and parse_index refuse a collection that runs past the end of
the file and an object that runs past the end of its collection.
tests/vl_heap_bounds.rs measures peak heap use with a counting
allocator: 129 MB and 350 MB live before on its two crafted files (64 KB
and 176 KB), 97 KB and 0.9 MB now. Conformance unchanged at 575 of 697.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
np.concatenate copies structured dtypes field by field into np.empty, so
the padding of ds[[0, 3, 6]] held process memory. The runs' bytes are
joined in Rust, whole elements at a time, before anything becomes numpy:
the padding is the file's bytes (h5py's) and the result is still a view
of the Rust buffer. The h5py comparisons now compare every byte of
structured values; the new test failed on the padding before.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
PanicException derives from BaseException, so `except Exception` let a
library bug through. Every call from the bindings into the library now
runs under catch_unwind and a panic becomes InternalError (RuntimeError)
naming the object. Tests: a hidden hook panics inside the guard; and the
v4 chunk indexes are compared with h5py from Python — with the library
fix reverted, ds[0:30] of the implicit-index dataset now raises
InternalError instead of aborting the test run.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
read_raw_data_selection's chunked fallback (taken when partial_read
declines, e.g. a bounding box over half the dataset) handed the layout's
chunk dimensions, element-size dimension included, to
generate_implicit_chunks, which indexed past the dataset rank. It then
decoded the whole dataset regardless, so the enumeration is gone: the
arm decodes and extracts for every chunk index.
The new test reads small and large hyperslabs of all five v4 indexes
written by h5py and compares with h5py's values; it panicked before.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The review measured the default pool unchanged (about 2900 MB/s at 16
threads before and after) and still short of 16 h5py processes; small
pools still make outside readers wait. Say so instead of marking the
scaling issue fixed.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Full reads of chunked datasets handed their chunks to rayon. With a
one-thread pool (concurrent_read --decode-threads 1, RAYON_NUM_THREADS=1)
every thread reading through a File queued behind that single worker, so
16 readers decoded on one core: per-thread CPU time showed one thread
doing all the decoding and the readers almost none, and full reads
stopped at about 2x one thread. The cached full-read path and the
uncached reader behind verify_provenance now decode inline when the pool
cannot parallelise (parallel_read::pool_can_parallelise).
The File's chunk cache was the suspect but not the cause: datasets over
its budget were already read without inserting, and skipping its lookups
gained only a few percent at 16 threads.
The regression test keeps a one-thread global pool's worker busy and
requires a full read and verify_provenance to finish anyway; before the
fix both waited for the worker (timed out).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Setting a group or root attribute twice wrote two attribute messages with
the same name, and h5py read back the first value: set_attr("w", 1) then
set_attr("w", "two") read as 1. The later value now replaces the earlier
one, as `attrs[name] = v` does in h5py, including when a group is merged
from two builders.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The writer marked every link name ASCII, so a name such as "größe" was
stored as UTF-8 bytes under the ASCII character set (h5py reports cset 0
for it). Names that are not plain ASCII now carry the UTF-8 flag, as h5py
writes them; ASCII names are unchanged.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
FileWriter wrote the root group plus one level of groups, and refused
path-like names. The writer now flattens its builders into a group tree
(writer_tree.rs) before layout:
- A name may be a path ("a/b/x", "/a/b/x" at the root); missing
intermediate groups are created as h5py does, and GroupBuilder gains
create_group/add_group so builders nest to any depth. A group added at a
path that already holds a group is merged into it (require_group);
any other repeated name, an empty or "." component, or an absolute path
below the root is an error.
- add_soft_link, add_hard_link and add_external_link on FileWriter,
FileBuilder and GroupBuilder. Hard-link targets are resolved to objects
at finish (through other hard links; a missing target, a soft link on the
way or a cycle of paths is an error). Objects with several hard links get
an Object Reference Count message so libhdf5 can delete one link without
freeing the object.
- track_order(true) per group, or as the file default, tracks and indexes
link creation order: Link Info flags and max order, the order in each
Link message, and a type-6 creation-order B-tree for dense groups.
- A group's link index is one B-tree leaf; more than 65535 links is an
error.
Groups are laid out depth-first from the root, datasets group by group,
and untracked groups keep writing datasets, then groups, then other links:
files with one level of groups are byte-identical to before.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
clawhdf5-format now depends on libc on Linux (huge-page advice for read
buffers); the probe's committed lockfile picks that up.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
VL-string datasets (h5py's default str dtype) failed read_string with
"type mismatch: expected String, got VariableLength". read_string now
reads fixed- and variable-length strings, with h5py's values (a string
ends at a NUL, a null element is ""). New:
- Dataset::read_string_bytes: each VL string's exact bytes;
- Dataset::read_string_selection: hyperslabs/points of either kind;
- Dataset::read_vlen::<T>() and read_vlen_selection::<T>(): VL sequences
of numbers as Vec<Vec<T>>, T in f64/f32/i64/i32/u64, converted like the
other typed readers;
- File::decode_strings / decode_string_bytes / decode_vlen: VL values in
compound fields and AttrValue::Raw attributes;
- MmapDataset and LazyDataset: read_string for VL strings,
read_string_bytes and read_vlen.
tests/vl_data_interop.rs checks every path against h5py with 8- and
4-byte offsets: scalar, 1-D and 2-D, ASCII and UTF-8, empty strings,
contiguous, compact, chunked with gzip and shuffle, unwritten and partly
written chunks, hyperslabs, compound members, attributes, a big-endian
base type, and a patched file with an embedded NUL and mis-sized heap
objects. NetCDF-4 string variables read too (netCDF4-python test).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
README gains a Python section (maturin develop into a venv, a reading
example that was run against an h5py-written file, the supported types
and keys, what writing covers). The crate README says the same in more
detail. QUICKSTART showed clawhdf5.open()/read_f64(), which never
existed; it now shows File(...)[...].
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A 256 x 256 hyperslab of a contiguous f32 dataset read at an eighth of
h5py's speed: partial_read copied the bounding box out of the file, the
extractor then walked it element by element (a recursive call and two
bounds checks per element) into a second buffer, and read_f32_selection
converted that into a third.
Selections of contiguous data are now copied straight from the file, one
memcpy per run of elements contiguous in the file (gather.rs: a block
along the last dimension, touching blocks as one range, whole rows
merged), with no zero-filled intermediate and no full copy for large
selections. The typed selection readers copy into their Vec<T> directly
when the dataset stores T natively (new data_read::read_selection_native
and sealed NativeElement trait, which the read_as_* fast paths now share;
read_as_u64 gains one) and convert as before otherwise. The general
extractor used by the chunked paths runs on the same run walker, keeping
its old handling of unvalidated selections.
Checked against h5py (contiguous_read_interop.rs) for strided, blocked,
adjacent-block and whole-row hyperslabs, points and empty selections of
every 1-8-byte type in both byte orders, ranks 1-4.
Also keeps the huge-page threshold constant out of no_std builds, where
it was unused.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
ci-test.sh gains a step that lints clawhdf5-py, builds its wheel with
maturin, unpacks it under target/ (the interpreter's environment is not
touched) and runs the pytest suite, which compares reads with h5py. It
skips without maturin/pytest, and fails instead under
CLAWHDF5_REQUIRE_INTEROP=1. The CI interop venv installs maturin and
pytest, so CI runs it.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Checked with h5py on a patched file:
- a VL string with an embedded NUL reads up to the NUL (libhdf5 converts
VL strings to C strings); read_vl_strings returned "a\0b";
- an element whose global heap object is not length x base size bytes is
an error ("Expected global heap object size does not match"); we
returned the object cut to the length;
- a heap address of 0 is a null element whatever its length.
vl_data::VlResolver does this, caching each parsed heap collection:
read_vl_strings parsed the whole collection again for every element.
read_vl_strings and read_vl_bytes use it; check_element_size refuses a VL
type whose stored element size is not 4 + offset size + 4. The
conformance probe resolves VL values through VlResolver instead of its
own lenient copy (575 of 697, unchanged).
The new unit tests fail against the old read_vl_strings.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
ds[key] read the whole dataset and sliced it in numpy, and knew six
dtypes. Keys (ints, positive-step slices, Ellipsis, one increasing index
list, compound field names) now map onto hyperslab selections, and the
facade's read_selection bytes become the numpy buffer without a copy
(PyArray::from_vec viewed as the dtype). dtype mapping follows h5py for
all integer/IEEE float widths and byte orders, bool, enum, complex, fixed
and variable-length strings, vlen sequences, opaque, array types and
(nested, padded) compounds; anything it cannot describe exactly is a
TypeError. Attributes return what h5py returns; groups and files gain
the rest of the h5py mapping interface. Reads run under py.detach.
tests/test_read_vs_h5py.py compares >500 reads with h5py 3.16 on an
h5py-written file, checks errors match, that a damaged chunk outside the
selection is never touched, and 8 threads reading at once.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
In a file with sizeof_addr = 4, a VL string attribute came back as
AttrValue::Raw, a compound's VL member failed with
GlobalHeapObjectNotFound and VL datasets failed with a size mismatch.
Two bugs: Datatype::type_size() said 16 for every VL type, while the
element is 4 + offset size + 4 bytes (12 here); and the global heap was
parsed without the padding libhdf5 puts after its collection and object
headers (both round up to 8), so with 4-byte lengths every object was
looked up 4 bytes early. Datatype::VariableLength now carries the size
its datatype message stores, and writes it back.
Checked against h5py in tests/vl_offset4_interop.rs (fails with either
fix reverted). Conformance unchanged at 575 of 697; in cve-2024-32608 a
VL attribute whose datatype claims 524304-byte elements is now an error
(h5py cannot iterate those attributes at all).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A full read of a contiguous dataset is one memcpy from the mapped file,
yet ran at a quarter of h5py's speed on one thread: the fresh output Vec
took a page fault and a kernel page clear for every 4 KiB page written,
16384 per 64 MiB, costing several times the copy (the benchmark spent
6.2 s of 8 s in the kernel, 4.3M minor faults). numpy, so h5py, madvises
MADV_HUGEPAGE on allocations of 4 MiB or more; the typed readers' output,
the raw contiguous read and the chunk assembly buffer now do the same
(Linux only, libc as a Linux-only dependency; no-op otherwise).
New h5py comparison tests cover full and selection reads of contiguous
data for every 1-8-byte integer and float type, both byte orders, ranks
1-4, empty selections, and datasets past the 4 MiB threshold.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
pyproject.toml named the distribution rustyhdf5 while the extension
module is clawhdf5, and the package's tests imported rustyhdf5, so
pytest failed at collection. Distribution, module-name and tests now
agree; the module gains __version__. maturin develop + pytest: 28 pass.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
libhdf5 reads a group's Group Info message before it inserts a link, and
FileWriter wrote none, so h5py in "r+" mode could not add a link to any
group we wrote: "Unable to create link (message type not found)". Each
group header now carries a version 0 Group Info message with the default
link-phase thresholds, as libhdf5 writes for a new group.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
h5py threads stay flat (global lock); clawhdf5 hyperslab reads of deflate
data scale to 1244 MB/s at 16 threads (9.7x h5py threads, 0.89x h5py
processes). Two deficits recorded as open issues: full chunked reads stop
scaling at ~4 threads (chunk cache suspected), and contiguous reads are
4x (full) to 8x (hyperslab) slower than h5py single-threaded.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
With the header checks merged, the library refuses truncated files and
misaligned chunk-index keys itself, so check reported only "file cannot
be opened" for a truncated file. It now reports the truncation (stored
end of file vs file length) or the library's error, and the misaligned
chunk test accepts the library's refusal of the key.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Measured on tank on the 150 cvefiles/ and fuzzerfiles/ of cve_hdf5 (the
earlier text said 180): check --data passes 16 (was 28), and h5dump
1.14.6 rejects 9 of those (was 21). It still flags none of the 418
conformance files both readers read in full.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
parallel_integration.rs declared `_sequential` and used `sequential`
under the parallel feature, which nothing in CI enabled for the facade.
ci-test.sh now lints and tests the facade with parallel on.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The review read libhdf5's H5O__chunk_deserialize as accepting a v2
message that runs up to 4 bytes into the chunk's checksum, since it
bounds message bodies by the whole chunk buffer. It does not accept it:
the message loop stops at the checksum, and the checksum read that
follows starts past it and overruns the chunk ("ran off end of input
buffer while decoding"). h5py refuses such files whether the message
runs 1, 4 or 5 bytes in, and so does clawhdf5, with its own error text.
No code change; the test pins the agreement and a comment records why.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The truncated-file check and the end-of-file clamp reached File,
LazyFile and MmapFile but not clawhdf5-io's readers, which still opened
truncated files and read past the recorded end of file. NativeVol
(open, and read_dataset for from_bytes), AsyncHDF5File::from_bytes and
MpiVol's collective read now view the file through the new
vol::hdf5_view: from the superblock to Superblock::data_end, refusing a
file shorter than that.
MpiVol's read is compiled only with the mpi-io feature, which needs an
MPI installation; it was not built here. The edit there only swaps its
two-line superblock setup for hdf5_view.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A chunked layout records the element size as its last dimension, and
libhdf5 refuses a dataset whose datatype has another size
(H5D__chunk_set_sizes: "stored datatype size in chunk layout does not
match datatype description"). clawhdf5 ignored the recorded size and
read the chunks anyway, for v3 and v4 layouts. The check runs on every
chunked read (read_chunked_data*, read_raw_data_selection) and compares
against the stored size: a variable-length element is 4 + offset size
+ 4 bytes, not Datatype::type_size's 16.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
libhdf5 encodes a version-4 layout's chunk dimensions in (log2(max) +
8) / 8 bytes, and HDF5 2.0.0 (h5py 3.16) refuses any other width:
"stored chunk dimension encoding length does not match value calculated
from chunk dimensions". The writer rounded 3 bytes up to 4, so h5py
could not open a dataset we wrote with a chunk dimension from 65 536 to
16 777 215, for every chunk index (single chunk, fixed and extensible
array, v2 B-tree). The three encoders now share push_v4_chunk_dims,
which writes the exact width.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A version-4 layout stores every chunk dimension in the fewest bytes that
hold the largest one (H5D__chunk_set_sizes: (log2(dim) + 8) / 8), so a
chunk dimension of 65 536 to 16 777 215 takes 3 bytes. Only widths 1, 2,
4 and 8 were decoded; an h5py file with chunks=(70000,) and
libver='latest' failed with UnexpectedEof. Widths 1-8 are decoded now;
0 and more than 8 are refused with libhdf5's "encoded chunk dimension
size is too large", and a dimension past u32 is refused, not truncated.
The review asked for libhdf5's check that the stored width matches the
one computed from the dimensions. HDF5 2.0.0 (h5py 3.16) refuses any
mismatch, but HDFGroup/hdf5@e124c36 ("Allow reading of files with chunk
dimensions encoded using more bytes than necessary", 2026-06-05) relaxed
it to refusing only a width too small for the dimensions, which cannot
happen once the dimensions have been decoded from that width. Follow
current libhdf5: a wider-than-needed encoding is read. clawhdf5's own
writer produces such layouts (the next commit fixes that).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The short-decoding chunk (wrong data, pre-existing), the Blosc header
underflow (crash) and filter 32023 registration, each with its date and
what it changes; the conformance count is unchanged at 573 of 697.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The compound overlap check measured each earlier member with
Datatype::type_size, which is a fixed 16 for a variable-length type. On
disk a VL member takes 4 + offset size + 4 bytes, 12 in a file with
4-byte offsets, so a member right after one was refused as "member
overlaps with previous member" (and with the type, every attribute of
the object). libhdf5 measures members by their decoded, stored size
(times a v1 member's array dimensions); so does this now.
Reading VL values in such files is a separate, older gap, now recorded
in known-issues.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
With the pcodec feature, 32023 was a built-in entry (the legacy reader for
the pcodec chunks clawhdf5 <= 2.7.0 wrote under that ID), so
register_filter(32023, ...) was refused as "built in", although
UnsupportedFilter(32023) names Granular BitRound as not implemented and
the registry docs point to register_filter for such IDs.
That entry is now shared: it claims only chunks whose filter is named
"pcodec"; any other chunk with ID 32023 goes to the registered codec (or,
with none registered, gets UnsupportedFilter as before), and writing
32023 uses the registered codec. Every other built-in ID still refuses
registration.
Test: a_codec_can_be_registered_for_granular_bitround (registers, round-
trips chunks with no name and other names, still reads a legacy "pcodec"
chunk with the built-in reader, and after unregistering reads nothing).
It fails without the change ("filter 32023 ... is built in and cannot be
re-registered"). It and the existing legacy-pcodec test share a lock,
since the registry is process-wide.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
dump_matches_h5dump requires byte identity with the h5dump on PATH, and
CI's rust:latest (Debian 13.7) installs hdf5-tools 1.14.5, not the 1.14.6
the test was written against. Ran the whole clawhdf5-tools suite in
rust:latest with Debian's hdf5-tools and pip h5py 3.16.0 (HDF5 2.0.0), as
CI sets it up, with CLAWHDF5_REQUIRE_INTEROP=1: 18 of 18 pass, so the
comparison needs no loosening. The version is now printed in the CI log
so a future Debian update that changes the output is easy to spot.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A null-padded fixed string inside a compound or an array member printed
trimmed ("" for three NULs, "a" for "a\0b"), where h5dump prints every
byte ("\000\000\000", "a\000b"); only top-level strings were shown in
full. DATA blocks now render elements through one function that keeps
the padding at any depth.
The README now lists the remaining known differences from h5dump:
nested compounds print inline, and long double values are printed as
errors (exit 1) with the datatype as an H5T_FLOAT block.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The README said check skips only "global heap collections other than
those a value read touches", but read_dataset returns the raw heap IDs,
so no collection was ever read: a file whose global heap collection
claims a 4 GiB object passed `check --data` with no problems, while
h5dump (and h5rs dump/diff) fail on it.
With --data, every variable-length element (strings and sequences, also
inside compounds, arrays and nested sequences) of every dataset and
attribute is followed into its collection. A collection that does not
parse, a missing heap object, or a sequence longer than its heap object
is a problem at the collection's address, once per object; the summary
counts the collections read.
Measured on tank, 2026-09-26: the 418 fully-read conformance ok files
still pass (scripts/h5rs-check-ok-files.sh --data, 0 flagged), and
`check --data` now flags 152 of the 180 CVE-corpus files (was 147); of
the 28 it passes, h5dump 1.14.6 rejects 21 (was 26 of 33).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
HDF5 stores every chunk at the full chunk size (edge chunks are padded
before filtering, and with "don't filter partial edge chunks" they are
stored raw at full size), so a filter pipeline that decodes to fewer bytes
means a corrupt chunk. Every chunk reader padded it with zeros and
returned it as data. libhdf5 returns the rest uninitialised, or fails when
the filter checks (Blosc with nbytes = 0).
New filters::decompress_chunk_exact decodes and then requires exactly the
chunk size, with the chunk's coordinates in the error
(ChunkedReadError "chunk at [16] decoded to 16 bytes, expected 32"). It
replaces decompress_chunk_masked at every chunk read path: the full read
(sequential and lane-partitioned), the cached read, the sweep read, the
planned-selection read, parallel_read's three decoders and partial_read's
box read. decompress_chunk_masked is unchanged (fractal-heap huge objects
already checked their own size). Blosc also rejects a frame declaring no
data where the chunk size is known.
Tests, each failing with the check disabled: filters and parallel_read
unit tests; h5py_short_decoded_chunk_is_an_error (gzip chunks rewritten
short with write_direct_chunk: 1-D, a 2-D edge chunk, and 40 chunks with
shuffle, read through File full/cached/selection reads, a selection that
avoids the chunk still reads, MmapFile and LazyFile, with and without the
parallel feature); plugin_filters_interop short_decoding_chunks_are_errors
(Blosc nbytes=0 and short, LZF and bzip2 short; the Blosc nbytes=0 case
read as 16 zeros before). The existing don't-filter-partial-edge-chunks
tests still pass. Conformance (tank, 2026-09-26): 573 of 697 ok, and no
file changed class, reader result or first issue against the pre-fix run.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The reader now refuses a compound with a repeated field name or no
fields and an enum member with an empty name, as libhdf5 does, but the
writer still wrote them: CompoundTypeBuilder and EnumTypeBuilder build
them without complaint, so clawhdf5 wrote files it could not read back.
They were never valid HDF5; h5py refuses them.
Datatype::check_encodable, which FileWriter::finish runs on every
dataset and attribute type, now parses the type's own encoding back and
refuses one the reader refuses, with the reader's reason. That keeps the
writer in step with every reader check, not only these three.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
-c meant "list at most N differences" in h5rs, but in h5diff -c is
--compare (a flag) and the count is -n/--count=N, so a script moved over
from h5diff behaved differently: `h5diff -r -c 2 A B` exits 2 (the 2 is
taken as a file name) while h5rs exited 1.
The count is now -n/--count, -c/--compare is accepted (h5rs always lists
objects that are not comparable), and the --count=N, --delta=D,
--relative=R forms are accepted; exit codes equal h5diff's on 7 cases.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
An OBJ that was a soft link was resolved and its target object compared,
so two files whose /g/s both point at /z differed when /z did: exit 1,
where h5diff (without --follow-symlinks) compares the links' target paths
and exits 0.
A soft link is now compared as a link wherever it is, OBJ included.
--follow-symlinks compares the objects soft links lead to instead, walks
into soft-linked groups, resolves relative targets against the link's
group, and treats two dangling links as the same; exit codes equal
h5diff's on 14 cases. External links are never followed (documented).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
libhdf5 refuses a chunk of 4 GiB or more only when a version-1 B-tree
indexes it (H5D__chunk_init: "chunk size must be < 4GB with v1 b-tree
index"). HDF5 2.0 writes larger chunks with layout version 5, and h5py
reads them; these were refused. chunk_geometry now takes the layout
version and applies the limit to layout version 3 and earlier only.
The interop test is ignored by default: h5py writes a 4 GiB chunk and
both libraries hold it in memory.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Audited LZF, bitshuffle, bzip2 and Blosc/BloscLZ for arithmetic on
header fields and unchecked slicing. The only live bug was the Blosc
frame-size underflow fixed in the previous commit; bzip2's output-growth
step now uses a saturating subtraction as well (the allocator may hand
back more capacity than asked for).
src/test_fuzz.rs (tests only) feeds each decoder random bytes, truncated
seeds and one-to-four-edit mutations of valid frames, biased towards
edge-case u32 values in size and offset fields, and asserts no panic and
no output over the limit (tests build with overflow checks and debug
assertions). Per decoder: LZF, bzip2, bitshuffle in all six mode/block
settings plus hostile cd_values, Blosc across four codecs, three shuffles,
stored frames and a hand-built BloscLZ frame, and BloscLZ streams alone.
With the previous commit's check removed, fuzzed_frames_never_panic panics
at the same subtraction. A 100x-iteration soak (different seed) found no
other panic.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
With a tolerance, integers were converted to f64 before comparing, so
int64/uint64 values above 2^53 that differ compared equal: -d 0 on 2^60
and 2^60 + 1 exited 0, where h5diff exits 1. Integer pairs are now
compared in i128 (the delta against floor(D), the relative quotient from
an exact difference), and the report prints the exact difference.
h5diff compares exactly when -p is below the f64 epsilon (2^60 and
2^60 + 1 differ at -p 1e-18, and nextafter(2, 0) and 2 at -p 1.5e-16);
h5rs now does the same.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The path walk skipped the second hard link to an object, so a file that
shares one dataset between /x and /y differed from a file holding two
identical copies: "</y> exists only in <B>", exit 1, where h5diff exits 0.
For a hard-linked group every member was reported the same way.
diff now enumerates every path below the start object (a hard link back
to an ancestor is recorded but not descended into), so each name is
compared. A group whose links cannot be read is now an error instead of
an empty group.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A hostile chunk whose header gave a compressed size below 16 bytes, not
stored raw, made the block-table check subtract past zero: a panic in any
build with overflow checks (cargo test, maturin develop, debug CLI). The
frame size is now checked against the header size, and the stream-length
read no longer adds to an untrusted offset.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
CHANGELOG (Correctness): the new header, datatype, chunk and truncation
checks, what is left out on purpose (checks HDF5 2.0 lacks; the two
v2.7.0 writer quirks), the conformance numbers and the new FormatError
variants. known-issues: the "Header checks" audit gap is fixed, with the
one CVE object and two CVE files libhdf5 still refuses and we read.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
h5py writes a dataset (libver earliest, so version-1 object headers) and
the script damages one field of a copy: a layout message flagged
shareable, a message size that is not a multiple of 8, a compound field
that repeats an earlier name or overlaps it, an empty enum member name, a
float exponent overlapping the mantissa. h5py refuses every damaged copy,
and clawhdf5 must refuse exactly those and read the valid files. All of
them but the unaligned one (then an UnexpectedEof) read before this
branch. The helper now reads any datatype
(File::read_multi) rather than only integers.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Two of the datatype checks added on this branch refused files clawhdf5
itself wrote up to v2.7.0: it put the sign bit of every float at
position 63 (so every f32 it wrote failed "sign bit position out of
bounds", including every agent store's embeddings), and wrote an
empty-string attribute with a size-0 string type ("invalid datatype
size", failing every attribute of the object). libhdf5 refuses both, but
neither decodes to wrong values (an IEEE float's sign position is not
used; a size-0 string is empty), so this reader keeps accepting them.
New fixtures written by clawhdf5 v2.7.0 (FileBuilder with every datatype,
layout and attribute kind it could write, and a FileWriter paged file)
and legacy_writer_files.rs, which reads every object of them. The agent's
v2.5.0 store fixture (float16_store) passes again.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Measured on the CVE corpus: check --data passes 33 of 180 files, and
h5dump 1.14.6 rejects 26 of those. Recorded under the open "Header checks"
gap and in the crate README.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The superblock records where the file's data ends. libhdf5 refuses to
open a file shorter than that ("truncated file", H5F__super_read) and
fails any read past it ("addr overflow" / "address plus size exceeds
file eoa"). clawhdf5 read whatever was left of a truncated file, and read
bytes after the recorded end as if they belonged to the file.
New Superblock::data_end: FormatError::TruncatedFile for a file shorter
than its recorded end, otherwise where the HDF5 data ends. As in libhdf5,
the recorded end moves with the superblock when its recorded base address
is not where it is (a user block added afterwards; cve-2021-36977, which
h5py reads, depends on it), and the check is skipped for a v3 superblock
still being written in SWMR mode. File, LazyFile, MmapFile and the
conformance probe refuse a truncated file and parse only up to the end.
Files clawhdf5 writes record their true length, and the v2.5.0 agent-store
fixture and files written by v2.7.0 (plain, paged, user-block free) pass
the check.
Interop test: h5py writes a file; a copy missing its last 8 bytes must be
refused by both, a copy with bytes appended and one moved behind a new
512-byte user block must read in both.
Conformance (cached corpus, tank): 570 -> 571 ok
(h5clear_fsm_persist_less.h5, whose data past the recorded end was being
read); ten files h5py refuses as truncated (cve-2018-13874,
cve-2018-13876, the family/multi/subfiling members, h5clear_fsm_persist_
greater/user_greater) are now refused at open instead of read.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The clawhdf5-tools interop tests compare h5rs with h5ls, h5stat, h5dump and
h5diff, and CLAWHDF5_REQUIRE_INTEROP=1 turns a missing tool into a failure.
Also hold clawhdf5-tools to the no-C-in-the-default-build check.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
scripts/h5rs-fuzz.sh runs every h5rs subcommand over every file of a corpus
(default: the HDF Group's CVE reproducers), optionally with byte-flipped
copies (MUTATE=N), under a timeout and a memory limit, with a debug build so
integer overflow panics instead of wrapping; any exit status above 2 (a
caught panic, a timeout, a signal) fails it. It found size*8 overflows in
the datatype names on cve-2021-46244.h5, cve-2024-29161.h5 and unknown-1.h5
(fixed in the crate before it landed).
scripts/h5rs-check-ok-files.sh runs check (--data) over the conformance
files that clawhdf5 and h5py both read in full; none may be flagged.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
New workspace crate clawhdf5-tools with one binary, h5rs, built only on the
clawhdf5 facade and clawhdf5-format (no libhdf5, no C):
- ls [-r] [-v] FILE[/path]: h5ls's listing (same text in its first two
columns) plus the datatype; -v adds address, link count, layout and chunk
index, chunk size, storage, filters, datatype and attributes.
- dump [--json] [-A] [-p] [-d PATH] FILE: h5dump DDL (byte-identical to
h5dump 1.14.6 on the test files) or hdf5-json.
- stat FILE: h5stat's object/link/rank/layout/filter/attribute counts, raw
data and total size.
- diff [-r] [-q] [-d D] [-p R] A B [OBJ1 [OBJ2]]: structural and value
differences, exit 0/1/2 like h5diff.
- check [--data] FILE: walks every object, parses every message, verifies
the checksums of every v2+ structure (including the fractal heap blocks
the library never checks), checks chunk indexes against their datasets
and raw data for out-of-file or overlapping extents; every problem with
its address.
Values over --max-bytes are reported, not read; dense-storage heaps are
verified before objects are read from them; panics are caught (exit 3).
Tests compare with h5ls, h5stat, h5dump and h5diff and with h5py's values,
and flip the checksum of every checksummed structure in a v1.14-format file.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A chunk dimension of 0 read a dataset as all fill values, 0x80000000 made
an 8 GiB chunk, and a chunk dimension the chunk index's offsets are not
multiples of read chunks at the wrong place (cve-2018-11205). libhdf5
refuses all of these; now so does clawhdf5:
- DataLayout::parse (H5O__layout_decode): no chunk dimension 0 ("bad chunk
dimension value"), at most 33 dimensions, and before layout v4 at least
2 ("bad dimensions for chunked storage"). New
FormatError::InvalidChunkDimensions.
- Reading a chunked dataset (H5D__chunk_init / H5D__chunk_set_sizes): the
chunk rank must match the dataspace's and a chunk must be under 4 GiB.
One chunked_read::chunk_geometry replaces the four copies of the rank
check.
- v1 B-tree chunk index (H5D__btree_decode_key): every key's offsets must
be multiples of the chunk dimensions, including the keys that only bound
a node, which is where cve-2018-11205's bad dimension shows. New
chunked_read::collect_chunk_info_checked; the chunked read and selection
paths use it.
New interop test header_validation_interop.rs: h5py writes chunked files
(layout v3 and v4), the script corrupts the chunk dimension, and
clawhdf5 must read exactly the copies h5py reads.
Conformance (cached corpus, tank): 570 ok, unchanged; cve-2018-11205 now
refuses the dataset h5py refuses; six more objects that already failed now
fail with libhdf5's reason.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
CHANGELOG (Unreleased): LZF, bitshuffle, bzip2 and Blosc read and write
in pure Rust, their features, the ChunkOptions::plugin field (breaking for
struct-literal construction), the filter registry, the named
UnsupportedFilter message, and Blosc2/ZFP still unimplemented.
README: the clawhdf5-format feature table gains lzf (default),
bitshuffle, bzip2, blosc and plugin-filters, with how to write them and
what is not implemented; no speed claims. docs/known-issues.md: the audit's
filter gap is marked fixed 2026-09-26 for LZF/bitshuffle/bzip2/Blosc,
Blosc2 and ZFP still open. CLAUDE.md: the clawhdf5-filters row no longer
says "No Blosc". clawhdf5-format's crate docs list the new features.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The check matches any *-sys crate, and clawhdf5-wasm pulls in js-sys,
wasm-bindgen's bindings to JavaScript, which compiles no C. Exempt it
by name so the check keeps catching real C for the wasm crate.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
scripts/ci-test.sh: the format feature matrix (clippy and tests) adds
plugin-filters; bitshuffle, bzip2 and blosc are each linted alone (blosc
and bitshuffle share code); the facade is linted with plugin-filters; and
the interop section runs tests/plugin_filters_interop.rs with it, against
h5py + hdf5plugin (CI's venv already installs hdf5plugin).
conformance/probe enables plugin-filters. Sweep (tank, 2026-09-26,
conformance/run.sh --no-fetch against the cached corpus): 573 of 697 ok
(baseline 569), no regressions; newly ok: h5ex_d_blosc.h5,
h5ex_d_bshuf.h5, h5ex_d_bzip2.h5, h5ex_d_lzf.h5. h5ex_d_blosc2.h5 and
h5ex_d_zfp.h5 remain UnsupportedFilter.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
libhdf5 1.14.4+ treats an integer, float or bit field wider than a byte
whose precision and offset leave more than half its bits unused as
corruption when the type sits in a header without a checksum (version 1),
unless the file is opened with H5Pset_relax_file_integrity_checks
(H5T_is_numeric_with_unusual_unused_bits). clawhdf5 read such types,
e.g. a 3-bit integer in 4 bytes (cve-2024-29162) or a 32-bit float in
65525 bytes (cve-2024-32614, tmisc38a.h5).
New Datatype::check_unused_bits (recursive) and Datatype::parse_in_header,
which applies it for version-1 headers. Dataset datatypes (facade File,
LazyFile, MmapFile; clawhdf5-io VOL, MPI VOL, async reader; the
conformance probe) and compact attributes in version-1 headers use it.
Conformance (cached corpus, tank): 570 ok, unchanged; cve-2024-29162,
cve-2024-32614 and tmisc38a.h5 now refuse the object h5py refuses, and
tmisc38b.h5 / unknown-1.h5 now fail with libhdf5's reason.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Blosc2 (32026) is out of reach for now: hdf5plugin's Blosc2 filter stores
each HDF5 chunk as a Blosc2 super-chunk frame (msgpack header, a compressed
chunk-offset index, trailer metalayers) and, for 2-D and larger chunks, as
a B2ND array whose n-D blocks have to be reassembled - on top of the Blosc2
chunk format itself (extended header, filter pipeline, special-value
chunks). ZFP (32013) is out of scope. Both keep failing with
UnsupportedFilter, and the message now says what the ID is:
"unsupported filter: 32026 (Blosc2, not implemented by clawhdf5)", or,
for a filter this build left out, "... (Blosc; this build lacks the
`blosc` feature)". filter_registry::known_filter exposes the table.
tests/plugin_filters_interop.rs: hdf5plugin's Blosc2 and ZFP datasets
read as an error naming the filter, never as data.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
hdf5plugin's Blosc (hdf5-blosc) failed with UnsupportedFilter(32001). The
new `blosc` feature decodes the Blosc 1 frame c-blosc 1.x writes: the
16-byte header, raw ("memcpyed") frames, the block table, blocks split
into one stream per byte plane (and the "do not split" flag), streams
stored raw, the byte shuffle and bit shuffle (whole 8-element groups, the
rest copied) - and every codec hdf5plugin offers: BloscLZ (implemented
here from c-blosc 1.21's blosclz_decompress, including its rejection of
malformed and truncated streams), LZ4/LZ4HC (lz4_flex), Snappy (snap),
Zlib (flate2) and Zstandard (ruzstd). Every stream must decode to exactly
its size and the frame to at most the chunk size; a frame of another
format version (Blosc 2) is a clear error.
It also encodes (DatasetBuilder::with_blosc(codec, level, shuffle)):
LZ4, Snappy, Zlib or Zstandard, with c-blosc's split rule, raw streams
where compression does not pay, and a stored frame for level 0 or
incompressible data. It cannot write BloscLZ (asking for it is an
error). `plugin-filters` enables LZF, bitshuffle, bzip2 and Blosc.
Interop: hdf5plugin writes all six codecs x {no, byte, bit} shuffle at
levels 5/9/1, plus level 0, over the 12-case matrix, read byte for byte;
our four codecs x four shuffle/level settings read back through
hdf5plugin. Both fail with the decoder removed. `cargo tree` with
`plugin-filters` has no -sys crate other than libbz2-rs-sys (pure Rust),
no cc and no cmake.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Datatype::parse now makes the checks of libhdf5's H5O__dtype_decode_helper
and fails with InvalidDatatype (libhdf5's own error text) instead of
decoding a corrupt type:
- size 0 ("invalid datatype size"), for every class;
- integer bit offset/precision outside the type, or precision 0;
- float sign/exponent/mantissa outside the type, empty, or overlapping;
normalization 3; bit 6 without bit 0 from version 3;
- compound with no members, a member outside the compound, a duplicate
name, or a member overlapping an earlier one;
- enum whose size differs from its base type's, or an empty member name;
- array of more than 32 dimensions or with a zero-sized one (v1 compound
array members now say so rather than InvalidDatatypeVersion);
- opaque tag length that is not a multiple of 8.
Bit 6 of a version-1/2 float's class bits used to be read as VAX order,
byte-swapping values; libhdf5 ignores it before version 3, and so does
this now.
Only checks HDF5 2.0 (h5py 3.16) makes are added: newer libhdf5 also
checks bit fields, the variable-length kind and array sizes, but h5py
opens files that fail those, so they are left out. Each check was
confirmed against h5py by corrupting a file it wrote.
The conformance probe now decodes committed datatypes, as h5py's f[name]
does. Conformance (cached corpus, tank): 570 ok, unchanged. Objects
libhdf5 refuses that clawhdf5 used to read: cve-2016-4332-mtime (/cmpnd),
cve-2017-17508, cve-2024-32616 (/type1), cve-2024-32618, cve-2026-34734,
bad_compound.h5 (/cmpnd, /dataset); eight more that already failed now
fail with libhdf5's reason (e.g. cve-2024-29163 "mantissa range out of
bounds").
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Changelog entry with the sizes measured on tank on 2026-09-26 (and the
h5wasm 0.10.3 comparison), the browser build's limits as a known-issues
entry, and where its tests run.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
cargo test --workspace unifies clawhdf5-format/zstd on (another member
enables it), so the native interop test read the Zstd dataset that the
wasm build refuses. The fixture now records its values plus the error
the wasm build must give; the native test accepts either, the Node test
of the real wasm package still requires the error.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Drop a file (or pass ?file=<url>&path=<object>), browse the tree lazily,
see a dataset's type, shape, max shape and attributes, and page through
its values as 50x12 hyperslab windows (leading dims of 3-D+ data held
at chosen indices). build.sh produces pkg/ (not committed) with
wasm-bindgen --target web and checks the CLI matches the crate version.
test/run.sh builds it and runs test.mjs under Node against the h5py/
netCDF4 fixture (250 checks: every dataset whole and as a strided
hyperslab, listings, attributes, error paths, the page's DOM-free
helpers), then browser.sh renders the page in headless Chromium for
eight objects and checks the DOM. The fixture gains LZ4 (read) and Zstd
(refused: links C) datasets and a compound attribute (value null plus
its type). ci-test.sh runs it when node and wasm-bindgen exist; the CI
container has neither, so CI relies on the native h5py_interop test.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
hdf5plugin's BZip2 failed with UnsupportedFilter(307). The new `bzip2`
feature decodes the single bzip2 stream H5Zbzip2.c stores, bounded by the
chunk size (a truncated stream is an error, not short data), and encodes
at block size cd_values[0] (DatasetBuilder::with_bzip2(level)). It uses the
bzip2 crate's default backend, libbz2-rs-sys, a pure-Rust port of
libbzip2: `cargo tree` shows no cc/cmake, and nothing is compiled from C.
Interop: hdf5plugin writes block sizes 9, 1 and 5+shuffle over the
12-case matrix, read byte for byte; ours at 9 (shuffled) and 1 (not)
reads back through hdf5plugin. Both fail with the decoder removed.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
hdf5plugin's Bitshuffle failed with UnsupportedFilter(32008). The new
`bitshuffle` feature (pure Rust: lz4_flex, and ruzstd for Zstandard — the
`zstd` feature's libzstd is not needed) decodes all three modes of
bshuf_h5filter.c — transpose only, LZ4 and Zstandard blocks behind the
12-byte header — including the default and explicit block sizes, the
shorter last block rounded down to a multiple of 8 elements, and the
untransposed trailing elements. Sizes read from the chunk are bounded by
the chunk size.
It also encodes: DatasetBuilder::with_bitshuffle(BitshuffleCompression)
or PluginFilter::Bitshuffle { block_size, compression } writes the filter
with hdf5plugin's cd_values and no automatic byte shuffle. ruzstd has one
compression level (about zstd's 1); the requested level is recorded.
The bit transpose is checked bit for bit against a one-bit-at-a-time model
(which matched hdf5plugin's output) and is shared with blosc next.
Interop: hdf5plugin writes none/LZ4/Zstandard at default and explicit
block sizes and levels over the 12-case matrix, read byte for byte; our
three modes at two block sizes read back through hdf5plugin. Both fail
with the decoder removed.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
ObjectHeader::parse now checks each header message the way libhdf5's
H5O__chunk_deserialize does, and fails with InvalidObjectHeader (libhdf5's
own error text) instead of reading objects out of a corrupt header:
- v1: every message in chunk 0 is read (not just the prefix's count) and
more messages than the prefix claims is "bad object header message
count"; message sizes must be multiples of 8; leftover bytes are a gap,
which only v2 allows; the prefix's chunk size must fit its count.
- v1 and v2: a message running past its chunk is an error (it used to end
the chunk quietly, dropping it and everything after); contradictory
message flags; a message of a class that cannot be shared flagged
shared/shareable; a reference-count message in a v1 header; malformed
continuation, reference-count and modification-time messages (libhdf5
decodes these while loading the header).
- v2: unknown header status flags, max_compact < min_dense, a chunk 0
smaller than a message header, a gap in a chunk that has NIL messages.
Conformance (cached corpus, tank): 569 -> 570 ok (h5stat_err_refcount.h5).
Objects libhdf5 refuses that clawhdf5 used to read: cve-2016-4332-mtime
(/dataset), cve-2016-4332-mtime-new, cve-2018-11204, cve-2018-13873,
cve-2024-32619, cve-2024-33873, cve-2024-33874, gh-4433-poc-08; seven more
CVE objects that already failed now fail with libhdf5's reason.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
h5py's built-in compression="lzf" failed with UnsupportedFilter(32000). The
new `lzf` feature (no dependencies, on by default in clawhdf5-format and
the facade) decodes the raw liblzf stream h5py's filter stores, bounded by
the chunk size, and encodes it: DatasetBuilder::with_lzf() (or
with_plugin_filter(PluginFilter::Lzf)) writes the filter with h5py's
cd_values (filter version 4, liblzf 0x0105, chunk size in bytes), flagged
optional as h5py does. ChunkOptions gains a `plugin` field for the plugin
filters; build_pipeline_for_chunk passes the chunk size to filters that
record it.
tests/plugin_filters_interop.rs: h5py writes LZF (alone, with shuffle, with
shuffle+fletcher32) over 12 dtype/shape/chunk/data cases with partial edge
chunks and incompressible data, and every dataset reads byte for byte equal
to its unfiltered twin; our LZF output (1-D and 2-D, edge chunks, with and
without shuffle) reads back in h5py. Both fail with the decoder removed.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
open(bytes) -> H5File with kind/list/info/attrs/attrErrors/read/
readHyperslab. Numeric data comes back in the typed array of the
stored width (Int16Array for i16, BigInt64Array for i64, Float32Array
for f32/f16, ...), strings and enum names as string arrays, array
datatypes flattened with their dims appended to the shape. Compound,
reference, opaque and VL-sequence datasets are refused with an error
naming the type; nothing is returned as reinterpreted bytes.
The logic is in a plain-Rust core module, tested natively: unit tests,
and h5py_interop, which compares every dataset, hyperslab, listing and
attribute of an h5py- and a netCDF4-written file with what libhdf5
reads back (generator shared with the Node test of the built package).
No mmap, no threads; lz4 is on, zstd/szip (C) are not. A
wasm-release profile (opt-level s, LTO) serves the browser build.
ci-test.sh lints the crate for wasm32 and checks it builds no C.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The facade already builds for the browser target without mmap (and
with it: memmap2 compiles there and File::open just fails, as std::fs
does). Nothing needed gating; keep it that way with a ci-test.sh step,
and install the target in the CI container.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
decompress_chunk_masked and compress_chunk matched on the filter ID. They
now look the ID up in filter_registry: a static table of the built-in
filters compiled into this build (a filter whose cargo feature is off is
simply absent), then the codecs an application registered at run time with
register_filter (a FilterCodec, or a plain decoding closure). Registered
codecs cannot shadow a built-in one, and their output is held to the same
per-stage bound as the built-in decoders. An ID in neither tier still fails
with UnsupportedFilter(id).
The "feature off" stub functions that returned UnsupportedFilter are gone:
the table leaves those filters out instead.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
concurrent_read reads one shared File from 1-16 threads: every dataset
in full (distinct datasets per thread) and random hyperslabs of one
dataset, over a deflate and a contiguous file it generates (or reuses
while manifest.json matches). It reports decoded MB/s and scaling
efficiency, warm or --cold (posix_fadvise) page cache, sizes the decode
pool with --decode-threads, and writes JSON.
scripts/concurrent_read_h5py.py runs the same workload on the same files
with h5py threads or spawned processes (same splitmix64 data and slab
stream, checked at spot elements), and compare_concurrent_read.py prints
one table and refuses runs with different workloads. A smoke test runs
all three end to end on tiny files (h5py half honours CLAWHDF5_PYTHON /
CLAWHDF5_REQUIRE_INTEROP).
BENCHMARKS.md gets a "Concurrent reads" section with the commands, marked
not yet measured.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
dtype() simplifies the type (no byte order, string padding or member
offsets), so callers could not decode read_selection's bytes for types
the typed read_* methods skip. raw_datatype() returns the parsed
Datatype, committed types resolved, for use with data_read.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
