libhdf5 fails to read a VL element whose global heap address is
undefined (all 0xff), even at length 0 ("addr undefined"); we returned
"" (or an empty sequence) in every reader. Checked with h5py first:
libhdf5 writes a null element with address 0, which still reads as
empty, and h5py writes "" as a zero-size heap object at a real address,
so no file they write relies on the old behaviour. read_vl_bytes now
treats address 0 as null whatever the length, as VlResolver does.
Tests, each failing before: vl_data unit test (8- and 4-byte offsets,
lengths 0 and 1); clawhdf5 vl_data_interop
a_vl_element_at_the_undefined_heap_address_fails_like_h5py (also checks
where h5py writes ""); h5rs dump --json and check --data on the patched
`undef` dataset; clawhdf5-wasm vl_strings.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The READMEs said ds[...] reads only the selected elements, and the
facade's read_selection docs that only intersecting chunks are
decompressed. The bounding-box path runs only when the box covers at
most half the dataset; larger boxes (any strided slice across the
dataset), compact, virtual and unwritten datasets and chunked ones with
a non-default fill value decode the whole dataset. The READMEs, the
facade and format docs, the bindings' docstrings and known-issues now
say so, and how index lists are read.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
h5rs (dump, ls, diff, check --data) kept its own lenient VL decoder:
a heap object longer than its element was cut to the element's length
(libhdf5 and h5py refuse it), a null string printed "" where h5dump
prints NULL, the stored element size was trusted, and every heap
collection was kept as an owned copy for the whole run. It now resolves
each element with VlResolver::element / string_element (new: one element
in place, borrowing from the file), and refuses a VL type whose stored
element size is not 4 + offset size + 4, as File does. H5::heap_object
and its cache are gone. h5diff compares a null VL string equal to an
empty one; so does h5rs diff.
clawhdf5-wasm already resolved VL strings with read_vl_strings; it now
uses VlResolver and checks the stored element size before reading, as
File::read_string does.
Tests (h5py writes the files, patched for "a\0b", a null element and
mis-sized heap objects, with 8- and 4-byte offsets):
- h5rs_interop dump_prints_vl_data_like_h5dump: byte-identical to h5dump;
- dump_json_vl_values_match_h5py: h5py's values, errors where h5py fails;
- check_data_flags_mis_sized_vl_heap_objects;
- clawhdf5-wasm tests/vl_strings.rs: wasm, File and h5py agree.
All four fail before. check --data over the 150 cve_hdf5 CVE and fuzzer
files now passes 15 (h5dump rejects 8 of them), was 16 and 9: the
stored-size check flags cve-2024-32608. h5rs-check-ok-files.sh --data:
0 of 422 flagged; h5rs-fuzz.sh: clean on 180 files.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
test_threads_read_the_same_file passed with the GIL held. The new
test_reads_release_the_gil measures the longest stall of a spinning
Python thread while another reads: with py.detach removed from the read
it stalled 0.062 s of a 0.064 s read and failed; with it, about 3 ms.
test_errors_match_h5py now compares the result whenever h5py reads the
key, instead of only checking that we raise when h5py raises, over a
longer key list.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
ds[np.array(1)] went down the index-list path, where tolist() returns a
scalar and extracting a list of indices raised a confusing TypeError.
h5py treats it as an integer index; so do we now. The h5py comparison
keys include 0-d arrays (signed and unsigned) on each axis; they failed
before.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
b0a1e4f fixed this for group and root attributes only. Setting a dataset
attribute twice still wrote two attribute messages with one name, and h5py
read back the first value: set_attr("a", 1) then set_attr("a", 2) read as
1, and list(attrs) was ["a", "a"]. DatasetBuilder::set_attr now replaces
the earlier value, compact or dense. Likewise, a hand-set attribute named
like a provenance attribute (_provenance_sha256, ...) is replaced by the
computed one instead of being written next to it and read first.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Every ds[...] and g[k] resolved the path from the root again, two or
three times per open, and resolving a name in a large group scans its
links: visiting a group was O(n^2). 4000 scalar datasets in one group
took 39 s (v1 group) and 131 s (dense) to list, read and re-read; now
0.3 s each. A Dataset keeps its object address, a Group (and the file's
root) its address and, after the first lookup, its link table.
New facade API File::dataset_at(address), tested in integration_tests.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A hard link's target may go through other hard links, and each was
resolved again every time a path went through it. With each link's
target naming the previous link twice (g/s{i} -> /g/s{i-1}/s{i-1}) the
work doubled per link: finish() took 46 s for 26 links in a debug build,
and 60 would never finish. Resolved links are now remembered, so the work
is linear in the links, and a hard link met again while it is being
resolved is reported as a cycle by name. The depth limit (64) still bounds
the recursion through links not yet resolved.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The link and attribute name indexes are one v2 B-tree leaf, sized to the
next power of two. libhdf5 takes a leaf's capacity from that node size,
but a leaf's record count is a 2-byte field. From about 47 700 links the
node had room for more than 65 535 records, so adding a link in h5py
overflowed the count: a group of 65 535 links crashed h5py, or could no
longer be listed ("unknown link class"). The node is now capped at a full
leaf of 65 535 records, so libhdf5 splits it instead.
Dense attributes now go through the same index builder. Their record
count was written modulo 65 536, without error; more than 65 535
attributes on one object are now refused, like links.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A message in dense storage is a fractal heap object, and an object must
fit one direct block: 65 515 bytes here, since the writer has no
huge-object path. A bigger one (a soft link with a 80 000-byte target in
a group of more than 8 links) was written without error, cut off at the
end of its block, and libhdf5 could not list the group ("object overruns
end of direct block"). finish() now fails with an error that names the
limit, for links and for dense attributes; a 65 001-byte soft link target
still works and h5py reads it back. The heap packer also skips a child
indirect block whose blocks are all too small for the next object instead
of walking it.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Each run of consecutive indices was its own uncached hyperslab read, so
a list over a compressed chunked dataset decoded the same chunk once per
run (d[range(0, 200000, 40)] over 20 gzip chunks: 8 s, h5py 0.014 s).
Plan::reads now groups the indices — a group ends only where a whole
chunk holds no selected index, or, unchunked, at a gap over 64 KiB — and
the selected rows are gathered from each group's block in Rust. Now
3.8 ms (h5py 4.1 ms, release, tank). The new test (1-D, 2-D and
contiguous, compared with h5py, 2 s bound) took 5.8 s before.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Dense link and attribute storage keeps its messages in a fractal heap. Its
root indirect block holds direct blocks up to 64 KiB, 512 KiB in all; rows
past that are child indirect blocks. The writer kept adding rows of direct
blocks instead, and libhdf5 and h5rs read them as indirect blocks: a group
with 20 000 links of 20-byte names was written without error and could not
be listed ("incorrect metadata checksum"), and 150 dense attributes of up
to 56 KB could not be opened. The heap writer now follows the doubling
table: rows past the direct ones hold child indirect blocks, each with its
own rows, nested as deep as the heap needs.
Two more heap bugs are fixed on the way. An object bigger than the next
block's free space was written into it anyway and cut off; the block is
now left unallocated and the object goes in the first block big enough, as
libhdf5 skips blocks. And the header's next-block offset was 0, so libhdf5
adding a link to such a group overwrote the heap's first block ("bad
version number for message"); it is now the offset after the last block.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
VlResolver kept an owned copy of every object of every heap collection
it parsed, for the whole read. Collections nested inside each other's
object data, 32 bytes apart with each element pointing at a different
one, made retained memory O(elements x file size): 1.58 GB for a 744 KB
file (read_vl_strings did the same before VlResolver). Chaining every
collection's objects into one shared run of tiny objects made parse
time O(elements x objects) as well. libhdf5 refuses these files.
- The cache records where each object lies (GlobalHeapCollection::
parse_index, new) instead of copying it, and is dropped past a 32 MiB
budget.
- A collection overlapping one already read is an error: libhdf5 gives
every collection its own block, so only a crafted file has them.
- parse and parse_index refuse a collection that runs past the end of
the file and an object that runs past the end of its collection.
tests/vl_heap_bounds.rs measures peak heap use with a counting
allocator: 129 MB and 350 MB live before on its two crafted files (64 KB
and 176 KB), 97 KB and 0.9 MB now. Conformance unchanged at 575 of 697.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
np.concatenate copies structured dtypes field by field into np.empty, so
the padding of ds[[0, 3, 6]] held process memory. The runs' bytes are
joined in Rust, whole elements at a time, before anything becomes numpy:
the padding is the file's bytes (h5py's) and the result is still a view
of the Rust buffer. The h5py comparisons now compare every byte of
structured values; the new test failed on the padding before.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
PanicException derives from BaseException, so `except Exception` let a
library bug through. Every call from the bindings into the library now
runs under catch_unwind and a panic becomes InternalError (RuntimeError)
naming the object. Tests: a hidden hook panics inside the guard; and the
v4 chunk indexes are compared with h5py from Python — with the library
fix reverted, ds[0:30] of the implicit-index dataset now raises
InternalError instead of aborting the test run.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
read_raw_data_selection's chunked fallback (taken when partial_read
declines, e.g. a bounding box over half the dataset) handed the layout's
chunk dimensions, element-size dimension included, to
generate_implicit_chunks, which indexed past the dataset rank. It then
decoded the whole dataset regardless, so the enumeration is gone: the
arm decodes and extracts for every chunk index.
The new test reads small and large hyperslabs of all five v4 indexes
written by h5py and compares with h5py's values; it panicked before.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Full reads of chunked datasets handed their chunks to rayon. With a
one-thread pool (concurrent_read --decode-threads 1, RAYON_NUM_THREADS=1)
every thread reading through a File queued behind that single worker, so
16 readers decoded on one core: per-thread CPU time showed one thread
doing all the decoding and the readers almost none, and full reads
stopped at about 2x one thread. The cached full-read path and the
uncached reader behind verify_provenance now decode inline when the pool
cannot parallelise (parallel_read::pool_can_parallelise).
The File's chunk cache was the suspect but not the cause: datasets over
its budget were already read without inserting, and skipping its lookups
gained only a few percent at 16 threads.
The regression test keeps a one-thread global pool's worker busy and
requires a full read and verify_provenance to finish anyway; before the
fix both waited for the worker (timed out).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Setting a group or root attribute twice wrote two attribute messages with
the same name, and h5py read back the first value: set_attr("w", 1) then
set_attr("w", "two") read as 1. The later value now replaces the earlier
one, as `attrs[name] = v` does in h5py, including when a group is merged
from two builders.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The writer marked every link name ASCII, so a name such as "größe" was
stored as UTF-8 bytes under the ASCII character set (h5py reports cset 0
for it). Names that are not plain ASCII now carry the UTF-8 flag, as h5py
writes them; ASCII names are unchanged.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
FileWriter wrote the root group plus one level of groups, and refused
path-like names. The writer now flattens its builders into a group tree
(writer_tree.rs) before layout:
- A name may be a path ("a/b/x", "/a/b/x" at the root); missing
intermediate groups are created as h5py does, and GroupBuilder gains
create_group/add_group so builders nest to any depth. A group added at a
path that already holds a group is merged into it (require_group);
any other repeated name, an empty or "." component, or an absolute path
below the root is an error.
- add_soft_link, add_hard_link and add_external_link on FileWriter,
FileBuilder and GroupBuilder. Hard-link targets are resolved to objects
at finish (through other hard links; a missing target, a soft link on the
way or a cycle of paths is an error). Objects with several hard links get
an Object Reference Count message so libhdf5 can delete one link without
freeing the object.
- track_order(true) per group, or as the file default, tracks and indexes
link creation order: Link Info flags and max order, the order in each
Link message, and a type-6 creation-order B-tree for dense groups.
- A group's link index is one B-tree leaf; more than 65535 links is an
error.
Groups are laid out depth-first from the root, datasets group by group,
and untracked groups keep writing datasets, then groups, then other links:
files with one level of groups are byte-identical to before.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
VL-string datasets (h5py's default str dtype) failed read_string with
"type mismatch: expected String, got VariableLength". read_string now
reads fixed- and variable-length strings, with h5py's values (a string
ends at a NUL, a null element is ""). New:
- Dataset::read_string_bytes: each VL string's exact bytes;
- Dataset::read_string_selection: hyperslabs/points of either kind;
- Dataset::read_vlen::<T>() and read_vlen_selection::<T>(): VL sequences
of numbers as Vec<Vec<T>>, T in f64/f32/i64/i32/u64, converted like the
other typed readers;
- File::decode_strings / decode_string_bytes / decode_vlen: VL values in
compound fields and AttrValue::Raw attributes;
- MmapDataset and LazyDataset: read_string for VL strings,
read_string_bytes and read_vlen.
tests/vl_data_interop.rs checks every path against h5py with 8- and
4-byte offsets: scalar, 1-D and 2-D, ASCII and UTF-8, empty strings,
contiguous, compact, chunked with gzip and shuffle, unwritten and partly
written chunks, hyperslabs, compound members, attributes, a big-endian
base type, and a patched file with an embedded NUL and mis-sized heap
objects. NetCDF-4 string variables read too (netCDF4-python test).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
README gains a Python section (maturin develop into a venv, a reading
example that was run against an h5py-written file, the supported types
and keys, what writing covers). The crate README says the same in more
detail. QUICKSTART showed clawhdf5.open()/read_f64(), which never
existed; it now shows File(...)[...].
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A 256 x 256 hyperslab of a contiguous f32 dataset read at an eighth of
h5py's speed: partial_read copied the bounding box out of the file, the
extractor then walked it element by element (a recursive call and two
bounds checks per element) into a second buffer, and read_f32_selection
converted that into a third.
Selections of contiguous data are now copied straight from the file, one
memcpy per run of elements contiguous in the file (gather.rs: a block
along the last dimension, touching blocks as one range, whole rows
merged), with no zero-filled intermediate and no full copy for large
selections. The typed selection readers copy into their Vec<T> directly
when the dataset stores T natively (new data_read::read_selection_native
and sealed NativeElement trait, which the read_as_* fast paths now share;
read_as_u64 gains one) and convert as before otherwise. The general
extractor used by the chunked paths runs on the same run walker, keeping
its old handling of unvalidated selections.
Checked against h5py (contiguous_read_interop.rs) for strided, blocked,
adjacent-block and whole-row hyperslabs, points and empty selections of
every 1-8-byte type in both byte orders, ranks 1-4.
Also keeps the huge-page threshold constant out of no_std builds, where
it was unused.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Checked with h5py on a patched file:
- a VL string with an embedded NUL reads up to the NUL (libhdf5 converts
VL strings to C strings); read_vl_strings returned "a\0b";
- an element whose global heap object is not length x base size bytes is
an error ("Expected global heap object size does not match"); we
returned the object cut to the length;
- a heap address of 0 is a null element whatever its length.
vl_data::VlResolver does this, caching each parsed heap collection:
read_vl_strings parsed the whole collection again for every element.
read_vl_strings and read_vl_bytes use it; check_element_size refuses a VL
type whose stored element size is not 4 + offset size + 4. The
conformance probe resolves VL values through VlResolver instead of its
own lenient copy (575 of 697, unchanged).
The new unit tests fail against the old read_vl_strings.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
ds[key] read the whole dataset and sliced it in numpy, and knew six
dtypes. Keys (ints, positive-step slices, Ellipsis, one increasing index
list, compound field names) now map onto hyperslab selections, and the
facade's read_selection bytes become the numpy buffer without a copy
(PyArray::from_vec viewed as the dtype). dtype mapping follows h5py for
all integer/IEEE float widths and byte orders, bool, enum, complex, fixed
and variable-length strings, vlen sequences, opaque, array types and
(nested, padded) compounds; anything it cannot describe exactly is a
TypeError. Attributes return what h5py returns; groups and files gain
the rest of the h5py mapping interface. Reads run under py.detach.
tests/test_read_vs_h5py.py compares >500 reads with h5py 3.16 on an
h5py-written file, checks errors match, that a damaged chunk outside the
selection is never touched, and 8 threads reading at once.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
In a file with sizeof_addr = 4, a VL string attribute came back as
AttrValue::Raw, a compound's VL member failed with
GlobalHeapObjectNotFound and VL datasets failed with a size mismatch.
Two bugs: Datatype::type_size() said 16 for every VL type, while the
element is 4 + offset size + 4 bytes (12 here); and the global heap was
parsed without the padding libhdf5 puts after its collection and object
headers (both round up to 8), so with 4-byte lengths every object was
looked up 4 bytes early. Datatype::VariableLength now carries the size
its datatype message stores, and writes it back.
Checked against h5py in tests/vl_offset4_interop.rs (fails with either
fix reverted). Conformance unchanged at 575 of 697; in cve-2024-32608 a
VL attribute whose datatype claims 524304-byte elements is now an error
(h5py cannot iterate those attributes at all).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A full read of a contiguous dataset is one memcpy from the mapped file,
yet ran at a quarter of h5py's speed on one thread: the fresh output Vec
took a page fault and a kernel page clear for every 4 KiB page written,
16384 per 64 MiB, costing several times the copy (the benchmark spent
6.2 s of 8 s in the kernel, 4.3M minor faults). numpy, so h5py, madvises
MADV_HUGEPAGE on allocations of 4 MiB or more; the typed readers' output,
the raw contiguous read and the chunk assembly buffer now do the same
(Linux only, libc as a Linux-only dependency; no-op otherwise).
New h5py comparison tests cover full and selection reads of contiguous
data for every 1-8-byte integer and float type, both byte orders, ranks
1-4, empty selections, and datasets past the 4 MiB threshold.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
pyproject.toml named the distribution rustyhdf5 while the extension
module is clawhdf5, and the package's tests imported rustyhdf5, so
pytest failed at collection. Distribution, module-name and tests now
agree; the module gains __version__. maturin develop + pytest: 28 pass.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
libhdf5 reads a group's Group Info message before it inserts a link, and
FileWriter wrote none, so h5py in "r+" mode could not add a link to any
group we wrote: "Unable to create link (message type not found)". Each
group header now carries a version 0 Group Info message with the default
link-phase thresholds, as libhdf5 writes for a new group.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
With the header checks merged, the library refuses truncated files and
misaligned chunk-index keys itself, so check reported only "file cannot
be opened" for a truncated file. It now reports the truncation (stored
end of file vs file length) or the library's error, and the misaligned
chunk test accepts the library's refusal of the key.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Measured on tank on the 150 cvefiles/ and fuzzerfiles/ of cve_hdf5 (the
earlier text said 180): check --data passes 16 (was 28), and h5dump
1.14.6 rejects 9 of those (was 21). It still flags none of the 418
conformance files both readers read in full.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
parallel_integration.rs declared `_sequential` and used `sequential`
under the parallel feature, which nothing in CI enabled for the facade.
ci-test.sh now lints and tests the facade with parallel on.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The review read libhdf5's H5O__chunk_deserialize as accepting a v2
message that runs up to 4 bytes into the chunk's checksum, since it
bounds message bodies by the whole chunk buffer. It does not accept it:
the message loop stops at the checksum, and the checksum read that
follows starts past it and overruns the chunk ("ran off end of input
buffer while decoding"). h5py refuses such files whether the message
runs 1, 4 or 5 bytes in, and so does clawhdf5, with its own error text.
No code change; the test pins the agreement and a comment records why.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The truncated-file check and the end-of-file clamp reached File,
LazyFile and MmapFile but not clawhdf5-io's readers, which still opened
truncated files and read past the recorded end of file. NativeVol
(open, and read_dataset for from_bytes), AsyncHDF5File::from_bytes and
MpiVol's collective read now view the file through the new
vol::hdf5_view: from the superblock to Superblock::data_end, refusing a
file shorter than that.
MpiVol's read is compiled only with the mpi-io feature, which needs an
MPI installation; it was not built here. The edit there only swaps its
two-line superblock setup for hdf5_view.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A chunked layout records the element size as its last dimension, and
libhdf5 refuses a dataset whose datatype has another size
(H5D__chunk_set_sizes: "stored datatype size in chunk layout does not
match datatype description"). clawhdf5 ignored the recorded size and
read the chunks anyway, for v3 and v4 layouts. The check runs on every
chunked read (read_chunked_data*, read_raw_data_selection) and compares
against the stored size: a variable-length element is 4 + offset size
+ 4 bytes, not Datatype::type_size's 16.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
libhdf5 encodes a version-4 layout's chunk dimensions in (log2(max) +
8) / 8 bytes, and HDF5 2.0.0 (h5py 3.16) refuses any other width:
"stored chunk dimension encoding length does not match value calculated
from chunk dimensions". The writer rounded 3 bytes up to 4, so h5py
could not open a dataset we wrote with a chunk dimension from 65 536 to
16 777 215, for every chunk index (single chunk, fixed and extensible
array, v2 B-tree). The three encoders now share push_v4_chunk_dims,
which writes the exact width.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A version-4 layout stores every chunk dimension in the fewest bytes that
hold the largest one (H5D__chunk_set_sizes: (log2(dim) + 8) / 8), so a
chunk dimension of 65 536 to 16 777 215 takes 3 bytes. Only widths 1, 2,
4 and 8 were decoded; an h5py file with chunks=(70000,) and
libver='latest' failed with UnexpectedEof. Widths 1-8 are decoded now;
0 and more than 8 are refused with libhdf5's "encoded chunk dimension
size is too large", and a dimension past u32 is refused, not truncated.
The review asked for libhdf5's check that the stored width matches the
one computed from the dimensions. HDF5 2.0.0 (h5py 3.16) refuses any
mismatch, but HDFGroup/hdf5@e124c36 ("Allow reading of files with chunk
dimensions encoded using more bytes than necessary", 2026-06-05) relaxed
it to refusing only a width too small for the dimensions, which cannot
happen once the dimensions have been decoded from that width. Follow
current libhdf5: a wider-than-needed encoding is read. clawhdf5's own
writer produces such layouts (the next commit fixes that).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The compound overlap check measured each earlier member with
Datatype::type_size, which is a fixed 16 for a variable-length type. On
disk a VL member takes 4 + offset size + 4 bytes, 12 in a file with
4-byte offsets, so a member right after one was refused as "member
overlaps with previous member" (and with the type, every attribute of
the object). libhdf5 measures members by their decoded, stored size
(times a v1 member's array dimensions); so does this now.
Reading VL values in such files is a separate, older gap, now recorded
in known-issues.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
With the pcodec feature, 32023 was a built-in entry (the legacy reader for
the pcodec chunks clawhdf5 <= 2.7.0 wrote under that ID), so
register_filter(32023, ...) was refused as "built in", although
UnsupportedFilter(32023) names Granular BitRound as not implemented and
the registry docs point to register_filter for such IDs.
That entry is now shared: it claims only chunks whose filter is named
"pcodec"; any other chunk with ID 32023 goes to the registered codec (or,
with none registered, gets UnsupportedFilter as before), and writing
32023 uses the registered codec. Every other built-in ID still refuses
registration.
Test: a_codec_can_be_registered_for_granular_bitround (registers, round-
trips chunks with no name and other names, still reads a legacy "pcodec"
chunk with the built-in reader, and after unregistering reads nothing).
It fails without the change ("filter 32023 ... is built in and cannot be
re-registered"). It and the existing legacy-pcodec test share a lock,
since the registry is process-wide.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
dump_matches_h5dump requires byte identity with the h5dump on PATH, and
CI's rust:latest (Debian 13.7) installs hdf5-tools 1.14.5, not the 1.14.6
the test was written against. Ran the whole clawhdf5-tools suite in
rust:latest with Debian's hdf5-tools and pip h5py 3.16.0 (HDF5 2.0.0), as
CI sets it up, with CLAWHDF5_REQUIRE_INTEROP=1: 18 of 18 pass, so the
comparison needs no loosening. The version is now printed in the CI log
so a future Debian update that changes the output is easy to spot.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A null-padded fixed string inside a compound or an array member printed
trimmed ("" for three NULs, "a" for "a\0b"), where h5dump prints every
byte ("\000\000\000", "a\000b"); only top-level strings were shown in
full. DATA blocks now render elements through one function that keeps
the padding at any depth.
The README now lists the remaining known differences from h5dump:
nested compounds print inline, and long double values are printed as
errors (exit 1) with the datatype as an H5T_FLOAT block.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
osobh