Fast contiguous and concurrent reads, VL data, nested groups and links, Python bindings #15

2026-09-26T14:29:21Z

osobh commented

2026-09-26 14:29:21 +00:00

Phase 2, first batch: the two measured read-performance deficits, variable-length data through File, nested groups and links in the writer, and working Python bindings. Five branches, each adversarially reviewed. Every review finding is fixed with a test that fails without the fix.

Performance: measured on an idle tank at `408f69e`, h5py re-run in the same session (`BENCHMARKS.md`)

Workload	Threads	before	now	h5py threads	h5py processes
Contiguous full read	1	2.5 GB/s	13.7 GB/s	9.5	8.8
Contiguous 256×256 hyperslabs	1	0.62 GB/s	32 GB/s	5.1	5.1
Deflate full reads	8	887 MB/s	2943 MB/s	428	3042
Deflate hyperslabs	16	1244 MB/s	1592 MB/s	128	1399
Deflate full reads (default pool)	16	2824 MB/s	2341 MB/s	375	3083

Contiguous reads: the fresh 64 MiB output buffer took about 16K 4 KiB page faults per read. Large read buffers now ask for transparent huge pages, as numpy does. Hyperslabs were copied three times, element by element; they are now copied run by run, straight from the file into the output. Values are checked against h5py for every numeric type in both byte orders, at ranks 1–4.
Full chunked reads: the cause wasn't the chunk cache. With a one-thread rayon pool, every reader queued behind its single worker; such reads now decode on the calling thread.
- Still open: 16-thread full chunked reads run at 0.69–0.76× h5py processes. This is recorded in docs/known-issues.md.
- The contiguous-hyperslab row mostly measures copies out of the CPU cache, and the 16-thread rows were noisy for every tool. Both caveats are written next to the table.

Variable-length data

VL-string datasets are now readable through File: read_string, read_string_bytes, and selections. Typed VL sequences are read with read_vlen::<T>().
Files with 4-byte offsets: VL values failed to read (type_size() hard-coded 16 bytes, and heap parsing assumed 8-byte offsets). Fixed.
One decoder: h5rs and the wasm reader now decode through the library's VlResolver, so every reader agrees with h5py. h5rs dump matches h5dump on VL data, including NULL strings.
Hostile heaps: a crafted global heap can no longer make memory balloon. One test went from 350 MB to 0.9 MB peak. Overlapping collections are refused, and an element pointing at the undefined address is an error, as in libhdf5.

Writer: groups and links

Groups nest to any depth; path names create the missing intermediate groups, as h5py does.
Soft, hard and external links can go anywhere, and link creation order can be tracked.
Files are checked with h5py, h5dump and h5rs check, including h5py adding and deleting links in r+ mode.
Output that uses one level of groups is byte-identical to before.
Bugs fixed on the way, several also present in v2.7.0:
- Missing Group Info message: h5py could not add a link to any group we wrote.
- Setting an attribute twice on a group or dataset: h5py read the first value.
- Non-ASCII link names were flagged as ASCII.
- Dense groups over about 512 KiB were written unreadable. The fractal heap writer now writes indirect blocks; this was tested with 65,535 links in a 7 MB heap.
- Heap objects bigger than a block's free space were truncated.
- Adding a link to a dense group from h5py:
  - once the link heap was over 64 KiB, h5py overwrote its first block;
  - at about 47,700 links, it overflowed the 2-byte link count, and h5py segfaulted.
- Hard-link resolution was exponential; each link is now resolved once, and cycles are reported by name.
Limits, now documented accurately: 65,535 links or dense attributes per object, and 65,515 bytes per dense entry. Past them the writer returns a clear error.

Python bindings (`crates/clawhdf5-py`)

Package name: the package installs and imports as clawhdf5; it used to conflict with the name rustyhdf5.
h5py-style reads: File, Group and Dataset behave like h5py's, and a selection decodes only the chunks it touches (with documented exceptions).
Types: every numeric type, plus bool, enums, complex, fixed and VL strings, VL sequences, opaque, array types and compounds.
Zero-copy: results are numpy arrays built on the Rust buffer, with no extra copy.
GIL: released during reads. A test detects a held GIL.
CI: builds the package with maturin and runs 47 pytest cases against h5py.
Review fixes:
- Library panic, reachable from Rust too: read_selection on a chunked dataset with an implicit chunk index, when the selection covered more than half of it. There is a new Rust test for every newer (layout v4) chunk-index type.
- Uninitialised padding: padding bytes in compound results leaked process memory.
- Index lists: a list was 585× slower than h5py; it's now on par (3.8 ms vs 4.1 ms).
- Big groups: O(n²) path resolution. A 4000-member group went from 39 s to 0.3 s.
- 0-d numpy index: ds[np.array(1)] now behaves like an integer index.
- Panics: a library panic now raises a catchable exception, not PanicException.

Verification (tank, 2026-09-26)

CLAWHDF5_REQUIRE_INTEROP=1 scripts/ci-test.sh: 22/22 pass, including the new Python package step.
Conformance: 575 of 697, the gate passes, no regressions. h5rs check --data flags 0 of the 422 fully-read files.
ClawBrainHub: 204 of 204 tests pass against this branch.

Breaking changes (in the CHANGELOG)

Python: fixed-length string attributes now read as numpy.bytes_, as in h5py.
Writer: path-like names now create nested groups; they used to be refused.
Writer: fractal heaps over 64 KiB are no longer byte-identical to earlier output, because the next-block offset in the heap header was fixed.
Reading VL data:
- a VL type whose stored element size doesn't match the file is refused;
- an element pointing at the undefined heap address is an error.

🤖 Generated with Claude Code

Phase 2, first batch: the two measured read-performance deficits, variable-length data through `File`, nested groups and links in the writer, and working Python bindings. Five branches, each adversarially reviewed. Every review finding is fixed with a test that fails without the fix. ## Performance: measured on an idle tank at `408f69e`, h5py re-run in the same session (`BENCHMARKS.md`) | Workload | Threads | before | **now** | h5py threads | h5py processes | |---|---:|---:|---:|---:|---:| | Contiguous full read | 1 | 2.5 GB/s | **13.7 GB/s** | 9.5 | 8.8 | | Contiguous 256×256 hyperslabs | 1 | 0.62 GB/s | **32 GB/s** | 5.1 | 5.1 | | Deflate full reads | 8 | 887 MB/s | **2943 MB/s** | 428 | 3042 | | Deflate hyperslabs | 16 | 1244 MB/s | **1592 MB/s** | 128 | 1399 | | Deflate full reads (default pool) | 16 | 2824 MB/s | 2341 MB/s | 375 | 3083 | - **Contiguous reads:** the fresh 64 MiB output buffer took about 16K 4 KiB page faults per read. Large read buffers now ask for transparent huge pages, as numpy does. Hyperslabs were copied three times, element by element; they are now copied run by run, straight from the file into the output. Values are checked against h5py for every numeric type in both byte orders, at ranks 1–4. - **Full chunked reads:** the cause wasn't the chunk cache. With a one-thread rayon pool, every reader queued behind its single worker; such reads now decode on the calling thread. - **Still open:** 16-thread full chunked reads run at 0.69–0.76× h5py processes. This is recorded in `docs/known-issues.md`. - The contiguous-hyperslab row mostly measures copies out of the CPU cache, and the 16-thread rows were noisy for every tool. Both caveats are written next to the table. ## Variable-length data - VL-string datasets are now readable through `File`: `read_string`, `read_string_bytes`, and selections. Typed VL sequences are read with `read_vlen::<T>()`. - **Files with 4-byte offsets:** VL values failed to read (`type_size()` hard-coded 16 bytes, and heap parsing assumed 8-byte offsets). Fixed. - **One decoder:** h5rs and the wasm reader now decode through the library's `VlResolver`, so every reader agrees with h5py. `h5rs dump` matches h5dump on VL data, including NULL strings. - **Hostile heaps:** a crafted global heap can no longer make memory balloon. One test went from 350 MB to 0.9 MB peak. Overlapping collections are refused, and an element pointing at the undefined address is an error, as in libhdf5. ## Writer: groups and links - Groups nest to any depth; path names create the missing intermediate groups, as h5py does. - Soft, hard and external links can go anywhere, and link creation order can be tracked. - Files are checked with h5py, h5dump and `h5rs check`, including h5py adding and deleting links in r+ mode. - Output that uses one level of groups is byte-identical to before. - **Bugs fixed on the way, several also present in v2.7.0:** - **Missing Group Info message:** h5py could not add a link to any group we wrote. - **Setting an attribute twice** on a group or dataset: h5py read the first value. - **Non-ASCII link names** were flagged as ASCII. - **Dense groups over about 512 KiB** were written unreadable. The fractal heap writer now writes indirect blocks; this was tested with 65,535 links in a 7 MB heap. - **Heap objects bigger than a block's free space** were truncated. - **Adding a link to a dense group from h5py:** - once the link heap was over 64 KiB, h5py overwrote its first block; - at about 47,700 links, it overflowed the 2-byte link count, and h5py segfaulted. - **Hard-link resolution** was exponential; each link is now resolved once, and cycles are reported by name. - **Limits, now documented accurately:** 65,535 links or dense attributes per object, and 65,515 bytes per dense entry. Past them the writer returns a clear error. ## Python bindings (`crates/clawhdf5-py`) - **Package name:** the package installs and imports as `clawhdf5`; it used to conflict with the name `rustyhdf5`. - **h5py-style reads:** File, Group and Dataset behave like h5py's, and a selection decodes only the chunks it touches (with documented exceptions). - **Types:** every numeric type, plus bool, enums, complex, fixed and VL strings, VL sequences, opaque, array types and compounds. - **Zero-copy:** results are numpy arrays built on the Rust buffer, with no extra copy. - **GIL:** released during reads. A test detects a held GIL. - **CI:** builds the package with maturin and runs 47 pytest cases against h5py. - **Review fixes:** - **Library panic, reachable from Rust too:** `read_selection` on a chunked dataset with an implicit chunk index, when the selection covered more than half of it. There is a new Rust test for every newer (layout v4) chunk-index type. - **Uninitialised padding:** padding bytes in compound results leaked process memory. - **Index lists:** a list was 585× slower than h5py; it's now on par (3.8 ms vs 4.1 ms). - **Big groups:** O(n²) path resolution. A 4000-member group went from 39 s to 0.3 s. - **0-d numpy index:** `ds[np.array(1)]` now behaves like an integer index. - **Panics:** a library panic now raises a catchable exception, not `PanicException`. ## Verification (tank, 2026-09-26) - `CLAWHDF5_REQUIRE_INTEROP=1 scripts/ci-test.sh`: **22/22 pass**, including the new Python package step. - Conformance: 575 of 697, the gate passes, no regressions. `h5rs check --data` flags 0 of the 422 fully-read files. - ClawBrainHub: 204 of 204 tests pass against this branch. ## Breaking changes (in the CHANGELOG) - **Python:** fixed-length string attributes now read as `numpy.bytes_`, as in h5py. - **Writer:** path-like names now create nested groups; they used to be refused. - **Writer:** fractal heaps over 64 KiB are no longer byte-identical to earlier output, because the next-block offset in the heap header was fixed. - **Reading VL data:** - a VL type whose stored element size doesn't match the file is refused; - an element pointing at the undefined heap address is an error. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

osobh added 41 commits 2026-09-26 14:29:22 +00:00

fix(format): write a Group Info message in every group 8cbbef3fae

libhdf5 reads a group's Group Info message before it inserts a link, and
FileWriter wrote none, so h5py in "r+" mode could not add a link to any
group we wrote: "Unable to create link (message type not found)". Each
group header now carries a version 0 Group Info message with the default
link-phase thresholds, as libhdf5 writes for a new group.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

fix(py): one name, clawhdf5, for the Python distribution and module 006bf3b131

pyproject.toml named the distribution rustyhdf5 while the extension
module is clawhdf5, and the package's tests imported rustyhdf5, so
pytest failed at collection. Distribution, module-name and tests now
agree; the module gains __version__. maturin develop + pytest: 28 pass.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

perf(format): back large read buffers with transparent huge pages 78c769f179

A full read of a contiguous dataset is one memcpy from the mapped file,
yet ran at a quarter of h5py's speed on one thread: the fresh output Vec
took a page fault and a kernel page clear for every 4 KiB page written,
16384 per 64 MiB, costing several times the copy (the benchmark spent
6.2 s of 8 s in the kernel, 4.3M minor faults). numpy, so h5py, madvises
MADV_HUGEPAGE on allocations of 4 MiB or more; the typed readers' output,
the raw contiguous read and the chunk assembly buffer now do the same
(Linux only, libc as a Linux-only dependency; no-op otherwise).

New h5py comparison tests cover full and selection reads of contiguous
data for every 1-8-byte integer and float type, both byte orders, ranks
1-4, empty selections, and datasets past the 4 MiB threshold.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

fix(format): read VL values in files with 4-byte offsets 10da8f0d09

In a file with sizeof_addr = 4, a VL string attribute came back as
AttrValue::Raw, a compound's VL member failed with
GlobalHeapObjectNotFound and VL datasets failed with a size mismatch.

Two bugs: Datatype::type_size() said 16 for every VL type, while the
element is 4 + offset size + 4 bytes (12 here); and the global heap was
parsed without the padding libhdf5 puts after its collection and object
headers (both round up to 8), so with 4-byte lengths every object was
looked up 4 bytes early. Datatype::VariableLength now carries the size
its datatype message stores, and writes it back.

Checked against h5py in tests/vl_offset4_interop.rs (fails with either
fix reverted). Conformance unchanged at 575 of 697; in cve-2024-32608 a
VL attribute whose datatype claims 524304-byte elements is now an error
(h5py cannot iterate those attributes at all).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

feat(py): h5py-style reads of only the selected elements, GIL released 2d4b211523

ds[key] read the whole dataset and sliced it in numpy, and knew six
dtypes. Keys (ints, positive-step slices, Ellipsis, one increasing index
list, compound field names) now map onto hyperslab selections, and the
facade's read_selection bytes become the numpy buffer without a copy
(PyArray::from_vec viewed as the dtype). dtype mapping follows h5py for
all integer/IEEE float widths and byte orders, bool, enum, complex, fixed
and variable-length strings, vlen sequences, opaque, array types and
(nested, padded) compounds; anything it cannot describe exactly is a
TypeError. Attributes return what h5py returns; groups and files gain
the rest of the h5py mapping interface. Reads run under py.detach.

tests/test_read_vs_h5py.py compares >500 reads with h5py 3.16 on an
h5py-written file, checks errors match, that a damaged chunk outside the
selection is never touched, and 8 threads reading at once.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

fix(format): resolve VL elements as libhdf5 does f99587c27d

Checked with h5py on a patched file:
- a VL string with an embedded NUL reads up to the NUL (libhdf5 converts
  VL strings to C strings); read_vl_strings returned "a\0b";
- an element whose global heap object is not length x base size bytes is
  an error ("Expected global heap object size does not match"); we
  returned the object cut to the length;
- a heap address of 0 is a null element whatever its length.

vl_data::VlResolver does this, caching each parsed heap collection:
read_vl_strings parsed the whole collection again for every element.
read_vl_strings and read_vl_bytes use it; check_element_size refuses a VL
type whose stored element size is not 4 + offset size + 4. The
conformance probe resolves VL values through VlResolver instead of its
own lenient copy (575 of 697, unchanged).

The new unit tests fail against the old read_vl_strings.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

ci: build the Python package with maturin and run its tests against h5py f7d88bb4fb

ci-test.sh gains a step that lints clawhdf5-py, builds its wheel with
maturin, unpacks it under target/ (the interpreter's environment is not
touched) and runs the pytest suite, which compares reads with h5py. It
skips without maturin/pytest, and fails instead under
CLAWHDF5_REQUIRE_INTEROP=1. The CI interop venv installs maturin and
pytest, so CI runs it.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

perf: copy contiguous hyperslab and point reads run by run 2bc4cb46a6

A 256 x 256 hyperslab of a contiguous f32 dataset read at an eighth of
h5py's speed: partial_read copied the bounding box out of the file, the
extractor then walked it element by element (a recursive call and two
bounds checks per element) into a second buffer, and read_f32_selection
converted that into a third.

Selections of contiguous data are now copied straight from the file, one
memcpy per run of elements contiguous in the file (gather.rs: a block
along the last dimension, touching blocks as one range, whole rows
merged), with no zero-filled intermediate and no full copy for large
selections. The typed selection readers copy into their Vec<T> directly
when the dataset stores T natively (new data_read::read_selection_native
and sealed NativeElement trait, which the read_as_* fast paths now share;
read_as_u64 gains one) and convert as before otherwise. The general
extractor used by the chunked paths runs on the same run walker, keeping
its old handling of unvalidated selections.

Checked against h5py (contiguous_read_interop.rs) for strided, blocked,
adjacent-block and whole-row hyperslabs, points and empty selections of
every 1-8-byte type in both byte orders, ranks 1-4.

Also keeps the huge-page threshold constant out of no_std builds, where
it was unused.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

docs: the Python package — install with maturin, h5py-style reading c3850a0b66

README gains a Python section (maturin develop into a venv, a reading
example that was run against an h5py-written file, the supported types
and keys, what writing covers). The crate README says the same in more
detail. QUICKSTART showed clawhdf5.open()/read_f64(), which never
existed; it now shows File(...)[...].

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

feat(facade): read VL strings and VL sequences through File 8ce6eca34d

VL-string datasets (h5py's default str dtype) failed read_string with
"type mismatch: expected String, got VariableLength". read_string now
reads fixed- and variable-length strings, with h5py's values (a string
ends at a NUL, a null element is ""). New:
- Dataset::read_string_bytes: each VL string's exact bytes;
- Dataset::read_string_selection: hyperslabs/points of either kind;
- Dataset::read_vlen::<T>() and read_vlen_selection::<T>(): VL sequences
  of numbers as Vec<Vec<T>>, T in f64/f32/i64/i32/u64, converted like the
  other typed readers;
- File::decode_strings / decode_string_bytes / decode_vlen: VL values in
  compound fields and AttrValue::Raw attributes;
- MmapDataset and LazyDataset: read_string for VL strings,
  read_string_bytes and read_vlen.

tests/vl_data_interop.rs checks every path against h5py with 8- and
4-byte offsets: scalar, 1-D and 2-D, ASCII and UTF-8, empty strings,
contiguous, compact, chunked with gzip and shuffle, unwritten and partly
written chunks, hyperslabs, compound members, attributes, a big-endian
base type, and a patched file with an embedded NUL and mis-sized heap
objects. NetCDF-4 string variables read too (netCDF4-python test).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

build: record libc in the conformance probe's lockfile 6e8421a81e

clawhdf5-format now depends on libc on Linux (huge-page advice for read
buffers); the probe's committed lockfile picks that up.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

feat(format): nested groups, soft/hard/external links and creation order in the writer d102c06306

FileWriter wrote the root group plus one level of groups, and refused
path-like names. The writer now flattens its builders into a group tree
(writer_tree.rs) before layout:

- A name may be a path ("a/b/x", "/a/b/x" at the root); missing
  intermediate groups are created as h5py does, and GroupBuilder gains
  create_group/add_group so builders nest to any depth. A group added at a
  path that already holds a group is merged into it (require_group);
  any other repeated name, an empty or "." component, or an absolute path
  below the root is an error.
- add_soft_link, add_hard_link and add_external_link on FileWriter,
  FileBuilder and GroupBuilder. Hard-link targets are resolved to objects
  at finish (through other hard links; a missing target, a soft link on the
  way or a cycle of paths is an error). Objects with several hard links get
  an Object Reference Count message so libhdf5 can delete one link without
  freeing the object.
- track_order(true) per group, or as the file default, tracks and indexes
  link creation order: Link Info flags and max order, the order in each
  Link message, and a type-6 creation-order B-tree for dense groups.
- A group's link index is one B-tree leaf; more than 65535 links is an
  error.

Groups are laid out depth-first from the root, datasets group by group,
and untracked groups keep writing datasets, then groups, then other links:
files with one level of groups are byte-identical to before.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

fix(format): flag non-ASCII link names as UTF-8 bd36fe883b

The writer marked every link name ASCII, so a name such as "größe" was
stored as UTF-8 bytes under the ASCII character set (h5py reports cset 0
for it). Names that are not plain ASCII now carry the UTF-8 flag, as h5py
writes them; ASCII names are unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

fix(format): a group attribute set again replaces the earlier value b0a1e4f9a6

Setting a group or root attribute twice wrote two attribute messages with
the same name, and h5py read back the first value: set_attr("w", 1) then
set_attr("w", "two") read as 1. The later value now replaces the earlier
one, as `attrs[name] = v` does in h5py, including when a group is merged
from two builders.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

fix(read): decode on the calling thread when rayon's pool has one thread a5e41c1a53

Full reads of chunked datasets handed their chunks to rayon. With a
one-thread pool (concurrent_read --decode-threads 1, RAYON_NUM_THREADS=1)
every thread reading through a File queued behind that single worker, so
16 readers decoded on one core: per-thread CPU time showed one thread
doing all the decoding and the readers almost none, and full reads
stopped at about 2x one thread. The cached full-read path and the
uncached reader behind verify_provenance now decode inline when the pool
cannot parallelise (parallel_read::pool_can_parallelise).

The File's chunk cache was the suspect but not the cause: datasets over
its budget were already read without inserting, and skipping its lookups
gained only a few percent at 16 threads.

The regression test keeps a one-thread global pool's worker busy and
requires a full read and verify_provenance to finish anyway; before the
fix both waited for the worker (timed out).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

docs: the rayon fix covers a one-thread pool, not the h5py-process gap 37770f594a

The review measured the default pool unchanged (about 2900 MB/s at 16
threads before and after) and still short of 16 h5py processes; small
pools still make outside readers wait. Say so instead of marking the
scaling issue fixed.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

fix(format): selections of v4 implicit-index chunked data no longer panic 3bcd443e63

read_raw_data_selection's chunked fallback (taken when partial_read
declines, e.g. a bounding box over half the dataset) handed the layout's
chunk dimensions, element-size dimension included, to
generate_implicit_chunks, which indexed past the dataset rank. It then
decoded the whole dataset regardless, so the enumeration is gone: the
arm decodes and extracts for every chunk index.

The new test reads small and large hyperslabs of all five v4 indexes
written by h5py and compares with h5py's values; it panicked before.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

fix(py): a panic in the library raises clawhdf5.InternalError, not PanicException 24412a0e59

PanicException derives from BaseException, so `except Exception` let a
library bug through. Every call from the bindings into the library now
runs under catch_unwind and a panic becomes InternalError (RuntimeError)
naming the object. Tests: a hidden hook panics inside the guard; and the
v4 chunk indexes are compared with h5py from Python — with the library
fix reverted, ds[0:30] of the implicit-index dataset now raises
InternalError instead of aborting the test run.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

fix(py): index lists of padded compounds no longer return uninitialised padding 8c51b05b9c

np.concatenate copies structured dtypes field by field into np.empty, so
the padding of ds[[0, 3, 6]] held process memory. The runs' bytes are
joined in Rust, whole elements at a time, before anything becomes numpy:
the padding is the file's bytes (h5py's) and the result is still a view
of the Rust buffer. The h5py comparisons now compare every byte of
structured values; the new test failed on the padding before.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

fix(format): bound what a VL read retains on a crafted global heap 41b7837d0a

VlResolver kept an owned copy of every object of every heap collection
it parsed, for the whole read. Collections nested inside each other's
object data, 32 bytes apart with each element pointing at a different
one, made retained memory O(elements x file size): 1.58 GB for a 744 KB
file (read_vl_strings did the same before VlResolver). Chaining every
collection's objects into one shared run of tiny objects made parse
time O(elements x objects) as well. libhdf5 refuses these files.

- The cache records where each object lies (GlobalHeapCollection::
  parse_index, new) instead of copying it, and is dropped past a 32 MiB
  budget.
- A collection overlapping one already read is an error: libhdf5 gives
  every collection its own block, so only a crafted file has them.
- parse and parse_index refuse a collection that runs past the end of
  the file and an object that runs past the end of its collection.

tests/vl_heap_bounds.rs measures peak heap use with a counting
allocator: 129 MB and 350 MB live before on its two crafted files (64 KB
and 176 KB), 97 KB and 0.9 MB now. Conformance unchanged at 575 of 697.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

fix(format): write child indirect blocks in big fractal heaps 81a0e8685d

Dense link and attribute storage keeps its messages in a fractal heap. Its
root indirect block holds direct blocks up to 64 KiB, 512 KiB in all; rows
past that are child indirect blocks. The writer kept adding rows of direct
blocks instead, and libhdf5 and h5rs read them as indirect blocks: a group
with 20 000 links of 20-byte names was written without error and could not
be listed ("incorrect metadata checksum"), and 150 dense attributes of up
to 56 KB could not be opened. The heap writer now follows the doubling
table: rows past the direct ones hold child indirect blocks, each with its
own rows, nested as deep as the heap needs.

Two more heap bugs are fixed on the way. An object bigger than the next
block's free space was written into it anyway and cut off; the block is
now left unallocated and the object goes in the first block big enough, as
libhdf5 skips blocks. And the header's next-block offset was 0, so libhdf5
adding a link to such a group overwrote the heap's first block ("bad
version number for message"); it is now the offset after the last block.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

perf(py): read an index list one group of chunks at a time b43bd2e67f

Each run of consecutive indices was its own uncached hyperslab read, so
a list over a compressed chunked dataset decoded the same chunk once per
run (d[range(0, 200000, 40)] over 20 gzip chunks: 8 s, h5py 0.014 s).
Plan::reads now groups the indices — a group ends only where a whole
chunk holds no selected index, or, unchunked, at a gap over 64 KiB — and
the selected rows are gathered from each group's block in Rust. Now
3.8 ms (h5py 4.1 ms, release, tank). The new test (1-D, 2-D and
contiguous, compared with h5py, 2 s bound) took 5.8 s before.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

fix(format): refuse a dense link or attribute too big for the heap 751edeb7e6

A message in dense storage is a fractal heap object, and an object must
fit one direct block: 65 515 bytes here, since the writer has no
huge-object path. A bigger one (a soft link with a 80 000-byte target in
a group of more than 8 links) was written without error, cut off at the
end of its block, and libhdf5 could not list the group ("object overruns
end of direct block"). finish() now fails with an error that names the
limit, for links and for dense attributes; a 65 001-byte soft link target
still works and h5py reads it back. The heap packer also skips a child
indirect block whose blocks are all too small for the next object instead
of walking it.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

fix(format): keep a dense index leaf within 65 535 records bd1d8f1a59

The link and attribute name indexes are one v2 B-tree leaf, sized to the
next power of two. libhdf5 takes a leaf's capacity from that node size,
but a leaf's record count is a 2-byte field. From about 47 700 links the
node had room for more than 65 535 records, so adding a link in h5py
overflowed the count: a group of 65 535 links crashed h5py, or could no
longer be listed ("unknown link class"). The node is now capped at a full
leaf of 65 535 records, so libhdf5 splits it instead.

Dense attributes now go through the same index builder. Their record
count was written modulo 65 536, without error; more than 65 535
attributes on one object are now refused, like links.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

fix(format): resolve each hard link once 400e3a9fec

A hard link's target may go through other hard links, and each was
resolved again every time a path went through it. With each link's
target naming the previous link twice (g/s{i} -> /g/s{i-1}/s{i-1}) the
work doubled per link: finish() took 46 s for 26 links in a debug build,
and 60 would never finish. Resolved links are now remembered, so the work
is linear in the links, and a hard link met again while it is being
resolved is reported as a cycle by name. The depth limit (64) still bounds
the recursion through links not yet resolved.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

perf(py): datasets and groups keep their address; groups their links f0ecae38b6

Every ds[...] and g[k] resolved the path from the root again, two or
three times per open, and resolving a name in a large group scans its
links: visiting a group was O(n^2). 4000 scalar datasets in one group
took 39 s (v1 group) and 131 s (dense) to list, read and re-read; now
0.3 s each. A Dataset keeps its object address, a Group (and the file's
root) its address and, after the first lookup, its link table.

New facade API File::dataset_at(address), tested in integration_tests.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

fix(format): a dataset attribute set again replaces the earlier value 8bcae3c78e

b0a1e4f fixed this for group and root attributes only. Setting a dataset
attribute twice still wrote two attribute messages with one name, and h5py
read back the first value: set_attr("a", 1) then set_attr("a", 2) read as
1, and list(attrs) was ["a", "a"]. DatasetBuilder::set_attr now replaces
the earlier value, compact or dense. Likewise, a hand-set attribute named
like a provenance attribute (_provenance_sha256, ...) is replaced by the
computed one instead of being written next to it and read first.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

fix(py): a 0-d integer array indexes like an int 05b0192a60

ds[np.array(1)] went down the index-list path, where tolist() returns a
scalar and extracting a list of indices raised a confusing TypeError.
h5py treats it as an integer index; so do we now. The h5py comparison
keys include 0-d arrays (signed and unsigned) on each axis; they failed
before.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

docs: dense storage fixes and the real limits of big groups 546fdb84fa

The changelog, known issues and README said a group holds up to 65 535
links while a group of about 17 000 was already unreadable. Record the
fixes (child indirect blocks, the next-block offset, the index leaf cap,
refusing oversized dense messages, hard-link memoisation, dataset
attribute overwrite) and the limits that remain true: 65 535 links or
dense attributes per object, and 65 515 bytes per dense message.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

test(py): detect a held GIL, and errors h5py does not raise 17edfe2cf0

test_threads_read_the_same_file passed with the GIL held. The new
test_reads_release_the_gil measures the longest stall of a spinning
Python thread while another reads: with py.detach removed from the read
it stalled 0.062 s of a 0.064 s read and failed; with it, about 3 ms.
test_errors_match_h5py now compares the result whenever h5py reads the
key, instead of only checking that we raise when h5py raises, over a
longer key list.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

fix(tools,wasm): resolve VL data through the library's VlResolver d345ffbf80

h5rs (dump, ls, diff, check --data) kept its own lenient VL decoder:
a heap object longer than its element was cut to the element's length
(libhdf5 and h5py refuse it), a null string printed "" where h5dump
prints NULL, the stored element size was trusted, and every heap
collection was kept as an owned copy for the whole run. It now resolves
each element with VlResolver::element / string_element (new: one element
in place, borrowing from the file), and refuses a VL type whose stored
element size is not 4 + offset size + 4, as File does. H5::heap_object
and its cache are gone. h5diff compares a null VL string equal to an
empty one; so does h5rs diff.

clawhdf5-wasm already resolved VL strings with read_vl_strings; it now
uses VlResolver and checks the stored element size before reading, as
File::read_string does.

Tests (h5py writes the files, patched for "a\0b", a null element and
mis-sized heap objects, with 8- and 4-byte offsets):
- h5rs_interop dump_prints_vl_data_like_h5dump: byte-identical to h5dump;
- dump_json_vl_values_match_h5py: h5py's values, errors where h5py fails;
- check_data_flags_mis_sized_vl_heap_objects;
- clawhdf5-wasm tests/vl_strings.rs: wasm, File and h5py agree.
All four fail before. check --data over the 150 cve_hdf5 CVE and fuzzer
files now passes 15 (h5dump rejects 8 of them), was 16 and 9: the
stored-size check flags cve-2024-32608. h5rs-check-ok-files.sh --data:
0 of 422 flagged; h5rs-fuzz.sh: clean on 180 files.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

docs: say when a selection read decodes more than the selection 45d617c39e

The READMEs said ds[...] reads only the selected elements, and the
facade's read_selection docs that only intersecting chunks are
decompressed. The bounding-box path runs only when the box covers at
most half the dataset; larger boxes (any strided slice across the
dataset), compact, virtual and unwritten datasets and chunked ones with
a non-default fill value decode the whole dataset. The READMEs, the
facade and format docs, the bindings' docstrings and known-issues now
say so, and how index lists are read.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

test: the v4 chunk-index selection test passes clippy -D warnings 8dcce084ca

A type alias for the hyperslab tuple, and as_chunks for the i32 decode.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

fix(format): a VL element at the undefined heap address is an error 5a202f3791

libhdf5 fails to read a VL element whose global heap address is
undefined (all 0xff), even at length 0 ("addr undefined"); we returned
"" (or an empty sequence) in every reader. Checked with h5py first:
libhdf5 writes a null element with address 0, which still reads as
empty, and h5py writes "" as a zero-size heap object at a real address,
so no file they write relies on the old behaviour. read_vl_bytes now
treats address 0 as null whatever the length, as VlResolver does.

Tests, each failing before: vl_data unit test (8- and 4-byte offsets,
lengths 0 and 1); clawhdf5 vl_data_interop
a_vl_element_at_the_undefined_heap_address_fails_like_h5py (also checks
where h5py writes ""); h5rs dump --json and check --data on the patched
`undef` dataset; clawhdf5-wasm vl_strings.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

Merge branch 'perf/p2-chunk-cache-scaling' into feat/p2-perf-coverage 20bd381c87

Merge branch 'perf/p2-contiguous-reads' into feat/p2-perf-coverage ca779b2864

# Conflicts:
#	CHANGELOG.md
#	docs/known-issues.md

Merge branch 'feat/p2-vl-strings' into feat/p2-perf-coverage 846c35455d

# Conflicts:
#	CHANGELOG.md

Merge branch 'feat/p2-writer-groups-links' into feat/p2-perf-coverage 956e55c76a

# Conflicts:
#	CHANGELOG.md
#	crates/clawhdf5-tools/tests/h5rs_interop.rs

Merge branch 'feat/p2-python-bindings' into feat/p2-perf-coverage 73a01f1256

# Conflicts:
#	CHANGELOG.md
#	README.md

docs: conformance report after the perf and coverage merges (575 of 697 ok) 408f69ec1d

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

bench: concurrent reads re-measured after the read fixes

CI / test-arm64 (pull_request) Successful in 1m34s

Details

CI / test (pull_request) Successful in 7m30s

Details

dda28d6c72

Idle tank at 408f69e, h5py re-run in the same session. Contiguous reads
went from 0.25x to 1.44x h5py (full) and 0.12x to 6.3x (256x256
hyperslabs) on one thread; deflate full reads at 8 threads 887 -> 2943
MB/s (h5py processes 3042). Full chunked reads at 16 threads are still
0.69x-0.76x h5py processes; the issue stays open.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

osobh merged commit de2a53f613 into main

2026-09-26 14:57:01 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: quantumclaw/clawhdf5#15