An attribute needing a heap block larger than the next one was refused
("skipping blocks too small for an object", "a first object too large for
the starting block"); once an object's move to dense storage was refused
it refused every new attribute, so 24% of set_attr calls in the review's
random workload failed.
Following H5HF__hdr_update_iter, H5HF__man_iblock_root_create/_double and
H5HF__hdr_skip_blocks, the smaller blocks are now skipped: the iterator
moves past them and they become an indirect free section with a first
row section (serialized, class 1, as H5HF__sect_indirect_serialize writes
it) and ghost normal rows, added as returned space so it merges with a
range skipped just before it (H5HF__sect_indirect_merge_row). Later
objects that best-fit a row section get a block created there
(H5HF__man_iblock_alloc_row / H5HF__sect_indirect_reduce_row: from the
start or end of the range, or from its middle, which splits it, with
libhdf5's span bookkeeping). Heaps with such sections, as libhdf5 writes
them, are now read too (they were refused at open).
dense_skipped_blocks_match_libhdf5 drives every path (merge, split, end,
last entry, row wrap) on earliest/v110/latest files against libhdf5
doing the same edits one session each; heaps, free sections and index
B-trees are equal after every phase. The refusal test now checks the
skip against libhdf5 and keeps a real refusal (last object in a block);
clawhdf5-written heaps get 1-4 KiB attributes too. The three tests fail
on the previous fheap.rs. Random workload refusals: 24% -> 2.2%, all the
documented last-object-in-a-block case.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
306 104 and 49 930 are h5repack of the editor's file; the text read as if
they were h5repack of libhdf5's, which measures 305 954 and 50 188. Both
are now given, from measure_append_waste rerun on 2026-09-26 (file sizes
unchanged).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Our checksum reduced its sums with `% 65535`; libhdf5's
H5_checksum_fletcher32 folds them with `(s & 0xffff) + (s >> 16)`, which
leaves 0xffff where the modulo leaves 0. On about one chunk in 32768
libhdf5 refused the chunks we wrote and we refused the chunks it wrote.
Every release since v2.1.0 is affected.
clawhdf5_format::checksum::fletcher32 is a port of H5_checksum_fletcher32
and the filter's only implementation. Verification also accepts the
byte-swapped form libhdf5 accepts (1.6.2 and earlier) and the `% 65535`
form earlier releases wrote, so their files stay readable.
The new interop test compares the checksum with libhdf5's own function
(ctypes) on every 1- and 2-byte input and 40 000 random and fold-heavy
inputs, and moves fold-case chunks between h5py and FileBuilder/FileEditor
in both directions; with the old filters.rs the three file tests fail.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
CHANGELOG, the clawhdf5-remote and h5rs READMEs and the remote-files
known issues: redirect rules, scaled timeouts (min_speed), URL redaction,
claimed lengths never allocated (download, --max-download), a 200 for a
small file accepted, and ObjectStoreStorage from any thread.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
It refused whenever Handle::try_current() was Ok, which is also the case
inside spawn_blocking threads — so the workaround its own error message
recommended failed the same way, and the backend could only be used from
a bare std::thread in a tokio application.
Reads are now spawned on the storage's own runtime and the caller waits on
a channel: the future never runs on the caller's thread, so neither a
spawn_blocking thread nor a current-thread runtime can deadlock or panic
(a read inside a runtime blocks that thread, like any blocking call; the
docs still recommend spawn_blocking there).
Tests: a read in spawn_blocking of a multi-thread runtime and a read inside
a current-thread runtime's task give File::open's values (both errors
before).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Only the full read split its chunk fetches into 64 MiB batches. The
selection path, the indexed read and the parallel_read decoders fetched
every chunk's stored bytes in one read_ranges call, each extent bounded only
by the file length, so a crafted chunk index pointing many chunks at one
large extent made File::open_storage hold chunks x extent bytes (3.3 GB from
a 16.8 MB file) before the first decode error.
- storage::for_each_extent_batch is now the one way raw-data reads fetch
chunk bytes: batches of at most RAW_BATCH_BYTES (now pub), each decoded
before the next is fetched. Used by the full, cached, indexed, selection
and parallel_read paths; the sweep read uses read_extent per chunk.
- ExtentReq carries each chunk's claimed extent (bounds-checked as before,
same errors) and the prefix actually fetched:
filters::stored_chunk_limit — the chunk size if unfiltered, else each
applied filter's worst-case growth (n + n/4 + 4096 per codec; unbounded
only for an application-registered codec). The in-memory path cuts the
slice it decodes the same way, so both paths still agree.
- tests/raw_fetch_bounds.rs: a crafted chunked_large.h5 (ten chunks all
claiming 20 MiB at one padding blob) read through every path over a
storage that records the largest single fetch; and 160 MiB of legitimate
unfiltered chunks fetched batch by batch. Before: one 80 MiB fetch
(selection) and one 160 MiB fetch; after: within the budget.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
README: "Reading remote files" (open_url, the range_server and read_url
examples with their real output against the fixtures, h5rs on URLs), the
crate in the crate map and the unreleased highlights. CHANGELOG: the
clawhdf5-remote crate, h5rs URLs, File::storage and
VlResolver::element_in, with the request counts over the conformance
corpus (tank, 2026-09-26, the command given). known-issues: the M2
range-read entry updated (the cache now exists; h5rs reads through
storage) and a new entry for the remote backends' limits (no Python or
browser URLs yet, fixed block size, cloud stores not run against a real
bucket, validators, credentials). Design doc: M3 status with the choices
that differ from the plan (a crate rather than a clawhdf5-io feature,
ureq for HTTP so the default build has no C) and the corpus counts.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
CHANGELOG (Unreleased): the new FileEditor operations, space reuse, and
the two reader fixes (implicit index grid, object-header continuation
chains). known-issues: the editor's remaining refusals (skipped heap
blocks, heaps with filters or child indirect blocks, freeing a heap
block, implicit-index insertions, ...) and the append-waste sizes before
and after reuse (measure_append_waste, tank 2026-09-26; file sizes are
deterministic). range-reads design: status note on the reader changes.
README and CLAUDE.md: what the editor covers and how to test it.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
CHANGELOG (Unreleased): File::open_storage, raw data and v2 B-trees over
Storage, the tests and their corpus results (2026-09-26, tank; conformance
600 of 697, results.json identical to 8f59b2e). known-issues: what
open_storage does not do yet (no remote backend or block cache, read_at
counts of a one-pass read, v1 group lookups, whole-file VDS sources,
zero-copy methods, SWMR growth, hash-order error choice on damaged chunked
datasets). Design: M2 status and the choices made.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
FileBuilder stored every chunk of an LZF or Blosc dataset through the
filter with filter mask 0. libhdf5 counts LZF and Blosc output no smaller
than the chunk as a failure of the optional filter and stores the chunk raw
with the filter's mask bit set. For an LZF chunk whose stream was exactly
the chunk's size, the first libhdf5 rewrite stored raw data at the same
size and kept our stale mask 0 in the index, and h5py could no longer read
the dataset.
precompress_chunks now runs chunks through compress_chunk_masked (as
FileEditor does since f7e2ab1), sequentially and on the parallel path, and
build_chunked_data_from_precompressed records each chunk's real mask in
every index the writer builds: single chunk (layout field), Fixed Array and
Extensible Array filtered elements, and version-2 B-tree type 11 records
(create_datasets_parallel goes through the same path). The writer builds
no version-1 B-tree or implicit index. PrecompressedChunks::chunks gains
the mask. Files whose chunks all compress are byte-identical.
Latent only in the unreleased LZF/Blosc writer (added 2026-09-26); no
tagged release writes either filter.
Tests:
- plugin_filters_interop skipped_optional_filters_are_masked_as_libhdf5_masks_them:
LZF, shuffle+LZF+fletcher32 and Blosc over random, compressible and
alternating chunks in every index; masks equal an h5py-written twin's;
h5py r+ rewrites and extends them; h5py, h5dump and our reader read
every value. Before: 20 of 24 datasets had masks other than h5py's, and
with that check disabled h5py failed to read the rewritten datasets
("filter returned failure during read").
- plugin_filters_interop files_whose_chunks_all_compress_are_unchanged:
pins the pre-fix bytes of five all-compressing files.
- chunked_write skipped_lzf_chunks_are_masked_in_every_index (fails before:
mask 0, want 2).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
- Per-file probe output is identical for 696 of 697 files, not all:
cve-2025-2310.h5's error string depends on which parallel chunk decode
fails first, at f2ff2c4 as on this branch.
- The parser cores are generic (S: Storage + ?Sized); provisional A/B
numbers against f2ff2c4, including the one bench that still shows
ObjectHeader::parse slower when old and new are separate binaries.
- Reads sized by untrusted fields are bounded; the harness only accepts
the known whole-file fallbacks.
- range-reads.md records why M1 went generic rather than &dyn.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A dataset whose filter this build cannot encode (scale-offset, N-Bit, SZIP;
a plugin filter the build lacks) failed with Error::Format("unsupported
filter: 6"), although the editor documents every refused edit as
Error::Unsupported, and the Python bindings raised ValueError rather than
NotImplementedError. Every edit now maps FormatError::UnsupportedFilter to
Error::Unsupported; the file is left untouched as before.
Test: edit_interop unencodable_filters_are_unsupported — h5py scale-offset
datasets (integer with chunks, integer never written, float D-scale):
Error::Unsupported naming the filter, and the file byte for byte unchanged.
Fails on the previous editor (Format(UnsupportedFilter(6))).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Three `chunk_info.address as usize` casts behind the `parallel` feature
survived the conversion, because check-32bit-casts.sh linted only default
features plus plugin-filters. On a 32-bit target with rayon a chunk address
past 4 GiB still wrapped onto another part of the file. They go through
addr::to_usize now, and the lane index (h % n, always < n) through
saturating_usize.
The script now lints no default features, default features, and every
optional feature but szip (wasm32; the set with zstd, which does not build
for wasm32, on the host, where the lint reports the same casts). With the
old parallel_read.rs/lane_partition.rs it fails listing the four casts; the
old script passed them. CHANGELOG and the design note give the exact count
(119) and what is not covered.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
known-issues records what the editor refuses, that freed space is never
reused (append-workload file sizes measured 2026-09-26 on tank with the
ignored measure_append_waste test; sizes are deterministic), and that there
is no journal.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Lists the parsers now reading through Storage, what still needs the
whole file (v2 B-tree-indexed structures: a clean error; raw data: M2),
the equivalence harness, and the evidence that nothing changed: existing
tests, a byte-identical conformance results.json and per-file probe
output against f2ff2c4, and identical slice-API transcripts over 748
files between the two builds.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
ZFP (32013) was the one plugin filter still listed as UnsupportedFilter.
Conformance on tank, `conformance/run.sh --no-fetch` (2026-09-26): 600 of
697 files ok (baseline 599); h5ex_d_zfp.h5 is newly ok.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Idle-start run on tank at c5334b1 (noisier than the last: compare ratios
within the run). Full reads of deflate data at 16 threads: 4944 MB/s vs
3135 for 16 h5py processes (1.58x; 0.69x-0.76x before). One thread with
the default pool: 6143 MB/s, 15x one h5py call. The concurrent-read
known issue is closed.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The scale-offset fix (d110b1d) was covered only by unit vectors from
CVE chunks, and documented as three corner cases. The review found it
is much bigger: of 1480 scale-offset datasets h5py writes (every
integer type i1..u8, f4 and f8, both byte orders, with and without a
fill value, scaleoffset 0..full width), v2.7.0's decoder read 332
differently from h5py: 151 returned wrong values with no error (82
integer datasets with scaleoffset=0 and a wide range, 51 full-width
i4/u4/i8/u8, 18 f4 D-scale datasets with a large range) and 181 failed
to read. The cause in every case is a chunk libhdf5 stores at full
width, whose elements were decoded as offsets from minval.
tests/scaleoffset_interop.rs generates that matrix with h5py at test
time, stores h5py's decoded values uncompressed next to it, and
compares every dataset's bytes. It passes on this branch; with the
filters.rs before d110b1d it reports "332 of 1480 scale-offset datasets
differ from h5py".
CHANGELOG: a Correctness entry stating this was silent wrong data in
every release that decoded scale-offset (v2.2.0 to v2.7.0), replacing
the corner-case wording. docs/known-issues.md: a fixed entry with the
affected cases.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
For a metadata cache image libhdf5 cannot load, libhdf5 opens the file
and fails the first metadata read (the image loads on the first
H5C_protect after open); h5py reports the error on the root group. The
conformance probe reported it that way, but File::open refused the file,
so the gate counted cve-2025-6269-1..4 and cve-2025-6516 as agreeing
with h5py for behaviour the library did not have.
The library now behaves as the probe reports: File (mmap, buffered and
from_bytes) and MmapFile open the file and every object lookup (dataset,
dataset_at, group, group listings and attributes, VL decoding) fails with
the image's error; LazyFile reads the root group's header at open, so
its open is that first read and fails. Probe and library take the
three-way decision (refuse at open / image loads / image cannot load)
from the same clawhdf5_format::superblock_ext::cache_image_state.
One deliberate difference from libhdf5 remains, documented: after the
failed first read libhdf5 reads the file's own metadata, which the image
was meant to replace and may be stale; here every lookup keeps failing.
File::cache_image_error / MmapFile::cache_image_error expose the error
to code that parses as_bytes() itself; h5rs checks it before reading any
object header (h5rs ls on cve-2025-6269-1 said "invalid object header
version: 0" from the stale bytes).
Test: metadata_cache_image.rs an_image_libhdf5_cannot_load_fails_every_object
(the fixture with its image signature broken; h5py opens that file and
fails the first read with "Bad metadata cache image header signature").
It fails on the previous commit, where File::open refuses the file.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Conformance on tank, conformance/run.sh --no-fetch (2026-09-26): 597 of
697 ok, 6 our-errors (4 corrupt objects HDF5 2.0 reads through a bug, the
Blosc2 and ZFP filters), 2 mismatches (the known h5py big-endian VL bug).
Closes the known-issues entries for metadata cache images,
cve-2024-32624, cve-2020-10810/10812, and unfiltered chunks of the wrong
size; the N-Bit / 64-bit scale-offset entry is recorded as not our bug.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
CHANGELOG entry for the chunked read changes, and the known-issues entry
on concurrent chunked reads updated: both causes it names (per-read page
faults, readers waiting on a small pool) are fixed; the 16-thread
comparison with h5py stays open until re-measured on an idle machine.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Record the `blosc2` feature in the changelog, the README's feature table
and the crate table, and mark the Blosc2 half of the known "Filters"
issue fixed (dated, with the conformance run that shows h5ex_d_blosc2
reading). What stays open: ZFP, writing Blosc2, and the Blosc2 features
hdf5plugin never writes (dictionaries, lazy chunks, variable-length
blocks, user-defined codecs and registered filters), which are errors.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
h5py's track_order=True orders attributes as well as links; the writer
tracked links only. A tracking object's header now sets the attribute
creation order tracked/indexed flags and carries per-message creation
orders, an Attribute Info message holds the next order (inline too),
and dense storage gets a type-9 creation-order index. The file default
applies to datasets, with DatasetBuilder::track_order per dataset; more
than 65 535 attributes on a tracking object is an error (libhdf5's
counter is 2 bytes). The reader lists such attributes in creation
order.
h5py lists them in order (inline, dense, 20 000 on one dataset) and
keeps numbering in r+ mode, including its inline-to-dense move.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Inventory of the 109 whole-file parser functions, a measurement of the
metadata ranges clawhdf5 touches on three corpus files against libhdf5,
options (storage trait, virtual slice, metadata prefetch, userfaultfd)
with how ros3, h5py+fsspec, pyfive, h5wasm and object_store do it, and
an incremental plan: M0 name-index lookups, M1 metadata over a Storage
trait, M2 raw data with batched ranges, M3 object_store backend,
M4 wasm fetch-driven lazy loading.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
inventory.py counts the functions and call sites that take the whole
file as &[u8]; range-trace (standalone crate, x86-64 Linux) records every
load clawhdf5 makes from a file by mprotect + single-step, unchanged
library code; libhdf5_reads.py counts libhdf5's reads through h5py's
fileobj driver and prints a dataset's chunk extents.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Dense link and attribute indexes and the chunk index for several
unlimited dimensions were single leaves, capping them at 65 535
records. btree_v2_write builds trees of any depth, with node capacities
and pointer widths from libhdf5's H5B2__hdr_init arithmetic (now shared
with the reader as btree_v2::node_info) and libhdf5's node sizes (512
dense, 2048 chunks). Indexes that fit the old one-leaf layout are
written byte for byte as before (compared for 10..65 535 links, attrs
and chunks, tracked and filtered).
Tests: 100 000 links (short names; long names with creation order),
70 000 attributes, 200 000 chunks (and 80 000 deflated), read by h5py,
h5dump and clawhdf5 and edited by h5py r+; h5rs check on the same
shapes, asserting depths 2-3.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Idle tank at 408f69e, h5py re-run in the same session. Contiguous reads
went from 0.25x to 1.44x h5py (full) and 0.12x to 6.3x (256x256
hyperslabs) on one thread; deflate full reads at 8 threads 887 -> 2943
MB/s (h5py processes 3042). Full chunked reads at 16 threads are still
0.69x-0.76x h5py processes; the issue stays open.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
libhdf5 fails to read a VL element whose global heap address is
undefined (all 0xff), even at length 0 ("addr undefined"); we returned
"" (or an empty sequence) in every reader. Checked with h5py first:
libhdf5 writes a null element with address 0, which still reads as
empty, and h5py writes "" as a zero-size heap object at a real address,
so no file they write relies on the old behaviour. read_vl_bytes now
treats address 0 as null whatever the length, as VlResolver does.
Tests, each failing before: vl_data unit test (8- and 4-byte offsets,
lengths 0 and 1); clawhdf5 vl_data_interop
a_vl_element_at_the_undefined_heap_address_fails_like_h5py (also checks
where h5py writes ""); h5rs dump --json and check --data on the patched
`undef` dataset; clawhdf5-wasm vl_strings.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The READMEs said ds[...] reads only the selected elements, and the
facade's read_selection docs that only intersecting chunks are
decompressed. The bounding-box path runs only when the box covers at
most half the dataset; larger boxes (any strided slice across the
dataset), compact, virtual and unwritten datasets and chunked ones with
a non-default fill value decode the whole dataset. The READMEs, the
facade and format docs, the bindings' docstrings and known-issues now
say so, and how index lists are read.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
h5rs (dump, ls, diff, check --data) kept its own lenient VL decoder:
a heap object longer than its element was cut to the element's length
(libhdf5 and h5py refuse it), a null string printed "" where h5dump
prints NULL, the stored element size was trusted, and every heap
collection was kept as an owned copy for the whole run. It now resolves
each element with VlResolver::element / string_element (new: one element
in place, borrowing from the file), and refuses a VL type whose stored
element size is not 4 + offset size + 4, as File does. H5::heap_object
and its cache are gone. h5diff compares a null VL string equal to an
empty one; so does h5rs diff.
clawhdf5-wasm already resolved VL strings with read_vl_strings; it now
uses VlResolver and checks the stored element size before reading, as
File::read_string does.
Tests (h5py writes the files, patched for "a\0b", a null element and
mis-sized heap objects, with 8- and 4-byte offsets):
- h5rs_interop dump_prints_vl_data_like_h5dump: byte-identical to h5dump;
- dump_json_vl_values_match_h5py: h5py's values, errors where h5py fails;
- check_data_flags_mis_sized_vl_heap_objects;
- clawhdf5-wasm tests/vl_strings.rs: wasm, File and h5py agree.
All four fail before. check --data over the 150 cve_hdf5 CVE and fuzzer
files now passes 15 (h5dump rejects 8 of them), was 16 and 9: the
stored-size check flags cve-2024-32608. h5rs-check-ok-files.sh --data:
0 of 422 flagged; h5rs-fuzz.sh: clean on 180 files.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The changelog, known issues and README said a group holds up to 65 535
links while a group of about 17 000 was already unreadable. Record the
fixes (child indirect blocks, the next-block offset, the index leaf cap,
refusing oversized dense messages, hard-link memoisation, dataset
attribute overwrite) and the limits that remain true: 65 535 links or
dense attributes per object, and 65 515 bytes per dense message.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
VlResolver kept an owned copy of every object of every heap collection
it parsed, for the whole read. Collections nested inside each other's
object data, 32 bytes apart with each element pointing at a different
one, made retained memory O(elements x file size): 1.58 GB for a 744 KB
file (read_vl_strings did the same before VlResolver). Chaining every
collection's objects into one shared run of tiny objects made parse
time O(elements x objects) as well. libhdf5 refuses these files.
- The cache records where each object lies (GlobalHeapCollection::
parse_index, new) instead of copying it, and is dropped past a 32 MiB
budget.
- A collection overlapping one already read is an error: libhdf5 gives
every collection its own block, so only a crafted file has them.
- parse and parse_index refuse a collection that runs past the end of
the file and an object that runs past the end of its collection.
tests/vl_heap_bounds.rs measures peak heap use with a counting
allocator: 129 MB and 350 MB live before on its two crafted files (64 KB
and 176 KB), 97 KB and 0.9 MB now. Conformance unchanged at 575 of 697.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The review measured the default pool unchanged (about 2900 MB/s at 16
threads before and after) and still short of 16 h5py processes; small
pools still make outside readers wait. Say so instead of marking the
scaling issue fixed.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Full reads of chunked datasets handed their chunks to rayon. With a
one-thread pool (concurrent_read --decode-threads 1, RAYON_NUM_THREADS=1)
every thread reading through a File queued behind that single worker, so
16 readers decoded on one core: per-thread CPU time showed one thread
doing all the decoding and the readers almost none, and full reads
stopped at about 2x one thread. The cached full-read path and the
uncached reader behind verify_provenance now decode inline when the pool
cannot parallelise (parallel_read::pool_can_parallelise).
The File's chunk cache was the suspect but not the cause: datasets over
its budget were already read without inserting, and skipping its lookups
gained only a few percent at 16 threads.
The regression test keeps a one-thread global pool's worker busy and
requires a full read and verify_provenance to finish anyway; before the
fix both waited for the worker (timed out).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
FileWriter wrote the root group plus one level of groups, and refused
path-like names. The writer now flattens its builders into a group tree
(writer_tree.rs) before layout:
- A name may be a path ("a/b/x", "/a/b/x" at the root); missing
intermediate groups are created as h5py does, and GroupBuilder gains
create_group/add_group so builders nest to any depth. A group added at a
path that already holds a group is merged into it (require_group);
any other repeated name, an empty or "." component, or an absolute path
below the root is an error.
- add_soft_link, add_hard_link and add_external_link on FileWriter,
FileBuilder and GroupBuilder. Hard-link targets are resolved to objects
at finish (through other hard links; a missing target, a soft link on the
way or a cycle of paths is an error). Objects with several hard links get
an Object Reference Count message so libhdf5 can delete one link without
freeing the object.
- track_order(true) per group, or as the file default, tracks and indexes
link creation order: Link Info flags and max order, the order in each
Link message, and a type-6 creation-order B-tree for dense groups.
- A group's link index is one B-tree leaf; more than 65535 links is an
error.
Groups are laid out depth-first from the root, datasets group by group,
and untracked groups keep writing datasets, then groups, then other links:
files with one level of groups are byte-identical to before.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
VL-string datasets (h5py's default str dtype) failed read_string with
"type mismatch: expected String, got VariableLength". read_string now
reads fixed- and variable-length strings, with h5py's values (a string
ends at a NUL, a null element is ""). New:
- Dataset::read_string_bytes: each VL string's exact bytes;
- Dataset::read_string_selection: hyperslabs/points of either kind;
- Dataset::read_vlen::<T>() and read_vlen_selection::<T>(): VL sequences
of numbers as Vec<Vec<T>>, T in f64/f32/i64/i32/u64, converted like the
other typed readers;
- File::decode_strings / decode_string_bytes / decode_vlen: VL values in
compound fields and AttrValue::Raw attributes;
- MmapDataset and LazyDataset: read_string for VL strings,
read_string_bytes and read_vlen.
tests/vl_data_interop.rs checks every path against h5py with 8- and
4-byte offsets: scalar, 1-D and 2-D, ASCII and UTF-8, empty strings,
contiguous, compact, chunked with gzip and shuffle, unwritten and partly
written chunks, hyperslabs, compound members, attributes, a big-endian
base type, and a patched file with an embedded NUL and mis-sized heap
objects. NetCDF-4 string variables read too (netCDF4-python test).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
osobh