It refused whenever Handle::try_current() was Ok, which is also the case
inside spawn_blocking threads — so the workaround its own error message
recommended failed the same way, and the backend could only be used from
a bare std::thread in a tokio application.
Reads are now spawned on the storage's own runtime and the caller waits on
a channel: the future never runs on the caller's thread, so neither a
spawn_blocking thread nor a current-thread runtime can deadlock or panic
(a read inside a runtime blocks that thread, like any blocking call; the
docs still recommend spawn_blocking there).
Tests: a read in spawn_blocking of a multi-thread runtime and a read inside
a current-thread runtime's task give File::open's values (both errors
before).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
timeout_global (60 s) covered a whole request, and a request can carry
8 MiB (max_request): below about 140 KB/s every block run timed out, was
retried from scratch and failed, so a slow link could not read remote
files at all.
HttpOptions::timeout (now 30 s) bounds connecting and receiving the
response headers; the body gets timeout + its size at the new
HttpOptions::min_speed (16 KiB/s by default: 94 s for a 1 MiB block).
A slow but moving link is not cut off; a stalled one still fails.
(ureq has no idle timeout; its body timeout is a total budget.)
The test server can throttle bodies and stall mid-body. Test: a 256 KiB
block at 256 KiB/s reads with a 300 ms timeout (it failed before), and a
body stalled for 20 s fails in under 5 s.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The first request asks for bytes=0-1048575. RFC 9110 lets a server answer
200 when the range covers the whole representation, so a file under
1 MiB on a server that does support ranges could be refused as 'does not
support range requests'. A 200 whose Content-Length (or, without one, its
body, read at most that far) is within the range asked for is now kept as
the whole file and read from memory; a longer one is still refused unless
allow_full_download is set.
Test: a 9968-byte file served with 200 opens in one request with the
transcript of File::open (it was refused before); with a 4096-byte first
request it is still refused.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
ureq's defaults followed up to 10 redirects, including from https to plain
http, and forwarded the custom HttpOptions::headers (X-Api-Key, Cookie,
...) to whatever host a redirect named — only Authorization was stripped.
HttpStorage now follows redirects itself (ureq's max_redirects is 0):
- at most HttpOptions::max_redirects per request (default 5; 0 refuses
any redirect), then RemoteError::Redirect;
- never from https to another scheme, nor to a non-http(s) URL;
- once a redirect leaves the URL's origin (scheme, host, port), none of
the custom headers is sent any more (Authorization included);
- each hop counts as a request; errors show the target redacted.
Tests: a redirect to another local port reads the right data and the
target never sees X-Api-Key or Authorization (it did before); a
same-origin redirect keeps them; a loop stops after 6 requests; 0 refuses;
unit tests for target resolution, the https downgrade and origins.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Every RemoteError message and HttpStorage's Debug output held the URL as
given, with any user:password@ and the query string — for a presigned
S3/GCS/Azure URL, its signature or token. An application logging the
error leaked the credential.
- New clawhdf5_remote::redact_url: no userinfo, no fragment, query values
replaced by REDACTED (plain key names kept).
- HttpStorage formats every message with the redacted URL, and scrubs the
URL's secret parts from errors of the HTTP client (whose texts can echo
the URI); Debug shows the redacted URL. storage_for_url's and the object
store URL errors are redacted too. HttpStorage::url() still returns the
URL as given, documented as not for logging.
- h5rs prints FILE arguments that are URLs redacted: in errors and in
dump/stat/check/diff output.
- The test server can force a status and send a wrong Content-Range.
Tests: 404, 403 (at open and on a read), wrong Content-Range (at open and
on a read), no range support, encoded body, ETag change, timeout,
connection closed and bad scheme errors, Display and Debug, contain none
of the secrets; h5rs likewise for every subcommand.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
h5rs check URL read the whole file with one read_at(0, len), len being
whatever Content-Range said. BlockCache listed every block index of the
span and preallocated len bytes: a server claiming 2^62 bytes for a 10 KB
file made h5rs abort (memory allocation of 35184372088832 bytes failed).
- BlockCache: a read spanning more than the budget (or eight max_requests)
is fetched piece by piece and not kept, its output growing only as
data arrives; read_ranges falls back to that per range; prefetch is
clamped to the budget.
- New clawhdf5_remote::download(storage, max_bytes): refuses a claimed
length above the limit (RemoteError::TooLarge) before any request, then
reads in 64 MiB steps. New RemoteError::Backend for read errors.
- h5rs check downloads through it, with --max-download N (default 1 GiB).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A server can claim any length in Content-Range. block_len computed
start + block_size, which overflowed in the last blocks of a file claimed
to be near u64::MAX (a panic in debug builds, a wrapped value in
release); insert() multiplied block indices unchecked. The cache's block
arithmetic is now saturating/checked, and a run that does not split into
whole blocks is an error instead of an endless loop or a slice panic.
The test server gains fake_total (claim a length, serve zeros past the
data); a test reads the last bytes of such files and opens a file whose
superblock EOF and root addresses sit near u64::MAX.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A local port scanner's GET / reached the test listeners and was counted,
failing the exact request budgets (and consuming injected 503s). Requests
for paths the server does not serve are now answered 404 without being
counted, delayed or failed; the query string is not part of the path.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A reader that waited for another reader's fetch of a block got "the
fetch of this block failed" when that fetch failed, not why: a file
replaced on the server while open was reported as FileChanged to one
thread and as an anonymous failure to the others. The fetch's error is
now handed to every reader waiting on it.
Regression test: four threads read the same block from a slow backend
whose fetches fail with a "changed while open" error; each gets that
error (it failed for the waiters before this change).
Also fixes the ignore-Range test, broken by the previous commit: the
test server now counts a body before sending it, so "the refused body
was not read" is checked as "refused at the first response".
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
README: "Reading remote files" (open_url, the range_server and read_url
examples with their real output against the fixtures, h5rs on URLs), the
crate in the crate map and the unreleased highlights. CHANGELOG: the
clawhdf5-remote crate, h5rs URLs, File::storage and
VlResolver::element_in, with the request counts over the conformance
corpus (tank, 2026-09-26, the command given). known-issues: the M2
range-read entry updated (the cache now exists; h5rs reads through
storage) and a new entry for the remote backends' limits (no Python or
browser URLs yet, fixed block size, cloud stores not run against a real
bucket, validators, credentials). Design doc: M3 status with the choices
that differ from the plan (a crate rather than a clawhdf5-io feature,
ureq for HTTP so the default build has no C) and the corpus counts.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The per-file report now separates a tree view (open, every group's
entries, every dataset's shape and type) from reading the largest
dataset under 64 MiB, and checks the budget the design's testing section
asks for: listing the IMERG file (file A of docs/design/range-reads.md
section 2) takes at most 3 requests when CLAWHDF5_REMOTE_CORPUS includes
it. The test server now counts a response's bytes before sending it: a
client could read a body and reset the counters before the server thread
had added it, so the counts of the next file were occasionally too high.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
With the `remote` feature (`remote-https` for https://), ls, dump, stat
and diff take an http(s):// (or s3://, gs://, az:// with those
clawhdf5-remote features) URL wherever they take a file, and read it by
range requests through clawhdf5-remote's block cache. check validates
every byte, so it downloads a remote file whole and checks it as before.
Without the feature a URL is a clean error naming it.
The tools read the file through File::storage instead of as_bytes: object
headers, shared messages, attributes, v1 and v2 group links, dense
storage (fractal heaps and v2 B-trees), path resolution, chunk listings
and variable-length values go through the format crate's *_in functions,
and the fractal-heap block verifier reads each block through the storage
(a read failure of a remote file is reported as a problem, not as "past
the end of the file"). A local file's storage is its mapped bytes, so its
reads are still slices. stat's file size comes from the opened file, so
it is right for a URL.
Tests: tests/remote.rs serves fixtures (old and new formats, a paged
file, a metadata cache image, a multi-block fractal heap, compounds, v1
groups) with the clawhdf5-remote test server and requires every
subcommand's output and exit status for the URL to equal the local
file's, and diff of the two to be clean; 404s, non-HDF5 bodies and
https without its feature are clean errors. Local output is unchanged:
the old and new h5rs print the same for ls -r -v, dump, stat and check
--data on the 747 conformance and CVE corpus files (tank, 2026-09-26;
the dumps of h5diff_hyper1/2.h5 were too large for the comparison
script, their ls, stat and check agree), except cve-2025-2310.h5, whose
dump error messages differ between runs of the old binary too (which
failing chunk is reported first).
ci-test.sh lints h5rs with remote-https, runs the URL tests and checks
h5rs with remote for C.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
VlResolver::element and string_element return slices of the whole file,
so they exist only for a resolver over &[u8]. Their *_in forms work for
any Storage (a remote file): the element's bytes borrowed from the
resolver's cache of heap collections, with the same null-element, NUL
and size checks. h5rs decodes variable-length values with them.
Test: over a read_at-only storage they give what element/string_element
give over the slice, for a string with an embedded NUL, a null element
and an element whose heap object has the wrong size.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The bytes File::as_bytes returns (from the superblock on, bounded by the
recorded end of file, a metadata cache image laid over), as a
&(dyn Storage + Send + Sync) for every backend. Code that parses the file
itself with the clawhdf5_format *_in functions — h5rs does — can then
read a file opened with File::open_storage (a remote file) as well as a
local one; in memory its as_contiguous() is as_bytes(), so local reads
stay slices.
Test: for every fixture, File::open's storage() is as_bytes() as its
contiguous view, and File::open_storage over a read_at-only storage gives
the same bytes through storage().read_at.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
ObjectStoreStorage (feature `object-store`, pure Rust) reads one object
of any object_store store by ranged get_opts, pinned at open by a head
request: If-Match with its ETag (and the ETag and size of every response
compared), else its version or modification time. A change is
RemoteError::FileChanged. object_store is async and Storage is not, so
the storage owns a small multi-threaded tokio runtime (two workers) and
blocks the calling thread on it; the ranges of one read_ranges call are
fetched concurrently (up to 8). From inside another tokio runtime it
refuses with RemoteError::Usage instead of blocking a worker, and it
shuts its runtime down in the background on drop so dropping it in async
code does not panic.
open_object(store, path, options) opens a file through a block cache
(first block prefetched); open_url accepts s3://, gs:// and az:// with
the `s3`, `gcs` and `azure` features, configured from the environment by
object_store's from_env builders. Those pull object_store's cloud clients
and aws-lc-rs (C), so they are opt-in; without them the URL is a clean
UnsupportedScheme error naming the feature.
Tests against object_store's in-memory and local-file stores (no cloud):
every fixture's transcript equals File::open's, a multi-block object is
fetched in coalesced block runs, an object replaced while open is an
error, and a missing object or a read from inside a runtime is a clean
error. ci-test.sh lints all backends, runs these tests (with s3 for its
URL parsing test) and checks object-store for C in the no-C step.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Range-read milestone M3, first half: a new crate with the block cache the
design makes mandatory for remote files and an HTTP backend, so
open_url("http://...") gives a clawhdf5::File over File::open_storage.
BlockCache wraps any Storage: aligned blocks (1 MiB by default, the size
docs/design/range-reads.md section 2 measured), LRU with a byte budget,
the missing blocks of one read_at/read_ranges fetched with one backend
read_ranges call as runs of consecutive blocks (a one-block gap filled to
merge runs, each request at most 8 MiB), and reads that miss more than
half the budget not kept. Thread-safe without holding the lock across a
fetch: a block being fetched is in flight, a second reader waits for it
instead of fetching it again, and a failed fetch fails its waiters and is
not cached. A backend holding the file in memory passes through.
HttpStorage (ureq, no TLS by default; `https` adds rustls with ring):
opening is one ranged GET of the first block, whose Content-Range gives
the length (the cache keeps the bytes). The file is pinned by a strong
ETag (If-Match), else Last-Modified (If-Unmodified-Since), and its length,
checked on every response: a change is RemoteError::FileChanged, never
mixed data. A server that ignores Range is refused without reading the
body unless a full download is allowed. Connection errors, timeouts,
408/429/5xx and short bodies are retried with exponential backoff;
Accept-Encoding: identity, and an encoded body is refused. read_ranges
fetches its ranges in parallel.
Tests (a std-only HTTP/1.1 server in tests/common/server.rs, also the
range_server example): every fixture read over HTTP gives File::open's
transcript (CLAWHDF5_REMOTE_CORPUS adds the conformance corpus), with
request counts per file with and without the cache; an h5py-written file
against libhdf5's values; a multi-block file fetched in whole blocks, each
once; a server ignoring Range; a file replaced mid-read (ETag,
Last-Modified, length only); truncated bodies and 503s (retried, then an
error, never cached); a slow server with 8 concurrent readers (no block
fetched twice); bad URLs, 404, encoded bodies, non-HDF5 data. The cache
has unit tests for coalescing, splitting, LRU order, large reads,
failures and concurrent in-flight dedup.
ci-test.sh: clawhdf5-remote joins the no-C default-build check, and its
https feature is linted.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
CHANGELOG (Unreleased): File::open_storage, raw data and v2 B-trees over
Storage, the tests and their corpus results (2026-09-26, tank; conformance
600 of 697, results.json identical to 8f59b2e). known-issues: what
open_storage does not do yet (no remote backend or block cache, read_at
counts of a one-pass read, v1 group lookups, whole-file VDS sources,
zero-copy methods, SWMR growth, hash-order error choice on damaged chunked
datasets). Design: M2 status and the choices made.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
File::open_storage(Arc<dyn Storage + Send + Sync>) opens a file served by
any backend: groups, datasets, attributes, read_*, selections, VL data
and virtual datasets (external sources through the new
File::set_vds_resolver) all read through Storage::read_at/read_ranges.
The file's view (user block skipped, bounded by the recorded end of
file) is itself a Storage; File::open and File::from_bytes keep their
mmap and in-memory paths, now as that view's as_contiguous() fast path.
A storage-backed file's metadata cache image is laid over each read it
covers (new CacheImage::entries), as libhdf5 loads it.
The typed readers keep their fast paths over any storage: a contiguous
dataset is read in one piece and converted (read_f64 and friends), and a
contiguous native selection reads only its runs
(data_read::read_selection_native_in). Zero-copy methods answer
ContiguousStorageRequired when the bytes are not in memory, and
File::as_bytes panics there (File::contiguous_bytes is the fallible
form).
Test: tests/storage_equivalence.rs reads every fixture, and with
CLAWHDF5_STORAGE_CORPUS every conformance-corpus file, through File::open
and through open_storage over a read_at-only CountingStorage — tree,
attributes, and every dataset's values several ways — and requires
identical transcripts; it prints the read_at calls and bytes a one-pass
read costs per file.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Every raw-data path has a generic *_in core, with the &[u8] functions as
thin wrappers: data_read (read_raw_data*, read_raw_data_selection,
read_chunked_native), chunked_read (the v1 B-tree chunk index, list_chunks,
the full, cached, sweep and indexed reads), parallel_read, partial_read,
fill_value (read_full_with_fill, apply_to_unallocated_chunks; and
dataset_fill_value_from_storage is now generic), vds (the virtual file
through Storage, external sources still through the resolver),
vl_data (VlResolver<'a, S = [u8]>, read_vl_strings_in, read_vl_bytes_in),
AttributeMessage::read_vl_strings_in and provenance::verify_dataset_in.
With the whole file in memory nothing changes: chunks and contiguous data
are sliced from it as before. Otherwise a chunked read lists its chunks,
fetches their stored bytes with one Storage::read_ranges call per 64 MiB
batch (chunks the cache already holds are not fetched), then decodes as
today; a selection fetches only the chunks it overlaps, and a contiguous
selection only its runs. Each extent's bounds error is the one the slice
code gave, reported when that extent is reached, so errors keep their
order.
Tests: the equivalence harness now reads every dataset's values (whole,
fill-aware, cached, indexed, three selections, VDS, VL strings and
sequences) through the read_at-only storage and requires the slice
results (all 653 corpus files agree); a misbehaving storage (a failing
Nth read, short reads) only ever yields errors or the right values; and
chunked reads are checked to use one read_ranges call.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
BTreeV2Header::parse_in, collect_btree_v2_records_in and
find_btree_v2_records_in read one bounded window per node (its size is
known from the parent before the node is read; a count stretched past
node_size is checked against the end of the file first), with the
whole-file bounds errors unchanged. With them, dense attributes, a SOHM
B-tree index and huge fractal-heap objects no longer answer
ContiguousStorageRequired, and group_v1/group_v2 listings, lookups and
path resolution get *_in cores (resolve_group_children_in,
resolve_child_in, resolve_path_any_in, ...). The &[u8] functions are
thin wrappers, as in M1.
The equivalence harness now fails on any ContiguousStorageRequired and
compares v2 B-tree headers, records and descents, group listings, child
lookups and paths; a unit test compares a two-level tree through a
read_at-only storage truncated at every length and with every node byte
flipped.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
+6-7% (about 4 ns per header) is real; symbol-table nodes -17%, group
B-tree walk -16%, facade listing -2.4%: local metadata reads are net
slightly faster than main.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Regenerated on tank: ok 599 -> 600 (h5ex_d_zfp.h5), our-error 4 -> 3,
mismatch 2, no panics, hangs, crashes or OOM. Baseline raised.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
FileBuilder stored every chunk of an LZF or Blosc dataset through the
filter with filter mask 0. libhdf5 counts LZF and Blosc output no smaller
than the chunk as a failure of the optional filter and stores the chunk raw
with the filter's mask bit set. For an LZF chunk whose stream was exactly
the chunk's size, the first libhdf5 rewrite stored raw data at the same
size and kept our stale mask 0 in the index, and h5py could no longer read
the dataset.
precompress_chunks now runs chunks through compress_chunk_masked (as
FileEditor does since f7e2ab1), sequentially and on the parallel path, and
build_chunked_data_from_precompressed records each chunk's real mask in
every index the writer builds: single chunk (layout field), Fixed Array and
Extensible Array filtered elements, and version-2 B-tree type 11 records
(create_datasets_parallel goes through the same path). The writer builds
no version-1 B-tree or implicit index. PrecompressedChunks::chunks gains
the mask. Files whose chunks all compress are byte-identical.
Latent only in the unreleased LZF/Blosc writer (added 2026-09-26); no
tagged release writes either filter.
Tests:
- plugin_filters_interop skipped_optional_filters_are_masked_as_libhdf5_masks_them:
LZF, shuffle+LZF+fletcher32 and Blosc over random, compressible and
alternating chunks in every index; masks equal an h5py-written twin's;
h5py r+ rewrites and extends them; h5py, h5dump and our reader read
every value. Before: 20 of 24 datasets had masks other than h5py's, and
with that check disabled h5py failed to read the rewritten datasets
("filter returned failure during read").
- plugin_filters_interop files_whose_chunks_all_compress_are_unchanged:
pins the pre-fix bytes of five all-compressing files.
- chunked_write skipped_lzf_chunks_are_masked_in_every_index (fails before:
mask 0, want 2).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
- Per-file probe output is identical for 696 of 697 files, not all:
cve-2025-2310.h5's error string depends on which parallel chunk decode
fails first, at f2ff2c4 as on this branch.
- The parser cores are generic (S: Storage + ?Sized); provisional A/B
numbers against f2ff2c4, including the one bench that still shows
ObjectHeader::parse slower when old and new are separate binaries.
- Reads sized by untrusted fields are bounded; the harness only accepts
the known whole-file fallbacks.
- range-reads.md records why M1 went generic rather than &dyn.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
On a backend without the file in memory, a structure read whose length
comes from untrusted header fields was clamped only by the end of the
file, so a crafted size made one read (and copy) of up to the rest of
the file. Each such read now covers what the parser actually uses:
- local heap names: read in growing pieces (64 bytes first, then 4x)
up to the end of the data segment, instead of the rest of the segment
per name (quadratic for a big symbol-table group);
- fractal heap indirect blocks: the doubling-table geometry locates the
entry covering the object, and the first read ends at that entry; only
if it is unallocated does the walk read the rest of the block (it
visits every entry then). One walk implementation serves both;
- paged fixed/extensible array data blocks over 1 MiB: the prefix and
page bitmap, then each page in use on its own (smaller blocks are
still one read);
- blocks under one checksum (non-paged array data blocks, extensible
array index and super blocks): the bounds check that comes first (the
checksum's; the page bitmap's for a super block) is made against the
file length before reading (Window::check_extent), so a block claimed
past the end of the file costs no read. With the checksum feature off
the parser has no such first check and the old read stands.
Other windows were already bounded (the superblock and object header
prefixes, the fractal heap header by a u16, SOHM tables by u8/u16
counts) or are exact reads checked against the file length first.
In memory nothing changes: the pieces are borrowed slices.
Tests: CountingStorage over a crafted heap (16 MiB file, width and rows
0xFFFF: under 1 KiB read, 16.7 MB before), a heap segment claiming 64 MiB
(one 64-byte read per short name), long names at every piece boundary,
a fixed array block claimed past the end of a 16 MiB file (under 64
bytes read), and in the equivalence harness an h5py file with a 2.4 MB
fixed array block and a >1 MiB extensible array block, whole and cut at
97 points: every chunk index agrees with the slice read and the largest
takes 205 KB (2.4 MB and 1.2 MB when read whole).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The harness counted ContiguousStorageRequired from any parser as an
allowed difference, so a converted module that wrongly fell back to the
whole file would still pass. It now accepts the error only from the
three sites that are not converted yet (dense attribute storage, a SOHM
B-tree index, huge fractal-heap objects, all found through a v2 B-tree)
and only in the checks that can reach them; anything else fails with
the check and the site named.
Checked by making LocalHeap::parse_in return the error first: the
fixture and h5py runs fail ("local heap fell back to the whole file").
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Every `*_in` core and the read helpers take `file: &S` with
`S: Storage + ?Sized` instead of `&dyn Storage`, and the `&[u8]`
wrappers pass the slice itself, so they compile to a `[u8]` instance:
`as_contiguous()` inlines to `Some(self)` and each structure read is the
slice code's bounds check again, with no indirect call. `&dyn Storage`
still works (`S = dyn Storage`); there is one parser implementation.
Also, so the structure reads cost no more than the slice checks did:
- ObjectHeader::parse_in reads the prefix once (signature included)
instead of the signature and then the prefix: two reads for a
one-chunk header instead of three on a range backend;
- the symbol-table node and group B-tree (v1) loops walk their entries
with chunks_exact over the bytes read, and the node's redundant second
bounds check is gone (the entries' read is the check, same error);
- a version-1 header's message list is sized from its (capped) count.
Same results and errors; the unit and equivalence tests are unchanged.
New Criterion bench `clawhdf5/benches/local_metadata_bench.rs` over a
400-group version-1 file written by h5py (new fixture
`v1_groups_400.h5`): ObjectHeader::parse, symbol-table nodes, the group
B-tree walk and a facade listing, using only APIs that exist at f2ff2c4
so it builds there for an A/B.
Provisional A/B against f2ff2c4 (busy machine, not for docs): both
builds linked into one binary and timed in alternation, 200 rounds;
median ratio new/old: facade listing -0.5% to -3.5% (was +14%),
ObjectHeader::parse +1% to +2% (was +25%), symbol-table nodes -18%,
group B-tree walk -18%, local-heap names and resolve_group_children
within +-1.5%. An old-vs-old-copy run shows +-2% from code layout alone.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Each edit planned over the reader's memory map of the file and committed
while that File, and the Image's &[u8] over the mapping, were still alive,
writing the same file through the editor's descriptor. Nothing read the
mapping during the writes, but a shared slice whose memory changes
underneath it is undefined behaviour under Rust's aliasing rules.
Image::into_plan now detaches the edit's writes (patches, end of
allocation) into a Plan that owns all of its bytes and borrows nothing;
edit() takes the user-block size, drops the File — unmapping the file —
and only then commits the Plan. The invariant is documented in the image
module and the editor's module docs.
Test: edit::tests::file_is_not_mapped_while_an_edit_writes_it checks
/proc/self/maps at the moment each commit starts (write, resize, set_attr):
never mapped. With the commit moved back before the reader is dropped (the
previous order) it reports all three commits with the file mapped.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A dataset whose filter this build cannot encode (scale-offset, N-Bit, SZIP;
a plugin filter the build lacks) failed with Error::Format("unsupported
filter: 6"), although the editor documents every refused edit as
Error::Unsupported, and the Python bindings raised ValueError rather than
NotImplementedError. Every edit now maps FormatError::UnsupportedFilter to
Error::Unsupported; the file is left untouched as before.
Test: edit_interop unencodable_filters_are_unsupported — h5py scale-offset
datasets (integer with chunks, integer never written, float D-scale):
Error::Unsupported naming the filter, and the file byte for byte unchanged.
Fails on the previous editor (Format(UnsupportedFilter(6))).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
libhdf5 counts a version-2 object header's attributes through its Attribute
Info message (0x15) and reports none when the header has none. set_attr gave
v110/latest groups, the root group and datasets without attributes an
attribute message only, so h5py listed the attribute but len(obj.attrs) and
H5Oget_info's num_attrs said 0, and stayed wrong after h5py r+ added more.
Like H5O__attr_create, the edit now adds the message when a version-2 header
lacks it, in the same planned edit: version 0, the header's creation-order
track/index flags, maximum creation index 0, undefined fractal heap and
B-tree addresses, message flag DONTSHARE — byte for byte what libhdf5
writes. It goes before the attribute (libhdf5's order) when free space
holds both, else after it, so a continuation chunk made for the attribute
also takes it.
Test: edit_interop attribute_count_in_version_2_headers — v110 and latest
files, attributes set on the root group, groups and datasets with and
without existing attributes: h5py's len/num_attrs/list/values, h5dump -A
and our reader agree, also after h5py r+ adds attributes up to and past the
compact limit. Fails on the previous editor (h5py len 0).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A valid group has one link per name, but a damaged or hand-made one can
have two. resolve_child followed the first soft link of the name, the
listing skipped a dangling one and listed the name via a later link, and
path resolution followed the last symbolic link: three answers. All now
take the first link of the name (header message order in a compact group,
name index order in a dense one) and ignore the rest, even if the first
dangles. That is libhdf5's rule for compact groups (H5G__compact_lookup
stops at the first Link message); h5py opens nothing for a dangling first
link although a later one resolves. For a dense group libhdf5
binary-searches the index and may land on another of several exact
duplicates; documented on first_link_named. find_symbolic_link's v2 branch
was dead (only v1 groups reach it) and is now v1-only.
Test: an h5py compact group with soft links dup_A (dangling, or to /d) and
dup_B (the other), dup_B renamed to dup_A in the header and re-checksummed.
Lookup, path and listing through all three readers match h5py for both
orders. With the old group_v2.rs the path lookup returned 42 where h5py
opens nothing.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
addr::to_usize's error branch only ran where usize is narrower than u64,
and no such target runs tests in CI, so on x86_64 the test checked only
that every u64 fits. to_usize and saturating_usize are now the usize
instances of generic to_index/saturating_index; the test runs the same
code with u32 standing in for a 32-bit usize: values past u32::MAX
(including one an `as` cast would wrap to 0x1234) are Overflow, and the
saturating form clamps. A mutant that truncates instead fails the test;
the old addr.rs does not provide the helper the test needs.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The editor stored every chunk through the whole pipeline with filter mask 0.
For LZF that did not shrink a chunk, h5py instead stores it raw with the
filter's mask bit set. A chunk the editor stored LZF-encoded at exactly the
raw size was then rewritten raw by libhdf5 at the same size; libhdf5 does not
touch the index entry when the size is unchanged, so the stale mask 0 stayed
and h5py (and h5dump) could no longer read the dataset.
clawhdf5_format::filters::compress_chunk_masked runs the pipeline as
H5Z_pipeline does: an optional filter (H5Z_FLAG_OPTIONAL) that fails is
skipped and its bit set, a mandatory one fails the write, and LZF/Blosc
output no smaller than the input counts as failure, as in the reference
filters (their output buffer is the input's size). Deflate, LZ4, Zstd,
bitshuffle and bzip2 never fail on size in libhdf5 and are kept as before.
Test: edit_interop optional_filters_that_fail_are_skipped — the reviewer's
repro at every libver: the editor stores the chunk exactly as h5py does
(mask 1, size 5; shuffle+LZF+fletcher32 mask 2), h5py r+ rewrites and
extends the datasets, and h5py, h5dump and our reader read every value.
Fails on the previous editor (mask 0; h5dump cannot read /u8).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Three `chunk_info.address as usize` casts behind the `parallel` feature
survived the conversion, because check-32bit-casts.sh linted only default
features plus plugin-filters. On a 32-bit target with rayon a chunk address
past 4 GiB still wrapped onto another part of the file. They go through
addr::to_usize now, and the lane index (h % n, always < n) through
saturating_usize.
The script now lints no default features, default features, and every
optional feature but szip (wasm32; the set with zstd, which does not build
for wasm32, on the host, where the lint reports the same casts). With the
old parallel_read.rs/lane_partition.rs it fails listing the four casts; the
old script passed them. CHANGELOG and the design note give the exact count
(119) and what is not covered.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Only leaves and the header were checked. Harmless while every lookup read
the whole tree, but the indexed lookup prunes children by the keys stored
in internal nodes, so one corrupted byte there could route a name to the
wrong child and report it missing with no error. A BTIN whose lookup3
checksum does not match is now ChecksumMismatch on every read (lookups and
full traversals), as in libhdf5.
Test: one byte of the root BTIN of the 35 001-link h5py group's name index
changed -> lookups, paths and listings through File, MmapFile and LazyFile
all fail with ChecksumMismatch, and h5py refuses both. Before, lookups
returned Ok.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
read_exact_at and read_upto (and so Window::read) ask as_contiguous()
first and slice the file directly when the backend holds it in memory:
one dynamic call per structure read instead of two or three (len,
read_at, then len again for errors). Same results and errors.
Provisional (busy machine, not for docs): a listing that walks 400
symbol-table groups through the facade went from about 18% to about 14%
slower than before the Storage conversion; the extra cost is a few tens
of nanoseconds per structure read, which the facade's per-lookup
re-listing (range-reads.md M0) multiplies.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
known-issues records what the editor refuses, that freed space is never
reused (append-workload file sizes measured 2026-09-26 on tank with the
ignored measure_append_waste test; sizes are deterministic), and that there
is no journal.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
New clawhdf5::FileEditor opens an HDF5 file (h5py-written at any libver,
HDF5 2.0 format included, or clawhdf5-written) under an exclusive flock
and changes only what an edit touches:
- write_selection/write_all/write_values: compact, contiguous (also
late-allocated) and chunked datasets, any selection. Chunks are decoded,
updated and re-encoded; a filtered chunk that no longer fits moves to the
end of the file unless it is the file's last structure, which grows in
place. New chunks go into v1 B-tree, Extensible Array (paged data blocks
included), Fixed Array and single-chunk indexes, created on first use.
- resize: grow chunked datasets up to maxshape.
- set_attr: add/replace compact attributes, in a NIL slot or a new
continuation chunk.
Each edit is planned in an in-memory image and refused whole
(Error::Unsupported) when any part is unsupported (v2 B-tree / implicit
new chunks, shrinking, vlen/reference data, dense or order-tracked
attributes, cache images, paged/persistent free space). Commit writes and
syncs new space before patching existing bytes. Layout v5 (HDF5 2.0)
array indexes use 8-byte filtered chunk sizes, as libhdf5 does.
Error gains Unsupported/InvalidArgument/Locked and is #[non_exhaustive];
the Python bindings map them. build_attr_message is public.
Tests (h5py, h5dump, h5rs check --data after every round; h5py r+
afterwards): appends crossing EA super/data blocks and B-tree splits, the
same B-tree node counts and EA statistics as libhdf5 for the same writes
(in order, reversed and shuffled; paged blocks), every layout and chunk
index overwritten under random selections, attributes to continuation
chunks, random operations against a model, refused edits leave the file
byte-identical, locking.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Every `u64 as usize` cast in clawhdf5-format (115 on wasm32) now goes
through addr::to_usize for values read from the file — addresses, lengths,
counts, dimensions: FormatError::Overflow where the value does not fit
instead of wrapping onto another part of the file on a 32-bit target — or
addr::saturating_usize for counts bounded by something in memory (codec
progress counters, writer sizes), which fail a bounds check or allocation
rather than wrap. A chunk whose offset does not fit lies outside the
dataset and is skipped; partial reads treat such an offset as out of the
buffers. On 64-bit targets nothing changes.
scripts/check-32bit-casts.sh (run by ci-test.sh) lints the wasm32 build
with clippy's cast_possible_truncation and fails on any u64 -> usize
finding; before this commit it listed 115.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Finding one link or attribute by name read every entry: Group::dataset and
Group::group (File, MmapFile, LazyFile) listed the whole group per call, and
path resolution scanned each group's links. Opening every child of a
35 001-link group by name decoded ~1.2e9 links.
Now a dense group's v2 B-tree name index (type 5, lookup3 hash of the
name) is descended to the records with the name's hash
(btree_v2::find_btree_v2_records reads only the nodes whose key interval
overlaps), and only those links are read and compared; all hash-equal
records are compared, so libhdf5's tie order does not matter. Dense
attributes the same through their type 8 index
(attribute::find_attribute_in_file, facade attr(name)); huge heap objects
through their ID-ordered index. group_v2::resolve_child returns what the
listing has under a name (soft links followed, dangling/external ones not
found). Group::entries and File::group_at hand out a listing's addresses.
The lookup-stats feature counts heap objects read. Tests: one lookup in
an h5py-written 35 001-link group with colliding hashes reads at most two
links (before: 35 001, failing), attribute lookups likewise (before: 3 000,
failing), every child opens through all three readers and matches h5py,
every link kind resolves as h5py resolves it in dense and compact groups,
300 huge attributes are found, and a range search matches a full scan at
every tree depth.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Lists the parsers now reading through Storage, what still needs the
whole file (v2 B-tree-indexed structures: a clean error; raw data: M2),
the equivalence harness, and the evidence that nothing changed: existing
tests, a byte-identical conformance results.json and per-file probe
output against f2ff2c4, and identical slice-API transcripts over 748
files between the two builds.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
osobh