is_transient_format counted every format error but a handful as transient,
so on an open_swmr handle a permanent failure (a file that is not HDF5, an
unsupported version or message, a truncated file) was retried 100 times,
about 0.9 s of pauses per failing operation. Now only these are retried:
a checksum mismatch; a read past the file's current end (UnexpectedEof;
libhdf5 reads zeros there, which fail the checksum); and an object header
prefix whose signature or version does not decode, which libhdf5's
H5C__load_entry also retries (a header garbled whole fails there before its
checksum). Everything else is returned at once.
Tests: a unit test that every permanent kind returns after one call within
50 ms and every transient kind is retried to the limit; open_storage_swmr
of a non-HDF5 buffer returns SignatureNotFound within 100 ms (0.87 s before)
and a missing name on a live file fails without retries. The torn-read and
live h5py-writer tests still pass.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A file opened with open_swmr whose superblock does not have the SWMR-write
flag (its writer has closed it) is now read exactly as File::open reads it:
bounded by its recorded end of file, through the chunk cache, each
operation tried once, and is_swmr_read() is false. Before, every file opened
with open_swmr ignored its recorded end of file, so a closed file whose end
of file is below its length (h5clear_fsm_persist_less.h5, or the mid-write
fixture with its flags cleared) listed and read objects that File::open and
libhdf5's plain reader refuse. Opening such a file is not retried once the
superblock has been read.
libhdf5's SWMR reader is looser than either (it reads the fixture's chunk
index past the end of file even with the flags cleared); a test pins what
h5py does and docs/design/swmr.md says why we follow the plain reader.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Range-read milestone M5 (docs/design/swmr.md): read a file a libhdf5
SWMR writer (h5py f.swmr_mode = True) is still appending to, as h5py's
File(path, 'r', swmr=True) does.
- File::open_swmr / open_storage_swmr: the file is read through the new
FileStorage (pread / seek_read, never mapped, len() the current length),
reads are bounded by the length at each read, and the chunk cache is not
used (a cached index hides new chunks; a cached edge chunk reads as fill
where the writer has since written). A file open for writing without
SWMR is refused with Error::Locked, as libhdf5 refuses it.
- Dataset::refresh re-reads the object header (H5Drefresh).
- Operations that fail with an error a racing write can cause (every
format error but a wrong path/selection, an unsupported feature or a bad
argument) are run again, up to 100 attempts (libhdf5's default metadata
read attempts for SWMR readers), 1 us doubling to 10 ms apart; nested
operations retry as a whole. swmr_retries() counts them,
swmr_writer_active() re-reads the superblock flags.
Tests: FileStorage growth and retry policy (unit); the mid-write copy
through open_swmr; a storage that garbles reads (retried, given up after
the attempts, never returned); and a live h5py writer appending to
Extensible-Array (plain and gzip) and v2-B-tree datasets for 2500 steps
while two Rust reader threads and h5py's SWMR reader check every value,
then the closed file read equal to h5py. Leaving the chunk cache on in
live mode fails the live test.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A SWMR writer does not keep the superblock's end-of-file address up to
date: a copy h5py made of its own file mid-write records 715 in a
6 030-byte file. Superblock::data_end only ignored the recorded end when
it lay past the end of the file, so every open path bounded reads at
715: the file listed, but every chunked read failed ("unexpected EOF:
need 787 bytes, have 715") and h5rs check reported the chunk indexes
past the end of the file. libhdf5's SWMR reader skips the
end-of-allocation check for every read (H5FD_read); for a v3 superblock
with the SWMR-write flag the data now ends at the end of the file.
Test: tests/swmr_interop.rs over the mid-write copy (fixture), through
File::open, open_buffered and from_bytes, and against h5py's SWMR reader.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
libhdf5's SWMR semantics as they matter to a reader (superblock v3 with
the SWMR-write flag and a stale EOF, append-only writer, flush
dependencies, refresh, 100 metadata read attempts), what clawhdf5 did
with such files, and the plan: data_end bounded by the file length for
SWMR-flagged files, a live File::open_swmr over positioned reads without
the chunk cache, Dataset::refresh, and bounded whole-operation retries.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Since the M2 merge the facade handed in-memory files to the generic
`*_in` parsers as `&[u8]` (`with_bytes!`), which instantiates them in
the facade crate, where the format crate's private helpers do not
inline without LTO: listing a 400-group v1 file through `File::open`
was 7-10% slower than main. `ObjectHeader::parse_in`,
`group_v2::{resolve_child_in, resolve_group_children_in,
resolve_path_any_in}` and `attribute::{extract_attributes_tolerant_in,
find_attribute_in}` now pass a storage with `as_contiguous()` to their
non-generic slice entry point, compiled once in the format crate; other
storages reach the same generic core as before.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Reading continuation chunks from a queue (7e5e920, a69c5be) allocated a
queue Vec and a BTreeSet of chunk starts for every header, and inlined
the per-chunk message loop into the generic parser: ObjectHeader::parse
over 401 version-1 headers went from 24.8 to 45.7 us.
ChunkSpans now keeps the first 8 chunks in an inline array (cycle check
by scan) and is also the read queue; only a header of more chunks
allocates (a boxed spill list and start set). The message loop of one
version-1 chunk is its own non-generic function. Same checks as before:
any number of chunks up to 65,536, cycles refused, chunks bounded by the
file size, one chunk buffer alive at a time, libhdf5 message order,
overlap allowed. The cycle test now also covers spilled chunk lists.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The M2 fetch bound gave every codec n + n/4 + 4096 stored bytes, but
ZFP's fixed-rate mode stores up to 64 bits per value, doubling 4-byte
types: 60 of the 2205 zfp_interop datasets (rate 64, f4/i4) failed with
'compressed stream ends early' once M2 and ZFP were merged.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
check-32bit-casts.sh flagged two casts added by the previous commits; both
values are bounded (checked by gather_storage's first walk, and built from
a usize fetch length), so they go through addr::saturating_usize.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The facade equivalence harness turned every data-read error into "Err", so
it could not see File::open_storage failing differently from File::open
(a Storage or ContiguousStorageRequired error where the mmap path gives a
decode error, say).
- value() keeps the whole error. The only allowance is for a line on which
File::open itself varies between opens — the chunk cache lists a damaged
dataset's chunks in hash-map order, so which failing chunk a full read
reports varies (cve-2025-2310.h5, the one corpus file where this shows):
both sides must fail there, and a fresh File::open (up to 64) must
reproduce the storage's exact error. Open errors were already compared
in full; they still agree.
- The storage transcript may not contain ContiguousStorageRequired.
- More selections: a strided hyperslab (every third row) through
read_f64_selection, and out-of-order points through read_selection and
read_i64_selection.
- harness_compares_errors_not_just_failures checks the harness itself:
two different errors are different values, and a difference File::open
does not produce is reported.
With full errors the harness passes on the 61 fixtures and on the corpus
(701 files, 621 open).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
An attribute needing a heap block larger than the next one was refused
("skipping blocks too small for an object", "a first object too large for
the starting block"); once an object's move to dense storage was refused
it refused every new attribute, so 24% of set_attr calls in the review's
random workload failed.
Following H5HF__hdr_update_iter, H5HF__man_iblock_root_create/_double and
H5HF__hdr_skip_blocks, the smaller blocks are now skipped: the iterator
moves past them and they become an indirect free section with a first
row section (serialized, class 1, as H5HF__sect_indirect_serialize writes
it) and ghost normal rows, added as returned space so it merges with a
range skipped just before it (H5HF__sect_indirect_merge_row). Later
objects that best-fit a row section get a block created there
(H5HF__man_iblock_alloc_row / H5HF__sect_indirect_reduce_row: from the
start or end of the range, or from its middle, which splits it, with
libhdf5's span bookkeeping). Heaps with such sections, as libhdf5 writes
them, are now read too (they were refused at open).
dense_skipped_blocks_match_libhdf5 drives every path (merge, split, end,
last entry, row wrap) on earliest/v110/latest files against libhdf5
doing the same edits one session each; heaps, free sections and index
B-trees are equal after every phase. The refusal test now checks the
skip against libhdf5 and keeps a real refusal (last object in a block);
clawhdf5-written heaps get 1-4 KiB attributes too. The three tests fail
on the previous fheap.rs. Random workload refusals: 24% -> 2.2%, all the
documented last-object-in-a-block case.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
306 104 and 49 930 are h5repack of the editor's file; the text read as if
they were h5repack of libhdf5's, which measures 305 954 and 50 188. Both
are now given, from measure_append_waste rerun on 2026-09-26 (file sizes
unchanged).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The version-1 chunk walk nested continuation chunks depth-first and kept
every enclosing chunk's buffer alive, up to 65 536 chunks. With storage
that hands out owned buffers (CountingStorage, the Storage trait, remote
storage) a crafted chain of chunks nested in each other read and held the
square of the file's size (a 192 KB file read 768 MB).
Chunks are now read from a FIFO queue of (address, length) pairs in the
order their continuation messages are found, as H5O_protect does and as
the editor's header walker already did, each buffer released before the
next read. In both header versions a chunk starting at an address seen
before (cycle) is refused, and so are chunks adding up to more than the
file, which bounds a header's reads by the file's size. Overlap itself is
allowed: libhdf5 reads cve-2025-7067.h5, whose continuation chunk overlaps
chunk 0 (refusing overlap cost that conformance file).
Tests: the nested chain is refused having read at most the file (it read
n^2 bytes before); a 3000-chunk chain reads each chunk once; a chunk's
messages follow the whole previous chunk (they were inserted at the
continuation message); an overlapping continuation chunk is read.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
prune_plan stored one Vec<u64> for every chunk coordinate of the region a
shrink cuts off, existing or not, so a sparse dataset exhausted memory
(about 62 bytes per coordinate; (4, 2e7) with chunks (1, 1) took 2.5 GB,
larger extents never finished). It now places each existing chunk in
H5D__chunk_prune_by_extent's walk (its pass, then its coordinates) and
sorts, which gives the same chunks, order and actions in memory and time
proportional to the chunks that exist.
A unit test checks the plan against the full walk (kept as the test's
reference) for 3000 random extents and chunk subsets. The interop test
shrinks a (4, 10^12) dataset with chunks (1, 1) and 9 chunks (v1 and v2
B-tree): 0.56 s and 43 MB peak; the old code aborted on allocation under an
8 GB limit.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Our checksum reduced its sums with `% 65535`; libhdf5's
H5_checksum_fletcher32 folds them with `(s & 0xffff) + (s >> 16)`, which
leaves 0xffff where the modulo leaves 0. On about one chunk in 32768
libhdf5 refused the chunks we wrote and we refused the chunks it wrote.
Every release since v2.1.0 is affected.
clawhdf5_format::checksum::fletcher32 is a port of H5_checksum_fletcher32
and the filter's only implementation. Verification also accepts the
byte-swapped form libhdf5 accepts (1.6.2 and earlier) and the `% 65535`
form earlier releases wrote, so their files stay readable.
The new interop test compares the checksum with libhdf5's own function
(ctypes) on every 1- and 2-byte input and 40 000 random and fold-heavy
inputs, and moves fold-case chunks between h5py and FileBuilder/FileEditor
in both directions; with the old filters.rs the three file tests fail.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
ExtentBytes and read_exact_at/read_upto rejected short results but passed
longer-than-asked ones through, and FileData forwarded them too, so a
Storage that broke read_at's contract by returning extra bytes had them
decoded or returned as data (a contiguous dataset read gained 37 junk
bytes). gather_storage alone trimmed.
- storage::exact_len (new, pub): a read of len bytes as exactly len — cut
when longer, an error when short. read_exact_at, read_upto and
ExtentBytes (so chunk fetches and selection gathers) go through it.
- FileData cuts a backend's answer to what it asked for before laying the
cache image over it.
- Tests: over a storage that appends 37 junk bytes to every read, every
format-crate fixture reads exactly as from the slice
(overlong_reads_are_cut_to_the_range_asked_for), and every facade
fixture opens and reads through File::open_storage as through File::open
(overlong_storage_reads_identically). Both failed before.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Since M2 the facade handed the metadata parsers its FileData view, so a
local file ran the parsers monomorphised for FileData, whose Storage impl
worked out contiguous() (patched and overlay checks, two range conversions)
on every structure read. A metadata walk of h5stat_newgrat.h5 (35,001
groups: open, entries and attrs of each) was about 4.5% slower than at
8f59b2e.
- FileData works out its contiguous slice once at open (a borrow of its own
heap/mapped buffer, kept as a pointer; see the SAFETY notes).
- with_bytes! hands the in-memory slice to the format parsers when the file
has one (header parsing, attributes, group listings and lookups, path
resolution, shared messages, VL decoding), so local files run the [u8]
parsers as before; storage-backed files still get FileData.
Provisional A/B on tank (load 5-11), best of 30, 5 alternating rounds:
walk 26.07-26.37 ms at 8f59b2e, 27.24-28.03 ms before this commit,
26.52-26.98 ms after. Caching alone did not move it (27.02-27.47 ms); the
dispatch did. File::open read_f32 on 32M f32 (contiguous, chunked, gzip)
stays within noise of 8f59b2e.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
checked_div in the test server's throttle, a slice for the single range
of fetch_first, and dead-code allowances for the redaction helpers in a
build with neither http nor a cloud store.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
CHANGELOG, the clawhdf5-remote and h5rs READMEs and the remote-files
known issues: redirect rules, scaled timeouts (min_speed), URL redaction,
claimed lengths never allocated (download, --max-download), a 200 for a
small file accepted, and ObjectStoreStorage from any thread.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
open_arg_whole opened and parsed the remote file through open_arg, then
opened it again to download it, so every `h5rs check URL` probed the
server twice. It now opens the storage once and downloads through the
same block cache (whose first block the probe already filled).
Test: check --data of a file within one block costs exactly one request
(two before).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
cached() mapped an error of its open-time prefetch (a network error, a
changed file) to Error::Hdf5(Format(Storage)), misclassifying it for
callers that match on the variant. It is now Error::Remote
(RemoteError::Backend with the backend's message). open_object and the
s3/gs/az URLs fetch the first block of an ObjectStoreStorage directly, so
their errors keep their kind (FileChanged, ObjectStore).
Test: cached() over a backend whose reads fail gives Error::Remote(Backend)
(Error::Hdf5 before).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
It refused whenever Handle::try_current() was Ok, which is also the case
inside spawn_blocking threads — so the workaround its own error message
recommended failed the same way, and the backend could only be used from
a bare std::thread in a tokio application.
Reads are now spawned on the storage's own runtime and the caller waits on
a channel: the future never runs on the caller's thread, so neither a
spawn_blocking thread nor a current-thread runtime can deadlock or panic
(a read inside a runtime blocks that thread, like any blocking call; the
docs still recommend spawn_blocking there).
Tests: a read in spawn_blocking of a multi-thread runtime and a read inside
a current-thread runtime's task give File::open's values (both errors
before).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
gather_storage merged only runs that touch, so a strided selection of a
contiguous dataset over a Storage became one range (and one owned Vec) per
element: a stride-2 read of 32M f32 through File::open_storage made
16,777,232 read_at calls, took 2.0 s and peaked at 2.09 GB.
The selection is now walked twice. The first walk checks the runs and plans
spans: runs in increasing order at most 4 KiB apart (GATHER_GAP_BYTES) are
read as one span up to 8 MiB (GATHER_SPAN_BYTES; a longer run is split), so
nothing is stored per run. The spans are fetched in RAW_BATCH_BYTES batches
while the second walk copies each run out of its span. Same checks and
errors as before.
The same read is now 32 reads and 0.31 s (File::open: 0.08 s).
contiguous_read_interop: every h5py-checked selection is also read through
File::open_storage and must give libhdf5's bytes; a new test bounds the
range reads of strided, blocked, column and point selections (stride 2: at
most 1 data read; 563,200 before).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
timeout_global (60 s) covered a whole request, and a request can carry
8 MiB (max_request): below about 140 KB/s every block run timed out, was
retried from scratch and failed, so a slow link could not read remote
files at all.
HttpOptions::timeout (now 30 s) bounds connecting and receiving the
response headers; the body gets timeout + its size at the new
HttpOptions::min_speed (16 KiB/s by default: 94 s for a 1 MiB block).
A slow but moving link is not cut off; a stalled one still fails.
(ureq has no idle timeout; its body timeout is a total budget.)
The test server can throttle bodies and stall mid-body. Test: a 256 KiB
block at 256 KiB/s reads with a 300 ms timeout (it failed before), and a
body stalled for 20 s fails in under 5 s.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The first request asks for bytes=0-1048575. RFC 9110 lets a server answer
200 when the range covers the whole representation, so a file under
1 MiB on a server that does support ranges could be refused as 'does not
support range requests'. A 200 whose Content-Length (or, without one, its
body, read at most that far) is within the range asked for is now kept as
the whole file and read from memory; a longer one is still refused unless
allow_full_download is set.
Test: a 9968-byte file served with 200 opens in one request with the
transcript of File::open (it was refused before); with a 4096-byte first
request it is still refused.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
ureq's defaults followed up to 10 redirects, including from https to plain
http, and forwarded the custom HttpOptions::headers (X-Api-Key, Cookie,
...) to whatever host a redirect named — only Authorization was stripped.
HttpStorage now follows redirects itself (ureq's max_redirects is 0):
- at most HttpOptions::max_redirects per request (default 5; 0 refuses
any redirect), then RemoteError::Redirect;
- never from https to another scheme, nor to a non-http(s) URL;
- once a redirect leaves the URL's origin (scheme, host, port), none of
the custom headers is sent any more (Authorization included);
- each hop counts as a request; errors show the target redacted.
Tests: a redirect to another local port reads the right data and the
target never sees X-Api-Key or Authorization (it did before); a
same-origin redirect keeps them; a loop stops after 6 requests; 0 refuses;
unit tests for target resolution, the https downgrade and origins.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Only the full read split its chunk fetches into 64 MiB batches. The
selection path, the indexed read and the parallel_read decoders fetched
every chunk's stored bytes in one read_ranges call, each extent bounded only
by the file length, so a crafted chunk index pointing many chunks at one
large extent made File::open_storage hold chunks x extent bytes (3.3 GB from
a 16.8 MB file) before the first decode error.
- storage::for_each_extent_batch is now the one way raw-data reads fetch
chunk bytes: batches of at most RAW_BATCH_BYTES (now pub), each decoded
before the next is fetched. Used by the full, cached, indexed, selection
and parallel_read paths; the sweep read uses read_extent per chunk.
- ExtentReq carries each chunk's claimed extent (bounds-checked as before,
same errors) and the prefix actually fetched:
filters::stored_chunk_limit — the chunk size if unfiltered, else each
applied filter's worst-case growth (n + n/4 + 4096 per codec; unbounded
only for an application-registered codec). The in-memory path cuts the
slice it decodes the same way, so both paths still agree.
- tests/raw_fetch_bounds.rs: a crafted chunked_large.h5 (ten chunks all
claiming 20 MiB at one padding blob) read through every path over a
storage that records the largest single fetch; and 160 MiB of legitimate
unfiltered chunks fetched batch by batch. Before: one 80 MiB fetch
(selection) and one 160 MiB fetch; after: within the budget.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Every RemoteError message and HttpStorage's Debug output held the URL as
given, with any user:password@ and the query string — for a presigned
S3/GCS/Azure URL, its signature or token. An application logging the
error leaked the credential.
- New clawhdf5_remote::redact_url: no userinfo, no fragment, query values
replaced by REDACTED (plain key names kept).
- HttpStorage formats every message with the redacted URL, and scrubs the
URL's secret parts from errors of the HTTP client (whose texts can echo
the URI); Debug shows the redacted URL. storage_for_url's and the object
store URL errors are redacted too. HttpStorage::url() still returns the
URL as given, documented as not for logging.
- h5rs prints FILE arguments that are URLs redacted: in errors and in
dump/stat/check/diff output.
- The test server can force a status and send a wrong Content-Range.
Tests: 404, 403 (at open and on a read), wrong Content-Range (at open and
on a read), no range support, encoded body, ETag change, timeout,
connection closed and bad scheme errors, Display and Debug, contain none
of the secrets; h5rs likewise for every subcommand.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
h5rs check URL read the whole file with one read_at(0, len), len being
whatever Content-Range said. BlockCache listed every block index of the
span and preallocated len bytes: a server claiming 2^62 bytes for a 10 KB
file made h5rs abort (memory allocation of 35184372088832 bytes failed).
- BlockCache: a read spanning more than the budget (or eight max_requests)
is fetched piece by piece and not kept, its output growing only as
data arrives; read_ranges falls back to that per range; prefetch is
clamped to the budget.
- New clawhdf5_remote::download(storage, max_bytes): refuses a claimed
length above the limit (RemoteError::TooLarge) before any request, then
reads in 64 MiB steps. New RemoteError::Backend for read errors.
- h5rs check downloads through it, with --max-download N (default 1 GiB).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A server can claim any length in Content-Range. block_len computed
start + block_size, which overflowed in the last blocks of a file claimed
to be near u64::MAX (a panic in debug builds, a wrapped value in
release); insert() multiplied block indices unchecked. The cache's block
arithmetic is now saturating/checked, and a run that does not split into
whole blocks is an error instead of an endless loop or a slice panic.
The test server gains fake_total (claim a length, serve zeros past the
data); a test reads the last bytes of such files and opens a file whose
superblock EOF and root addresses sit near u64::MAX.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A local port scanner's GET / reached the test listeners and was counted,
failing the exact request budgets (and consuming injected 503s). Requests
for paths the server does not serve are now answered 404 without being
counted, delayed or failed; the query string is not part of the path.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A reader that waited for another reader's fetch of a block got "the
fetch of this block failed" when that fetch failed, not why: a file
replaced on the server while open was reported as FileChanged to one
thread and as an anonymous failure to the others. The fetch's error is
now handed to every reader waiting on it.
Regression test: four threads read the same block from a slow backend
whose fetches fail with a "changed while open" error; each gets that
error (it failed for the waiters before this change).
Also fixes the ignore-Range test, broken by the previous commit: the
test server now counts a body before sending it, so "the refused body
was not read" is checked as "refused at the first response".
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
README: "Reading remote files" (open_url, the range_server and read_url
examples with their real output against the fixtures, h5rs on URLs), the
crate in the crate map and the unreleased highlights. CHANGELOG: the
clawhdf5-remote crate, h5rs URLs, File::storage and
VlResolver::element_in, with the request counts over the conformance
corpus (tank, 2026-09-26, the command given). known-issues: the M2
range-read entry updated (the cache now exists; h5rs reads through
storage) and a new entry for the remote backends' limits (no Python or
browser URLs yet, fixed block size, cloud stores not run against a real
bucket, validators, credentials). Design doc: M3 status with the choices
that differ from the plan (a crate rather than a clawhdf5-io feature,
ureq for HTTP so the default build has no C) and the corpus counts.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The per-file report now separates a tree view (open, every group's
entries, every dataset's shape and type) from reading the largest
dataset under 64 MiB, and checks the budget the design's testing section
asks for: listing the IMERG file (file A of docs/design/range-reads.md
section 2) takes at most 3 requests when CLAWHDF5_REMOTE_CORPUS includes
it. The test server now counts a response's bytes before sending it: a
client could read a body and reset the counters before the server thread
had added it, so the counts of the next file were occasionally too high.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Datasets with early allocation and unlimited dimensions (Extensible
Array, version-2 B-tree, version-1 B-tree under earliest), unfiltered
and deflated: the random resize workload gives the values h5py gets and
the same chunk index shape, with every chunk a growth brings in
allocated and filled as H5D__chunk_allocate does.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
With the `remote` feature (`remote-https` for https://), ls, dump, stat
and diff take an http(s):// (or s3://, gs://, az:// with those
clawhdf5-remote features) URL wherever they take a file, and read it by
range requests through clawhdf5-remote's block cache. check validates
every byte, so it downloads a remote file whole and checks it as before.
Without the feature a URL is a clean error naming it.
The tools read the file through File::storage instead of as_bytes: object
headers, shared messages, attributes, v1 and v2 group links, dense
storage (fractal heaps and v2 B-trees), path resolution, chunk listings
and variable-length values go through the format crate's *_in functions,
and the fractal-heap block verifier reads each block through the storage
(a read failure of a remote file is reported as a problem, not as "past
the end of the file"). A local file's storage is its mapped bytes, so its
reads are still slices. stat's file size comes from the opened file, so
it is right for a URL.
Tests: tests/remote.rs serves fixtures (old and new formats, a paged
file, a metadata cache image, a multi-block fractal heap, compounds, v1
groups) with the clawhdf5-remote test server and requires every
subcommand's output and exit status for the URL to equal the local
file's, and diff of the two to be clean; 404s, non-HDF5 bodies and
https without its feature are clean errors. Local output is unchanged:
the old and new h5rs print the same for ls -r -v, dump, stat and check
--data on the 747 conformance and CVE corpus files (tank, 2026-09-26;
the dumps of h5diff_hyper1/2.h5 were too large for the comparison
script, their ls, stat and check agree), except cve-2025-2310.h5, whose
dump error messages differ between runs of the old binary too (which
failing chunk is reported first).
ci-test.sh lints h5rs with remote-https, runs the URL tests and checks
h5rs with remote for C.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
VlResolver::element and string_element return slices of the whole file,
so they exist only for a resolver over &[u8]. Their *_in forms work for
any Storage (a remote file): the element's bytes borrowed from the
resolver's cache of heap collections, with the same null-element, NUL
and size checks. h5rs decodes variable-length values with them.
Test: over a read_at-only storage they give what element/string_element
give over the slice, for a string with an embedded NUL, a null element
and an element whose heap object has the wrong size.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The bytes File::as_bytes returns (from the superblock on, bounded by the
recorded end of file, a metadata cache image laid over), as a
&(dyn Storage + Send + Sync) for every backend. Code that parses the file
itself with the clawhdf5_format *_in functions — h5rs does — can then
read a file opened with File::open_storage (a remote file) as well as a
local one; in memory its as_contiguous() is as_bytes(), so local reads
stay slices.
Test: for every fixture, File::open's storage() is as_bytes() as its
contiguous view, and File::open_storage over a read_at-only storage gives
the same bytes through storage().read_at.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
ObjectStoreStorage (feature `object-store`, pure Rust) reads one object
of any object_store store by ranged get_opts, pinned at open by a head
request: If-Match with its ETag (and the ETag and size of every response
compared), else its version or modification time. A change is
RemoteError::FileChanged. object_store is async and Storage is not, so
the storage owns a small multi-threaded tokio runtime (two workers) and
blocks the calling thread on it; the ranges of one read_ranges call are
fetched concurrently (up to 8). From inside another tokio runtime it
refuses with RemoteError::Usage instead of blocking a worker, and it
shuts its runtime down in the background on drop so dropping it in async
code does not panic.
open_object(store, path, options) opens a file through a block cache
(first block prefetched); open_url accepts s3://, gs:// and az:// with
the `s3`, `gcs` and `azure` features, configured from the environment by
object_store's from_env builders. Those pull object_store's cloud clients
and aws-lc-rs (C), so they are opt-in; without them the URL is a clean
UnsupportedScheme error naming the feature.
Tests against object_store's in-memory and local-file stores (no cloud):
every fixture's transcript equals File::open's, a multi-block object is
fetched in coalesced block runs, an object replaced while open is an
error, and a missing object or a read from inside a runtime is a clean
error. ci-test.sh lints all backends, runs these tests (with s3 for its
URL parsing test) and checks object-store for C in the no-C step.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
CHANGELOG (Unreleased): the new FileEditor operations, space reuse, and
the two reader fixes (implicit index grid, object-header continuation
chains). known-issues: the editor's remaining refusals (skipped heap
blocks, heaps with filters or child indirect blocks, freeing a heap
block, implicit-index insertions, ...) and the append-waste sizes before
and after reuse (measure_append_waste, tank 2026-09-26; file sizes are
deterministic). range-reads design: status note on the reader changes.
README and CLAUDE.md: what the editor covers and how to test it.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Range-read milestone M3, first half: a new crate with the block cache the
design makes mandatory for remote files and an HTTP backend, so
open_url("http://...") gives a clawhdf5::File over File::open_storage.
BlockCache wraps any Storage: aligned blocks (1 MiB by default, the size
docs/design/range-reads.md section 2 measured), LRU with a byte budget,
the missing blocks of one read_at/read_ranges fetched with one backend
read_ranges call as runs of consecutive blocks (a one-block gap filled to
merge runs, each request at most 8 MiB), and reads that miss more than
half the budget not kept. Thread-safe without holding the lock across a
fetch: a block being fetched is in flight, a second reader waits for it
instead of fetching it again, and a failed fetch fails its waiters and is
not cached. A backend holding the file in memory passes through.
HttpStorage (ureq, no TLS by default; `https` adds rustls with ring):
opening is one ranged GET of the first block, whose Content-Range gives
the length (the cache keeps the bytes). The file is pinned by a strong
ETag (If-Match), else Last-Modified (If-Unmodified-Since), and its length,
checked on every response: a change is RemoteError::FileChanged, never
mixed data. A server that ignores Range is refused without reading the
body unless a full download is allowed. Connection errors, timeouts,
408/429/5xx and short bodies are retried with exponential backoff;
Accept-Encoding: identity, and an encoded body is refused. read_ranges
fetches its ranges in parallel.
Tests (a std-only HTTP/1.1 server in tests/common/server.rs, also the
range_server example): every fixture read over HTTP gives File::open's
transcript (CLAWHDF5_REMOTE_CORPUS adds the conformance corpus), with
request counts per file with and without the cache; an h5py-written file
against libhdf5's values; a multi-block file fetched in whole blocks, each
once; a server ignoring Range; a file replaced mid-read (ETag,
Last-Modified, length only); truncated bodies and 503s (retried, then an
error, never cached); a slow server with 8 concurrent readers (no block
fetched twice); bad URLs, 404, encoded bodies, non-HDF5 data. The cache
has unit tests for coalescing, splitting, LRU order, large reads,
failures and concurrent in-flight dedup.
ci-test.sh: clawhdf5-remote joins the no-C default-build check, and its
https feature is linted.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Version-2 B-tree (its writer's own node size and single-leaf layout),
Extensible Array and Fixed Array datasets written by FileBuilder, with
and without deflate, resized up and down along both dimensions and
written at random against a model; h5py, h5dump and h5rs check read the
result, and h5py resizes and rewrites every dataset afterwards.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
random_operations_match_a_model (edit_interop) now drives, on every
libver (earliest, v114, latest) and filter set (none, gzip + shuffle +
fletcher32, LZF):
- a dataset with one unlimited dimension and one with two (a version-2
B-tree chunk index under v114/latest), resized to random shapes that
shrink and grow any resizable dimension, with block and point writes;
- attributes on the first dataset under 20 names with values of random
types and sizes (scalars, int64 arrays, short strings, strings above
the heap's managed limit), so they move to dense storage on version-2
headers and are replaced by values of other sizes;
against a model where shrunk-away elements that come back read as the
fill value, compared with our reader and with h5py/numpy every 40 steps,
with h5dump and h5rs check; h5py then grows both datasets and adds an
attribute. The attribute check also compares h5py's attribute count with
libhdf5's object info. CLAWHDF5_EDIT_SEED reruns the workloads with other
random choices (seeds 1000-4000 pass).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
libhdf5 deletes a fractal heap's huge-object B-tree when the heap is
closed with no huge object left (H5HF__huge_term) and starts huge IDs
over. The editor left the empty tree, and a read-only libhdf5 then
failed to list the object's attributes: closing the heap tried to delete
the tree ("no write intent on file"), and h5dump failed the same way.
Replacing an object's last huge attribute (one above the heap's 4 KiB
managed limit) now deletes the tree (header and nodes freed), resets the
next huge ID and the wrapped flag, as libhdf5 does; a later huge
attribute creates a new tree. Bt2::delete frees a whole tree.
Found by the extended random-operation test; regression:
last_huge_attribute_replaced (fails before: h5dump, h5py listing), which
also compares the heap with libhdf5's after the same replacement.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
osobh