Every RemoteError message and HttpStorage's Debug output held the URL as
given, with any user:password@ and the query string — for a presigned
S3/GCS/Azure URL, its signature or token. An application logging the
error leaked the credential.
- New clawhdf5_remote::redact_url: no userinfo, no fragment, query values
replaced by REDACTED (plain key names kept).
- HttpStorage formats every message with the redacted URL, and scrubs the
URL's secret parts from errors of the HTTP client (whose texts can echo
the URI); Debug shows the redacted URL. storage_for_url's and the object
store URL errors are redacted too. HttpStorage::url() still returns the
URL as given, documented as not for logging.
- h5rs prints FILE arguments that are URLs redacted: in errors and in
dump/stat/check/diff output.
- The test server can force a status and send a wrong Content-Range.
Tests: 404, 403 (at open and on a read), wrong Content-Range (at open and
on a read), no range support, encoded body, ETag change, timeout,
connection closed and bad scheme errors, Display and Debug, contain none
of the secrets; h5rs likewise for every subcommand.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
With the `remote` feature (`remote-https` for https://), ls, dump, stat
and diff take an http(s):// (or s3://, gs://, az:// with those
clawhdf5-remote features) URL wherever they take a file, and read it by
range requests through clawhdf5-remote's block cache. check validates
every byte, so it downloads a remote file whole and checks it as before.
Without the feature a URL is a clean error naming it.
The tools read the file through File::storage instead of as_bytes: object
headers, shared messages, attributes, v1 and v2 group links, dense
storage (fractal heaps and v2 B-trees), path resolution, chunk listings
and variable-length values go through the format crate's *_in functions,
and the fractal-heap block verifier reads each block through the storage
(a read failure of a remote file is reported as a problem, not as "past
the end of the file"). A local file's storage is its mapped bytes, so its
reads are still slices. stat's file size comes from the opened file, so
it is right for a URL.
Tests: tests/remote.rs serves fixtures (old and new formats, a paged
file, a metadata cache image, a multi-block fractal heap, compounds, v1
groups) with the clawhdf5-remote test server and requires every
subcommand's output and exit status for the URL to equal the local
file's, and diff of the two to be clean; 404s, non-HDF5 bodies and
https without its feature are clean errors. Local output is unchanged:
the old and new h5rs print the same for ls -r -v, dump, stat and check
--data on the 747 conformance and CVE corpus files (tank, 2026-09-26;
the dumps of h5diff_hyper1/2.h5 were too large for the comparison
script, their ls, stat and check agree), except cve-2025-2310.h5, whose
dump error messages differ between runs of the old binary too (which
failing chunk is reported first).
ci-test.sh lints h5rs with remote-https, runs the URL tests and checks
h5rs with remote for C.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
h5rs (dump, ls, diff, check --data) kept its own lenient VL decoder:
a heap object longer than its element was cut to the element's length
(libhdf5 and h5py refuse it), a null string printed "" where h5dump
prints NULL, the stored element size was trusted, and every heap
collection was kept as an owned copy for the whole run. It now resolves
each element with VlResolver::element / string_element (new: one element
in place, borrowing from the file), and refuses a VL type whose stored
element size is not 4 + offset size + 4, as File does. H5::heap_object
and its cache are gone. h5diff compares a null VL string equal to an
empty one; so does h5rs diff.
clawhdf5-wasm already resolved VL strings with read_vl_strings; it now
uses VlResolver and checks the stored element size before reading, as
File::read_string does.
Tests (h5py writes the files, patched for "a\0b", a null element and
mis-sized heap objects, with 8- and 4-byte offsets):
- h5rs_interop dump_prints_vl_data_like_h5dump: byte-identical to h5dump;
- dump_json_vl_values_match_h5py: h5py's values, errors where h5py fails;
- check_data_flags_mis_sized_vl_heap_objects;
- clawhdf5-wasm tests/vl_strings.rs: wasm, File and h5py agree.
All four fail before. check --data over the 150 cve_hdf5 CVE and fuzzer
files now passes 15 (h5dump rejects 8 of them), was 16 and 9: the
stored-size check flags cve-2024-32608. h5rs-check-ok-files.sh --data:
0 of 422 flagged; h5rs-fuzz.sh: clean on 180 files.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
-c meant "list at most N differences" in h5rs, but in h5diff -c is
--compare (a flag) and the count is -n/--count=N, so a script moved over
from h5diff behaved differently: `h5diff -r -c 2 A B` exits 2 (the 2 is
taken as a file name) while h5rs exited 1.
The count is now -n/--count, -c/--compare is accepted (h5rs always lists
objects that are not comparable), and the --count=N, --delta=D,
--relative=R forms are accepted; exit codes equal h5diff's on 7 cases.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
An OBJ that was a soft link was resolved and its target object compared,
so two files whose /g/s both point at /z differed when /z did: exit 1,
where h5diff (without --follow-symlinks) compares the links' target paths
and exits 0.
A soft link is now compared as a link wherever it is, OBJ included.
--follow-symlinks compares the objects soft links lead to instead, walks
into soft-linked groups, resolves relative targets against the link's
group, and treats two dangling links as the same; exit codes equal
h5diff's on 14 cases. External links are never followed (documented).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
With a tolerance, integers were converted to f64 before comparing, so
int64/uint64 values above 2^53 that differ compared equal: -d 0 on 2^60
and 2^60 + 1 exited 0, where h5diff exits 1. Integer pairs are now
compared in i128 (the delta against floor(D), the relative quotient from
an exact difference), and the report prints the exact difference.
h5diff compares exactly when -p is below the f64 epsilon (2^60 and
2^60 + 1 differ at -p 1e-18, and nextafter(2, 0) and 2 at -p 1.5e-16);
h5rs now does the same.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The path walk skipped the second hard link to an object, so a file that
shares one dataset between /x and /y differed from a file holding two
identical copies: "</y> exists only in <B>", exit 1, where h5diff exits 0.
For a hard-linked group every member was reported the same way.
diff now enumerates every path below the start object (a hard link back
to an ancestor is recorded but not descended into), so each name is
compared. A group whose links cannot be read is now an error instead of
an empty group.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
New workspace crate clawhdf5-tools with one binary, h5rs, built only on the
clawhdf5 facade and clawhdf5-format (no libhdf5, no C):
- ls [-r] [-v] FILE[/path]: h5ls's listing (same text in its first two
columns) plus the datatype; -v adds address, link count, layout and chunk
index, chunk size, storage, filters, datatype and attributes.
- dump [--json] [-A] [-p] [-d PATH] FILE: h5dump DDL (byte-identical to
h5dump 1.14.6 on the test files) or hdf5-json.
- stat FILE: h5stat's object/link/rank/layout/filter/attribute counts, raw
data and total size.
- diff [-r] [-q] [-d D] [-p R] A B [OBJ1 [OBJ2]]: structural and value
differences, exit 0/1/2 like h5diff.
- check [--data] FILE: walks every object, parses every message, verifies
the checksums of every v2+ structure (including the fractal heap blocks
the library never checks), checks chunk indexes against their datasets
and raw data for out-of-file or overlapping extents; every problem with
its address.
Values over --max-bytes are reported, not read; dense-storage heaps are
verified before objects are read from them; panics are caught (exit 3).
Tests compare with h5ls, h5stat, h5dump and h5diff and with h5py's values,
and flip the checksum of every checksummed structure in a v1.14-format file.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
