quantumclaw/clawhdf5
3
1
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases 1 Wiki Activity

h5rs: URLs as FILE arguments (feature remote)

Browse Source 
With the `remote` feature (`remote-https` for https://), ls, dump, stat
and diff take an http(s):// (or s3://, gs://, az:// with those
clawhdf5-remote features) URL wherever they take a file, and read it by
range requests through clawhdf5-remote's block cache. check validates
every byte, so it downloads a remote file whole and checks it as before.
Without the feature a URL is a clean error naming it.

The tools read the file through File::storage instead of as_bytes: object
headers, shared messages, attributes, v1 and v2 group links, dense
storage (fractal heaps and v2 B-trees), path resolution, chunk listings
and variable-length values go through the format crate's *_in functions,
and the fractal-heap block verifier reads each block through the storage
(a read failure of a remote file is reported as a problem, not as "past
the end of the file"). A local file's storage is its mapped bytes, so its
reads are still slices. stat's file size comes from the opened file, so
it is right for a URL.

Tests: tests/remote.rs serves fixtures (old and new formats, a paged
file, a metadata cache image, a multi-block fractal heap, compounds, v1
groups) with the clawhdf5-remote test server and requires every
subcommand's output and exit status for the URL to equal the local
file's, and diff of the two to be clean; 404s, non-HDF5 bodies and
https without its feature are clean errors. Local output is unchanged:
the old and new h5rs print the same for ls -r -v, dump, stat and check
--data on the 747 conformance and CVE corpus files (tank, 2026-09-26;
the dumps of h5diff_hyper1/2.h5 were too large for the comparison
script, their ls, stat and check agree), except cve-2025-2310.h5, whose
dump error messages differ between runs of the old binary too (which
failing chunk is reported first).

ci-test.sh lints h5rs with remote-https, runs the URL tests and checks
h5rs with remote for C.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-26 17:23:53 -05:00
co-authored by Claude Opus 5.5
parent a4f586e657
commit c513f7e6d7
13 changed files with 357 additions and 61 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/clawhdf5-tools/README.md
+20
View File
@@ -24,6 +24,26 @@ Every command takes `--max-bytes N` where it reads values (default 1 GiB): a
dataset whose dataspace claims more than that is reported instead of read, so
a corrupt size cannot exhaust memory.
### Remote files
Built with the `remote` feature, every FILE argument may be an `http://`
URL (`remote-https` adds `https://`, through rustls and ring, which compiles
C; the default build has neither). The file is read by range requests
through [clawhdf5-remote](../clawhdf5-remote/README.md)'s block cache, so
`ls` of a large file fetches its metadata blocks, not the file:
```bash
cargo install --path crates/clawhdf5-tools --features remote
h5rs ls -r -v http://127.0.0.1:8000/file.h5
h5rs dump http://127.0.0.1:8000/file.h5
h5rs diff local.h5 http://127.0.0.1:8000/file.h5
```
A URL names the whole file (`FILE/OBJECT` suffixes are for local paths).
`check` validates every byte, so it downloads a remote file whole first.
The output is the local file's (`tests/remote.rs` compares every
subcommand).
## `h5rs ls`
```console