h5rs: URLs as FILE arguments (feature remote)
With the `remote` feature (`remote-https` for https://), ls, dump, stat and diff take an http(s):// (or s3://, gs://, az:// with those clawhdf5-remote features) URL wherever they take a file, and read it by range requests through clawhdf5-remote's block cache. check validates every byte, so it downloads a remote file whole and checks it as before. Without the feature a URL is a clean error naming it. The tools read the file through File::storage instead of as_bytes: object headers, shared messages, attributes, v1 and v2 group links, dense storage (fractal heaps and v2 B-trees), path resolution, chunk listings and variable-length values go through the format crate's *_in functions, and the fractal-heap block verifier reads each block through the storage (a read failure of a remote file is reported as a problem, not as "past the end of the file"). A local file's storage is its mapped bytes, so its reads are still slices. stat's file size comes from the opened file, so it is right for a URL. Tests: tests/remote.rs serves fixtures (old and new formats, a paged file, a metadata cache image, a multi-block fractal heap, compounds, v1 groups) with the clawhdf5-remote test server and requires every subcommand's output and exit status for the URL to equal the local file's, and diff of the two to be clean; 404s, non-HDF5 bodies and https without its feature are clean errors. Local output is unchanged: the old and new h5rs print the same for ls -r -v, dump, stat and check --data on the 747 conformance and CVE corpus files (tank, 2026-09-26; the dumps of h5diff_hyper1/2.h5 were too large for the comparison script, their ls, stat and check agree), except cve-2025-2310.h5, whose dump error messages differ between runs of the old binary too (which failing chunk is reported first). ci-test.sh lints h5rs with remote-https, runs the URL tests and checks h5rs with remote for C. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
+15
-3
@@ -117,21 +117,28 @@ run_step "cargo clippy (remote, all backends)" cargo clippy \
|
||||
--features object-store,https,s3,gcs,azure \
|
||||
-- -D warnings
|
||||
|
||||
# h5rs with URL arguments.
|
||||
run_step "cargo clippy (h5rs remote)" cargo clippy \
|
||||
-p clawhdf5-tools \
|
||||
--all-targets \
|
||||
--features remote-https \
|
||||
-- -D warnings
|
||||
|
||||
# The README promises that the core crates build no C by default. Hold it to
|
||||
# that: fail if a crate that compiles C (a *-sys crate, cc or cmake) enters the
|
||||
# default dependency tree of any of them. clawhdf5-migrate (bundled SQLite),
|
||||
# clawhdf5-napi (Node) and clawhdf5-gpu (graphics drivers) are exempt.
|
||||
# js-sys (clawhdf5-wasm's bindings to JavaScript) builds no C.
|
||||
# clawhdf5-remote is checked by default (plain HTTP) and with its
|
||||
# object-store feature; its https (ring) and s3/gcs/azure (aws-lc-rs)
|
||||
# features build C and are opt-in.
|
||||
# object-store feature, and h5rs with URL support (remote); the https
|
||||
# (ring) and s3/gcs/azure (aws-lc-rs) features build C and are opt-in.
|
||||
no_c_in_default_build() {
|
||||
local entry crate features found=0
|
||||
for entry in clawhdf5-format clawhdf5-io clawhdf5-filters clawhdf5 \
|
||||
clawhdf5-agent clawhdf5-ann clawhdf5-accel clawhdf5-netcdf4 clawhdf5-cli \
|
||||
clawhdf5-tools \
|
||||
clawhdf5-wasm \
|
||||
clawhdf5-remote clawhdf5-remote:object-store; do
|
||||
clawhdf5-remote clawhdf5-remote:object-store clawhdf5-tools:remote; do
|
||||
crate=${entry%%:*}
|
||||
features=()
|
||||
[ "$entry" != "$crate" ] && features=(--features "${entry#*:}")
|
||||
@@ -210,6 +217,11 @@ run_step "cargo test (remote, object_store backend, s3 URLs)" cargo test \
|
||||
-p clawhdf5-remote \
|
||||
--features object-store,s3
|
||||
|
||||
run_step "cargo test (h5rs on URLs)" cargo test \
|
||||
-p clawhdf5-tools \
|
||||
--features remote \
|
||||
--test remote
|
||||
|
||||
run_step "cargo test (ann parallel)" cargo test \
|
||||
-p clawhdf5-ann \
|
||||
--features parallel
|
||||
|
||||
Reference in New Issue
Block a user