h5rs: URLs as FILE arguments (feature remote)

With the `remote` feature (`remote-https` for https://), ls, dump, stat
and diff take an http(s):// (or s3://, gs://, az:// with those
clawhdf5-remote features) URL wherever they take a file, and read it by
range requests through clawhdf5-remote's block cache. check validates
every byte, so it downloads a remote file whole and checks it as before.
Without the feature a URL is a clean error naming it.

The tools read the file through File::storage instead of as_bytes: object
headers, shared messages, attributes, v1 and v2 group links, dense
storage (fractal heaps and v2 B-trees), path resolution, chunk listings
and variable-length values go through the format crate's *_in functions,
and the fractal-heap block verifier reads each block through the storage
(a read failure of a remote file is reported as a problem, not as "past
the end of the file"). A local file's storage is its mapped bytes, so its
reads are still slices. stat's file size comes from the opened file, so
it is right for a URL.

Tests: tests/remote.rs serves fixtures (old and new formats, a paged
file, a metadata cache image, a multi-block fractal heap, compounds, v1
groups) with the clawhdf5-remote test server and requires every
subcommand's output and exit status for the URL to equal the local
file's, and diff of the two to be clean; 404s, non-HDF5 bodies and
https without its feature are clean errors. Local output is unchanged:
the old and new h5rs print the same for ls -r -v, dump, stat and check
--data on the 747 conformance and CVE corpus files (tank, 2026-09-26;
the dumps of h5diff_hyper1/2.h5 were too large for the comparison
script, their ls, stat and check agree), except cve-2025-2310.h5, whose
dump error messages differ between runs of the old binary too (which
failing chunk is reported first).

ci-test.sh lints h5rs with remote-https, runs the URL tests and checks
h5rs with remote for C.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-26 17:23:53 -05:00
co-authored by Claude Opus 5.5
parent a4f586e657
commit c513f7e6d7
13 changed files with 357 additions and 61 deletions
+6 -5
View File
@@ -151,14 +151,14 @@ pub struct Decoder<'a> {
pub h5: &'a H5,
/// Variable-length elements are resolved as the library resolves them
/// (so as libhdf5 does), not by a decoder of our own.
vl: RefCell<VlResolver<'a>>,
vl: RefCell<VlResolver<'a, dyn clawhdf5_format::storage::Storage + 'a>>,
}
impl<'a> Decoder<'a> {
pub fn new(h5: &'a H5) -> Self {
Self {
h5,
vl: RefCell::new(VlResolver::new(h5.data(), h5.os(), h5.ls())),
vl: RefCell::new(VlResolver::new_in(h5.store(), h5.os(), h5.ls())),
}
}
@@ -268,7 +268,7 @@ impl<'a> Decoder<'a> {
/// has it.
fn decode_vlen(&self, is_string: bool, base: &Datatype, b: &[u8], depth: u32) -> Value {
if is_string {
return match self.vl.borrow_mut().string_element(b) {
return match self.vl.borrow_mut().string_element_in(b) {
Ok(Some(s)) => Value::Str(String::from_utf8_lossy(s).into_owned()),
Ok(None) => Value::NullStr,
Err(e) => Value::Error(e.to_string()),
@@ -278,8 +278,9 @@ impl<'a> Decoder<'a> {
if bs == 0 {
return Value::Error("VL base type of size 0".into());
}
let obj = match self.vl.borrow_mut().element(b, bs) {
Ok(o) => o.unwrap_or(&[]),
// Copied out: decoding an element may resolve nested ones.
let obj = match self.vl.borrow_mut().element_in(b, bs) {
Ok(o) => o.unwrap_or(&[]).to_vec(),
Err(e) => return Value::Error(e.to_string()),
};
Value::Seq(