clawhdf5-remote, h5rs: never allocate a length the server only claims

h5rs check URL read the whole file with one read_at(0, len), len being
whatever Content-Range said. BlockCache listed every block index of the
span and preallocated len bytes: a server claiming 2^62 bytes for a 10 KB
file made h5rs abort (memory allocation of 35184372088832 bytes failed).

- BlockCache: a read spanning more than the budget (or eight max_requests)
  is fetched piece by piece and not kept, its output growing only as
  data arrives; read_ranges falls back to that per range; prefetch is
  clamped to the budget.
- New clawhdf5_remote::download(storage, max_bytes): refuses a claimed
  length above the limit (RemoteError::TooLarge) before any request, then
  reads in 64 MiB steps. New RemoteError::Backend for read errors.
- h5rs check downloads through it, with --max-download N (default 1 GiB).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

crates/clawhdf5-remote/src/lib.rs

@@ -16,6 +16,7 @@
 //!   `gcs`, `azure`) → a [`clawhdf5::File`] with the whole read API.
 //! - [`storage_for_url`] gives the cached storage itself, to open with
 //!   [`clawhdf5::File::open_storage`] and to read its [`CacheStats`].
+//! - [`download`] reads a whole remote file into memory, up to a limit.
 //! - [`HttpStorage`] (range `GET`s, pinned by ETag/Last-Modified, retried
 //!   with backoff), [`ObjectStoreStorage`] (any `object_store` store,
 //!   feature `object-store`), and [`BlockCache`] over any
@@ -151,6 +152,47 @@ pub fn cached(backend: Backend, options: &Options) -> Result<RemoteStorage, Erro
     Ok(cache)
 }
 
+/// Default limit of [`download`]: 1 GiB.
+pub const DEFAULT_MAX_DOWNLOAD: u64 = 1 << 30;
+
+/// The whole file behind `storage`, read into memory — at most
+/// `max_bytes` of it (for example [`DEFAULT_MAX_DOWNLOAD`]).
+///
+/// The length is only what the server claims, so it is never used to
+/// allocate: a file longer than `max_bytes` is refused with
+/// [`RemoteError::TooLarge`] before anything is read, and the buffer grows
+/// only as bytes arrive (64 MiB per step, fetched as parallel requests by a
+/// [`BlockCache`]). A read that comes back short is an error.
+pub fn download(storage: &dyn Storage, max_bytes: u64) -> Result<Vec<u8>, Error> {
+    let len = storage.len();
+    if len > max_bytes {
+        return Err(RemoteError::TooLarge {
+            len,
+            limit: max_bytes,
+        }
+        .into());
+    }
+    const STEP: u64 = 64 << 20;
+    let mut out = Vec::new();
+    let mut pos = 0u64;
+    while pos < len {
+        let want = (len - pos).min(STEP);
+        let got = storage
+            .read_at(pos, want as usize)
+            .map_err(|e| RemoteError::Backend(e.to_string()))?;
+        if got.len() as u64 != want {
+            return Err(RemoteError::BadResponse(format!(
+                "{} bytes at offset {pos} instead of {want}",
+                got.len()
+            ))
+            .into());
+        }
+        out.extend_from_slice(&got);
+        pos += want;
+    }
+    Ok(out)
+}
+
 /// Open the object at `path` of any `object_store` store (in memory, local
 /// files, or a cloud store you configured) through a block cache.
 #[cfg(feature = "object-store")]