clawhdf5-remote, h5rs: never allocate a length the server only claims

h5rs check URL read the whole file with one read_at(0, len), len being
whatever Content-Range said. BlockCache listed every block index of the
span and preallocated len bytes: a server claiming 2^62 bytes for a 10 KB
file made h5rs abort (memory allocation of 35184372088832 bytes failed).

- BlockCache: a read spanning more than the budget (or eight max_requests)
  is fetched piece by piece and not kept, its output growing only as
  data arrives; read_ranges falls back to that per range; prefetch is
  clamped to the budget.
- New clawhdf5_remote::download(storage, max_bytes): refuses a claimed
  length above the limit (RemoteError::TooLarge) before any request, then
  reads in 64 MiB steps. New RemoteError::Backend for read errors.
- h5rs check downloads through it, with --max-download N (default 1 GiB).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-26 18:27:14 -05:00
co-authored by Claude Opus 5.5
parent e8aaf050be
commit 8df5b209a7
8 changed files with 239 additions and 20 deletions
+17 -2
View File
@@ -38,9 +38,19 @@ pub enum RemoteError {
Transport(String),
/// An error from the object store.
ObjectStore(String),
/// Called in a way the backend cannot serve (for example a blocking
/// read from inside an async runtime).
/// Called in a way the backend cannot serve.
Usage(String),
/// The file is larger than a download was allowed to be
/// ([`download`](crate::download)).
TooLarge {
/// The file's length, as the server reports it.
len: u64,
/// The limit.
limit: u64,
},
/// A read through a backend failed; its error, as text (the form a
/// [`clawhdf5::File`] read reports it in).
Backend(String),
}
impl RemoteError {
@@ -71,6 +81,11 @@ impl std::fmt::Display for RemoteError {
RemoteError::Transport(s) => write!(f, "network error: {s}"),
RemoteError::ObjectStore(s) => write!(f, "object store: {s}"),
RemoteError::Usage(s) => write!(f, "{s}"),
RemoteError::TooLarge { len, limit } => write!(
f,
"the remote file is {len} bytes, more than the download limit of {limit} bytes"
),
RemoteError::Backend(s) => write!(f, "{s}"),
}
}
}