clawhdf5-remote, h5rs: never allocate a length the server only claims
h5rs check URL read the whole file with one read_at(0, len), len being whatever Content-Range said. BlockCache listed every block index of the span and preallocated len bytes: a server claiming 2^62 bytes for a 10 KB file made h5rs abort (memory allocation of 35184372088832 bytes failed). - BlockCache: a read spanning more than the budget (or eight max_requests) is fetched piece by piece and not kept, its output growing only as data arrives; read_ranges falls back to that per range; prefetch is clamped to the budget. - New clawhdf5_remote::download(storage, max_bytes): refuses a claimed length above the limit (RemoteError::TooLarge) before any request, then reads in 64 MiB steps. New RemoteError::Backend for read errors. - h5rs check downloads through it, with --max-download N (default 1 GiB). Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
@@ -38,9 +38,19 @@ pub enum RemoteError {
|
||||
Transport(String),
|
||||
/// An error from the object store.
|
||||
ObjectStore(String),
|
||||
/// Called in a way the backend cannot serve (for example a blocking
|
||||
/// read from inside an async runtime).
|
||||
/// Called in a way the backend cannot serve.
|
||||
Usage(String),
|
||||
/// The file is larger than a download was allowed to be
|
||||
/// ([`download`](crate::download)).
|
||||
TooLarge {
|
||||
/// The file's length, as the server reports it.
|
||||
len: u64,
|
||||
/// The limit.
|
||||
limit: u64,
|
||||
},
|
||||
/// A read through a backend failed; its error, as text (the form a
|
||||
/// [`clawhdf5::File`] read reports it in).
|
||||
Backend(String),
|
||||
}
|
||||
|
||||
impl RemoteError {
|
||||
@@ -71,6 +81,11 @@ impl std::fmt::Display for RemoteError {
|
||||
RemoteError::Transport(s) => write!(f, "network error: {s}"),
|
||||
RemoteError::ObjectStore(s) => write!(f, "object store: {s}"),
|
||||
RemoteError::Usage(s) => write!(f, "{s}"),
|
||||
RemoteError::TooLarge { len, limit } => write!(
|
||||
f,
|
||||
"the remote file is {len} bytes, more than the download limit of {limit} bytes"
|
||||
),
|
||||
RemoteError::Backend(s) => write!(f, "{s}"),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user