clawhdf5-remote, h5rs: never allocate a length the server only claims

h5rs check URL read the whole file with one read_at(0, len), len being
whatever Content-Range said. BlockCache listed every block index of the
span and preallocated len bytes: a server claiming 2^62 bytes for a 10 KB
file made h5rs abort (memory allocation of 35184372088832 bytes failed).

- BlockCache: a read spanning more than the budget (or eight max_requests)
  is fetched piece by piece and not kept, its output growing only as
  data arrives; read_ranges falls back to that per range; prefetch is
  clamped to the budget.
- New clawhdf5_remote::download(storage, max_bytes): refuses a claimed
  length above the limit (RemoteError::TooLarge) before any request, then
  reads in 64 MiB steps. New RemoteError::Backend for read errors.
- h5rs check downloads through it, with --max-download N (default 1 GiB).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-26 18:27:14 -05:00
co-authored by Claude Opus 5.5
parent e8aaf050be
commit 8df5b209a7
8 changed files with 239 additions and 20 deletions
+21
View File
@@ -98,3 +98,24 @@ fn url_errors_are_clean() {
assert!(out.contains("`https` feature"), "{out}");
}
}
/// `check` downloads a remote file whole, but never trusts the length the
/// server claims: a server claiming 2^62 bytes for a small file is refused
/// before anything is allocated (it used to abort the process), and
/// `--max-download` caps real files too.
#[test]
fn check_refuses_a_remote_file_beyond_the_download_limit() {
use std::sync::atomic::Ordering;
let tall = Path::new(env!("CARGO_MANIFEST_DIR")).join("../clawhdf5/tests/fixtures/tall.h5");
let server = server::Server::start(vec![("/t.h5".into(), std::fs::read(&tall).unwrap())]);
let url = server.url("/t.h5");
let (out, rc) = h5rs(&["check", &url]);
assert_eq!(rc, 0, "{out}");
let (out, rc) = h5rs(&["check", "--max-download", "1000", &url]);
assert_eq!(rc, 2, "{out}");
assert!(out.contains("download limit of 1000 bytes"), "{out}");
server.shared.fake_total.store(1 << 62, Ordering::SeqCst);
let (out, rc) = h5rs(&["check", &url]);
assert_eq!(rc, 2, "{out}");
assert!(out.contains("more than the download limit"), "{out}");
}