clawhdf5-remote, h5rs: never allocate a length the server only claims
h5rs check URL read the whole file with one read_at(0, len), len being whatever Content-Range said. BlockCache listed every block index of the span and preallocated len bytes: a server claiming 2^62 bytes for a 10 KB file made h5rs abort (memory allocation of 35184372088832 bytes failed). - BlockCache: a read spanning more than the budget (or eight max_requests) is fetched piece by piece and not kept, its output growing only as data arrives; read_ranges falls back to that per range; prefetch is clamped to the budget. - New clawhdf5_remote::download(storage, max_bytes): refuses a claimed length above the limit (RemoteError::TooLarge) before any request, then reads in 64 MiB steps. New RemoteError::Backend for read errors. - h5rs check downloads through it, with --max-download N (default 1 GiB). Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
@@ -98,3 +98,24 @@ fn url_errors_are_clean() {
|
||||
assert!(out.contains("`https` feature"), "{out}");
|
||||
}
|
||||
}
|
||||
|
||||
/// `check` downloads a remote file whole, but never trusts the length the
|
||||
/// server claims: a server claiming 2^62 bytes for a small file is refused
|
||||
/// before anything is allocated (it used to abort the process), and
|
||||
/// `--max-download` caps real files too.
|
||||
#[test]
|
||||
fn check_refuses_a_remote_file_beyond_the_download_limit() {
|
||||
use std::sync::atomic::Ordering;
|
||||
let tall = Path::new(env!("CARGO_MANIFEST_DIR")).join("../clawhdf5/tests/fixtures/tall.h5");
|
||||
let server = server::Server::start(vec![("/t.h5".into(), std::fs::read(&tall).unwrap())]);
|
||||
let url = server.url("/t.h5");
|
||||
let (out, rc) = h5rs(&["check", &url]);
|
||||
assert_eq!(rc, 0, "{out}");
|
||||
let (out, rc) = h5rs(&["check", "--max-download", "1000", &url]);
|
||||
assert_eq!(rc, 2, "{out}");
|
||||
assert!(out.contains("download limit of 1000 bytes"), "{out}");
|
||||
server.shared.fake_total.store(1 << 62, Ordering::SeqCst);
|
||||
let (out, rc) = h5rs(&["check", &url]);
|
||||
assert_eq!(rc, 2, "{out}");
|
||||
assert!(out.contains("more than the download limit"), "{out}");
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user