quantumclaw/clawhdf5
3
1
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases 1 Wiki Activity

clawhdf5-remote, h5rs: never allocate a length the server only claims

Browse Source 
h5rs check URL read the whole file with one read_at(0, len), len being
whatever Content-Range said. BlockCache listed every block index of the
span and preallocated len bytes: a server claiming 2^62 bytes for a 10 KB
file made h5rs abort (memory allocation of 35184372088832 bytes failed).

- BlockCache: a read spanning more than the budget (or eight max_requests)
  is fetched piece by piece and not kept, its output growing only as
  data arrives; read_ranges falls back to that per range; prefetch is
  clamped to the budget.
- New clawhdf5_remote::download(storage, max_bytes): refuses a claimed
  length above the limit (RemoteError::TooLarge) before any request, then
  reads in 64 MiB steps. New RemoteError::Backend for read errors.
- h5rs check downloads through it, with --max-download N (default 1 GiB).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-26 18:27:14 -05:00
co-authored by Claude Opus 5.5
parent e8aaf050be
commit 8df5b209a7
8 changed files with 239 additions and 20 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/clawhdf5-tools/README.md
+3 -1
View File
@@ -40,7 +40,9 @@ h5rs diff local.h5 http://127.0.0.1:8000/file.h5
```
A URL names the whole file (`FILE/OBJECT` suffixes are for local paths).
`check` validates every byte, so it downloads a remote file whole first.
`check` validates every byte, so it downloads a remote file whole first —
up to `--max-download N` bytes (default 1 GiB), refusing a longer file
before reading any of it.
The output is the local file's (`tests/remote.rs` compares every
subcommand).