docs: range-read milestone M3 — remote files

README: "Reading remote files" (open_url, the range_server and read_url
examples with their real output against the fixtures, h5rs on URLs), the
crate in the crate map and the unreleased highlights. CHANGELOG: the
clawhdf5-remote crate, h5rs URLs, File::storage and
VlResolver::element_in, with the request counts over the conformance
corpus (tank, 2026-09-26, the command given). known-issues: the M2
range-read entry updated (the cache now exists; h5rs reads through
storage) and a new entry for the remote backends' limits (no Python or
browser URLs yet, fixed block size, cloud stores not run against a real
bucket, validators, credentials). Design doc: M3 status with the choices
that differ from the plan (a crate rather than a clawhdf5-io feature,
ureq for HTTP so the default build has no C) and the corpus counts.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-26 17:29:09 -05:00
co-authored by Claude Opus 5.5
parent ebe51f8e97
commit 955dd1c691
4 changed files with 203 additions and 20 deletions
+56
View File
@@ -2,6 +2,62 @@
## Unreleased
### Range reads, milestone M3: remote files (2026-09-26)
- **New crate `clawhdf5-remote`.** `open_url("http://host/file.h5")` gives
a `clawhdf5::File` (through `File::open_storage`) that reads the file by
HTTP `Range` requests; `storage_for_url` returns the cached storage to
read its statistics. Features: `http` (default; `ureq` without TLS, no
C), `https` (rustls with ring, which compiles C), `object-store` (any
`object_store` store, pure Rust), `s3`/`gcs`/`azure` (`s3://`, `gs://`,
`az://` URLs, configured from the environment; object_store's cloud
clients use aws-lc-rs, C).
- **`BlockCache`** wraps any `Storage`: aligned 1 MiB blocks (the size
`docs/design/range-reads.md` §2 measured), LRU with a byte budget
(64 MiB), the blocks one read misses fetched with one backend
`read_ranges` call as runs of consecutive blocks (a one-block gap is
fetched to merge two runs; at most 8 MiB per request). Readers on
several threads share it without holding its lock across a request, and
a block being fetched is waited for, not fetched again. A read that
misses more than half the budget is not kept (a large dataset does not
evict the metadata). A failed fetch is an error for every reader waiting
on it and is not cached.
- **`HttpStorage`**: opening is one ranged `GET` of the first block,
whose `Content-Range` gives the length. The file is pinned by its strong
`ETag` (`If-Match`) or else `Last-Modified` (`If-Unmodified-Since`), and
its length, checked on every response: a file changed while open is
`RemoteError::FileChanged`, never mixed data. A server that ignores
`Range` is refused without reading the body unless
`HttpOptions::allow_full_download`. Connection failures, timeouts,
`408`/`429`/`5xx` and cut-off bodies are retried with exponential
backoff; bodies are asked for with `Accept-Encoding: identity` and an
encoded one is refused. The ranges of one call are fetched in parallel.
- **`ObjectStoreStorage`** reads one object of any `object_store` store,
pinned by ETag (else version or modification time) and size. It blocks
on a small tokio runtime it owns; called from inside another runtime it
refuses (`RemoteError::Usage`) instead of blocking a worker.
`open_object(store, path, options)` opens a file through a block cache.
- Counted on the conformance corpus (tank, 2026-09-26,
`CLAWHDF5_REMOTE_CORPUS=conformance/.cache/corpus CLAWHDF5_REMOTE_REPORT=1
cargo test --release -p clawhdf5-remote --test http -- --nocapture corpus`):
the 621 files that open (254 MB) read over HTTP exactly as through
`File::open`; opening and listing them all (every group's entries, every
dataset's shape and type) took 640 requests and 55.5 MB, and then
reading each file's largest dataset under 64 MiB 96 more requests
(171 MB in all). Without the cache the same work is 141 936 requests.
Listing the 7.7 MB IMERG file (file A of the design's §2) takes 2
requests; the tests hold it to at most 3.
- **`h5rs` takes URLs** with the new `remote` feature (`remote-https` for
`https://`): `ls`, `dump`, `stat` and `diff` read by range requests;
`check` downloads the file whole, since it validates every byte. The
tools now read through `File::storage` and the format crate's `*_in`
functions; local output is unchanged.
- **`File::storage()`** (facade) returns the file's bytes from the
superblock on (the `as_bytes` view, cache image laid over) as a
`&(dyn Storage + Send + Sync)` for every backend, so code that parses a
file itself works on remote files too.
- **`VlResolver::element_in` / `string_element_in`** (format): the
`element`/`string_element` lookups over any `Storage`.
### Range reads, milestone M2: raw data and `File::open_storage` (2026-09-26)
- **`clawhdf5::File::open_storage(Arc<dyn Storage + Send + Sync>)`** opens
a file served by any `clawhdf5_format::storage::Storage` and gives the