docs: range-read milestone M3 — remote files
README: "Reading remote files" (open_url, the range_server and read_url examples with their real output against the fixtures, h5rs on URLs), the crate in the crate map and the unreleased highlights. CHANGELOG: the clawhdf5-remote crate, h5rs URLs, File::storage and VlResolver::element_in, with the request counts over the conformance corpus (tank, 2026-09-26, the command given). known-issues: the M2 range-read entry updated (the cache now exists; h5rs reads through storage) and a new entry for the remote backends' limits (no Python or browser URLs yet, fixed block size, cloud stores not run against a real bucket, validators, credentials). Design doc: M3 status with the choices that differ from the plan (a crate rather than a clawhdf5-io feature, ureq for HTTP so the default build has no C) and the corpus counts. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
@@ -2,6 +2,62 @@
|
||||
|
||||
## Unreleased
|
||||
|
||||
### Range reads, milestone M3: remote files (2026-09-26)
|
||||
- **New crate `clawhdf5-remote`.** `open_url("http://host/file.h5")` gives
|
||||
a `clawhdf5::File` (through `File::open_storage`) that reads the file by
|
||||
HTTP `Range` requests; `storage_for_url` returns the cached storage to
|
||||
read its statistics. Features: `http` (default; `ureq` without TLS, no
|
||||
C), `https` (rustls with ring, which compiles C), `object-store` (any
|
||||
`object_store` store, pure Rust), `s3`/`gcs`/`azure` (`s3://`, `gs://`,
|
||||
`az://` URLs, configured from the environment; object_store's cloud
|
||||
clients use aws-lc-rs, C).
|
||||
- **`BlockCache`** wraps any `Storage`: aligned 1 MiB blocks (the size
|
||||
`docs/design/range-reads.md` §2 measured), LRU with a byte budget
|
||||
(64 MiB), the blocks one read misses fetched with one backend
|
||||
`read_ranges` call as runs of consecutive blocks (a one-block gap is
|
||||
fetched to merge two runs; at most 8 MiB per request). Readers on
|
||||
several threads share it without holding its lock across a request, and
|
||||
a block being fetched is waited for, not fetched again. A read that
|
||||
misses more than half the budget is not kept (a large dataset does not
|
||||
evict the metadata). A failed fetch is an error for every reader waiting
|
||||
on it and is not cached.
|
||||
- **`HttpStorage`**: opening is one ranged `GET` of the first block,
|
||||
whose `Content-Range` gives the length. The file is pinned by its strong
|
||||
`ETag` (`If-Match`) or else `Last-Modified` (`If-Unmodified-Since`), and
|
||||
its length, checked on every response: a file changed while open is
|
||||
`RemoteError::FileChanged`, never mixed data. A server that ignores
|
||||
`Range` is refused without reading the body unless
|
||||
`HttpOptions::allow_full_download`. Connection failures, timeouts,
|
||||
`408`/`429`/`5xx` and cut-off bodies are retried with exponential
|
||||
backoff; bodies are asked for with `Accept-Encoding: identity` and an
|
||||
encoded one is refused. The ranges of one call are fetched in parallel.
|
||||
- **`ObjectStoreStorage`** reads one object of any `object_store` store,
|
||||
pinned by ETag (else version or modification time) and size. It blocks
|
||||
on a small tokio runtime it owns; called from inside another runtime it
|
||||
refuses (`RemoteError::Usage`) instead of blocking a worker.
|
||||
`open_object(store, path, options)` opens a file through a block cache.
|
||||
- Counted on the conformance corpus (tank, 2026-09-26,
|
||||
`CLAWHDF5_REMOTE_CORPUS=conformance/.cache/corpus CLAWHDF5_REMOTE_REPORT=1
|
||||
cargo test --release -p clawhdf5-remote --test http -- --nocapture corpus`):
|
||||
the 621 files that open (254 MB) read over HTTP exactly as through
|
||||
`File::open`; opening and listing them all (every group's entries, every
|
||||
dataset's shape and type) took 640 requests and 55.5 MB, and then
|
||||
reading each file's largest dataset under 64 MiB 96 more requests
|
||||
(171 MB in all). Without the cache the same work is 141 936 requests.
|
||||
Listing the 7.7 MB IMERG file (file A of the design's §2) takes 2
|
||||
requests; the tests hold it to at most 3.
|
||||
- **`h5rs` takes URLs** with the new `remote` feature (`remote-https` for
|
||||
`https://`): `ls`, `dump`, `stat` and `diff` read by range requests;
|
||||
`check` downloads the file whole, since it validates every byte. The
|
||||
tools now read through `File::storage` and the format crate's `*_in`
|
||||
functions; local output is unchanged.
|
||||
- **`File::storage()`** (facade) returns the file's bytes from the
|
||||
superblock on (the `as_bytes` view, cache image laid over) as a
|
||||
`&(dyn Storage + Send + Sync)` for every backend, so code that parses a
|
||||
file itself works on remote files too.
|
||||
- **`VlResolver::element_in` / `string_element_in`** (format): the
|
||||
`element`/`string_element` lookups over any `Storage`.
|
||||
|
||||
### Range reads, milestone M2: raw data and `File::open_storage` (2026-09-26)
|
||||
- **`clawhdf5::File::open_storage(Arc<dyn Storage + Send + Sync>)`** opens
|
||||
a file served by any `clawhdf5_format::storage::Storage` and gives the
|
||||
|
||||
Reference in New Issue
Block a user