docs: range-read milestone M3 — remote files

README: "Reading remote files" (open_url, the range_server and read_url
examples with their real output against the fixtures, h5rs on URLs), the
crate in the crate map and the unreleased highlights. CHANGELOG: the
clawhdf5-remote crate, h5rs URLs, File::storage and
VlResolver::element_in, with the request counts over the conformance
corpus (tank, 2026-09-26, the command given). known-issues: the M2
range-read entry updated (the cache now exists; h5rs reads through
storage) and a new entry for the remote backends' limits (no Python or
browser URLs yet, fixed block size, cloud stores not run against a real
bucket, validators, credentials). Design doc: M3 status with the choices
that differ from the plan (a crate rather than a clawhdf5-io feature,
ureq for HTTP so the default build has no C) and the corpus counts.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

docs/design/range-reads.md

@@ -5,8 +5,10 @@ change. Progress: M0 and M1 are done, and so is M2 (branch
 `feat/p3-m2-raw-data`): every read path of the format crate works through
 `Storage`, v2 B-trees, dense groups and raw data included, and
 `File::open_storage` gives the facade's read API over any `Storage` (see
-`CHANGELOG.md`, "Range reads, milestone M2"). M3 (a remote backend with
-its block cache) is next. Every count in §1–§2 was
+`CHANGELOG.md`, "Range reads, milestone M2"). M3 is done on branch
+`feat/p3-m3-remote`: the `clawhdf5-remote` crate (block cache, HTTP(S),
+object stores) and URLs in `h5rs` (see the M3 status below). M4 (wasm) is
+next. Every count in §1–§2 was
 taken on `tank` on 2026-09-26 at commit `de2a53f`, with the commands given
 next to it. No timing numbers appear here on purpose: the machine was shared
 with other build jobs when this was written.
@@ -458,6 +460,43 @@ fast path within benchmark noise.
   §2; the page size for paged files; the first block prefetched on open) and
   a request counter exposed for tests and users.
 - Python bindings: `clawhdf5.File("s3://…")` / `https://` through it.
+- *Status 2026-09-26:* done on branch `feat/p3-m3-remote`, except the
+  Python bindings, with these choices:
+  - A new crate, `clawhdf5-remote`, instead of a `remote` feature of
+    `clawhdf5-io`: `open_url` returns a `clawhdf5::File`, and `clawhdf5-io`
+    sits below the facade.
+  - HTTP(S) through `ureq` (`HttpStorage`), not object_store's HTTP store:
+    that one pulls reqwest with aws-lc-rs (C), while plain HTTP through
+    ureq builds no C, so it is the default feature; `https` adds rustls
+    with ring. S3/GCS/Azure go through `object_store` (`ObjectStoreStorage`,
+    features `s3`/`gcs`/`azure`, opt-in because of aws-lc-rs); the
+    `object-store` feature alone (in-memory, local files, a store you
+    build) is pure Rust. object_store is async: the storage blocks on a
+    two-thread tokio runtime of its own, and refuses to run inside another
+    runtime.
+  - `BlockCache` (any `Storage`): 1 MiB blocks and a 64 MiB LRU budget by
+    default, the first block fetched at open (for HTTP by the request
+    that learns the length), the missing blocks of one read fetched as
+    runs of consecutive blocks in one parallel batch, per-block in-flight
+    deduplication across threads, and reads that miss more than half the
+    budget not kept. The page size of paged files is not used as the block
+    size yet. `CacheStats` and `HttpStats` count requests and bytes.
+  - The file is pinned at open by ETag (else Last-Modified, or the object's
+    version) and length; a change is an error, not mixed data.
+  - `h5rs` (feature `remote`) reads through the new `File::storage()`, the
+    file's view as a `Storage`, so its parsing works on remote files; its
+    `check` downloads the file whole.
+  - Measured with `crates/clawhdf5-remote/tests/http.rs` (tank, 2026-09-26,
+    `CLAWHDF5_REMOTE_CORPUS=conformance/.cache/corpus
+    CLAWHDF5_REMOTE_REPORT=1 cargo test --release -p clawhdf5-remote --test
+    http -- --nocapture corpus`), requests as the test server counted
+    them: the 621 corpus files that open read over HTTP exactly as through
+    `File::open`. Open + list (every group's entries, every dataset's shape
+    and type) of all of them: 640 requests, 55.5 MB of 254 MB; then reading
+    each file's largest dataset under 64 MiB: 96 more (171 MB in all). The
+    same work without a cache: 141 936 requests. Per file: A lists in 2
+    requests (§2 predicted 2 blocks of 1 MiB), B in 1, C in 7 (its whole
+    6.4 MB: 35 001 object headers spread over the file).
 
 **M4 — wasm lazy loading (1–2 weeks).**
 - `clawhdf5-wasm`: `openUrl(url) -> Promise<H5File>` backed by `fetch` with a

docs/known-issues.md

@@ -766,20 +766,22 @@ which is what libhdf5 itself writes.
 ## Range reads (`File::open_storage`) limits
 
 **Status:** open (added 2026-09-26, milestone M2 of
-`docs/design/range-reads.md`). `File::open_storage` reads any
-`clawhdf5_format::storage::Storage` through the whole read API, and every
-format-crate read path works through `Storage::read_at`/`read_ranges`, but:
+`docs/design/range-reads.md`; remote backends added by M3). `File::open_storage`
+reads any `clawhdf5_format::storage::Storage` through the whole read API,
+every format-crate read path works through `Storage::read_at`/`read_ranges`,
+and `clawhdf5-remote` serves HTTP(S) and object-store files through a block
+cache, but:
 
-- **No remote backend and no block cache yet** (milestone M3). A `Storage`
-  is asked for each structure as the parsers need it, several times over
-  for some (an object header is re-read by each lookup through it): one
-  pass over the conformance corpus — open, list, every attribute, every
-  dataset once — is 176 092 `read_at` calls for 621 files, 92 489 of them
-  for the 35 001-group `h5stat_newgrat.h5` (2026-09-26, tank,
-  `crates/clawhdf5/tests/storage_equivalence.rs` with
-  `CLAWHDF5_STORAGE_CORPUS`). A backend over a network needs a cache in
-  front of it. `Storage::read_ranges` defaults to one `read_at` per range;
-  coalescing is the backend's job.
+- **A `Storage` without a cache is asked for each structure as the parsers
+  need it**, several times over for some (an object header is re-read by
+  each lookup through it): one pass over the conformance corpus — open,
+  list, every attribute, every dataset once — is 176 092 `read_at` calls
+  for 621 files, 92 489 of them for the 35 001-group `h5stat_newgrat.h5`
+  (2026-09-26, tank, `crates/clawhdf5/tests/storage_equivalence.rs` with
+  `CLAWHDF5_STORAGE_CORPUS`). A backend of your own over a network needs a
+  cache in front of it: wrap it in `clawhdf5_remote::BlockCache`, as
+  `open_url` does. `Storage::read_ranges` defaults to one `read_at` per
+  range; coalescing is the backend's (or the cache's) job.
 - A group lookup by name in a version-1 (symbol-table) group lists the whole
   group (dense groups use their name index). Over a range backend that is
   one read per symbol-table node and name, per lookup.
@@ -790,10 +792,12 @@ format-crate read path works through `Storage::read_at`/`read_ranges`, but:
   `read_*_zerocopy`) need the file in memory and answer
   `FormatError::ContiguousStorageRequired` otherwise; `File::as_bytes()`
   panics for such a file (`File::contiguous_bytes()` is the fallible form).
-  `LazyFile`, `MmapFile`, the Python and wasm bindings and `h5rs` still read
-  a whole file.
+  `LazyFile`, `MmapFile` and the Python and wasm bindings still read a
+  whole file (`h5rs` reads through `File::storage`, and takes URLs with its
+  `remote` feature).
 - The file's length is read once, at open: a growing file (SWMR) is not
-  followed (milestone M5).
+  followed (milestone M5). A remote file is pinned at open, so one that
+  grows is `RemoteError::FileChanged`.
 - Not new, but visible through the equivalence tests: a full read through
   the file's chunk cache (`read_raw_data_cached`, `read_raw_data_indexed`,
   and so `Dataset::read_*`) lists a damaged dataset's chunks in hash-map
@@ -801,6 +805,38 @@ format-crate read path works through `Storage::read_at`/`read_ranges`, but:
   the next (`cve-2025-2310.h5`); the values of a dataset that reads are
   not affected.
 
+## Remote files (`clawhdf5-remote`) limits
+
+**Status:** open (added 2026-09-26, milestone M3 of
+`docs/design/range-reads.md`).
+
+- **Python and the browser cannot open URLs yet.** `clawhdf5.File` (PyO3)
+  parses through `File::as_bytes`, which a remote file does not have; the
+  wasm reader's `openUrl` is milestone M4.
+- **The block size is fixed** (1 MiB unless `CacheConfig` says otherwise).
+  The design's policy of using a paged file's page size as the block size
+  is not implemented, and only the first block is read ahead.
+- **Checked against local servers and one public one.** The tests use an
+  in-process HTTP/1.1 server and object_store's in-memory and local-file
+  stores. HTTPS was checked by hand against `raw.githubusercontent.com`
+  (2026-09-26, tank: `h5rs dump` of h5py's `vlen_string_dset.h5` by URL
+  equals the downloaded file's). The `s3`, `gcs` and `azure` backends are
+  built and their URL parsing tested, but they have not been run against a
+  real bucket.
+- **A server with neither a strong ETag nor Last-Modified** can only be
+  checked by length, so a same-length replacement mid-read would go
+  unnoticed; `HttpOptions::require_validator` refuses such servers. A weak
+  ETag (`W/"…"`) cannot be sent as `If-Match`, so it counts as none.
+- **Credentials:** HTTP takes extra headers (`HttpOptions::headers`, e.g.
+  `Authorization`); `h5rs` has no option for them. The cloud stores read
+  credentials from the environment only.
+- Each `ObjectStoreStorage` owns a tokio runtime with two worker threads.
+- `h5rs check` downloads a remote file whole (it validates every byte), and
+  a URL cannot carry a `FILE/OBJECT` suffix; `h5rs` uses the default cache
+  settings.
+- The zero-copy methods and `File::as_bytes` are unavailable on a remote
+  file (see the range-read limits above).
+
 ## `clawhdf5-wasm` (browser) limits
 
 **Status:** open (by design for now; added 2026-09-26).