From 955dd1c691fa127237358bb8fba50119f9fcad66 Mon Sep 17 00:00:00 2001 From: osobh Date: Sat, 26 Sep 2026 17:29:09 -0500 Subject: [PATCH] =?UTF-8?q?docs:=20range-read=20milestone=20M3=20=E2=80=94?= =?UTF-8?q?=20remote=20files?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit README: "Reading remote files" (open_url, the range_server and read_url examples with their real output against the fixtures, h5rs on URLs), the crate in the crate map and the unreleased highlights. CHANGELOG: the clawhdf5-remote crate, h5rs URLs, File::storage and VlResolver::element_in, with the request counts over the conformance corpus (tank, 2026-09-26, the command given). known-issues: the M2 range-read entry updated (the cache now exists; h5rs reads through storage) and a new entry for the remote backends' limits (no Python or browser URLs yet, fixed block size, cloud stores not run against a real bucket, validators, credentials). Design doc: M3 status with the choices that differ from the plan (a crate rather than a clawhdf5-io feature, ureq for HTTP so the default build has no C) and the corpus counts. Co-Authored-By: Claude Opus 5.5 (1M context) --- CHANGELOG.md | 56 +++++++++++++++++++++++++++++++ README.md | 56 +++++++++++++++++++++++++++++-- docs/design/range-reads.md | 43 ++++++++++++++++++++++-- docs/known-issues.md | 68 +++++++++++++++++++++++++++++--------- 4 files changed, 203 insertions(+), 20 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 661a2eb..69941c3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,62 @@ ## Unreleased +### Range reads, milestone M3: remote files (2026-09-26) +- **New crate `clawhdf5-remote`.** `open_url("http://host/file.h5")` gives + a `clawhdf5::File` (through `File::open_storage`) that reads the file by + HTTP `Range` requests; `storage_for_url` returns the cached storage to + read its statistics. Features: `http` (default; `ureq` without TLS, no + C), `https` (rustls with ring, which compiles C), `object-store` (any + `object_store` store, pure Rust), `s3`/`gcs`/`azure` (`s3://`, `gs://`, + `az://` URLs, configured from the environment; object_store's cloud + clients use aws-lc-rs, C). + - **`BlockCache`** wraps any `Storage`: aligned 1 MiB blocks (the size + `docs/design/range-reads.md` §2 measured), LRU with a byte budget + (64 MiB), the blocks one read misses fetched with one backend + `read_ranges` call as runs of consecutive blocks (a one-block gap is + fetched to merge two runs; at most 8 MiB per request). Readers on + several threads share it without holding its lock across a request, and + a block being fetched is waited for, not fetched again. A read that + misses more than half the budget is not kept (a large dataset does not + evict the metadata). A failed fetch is an error for every reader waiting + on it and is not cached. + - **`HttpStorage`**: opening is one ranged `GET` of the first block, + whose `Content-Range` gives the length. The file is pinned by its strong + `ETag` (`If-Match`) or else `Last-Modified` (`If-Unmodified-Since`), and + its length, checked on every response: a file changed while open is + `RemoteError::FileChanged`, never mixed data. A server that ignores + `Range` is refused without reading the body unless + `HttpOptions::allow_full_download`. Connection failures, timeouts, + `408`/`429`/`5xx` and cut-off bodies are retried with exponential + backoff; bodies are asked for with `Accept-Encoding: identity` and an + encoded one is refused. The ranges of one call are fetched in parallel. + - **`ObjectStoreStorage`** reads one object of any `object_store` store, + pinned by ETag (else version or modification time) and size. It blocks + on a small tokio runtime it owns; called from inside another runtime it + refuses (`RemoteError::Usage`) instead of blocking a worker. + `open_object(store, path, options)` opens a file through a block cache. + - Counted on the conformance corpus (tank, 2026-09-26, + `CLAWHDF5_REMOTE_CORPUS=conformance/.cache/corpus CLAWHDF5_REMOTE_REPORT=1 + cargo test --release -p clawhdf5-remote --test http -- --nocapture corpus`): + the 621 files that open (254 MB) read over HTTP exactly as through + `File::open`; opening and listing them all (every group's entries, every + dataset's shape and type) took 640 requests and 55.5 MB, and then + reading each file's largest dataset under 64 MiB 96 more requests + (171 MB in all). Without the cache the same work is 141 936 requests. + Listing the 7.7 MB IMERG file (file A of the design's §2) takes 2 + requests; the tests hold it to at most 3. +- **`h5rs` takes URLs** with the new `remote` feature (`remote-https` for + `https://`): `ls`, `dump`, `stat` and `diff` read by range requests; + `check` downloads the file whole, since it validates every byte. The + tools now read through `File::storage` and the format crate's `*_in` + functions; local output is unchanged. +- **`File::storage()`** (facade) returns the file's bytes from the + superblock on (the `as_bytes` view, cache image laid over) as a + `&(dyn Storage + Send + Sync)` for every backend, so code that parses a + file itself works on remote files too. +- **`VlResolver::element_in` / `string_element_in`** (format): the + `element`/`string_element` lookups over any `Storage`. + ### Range reads, milestone M2: raw data and `File::open_storage` (2026-09-26) - **`clawhdf5::File::open_storage(Arc)`** opens a file served by any `clawhdf5_format::storage::Storage` and gives the diff --git a/README.md b/README.md index 88ad61b..cfa82d4 100644 --- a/README.md +++ b/README.md @@ -96,6 +96,12 @@ breaking change, are in [CHANGELOG.md](CHANGELOG.md). filtered top-k, never slower than unfiltered), and opt-in re-ranking and confidence rejection, which used to be reachable only through `ClawhdfBackend`. +**Remote files (unreleased)** +- New crate `clawhdf5-remote`: `open_url("http://…")` reads a file on an + HTTP server (or in S3/GCS/Azure, opt-in) by range requests through a + block cache, without downloading it; `h5rs` takes URLs with its `remote` + feature. See [Reading remote files](#reading-remote-files). + **Tooling** - CI now runs the h5py/netCDF4 interop suites for real (they had been skipping silently) and runs an aarch64 job for the NEON kernels. @@ -450,6 +456,50 @@ Each call changes the file in place (no rewrite) and syncs it. What it cannot change safely is refused before anything is written; see [known issues](docs/known-issues.md) for the limits. +### Reading remote files + +[`clawhdf5-remote`](crates/clawhdf5-remote/README.md) opens a file on an +HTTP server (or, with its `s3`/`gcs`/`azure` features, in an object store) +without downloading it: the read API is the same `clawhdf5::File`, and +only the bytes an operation needs are fetched, by `Range` requests through +a block cache (1 MiB blocks; opening fetches the first one). A file that +changes on the server while it is open is an error, never a mix of old and +new bytes. + +```rust +let file = clawhdf5_remote::open_url("http://127.0.0.1:8000/tall.h5")?; +let values = file.dataset("/g2/dset2.1")?.read_f64()?; +``` + +To try it without a server of your own, the crate's test server serves a +directory with range support: + +```bash +cargo run -p clawhdf5-remote --example range_server -- crates/clawhdf5/tests/fixtures 127.0.0.1:8000 +# in another shell: list the file, read one dataset, print what it cost +cargo run -p clawhdf5-remote --example read_url -- http://127.0.0.1:8000/tall.h5 /g2/dset2.1 +``` + +```text +/g1 group +/g2 group +/g2/dset2.1 dataset [10] F32 +/g2/dset2.2 dataset [3, 5] F32 +/g1/g1.1 group +/g1/g1.2 group +/g1/g1.2/g1.2.1 group +/g1/g1.1/dset1.1.1 dataset [10, 10] I32 +/g1/g1.1/dset1.1.2 dataset [20] I32 +/g2/dset2.1: 10 values, first [1.0, 1.100000023841858, 1.2000000476837158, ...] +1 range requests (the one at open included), 9968 bytes fetched, 9968 bytes cached +``` + +(`tall.h5` is 9 968 bytes, so the first block holds all of it.) `h5rs` +built with `--features remote` takes the same URLs: +`h5rs ls -r http://127.0.0.1:8000/tall.h5`. Plain HTTP builds no C; +`https://` is the `https` feature (rustls with ring, which compiles C). +Limits are in [known issues](docs/known-issues.md). + ### Python `crates/clawhdf5-py` is a Python package (PyO3 + numpy) that reads HDF5 with @@ -678,7 +728,7 @@ let exported = backend.export_markdown("MEMORY.md")?; ## Crate Map ``` -clawhdf5 workspace (17 crates, ~86K lines of Rust in src/, ~104K with tests +clawhdf5 workspace (19 crates, ~86K lines of Rust in src/, ~104K with tests and benches; plus libaec-sys, an internal FFI bindings crate for the optional szip feature) │ @@ -690,7 +740,8 @@ clawhdf5 workspace (17 crates, ~86K lines of Rust in src/, ~104K with tests │ ├── clawhdf5 — High-level API │ ├── clawhdf5-netcdf4 — NetCDF-4 support │ ├── clawhdf5-accel — SIMD (AVX2, NEON incl. SDOT int8; AVX-512 behind `avx512`) -│ └── clawhdf5-gpu — GPU compute (wgpu, hand-written WGSL compute shaders) +│ ├── clawhdf5-gpu — GPU compute (wgpu, hand-written WGSL compute shaders) +│ └── clawhdf5-remote — Remote files: HTTP(S) range requests, object stores, block cache │ ├── Agent Memory │ ├── clawhdf5-agent — Memory engine (24.7K lines, 32 modules; chained-CRC WAL) @@ -705,6 +756,7 @@ clawhdf5 workspace (17 crates, ~86K lines of Rust in src/, ~104K with tests │ └── clawhdf5-wasm — Browser (WebAssembly, wasm-bindgen; read-only) │ └── Tooling + ├── clawhdf5-tools — h5rs: ls, dump, stat, diff, check └── clawhdf5-bench — Benchmark suite ``` diff --git a/docs/design/range-reads.md b/docs/design/range-reads.md index 1098922..2c7d1a4 100644 --- a/docs/design/range-reads.md +++ b/docs/design/range-reads.md @@ -5,8 +5,10 @@ change. Progress: M0 and M1 are done, and so is M2 (branch `feat/p3-m2-raw-data`): every read path of the format crate works through `Storage`, v2 B-trees, dense groups and raw data included, and `File::open_storage` gives the facade's read API over any `Storage` (see -`CHANGELOG.md`, "Range reads, milestone M2"). M3 (a remote backend with -its block cache) is next. Every count in §1–§2 was +`CHANGELOG.md`, "Range reads, milestone M2"). M3 is done on branch +`feat/p3-m3-remote`: the `clawhdf5-remote` crate (block cache, HTTP(S), +object stores) and URLs in `h5rs` (see the M3 status below). M4 (wasm) is +next. Every count in §1–§2 was taken on `tank` on 2026-09-26 at commit `de2a53f`, with the commands given next to it. No timing numbers appear here on purpose: the machine was shared with other build jobs when this was written. @@ -458,6 +460,43 @@ fast path within benchmark noise. §2; the page size for paged files; the first block prefetched on open) and a request counter exposed for tests and users. - Python bindings: `clawhdf5.File("s3://…")` / `https://` through it. +- *Status 2026-09-26:* done on branch `feat/p3-m3-remote`, except the + Python bindings, with these choices: + - A new crate, `clawhdf5-remote`, instead of a `remote` feature of + `clawhdf5-io`: `open_url` returns a `clawhdf5::File`, and `clawhdf5-io` + sits below the facade. + - HTTP(S) through `ureq` (`HttpStorage`), not object_store's HTTP store: + that one pulls reqwest with aws-lc-rs (C), while plain HTTP through + ureq builds no C, so it is the default feature; `https` adds rustls + with ring. S3/GCS/Azure go through `object_store` (`ObjectStoreStorage`, + features `s3`/`gcs`/`azure`, opt-in because of aws-lc-rs); the + `object-store` feature alone (in-memory, local files, a store you + build) is pure Rust. object_store is async: the storage blocks on a + two-thread tokio runtime of its own, and refuses to run inside another + runtime. + - `BlockCache` (any `Storage`): 1 MiB blocks and a 64 MiB LRU budget by + default, the first block fetched at open (for HTTP by the request + that learns the length), the missing blocks of one read fetched as + runs of consecutive blocks in one parallel batch, per-block in-flight + deduplication across threads, and reads that miss more than half the + budget not kept. The page size of paged files is not used as the block + size yet. `CacheStats` and `HttpStats` count requests and bytes. + - The file is pinned at open by ETag (else Last-Modified, or the object's + version) and length; a change is an error, not mixed data. + - `h5rs` (feature `remote`) reads through the new `File::storage()`, the + file's view as a `Storage`, so its parsing works on remote files; its + `check` downloads the file whole. + - Measured with `crates/clawhdf5-remote/tests/http.rs` (tank, 2026-09-26, + `CLAWHDF5_REMOTE_CORPUS=conformance/.cache/corpus + CLAWHDF5_REMOTE_REPORT=1 cargo test --release -p clawhdf5-remote --test + http -- --nocapture corpus`), requests as the test server counted + them: the 621 corpus files that open read over HTTP exactly as through + `File::open`. Open + list (every group's entries, every dataset's shape + and type) of all of them: 640 requests, 55.5 MB of 254 MB; then reading + each file's largest dataset under 64 MiB: 96 more (171 MB in all). The + same work without a cache: 141 936 requests. Per file: A lists in 2 + requests (§2 predicted 2 blocks of 1 MiB), B in 1, C in 7 (its whole + 6.4 MB: 35 001 object headers spread over the file). **M4 — wasm lazy loading (1–2 weeks).** - `clawhdf5-wasm`: `openUrl(url) -> Promise` backed by `fetch` with a diff --git a/docs/known-issues.md b/docs/known-issues.md index 313bc8e..860e1eb 100644 --- a/docs/known-issues.md +++ b/docs/known-issues.md @@ -766,20 +766,22 @@ which is what libhdf5 itself writes. ## Range reads (`File::open_storage`) limits **Status:** open (added 2026-09-26, milestone M2 of -`docs/design/range-reads.md`). `File::open_storage` reads any -`clawhdf5_format::storage::Storage` through the whole read API, and every -format-crate read path works through `Storage::read_at`/`read_ranges`, but: +`docs/design/range-reads.md`; remote backends added by M3). `File::open_storage` +reads any `clawhdf5_format::storage::Storage` through the whole read API, +every format-crate read path works through `Storage::read_at`/`read_ranges`, +and `clawhdf5-remote` serves HTTP(S) and object-store files through a block +cache, but: -- **No remote backend and no block cache yet** (milestone M3). A `Storage` - is asked for each structure as the parsers need it, several times over - for some (an object header is re-read by each lookup through it): one - pass over the conformance corpus — open, list, every attribute, every - dataset once — is 176 092 `read_at` calls for 621 files, 92 489 of them - for the 35 001-group `h5stat_newgrat.h5` (2026-09-26, tank, - `crates/clawhdf5/tests/storage_equivalence.rs` with - `CLAWHDF5_STORAGE_CORPUS`). A backend over a network needs a cache in - front of it. `Storage::read_ranges` defaults to one `read_at` per range; - coalescing is the backend's job. +- **A `Storage` without a cache is asked for each structure as the parsers + need it**, several times over for some (an object header is re-read by + each lookup through it): one pass over the conformance corpus — open, + list, every attribute, every dataset once — is 176 092 `read_at` calls + for 621 files, 92 489 of them for the 35 001-group `h5stat_newgrat.h5` + (2026-09-26, tank, `crates/clawhdf5/tests/storage_equivalence.rs` with + `CLAWHDF5_STORAGE_CORPUS`). A backend of your own over a network needs a + cache in front of it: wrap it in `clawhdf5_remote::BlockCache`, as + `open_url` does. `Storage::read_ranges` defaults to one `read_at` per + range; coalescing is the backend's (or the cache's) job. - A group lookup by name in a version-1 (symbol-table) group lists the whole group (dense groups use their name index). Over a range backend that is one read per symbol-table node and name, per lookup. @@ -790,10 +792,12 @@ format-crate read path works through `Storage::read_at`/`read_ranges`, but: `read_*_zerocopy`) need the file in memory and answer `FormatError::ContiguousStorageRequired` otherwise; `File::as_bytes()` panics for such a file (`File::contiguous_bytes()` is the fallible form). - `LazyFile`, `MmapFile`, the Python and wasm bindings and `h5rs` still read - a whole file. + `LazyFile`, `MmapFile` and the Python and wasm bindings still read a + whole file (`h5rs` reads through `File::storage`, and takes URLs with its + `remote` feature). - The file's length is read once, at open: a growing file (SWMR) is not - followed (milestone M5). + followed (milestone M5). A remote file is pinned at open, so one that + grows is `RemoteError::FileChanged`. - Not new, but visible through the equivalence tests: a full read through the file's chunk cache (`read_raw_data_cached`, `read_raw_data_indexed`, and so `Dataset::read_*`) lists a damaged dataset's chunks in hash-map @@ -801,6 +805,38 @@ format-crate read path works through `Storage::read_at`/`read_ranges`, but: the next (`cve-2025-2310.h5`); the values of a dataset that reads are not affected. +## Remote files (`clawhdf5-remote`) limits + +**Status:** open (added 2026-09-26, milestone M3 of +`docs/design/range-reads.md`). + +- **Python and the browser cannot open URLs yet.** `clawhdf5.File` (PyO3) + parses through `File::as_bytes`, which a remote file does not have; the + wasm reader's `openUrl` is milestone M4. +- **The block size is fixed** (1 MiB unless `CacheConfig` says otherwise). + The design's policy of using a paged file's page size as the block size + is not implemented, and only the first block is read ahead. +- **Checked against local servers and one public one.** The tests use an + in-process HTTP/1.1 server and object_store's in-memory and local-file + stores. HTTPS was checked by hand against `raw.githubusercontent.com` + (2026-09-26, tank: `h5rs dump` of h5py's `vlen_string_dset.h5` by URL + equals the downloaded file's). The `s3`, `gcs` and `azure` backends are + built and their URL parsing tested, but they have not been run against a + real bucket. +- **A server with neither a strong ETag nor Last-Modified** can only be + checked by length, so a same-length replacement mid-read would go + unnoticed; `HttpOptions::require_validator` refuses such servers. A weak + ETag (`W/"…"`) cannot be sent as `If-Match`, so it counts as none. +- **Credentials:** HTTP takes extra headers (`HttpOptions::headers`, e.g. + `Authorization`); `h5rs` has no option for them. The cloud stores read + credentials from the environment only. +- Each `ObjectStoreStorage` owns a tokio runtime with two worker threads. +- `h5rs check` downloads a remote file whole (it validates every byte), and + a URL cannot carry a `FILE/OBJECT` suffix; `h5rs` uses the default cache + settings. +- The zero-copy methods and `File::as_bytes` are unavailable on a remote + file (see the range-read limits above). + ## `clawhdf5-wasm` (browser) limits **Status:** open (by design for now; added 2026-09-26).