quantumclaw/clawhdf5
3
1
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases 1 Wiki Activity

docs: range-read milestone M3 — remote files

Browse Source 
README: "Reading remote files" (open_url, the range_server and read_url
examples with their real output against the fixtures, h5rs on URLs), the
crate in the crate map and the unreleased highlights. CHANGELOG: the
clawhdf5-remote crate, h5rs URLs, File::storage and
VlResolver::element_in, with the request counts over the conformance
corpus (tank, 2026-09-26, the command given). known-issues: the M2
range-read entry updated (the cache now exists; h5rs reads through
storage) and a new entry for the remote backends' limits (no Python or
browser URLs yet, fixed block size, cloud stores not run against a real
bucket, validators, credentials). Design doc: M3 status with the choices
that differ from the plan (a crate rather than a clawhdf5-io feature,
ureq for HTTP so the default build has no C) and the corpus counts.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-26 17:29:09 -05:00
co-authored by Claude Opus 5.5
parent ebe51f8e97
commit 955dd1c691
4 changed files with 203 additions and 20 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+56
View File
@@ -2,6 +2,62 @@
## Unreleased
### Range reads, milestone M3: remote files (2026-09-26)
- **New crate `clawhdf5-remote`.** `open_url("http://host/file.h5")` gives
a `clawhdf5::File` (through `File::open_storage`) that reads the file by
HTTP `Range` requests; `storage_for_url` returns the cached storage to
read its statistics. Features: `http` (default; `ureq` without TLS, no
C), `https` (rustls with ring, which compiles C), `object-store` (any
`object_store` store, pure Rust), `s3`/`gcs`/`azure` (`s3://`, `gs://`,
`az://` URLs, configured from the environment; object_store's cloud
clients use aws-lc-rs, C).
- **`BlockCache`** wraps any `Storage`: aligned 1 MiB blocks (the size
`docs/design/range-reads.md` §2 measured), LRU with a byte budget
(64 MiB), the blocks one read misses fetched with one backend
`read_ranges` call as runs of consecutive blocks (a one-block gap is
fetched to merge two runs; at most 8 MiB per request). Readers on
several threads share it without holding its lock across a request, and
a block being fetched is waited for, not fetched again. A read that
misses more than half the budget is not kept (a large dataset does not
evict the metadata). A failed fetch is an error for every reader waiting
on it and is not cached.
- **`HttpStorage`**: opening is one ranged `GET` of the first block,
whose `Content-Range` gives the length. The file is pinned by its strong
`ETag` (`If-Match`) or else `Last-Modified` (`If-Unmodified-Since`), and
its length, checked on every response: a file changed while open is
`RemoteError::FileChanged`, never mixed data. A server that ignores
`Range` is refused without reading the body unless
`HttpOptions::allow_full_download`. Connection failures, timeouts,
`408`/`429`/`5xx` and cut-off bodies are retried with exponential
backoff; bodies are asked for with `Accept-Encoding: identity` and an
encoded one is refused. The ranges of one call are fetched in parallel.
- **`ObjectStoreStorage`** reads one object of any `object_store` store,
pinned by ETag (else version or modification time) and size. It blocks
on a small tokio runtime it owns; called from inside another runtime it
refuses (`RemoteError::Usage`) instead of blocking a worker.
`open_object(store, path, options)` opens a file through a block cache.
- Counted on the conformance corpus (tank, 2026-09-26,
`CLAWHDF5_REMOTE_CORPUS=conformance/.cache/corpus CLAWHDF5_REMOTE_REPORT=1
cargo test --release -p clawhdf5-remote --test http -- --nocapture corpus`):
the 621 files that open (254 MB) read over HTTP exactly as through
`File::open`; opening and listing them all (every group's entries, every
dataset's shape and type) took 640 requests and 55.5 MB, and then
reading each file's largest dataset under 64 MiB 96 more requests
(171 MB in all). Without the cache the same work is 141 936 requests.
Listing the 7.7 MB IMERG file (file A of the design's §2) takes 2
requests; the tests hold it to at most 3.
- **`h5rs` takes URLs** with the new `remote` feature (`remote-https` for
`https://`): `ls`, `dump`, `stat` and `diff` read by range requests;
`check` downloads the file whole, since it validates every byte. The
tools now read through `File::storage` and the format crate's `*_in`
functions; local output is unchanged.
- **`File::storage()`** (facade) returns the file's bytes from the
superblock on (the `as_bytes` view, cache image laid over) as a
`&(dyn Storage + Send + Sync)` for every backend, so code that parses a
file itself works on remote files too.
- **`VlResolver::element_in` / `string_element_in`** (format): the
`element`/`string_element` lookups over any `Storage`.
### Range reads, milestone M2: raw data and `File::open_storage` (2026-09-26)
- **`clawhdf5::File::open_storage(Arc<dyn Storage + Send + Sync>)`** opens
a file served by any `clawhdf5_format::storage::Storage` and gives the
README.md
+54 -2
View File
@@ -96,6 +96,12 @@ breaking change, are in [CHANGELOG.md](CHANGELOG.md).
filtered top-k, never slower than unfiltered), and opt-in re-ranking and
confidence rejection, which used to be reachable only through `ClawhdfBackend`.
**Remote files (unreleased)**
- New crate `clawhdf5-remote`: `open_url("http://…")` reads a file on an
HTTP server (or in S3/GCS/Azure, opt-in) by range requests through a
block cache, without downloading it; `h5rs` takes URLs with its `remote`
feature. See [Reading remote files](#reading-remote-files).
**Tooling**
- CI now runs the h5py/netCDF4 interop suites for real (they had been skipping
silently) and runs an aarch64 job for the NEON kernels.
@@ -450,6 +456,50 @@ Each call changes the file in place (no rewrite) and syncs it. What it
cannot change safely is refused before anything is written; see
[known issues](docs/known-issues.md) for the limits.
### Reading remote files
[`clawhdf5-remote`](crates/clawhdf5-remote/README.md) opens a file on an
HTTP server (or, with its `s3`/`gcs`/`azure` features, in an object store)
without downloading it: the read API is the same `clawhdf5::File`, and
only the bytes an operation needs are fetched, by `Range` requests through
a block cache (1 MiB blocks; opening fetches the first one). A file that
changes on the server while it is open is an error, never a mix of old and
new bytes.
```rust
let file = clawhdf5_remote::open_url("http://127.0.0.1:8000/tall.h5")?;
let values = file.dataset("/g2/dset2.1")?.read_f64()?;
```
To try it without a server of your own, the crate's test server serves a
directory with range support:
```bash
cargo run -p clawhdf5-remote --example range_server -- crates/clawhdf5/tests/fixtures 127.0.0.1:8000
# in another shell: list the file, read one dataset, print what it cost
cargo run -p clawhdf5-remote --example read_url -- http://127.0.0.1:8000/tall.h5 /g2/dset2.1
```
```text
/g1 group
/g2 group
/g2/dset2.1 dataset [10] F32
/g2/dset2.2 dataset [3, 5] F32
/g1/g1.1 group
/g1/g1.2 group
/g1/g1.2/g1.2.1 group
/g1/g1.1/dset1.1.1 dataset [10, 10] I32
/g1/g1.1/dset1.1.2 dataset [20] I32
/g2/dset2.1: 10 values, first [1.0, 1.100000023841858, 1.2000000476837158, ...]
1 range requests (the one at open included), 9968 bytes fetched, 9968 bytes cached
```
(`tall.h5` is 9 968 bytes, so the first block holds all of it.) `h5rs`
built with `--features remote` takes the same URLs:
`h5rs ls -r http://127.0.0.1:8000/tall.h5`. Plain HTTP builds no C;
`https://` is the `https` feature (rustls with ring, which compiles C).
Limits are in [known issues](docs/known-issues.md).
### Python
`crates/clawhdf5-py` is a Python package (PyO3 + numpy) that reads HDF5 with
@@ -678,7 +728,7 @@ let exported = backend.export_markdown("MEMORY.md")?;
## Crate Map
```
clawhdf5 workspace (17 crates, ~86K lines of Rust in src/, ~104K with tests
clawhdf5 workspace (19 crates, ~86K lines of Rust in src/, ~104K with tests
and benches; plus libaec-sys, an internal FFI bindings
crate for the optional szip feature)
│
@@ -690,7 +740,8 @@ clawhdf5 workspace (17 crates, ~86K lines of Rust in src/, ~104K with tests
│ ├── clawhdf5 — High-level API
│ ├── clawhdf5-netcdf4 — NetCDF-4 support
│ ├── clawhdf5-accel — SIMD (AVX2, NEON incl. SDOT int8; AVX-512 behind `avx512`)
│ └── clawhdf5-gpu — GPU compute (wgpu, hand-written WGSL compute shaders)
│ ├── clawhdf5-gpu — GPU compute (wgpu, hand-written WGSL compute shaders)
│ └── clawhdf5-remote — Remote files: HTTP(S) range requests, object stores, block cache
│
├── Agent Memory
│ ├── clawhdf5-agent — Memory engine (24.7K lines, 32 modules; chained-CRC WAL)
@@ -705,6 +756,7 @@ clawhdf5 workspace (17 crates, ~86K lines of Rust in src/, ~104K with tests
│ └── clawhdf5-wasm — Browser (WebAssembly, wasm-bindgen; read-only)
│
└── Tooling
├── clawhdf5-tools — h5rs: ls, dump, stat, diff, check
└── clawhdf5-bench — Benchmark suite
```
docs/design/range-reads.md
+41 -2
View File
@@ -5,8 +5,10 @@ change. Progress: M0 and M1 are done, and so is M2 (branch
`feat/p3-m2-raw-data`): every read path of the format crate works through
`Storage`, v2 B-trees, dense groups and raw data included, and
`File::open_storage` gives the facade's read API over any `Storage` (see
`CHANGELOG.md`, "Range reads, milestone M2"). M3 (a remote backend with
its block cache) is next. Every count in §1–§2 was
`CHANGELOG.md`, "Range reads, milestone M2"). M3 is done on branch
`feat/p3-m3-remote`: the `clawhdf5-remote` crate (block cache, HTTP(S),
object stores) and URLs in `h5rs` (see the M3 status below). M4 (wasm) is
next. Every count in §1–§2 was
taken on `tank` on 2026-09-26 at commit `de2a53f`, with the commands given
next to it. No timing numbers appear here on purpose: the machine was shared
with other build jobs when this was written.
@@ -458,6 +460,43 @@ fast path within benchmark noise.
§2; the page size for paged files; the first block prefetched on open) and
a request counter exposed for tests and users.
- Python bindings: `clawhdf5.File("s3://…")` / `https://` through it.
- *Status 2026-09-26:* done on branch `feat/p3-m3-remote`, except the
Python bindings, with these choices:
- A new crate, `clawhdf5-remote`, instead of a `remote` feature of
`clawhdf5-io`: `open_url` returns a `clawhdf5::File`, and `clawhdf5-io`
sits below the facade.
- HTTP(S) through `ureq` (`HttpStorage`), not object_store's HTTP store:
that one pulls reqwest with aws-lc-rs (C), while plain HTTP through
ureq builds no C, so it is the default feature; `https` adds rustls
with ring. S3/GCS/Azure go through `object_store` (`ObjectStoreStorage`,
features `s3`/`gcs`/`azure`, opt-in because of aws-lc-rs); the
`object-store` feature alone (in-memory, local files, a store you
build) is pure Rust. object_store is async: the storage blocks on a
two-thread tokio runtime of its own, and refuses to run inside another
runtime.
- `BlockCache` (any `Storage`): 1 MiB blocks and a 64 MiB LRU budget by
default, the first block fetched at open (for HTTP by the request
that learns the length), the missing blocks of one read fetched as
runs of consecutive blocks in one parallel batch, per-block in-flight
deduplication across threads, and reads that miss more than half the
budget not kept. The page size of paged files is not used as the block
size yet. `CacheStats` and `HttpStats` count requests and bytes.
- The file is pinned at open by ETag (else Last-Modified, or the object's
version) and length; a change is an error, not mixed data.
- `h5rs` (feature `remote`) reads through the new `File::storage()`, the
file's view as a `Storage`, so its parsing works on remote files; its
`check` downloads the file whole.
- Measured with `crates/clawhdf5-remote/tests/http.rs` (tank, 2026-09-26,
`CLAWHDF5_REMOTE_CORPUS=conformance/.cache/corpus
CLAWHDF5_REMOTE_REPORT=1 cargo test --release -p clawhdf5-remote --test
http -- --nocapture corpus`), requests as the test server counted
them: the 621 corpus files that open read over HTTP exactly as through
`File::open`. Open + list (every group's entries, every dataset's shape
and type) of all of them: 640 requests, 55.5 MB of 254 MB; then reading
each file's largest dataset under 64 MiB: 96 more (171 MB in all). The
same work without a cache: 141 936 requests. Per file: A lists in 2
requests (§2 predicted 2 blocks of 1 MiB), B in 1, C in 7 (its whole
6.4 MB: 35 001 object headers spread over the file).
**M4 — wasm lazy loading (1–2 weeks).**
- `clawhdf5-wasm`: `openUrl(url) -> Promise<H5File>` backed by `fetch` with a
docs/known-issues.md
+52 -16
View File
@@ -766,20 +766,22 @@ which is what libhdf5 itself writes.
## Range reads (`File::open_storage`) limits
**Status:** open (added 2026-09-26, milestone M2 of
`docs/design/range-reads.md`). `File::open_storage` reads any
`clawhdf5_format::storage::Storage` through the whole read API, and every
format-crate read path works through `Storage::read_at`/`read_ranges`, but:
`docs/design/range-reads.md`; remote backends added by M3). `File::open_storage`
reads any `clawhdf5_format::storage::Storage` through the whole read API,
every format-crate read path works through `Storage::read_at`/`read_ranges`,
and `clawhdf5-remote` serves HTTP(S) and object-store files through a block
cache, but:
- **No remote backend and no block cache yet** (milestone M3). A `Storage`
is asked for each structure as the parsers need it, several times over
for some (an object header is re-read by each lookup through it): one
pass over the conformance corpus — open, list, every attribute, every
dataset once — is 176 092 `read_at` calls for 621 files, 92 489 of them
for the 35 001-group `h5stat_newgrat.h5` (2026-09-26, tank,
`crates/clawhdf5/tests/storage_equivalence.rs` with
`CLAWHDF5_STORAGE_CORPUS`). A backend over a network needs a cache in
front of it. `Storage::read_ranges` defaults to one `read_at` per range;
coalescing is the backend's job.
- **A `Storage` without a cache is asked for each structure as the parsers
need it**, several times over for some (an object header is re-read by
each lookup through it): one pass over the conformance corpus — open,
list, every attribute, every dataset once — is 176 092 `read_at` calls
for 621 files, 92 489 of them for the 35 001-group `h5stat_newgrat.h5`
(2026-09-26, tank, `crates/clawhdf5/tests/storage_equivalence.rs` with
`CLAWHDF5_STORAGE_CORPUS`). A backend of your own over a network needs a
cache in front of it: wrap it in `clawhdf5_remote::BlockCache`, as
`open_url` does. `Storage::read_ranges` defaults to one `read_at` per
range; coalescing is the backend's (or the cache's) job.
- A group lookup by name in a version-1 (symbol-table) group lists the whole
group (dense groups use their name index). Over a range backend that is
one read per symbol-table node and name, per lookup.
@@ -790,10 +792,12 @@ format-crate read path works through `Storage::read_at`/`read_ranges`, but:
`read_*_zerocopy`) need the file in memory and answer
`FormatError::ContiguousStorageRequired` otherwise; `File::as_bytes()`
panics for such a file (`File::contiguous_bytes()` is the fallible form).
`LazyFile`, `MmapFile`, the Python and wasm bindings and `h5rs` still read
a whole file.
`LazyFile`, `MmapFile` and the Python and wasm bindings still read a
whole file (`h5rs` reads through `File::storage`, and takes URLs with its
`remote` feature).
- The file's length is read once, at open: a growing file (SWMR) is not
followed (milestone M5).
followed (milestone M5). A remote file is pinned at open, so one that
grows is `RemoteError::FileChanged`.
- Not new, but visible through the equivalence tests: a full read through
the file's chunk cache (`read_raw_data_cached`, `read_raw_data_indexed`,
and so `Dataset::read_*`) lists a damaged dataset's chunks in hash-map
@@ -801,6 +805,38 @@ format-crate read path works through `Storage::read_at`/`read_ranges`, but:
the next (`cve-2025-2310.h5`); the values of a dataset that reads are
not affected.
## Remote files (`clawhdf5-remote`) limits
**Status:** open (added 2026-09-26, milestone M3 of
`docs/design/range-reads.md`).
- **Python and the browser cannot open URLs yet.** `clawhdf5.File` (PyO3)
parses through `File::as_bytes`, which a remote file does not have; the
wasm reader's `openUrl` is milestone M4.
- **The block size is fixed** (1 MiB unless `CacheConfig` says otherwise).
The design's policy of using a paged file's page size as the block size
is not implemented, and only the first block is read ahead.
- **Checked against local servers and one public one.** The tests use an
in-process HTTP/1.1 server and object_store's in-memory and local-file
stores. HTTPS was checked by hand against `raw.githubusercontent.com`
(2026-09-26, tank: `h5rs dump` of h5py's `vlen_string_dset.h5` by URL
equals the downloaded file's). The `s3`, `gcs` and `azure` backends are
built and their URL parsing tested, but they have not been run against a
real bucket.
- **A server with neither a strong ETag nor Last-Modified** can only be
checked by length, so a same-length replacement mid-read would go
unnoticed; `HttpOptions::require_validator` refuses such servers. A weak
ETag (`W/"…"`) cannot be sent as `If-Match`, so it counts as none.
- **Credentials:** HTTP takes extra headers (`HttpOptions::headers`, e.g.
`Authorization`); `h5rs` has no option for them. The cloud stores read
credentials from the environment only.
- Each `ObjectStoreStorage` owns a tokio runtime with two worker threads.
- `h5rs check` downloads a remote file whole (it validates every byte), and
a URL cannot carry a `FILE/OBJECT` suffix; `h5rs` uses the default cache
settings.
- The zero-copy methods and `File::as_bytes` are unavailable on a remote
file (see the range-read limits above).
## `clawhdf5-wasm` (browser) limits
**Status:** open (by design for now; added 2026-09-26).