docs: range-read milestone M3 — remote files
README: "Reading remote files" (open_url, the range_server and read_url examples with their real output against the fixtures, h5rs on URLs), the crate in the crate map and the unreleased highlights. CHANGELOG: the clawhdf5-remote crate, h5rs URLs, File::storage and VlResolver::element_in, with the request counts over the conformance corpus (tank, 2026-09-26, the command given). known-issues: the M2 range-read entry updated (the cache now exists; h5rs reads through storage) and a new entry for the remote backends' limits (no Python or browser URLs yet, fixed block size, cloud stores not run against a real bucket, validators, credentials). Design doc: M3 status with the choices that differ from the plan (a crate rather than a clawhdf5-io feature, ureq for HTTP so the default build has no C) and the corpus counts. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
@@ -96,6 +96,12 @@ breaking change, are in [CHANGELOG.md](CHANGELOG.md).
|
||||
filtered top-k, never slower than unfiltered), and opt-in re-ranking and
|
||||
confidence rejection, which used to be reachable only through `ClawhdfBackend`.
|
||||
|
||||
**Remote files (unreleased)**
|
||||
- New crate `clawhdf5-remote`: `open_url("http://…")` reads a file on an
|
||||
HTTP server (or in S3/GCS/Azure, opt-in) by range requests through a
|
||||
block cache, without downloading it; `h5rs` takes URLs with its `remote`
|
||||
feature. See [Reading remote files](#reading-remote-files).
|
||||
|
||||
**Tooling**
|
||||
- CI now runs the h5py/netCDF4 interop suites for real (they had been skipping
|
||||
silently) and runs an aarch64 job for the NEON kernels.
|
||||
@@ -450,6 +456,50 @@ Each call changes the file in place (no rewrite) and syncs it. What it
|
||||
cannot change safely is refused before anything is written; see
|
||||
[known issues](docs/known-issues.md) for the limits.
|
||||
|
||||
### Reading remote files
|
||||
|
||||
[`clawhdf5-remote`](crates/clawhdf5-remote/README.md) opens a file on an
|
||||
HTTP server (or, with its `s3`/`gcs`/`azure` features, in an object store)
|
||||
without downloading it: the read API is the same `clawhdf5::File`, and
|
||||
only the bytes an operation needs are fetched, by `Range` requests through
|
||||
a block cache (1 MiB blocks; opening fetches the first one). A file that
|
||||
changes on the server while it is open is an error, never a mix of old and
|
||||
new bytes.
|
||||
|
||||
```rust
|
||||
let file = clawhdf5_remote::open_url("http://127.0.0.1:8000/tall.h5")?;
|
||||
let values = file.dataset("/g2/dset2.1")?.read_f64()?;
|
||||
```
|
||||
|
||||
To try it without a server of your own, the crate's test server serves a
|
||||
directory with range support:
|
||||
|
||||
```bash
|
||||
cargo run -p clawhdf5-remote --example range_server -- crates/clawhdf5/tests/fixtures 127.0.0.1:8000
|
||||
# in another shell: list the file, read one dataset, print what it cost
|
||||
cargo run -p clawhdf5-remote --example read_url -- http://127.0.0.1:8000/tall.h5 /g2/dset2.1
|
||||
```
|
||||
|
||||
```text
|
||||
/g1 group
|
||||
/g2 group
|
||||
/g2/dset2.1 dataset [10] F32
|
||||
/g2/dset2.2 dataset [3, 5] F32
|
||||
/g1/g1.1 group
|
||||
/g1/g1.2 group
|
||||
/g1/g1.2/g1.2.1 group
|
||||
/g1/g1.1/dset1.1.1 dataset [10, 10] I32
|
||||
/g1/g1.1/dset1.1.2 dataset [20] I32
|
||||
/g2/dset2.1: 10 values, first [1.0, 1.100000023841858, 1.2000000476837158, ...]
|
||||
1 range requests (the one at open included), 9968 bytes fetched, 9968 bytes cached
|
||||
```
|
||||
|
||||
(`tall.h5` is 9 968 bytes, so the first block holds all of it.) `h5rs`
|
||||
built with `--features remote` takes the same URLs:
|
||||
`h5rs ls -r http://127.0.0.1:8000/tall.h5`. Plain HTTP builds no C;
|
||||
`https://` is the `https` feature (rustls with ring, which compiles C).
|
||||
Limits are in [known issues](docs/known-issues.md).
|
||||
|
||||
### Python
|
||||
|
||||
`crates/clawhdf5-py` is a Python package (PyO3 + numpy) that reads HDF5 with
|
||||
@@ -678,7 +728,7 @@ let exported = backend.export_markdown("MEMORY.md")?;
|
||||
## Crate Map
|
||||
|
||||
```
|
||||
clawhdf5 workspace (17 crates, ~86K lines of Rust in src/, ~104K with tests
|
||||
clawhdf5 workspace (19 crates, ~86K lines of Rust in src/, ~104K with tests
|
||||
and benches; plus libaec-sys, an internal FFI bindings
|
||||
crate for the optional szip feature)
|
||||
│
|
||||
@@ -690,7 +740,8 @@ clawhdf5 workspace (17 crates, ~86K lines of Rust in src/, ~104K with tests
|
||||
│ ├── clawhdf5 — High-level API
|
||||
│ ├── clawhdf5-netcdf4 — NetCDF-4 support
|
||||
│ ├── clawhdf5-accel — SIMD (AVX2, NEON incl. SDOT int8; AVX-512 behind `avx512`)
|
||||
│ └── clawhdf5-gpu — GPU compute (wgpu, hand-written WGSL compute shaders)
|
||||
│ ├── clawhdf5-gpu — GPU compute (wgpu, hand-written WGSL compute shaders)
|
||||
│ └── clawhdf5-remote — Remote files: HTTP(S) range requests, object stores, block cache
|
||||
│
|
||||
├── Agent Memory
|
||||
│ ├── clawhdf5-agent — Memory engine (24.7K lines, 32 modules; chained-CRC WAL)
|
||||
@@ -705,6 +756,7 @@ clawhdf5 workspace (17 crates, ~86K lines of Rust in src/, ~104K with tests
|
||||
│ └── clawhdf5-wasm — Browser (WebAssembly, wasm-bindgen; read-only)
|
||||
│
|
||||
└── Tooling
|
||||
├── clawhdf5-tools — h5rs: ls, dump, stat, diff, check
|
||||
└── clawhdf5-bench — Benchmark suite
|
||||
```
|
||||
|
||||
|
||||
Reference in New Issue
Block a user