py: remote files (clawhdf5.File(url), File.open_url) through File::storage()
The Python bindings could not open a remote file: they parsed through File::as_bytes() in eight places (path lookups, object headers, dataspaces, attributes, group listings, the global heap of variable-length data), which a storage-backed file does not have. - Every object of a File now shares one handle (src/handle.rs) that runs all file access, metadata included, with the GIL released and parses through File::storage() and the clawhdf5_format *_in functions. Local files take the same path (their storage is the mmap). - clawhdf5.File(url) opens any scheme://... through clawhdf5_remote::storage_for_url (read-only; another mode is a ValueError). File.open_url(url, **options) takes the cache and HTTP options (block_size, cache_size, headers, retries, timeout, allow_full_download, max_full_download, require_validator, max_redirects, max_parallel); File.remote_stats gives the block cache's counters. - Default build: plain HTTP only, no C. https (rustls/ring) and s3/gcs/azure (aws-lc-rs) are opt-in features of clawhdf5-py, and ci-test.sh's no-C check now covers the crate. - A failed storage read (network error, file changed on the server) is an OSError, never KeyError/ValueError and never data; `key in group` raises it instead of answering False. Tests: the read-vs-h5py suite runs locally and over HTTP (1 MiB and 1 KiB blocks) against a range-capable http.server in the test process (conftest.RangeServer); test_remote.py covers request counts, cache hits, a server without Range support, a changed file, a server that hangs up, 16 threads, and a spinning thread that keeps running while a read waits on 0.2 s requests. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
@@ -844,9 +844,9 @@ cache, but:
|
||||
`read_*_zerocopy`) need the file in memory and answer
|
||||
`FormatError::ContiguousStorageRequired` otherwise; `File::as_bytes()`
|
||||
panics for such a file (`File::contiguous_bytes()` is the fallible form).
|
||||
`LazyFile`, `MmapFile` and the Python and wasm bindings still read a
|
||||
whole file (`h5rs` reads through `File::storage`, and takes URLs with its
|
||||
`remote` feature).
|
||||
`LazyFile`, `MmapFile` and the wasm bindings still read a whole file
|
||||
(`h5rs` and the Python bindings read through `File::storage`, and take
|
||||
URLs: `h5rs` with its `remote` feature, Python with `clawhdf5.File(url)`).
|
||||
- The file's length is read once, at open: a growing file (SWMR) is not
|
||||
followed (milestone M5). A remote file is pinned at open, so one that
|
||||
grows is `RemoteError::FileChanged`.
|
||||
@@ -862,9 +862,12 @@ cache, but:
|
||||
**Status:** open (added 2026-09-26, milestone M3 of
|
||||
`docs/design/range-reads.md`).
|
||||
|
||||
- **Python and the browser cannot open URLs yet.** `clawhdf5.File` (PyO3)
|
||||
parses through `File::as_bytes`, which a remote file does not have; the
|
||||
wasm reader's `openUrl` is milestone M4.
|
||||
- **The browser cannot open URLs yet**: the wasm reader's `openUrl` is
|
||||
milestone M4. Python can (`clawhdf5.File(url)`, since 2026-09-27), but
|
||||
the default wheel reads plain `http://` only: `https://` needs a wheel
|
||||
built with `--features https` (rustls with ring, which compiles C), and
|
||||
`s3://`, `gs://`, `az://` the `s3`, `gcs`, `azure` features (aws-lc-rs).
|
||||
The Python tests run against an in-process `http.server` only.
|
||||
- **The block size is fixed** (1 MiB unless `CacheConfig` says otherwise).
|
||||
The design's policy of using a paged file's page size as the block size
|
||||
is not implemented, and only the first block is read ahead.
|
||||
|
||||
Reference in New Issue
Block a user