docs: SWMR reader (range-read M5) in CHANGELOG, README, known issues and designs

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-27 06:49:04 -05:00
co-authored by Claude Opus 5.5
parent 21f104bb77
commit 3f45755d58
5 changed files with 157 additions and 13 deletions
+16 -1
View File
@@ -7,7 +7,9 @@ change. Progress: M0 and M1 are done, and so is M2 (branch
`File::open_storage` gives the facade's read API over any `Storage` (see
`CHANGELOG.md`, "Range reads, milestone M2"). M3 is done on branch
`feat/p3-m3-remote`: the `clawhdf5-remote` crate (block cache, HTTP(S),
object stores) and URLs in `h5rs` (see the M3 status below). M4 (wasm) is
object stores) and URLs in `h5rs` (see the M3 status below). M5 (SWMR) is
done on branch `feat/p3-m5-swmr-reader`, with its own design in
[`swmr.md`](swmr.md) (see the M5 status below). M4 (wasm) is
next. Every count in §1–§2 was
change. Progress: M1, first part (the `Storage` trait and the metadata
@@ -523,6 +525,19 @@ fast path within benchmark noise.
**M5 — SWMR and growth (later, separate design).** `Storage::len()` may grow;
add `File::refresh()` that re-reads the superblock/EOF and invalidates cached
blocks past the old end. Needs libhdf5 SWMR semantics research first.
- *Status 2026-09-27:* done on branch `feat/p3-m5-swmr-reader`; design and
libhdf5 research in [`swmr.md`](swmr.md). Differences from the sketch
above: the refresh is per dataset (`Dataset::refresh`, as libhdf5's
`H5Drefresh`), not per file — a SWMR writer only grows datasets, and the
superblock's EOF is not kept up to date by it, so there is nothing to
re-read there (`File::swmr_writer_active` re-reads its flags). A live
file (`File::open_swmr`) reads through `FileStorage` (positioned reads,
`len()` the current length) with no block or chunk cache, rather than
invalidating cached blocks: `BlockCache`/`HttpStorage` stay snapshot
readers. Operations that fail with an error a racing write can cause are
retried up to 100 times (libhdf5's default for SWMR readers).
Tested against a live h5py writer and h5py's SWMR reader
(`crates/clawhdf5/tests/swmr_interop.rs`).
Total: roughly 6–10 engineer-weeks for M0–M4 (estimate, not measured).