docs: SWMR reader (range-read M5) in CHANGELOG, README, known issues and designs

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

docs/design/range-reads.md

@@ -7,7 +7,9 @@ change. Progress: M0 and M1 are done, and so is M2 (branch
 `File::open_storage` gives the facade's read API over any `Storage` (see
 `CHANGELOG.md`, "Range reads, milestone M2"). M3 is done on branch
 `feat/p3-m3-remote`: the `clawhdf5-remote` crate (block cache, HTTP(S),
-object stores) and URLs in `h5rs` (see the M3 status below). M4 (wasm) is
+object stores) and URLs in `h5rs` (see the M3 status below). M5 (SWMR) is
+done on branch `feat/p3-m5-swmr-reader`, with its own design in
+[`swmr.md`](swmr.md) (see the M5 status below). M4 (wasm) is
 next. Every count in §1–§2 was
 
 change. Progress: M1, first part (the `Storage` trait and the metadata
@@ -523,6 +525,19 @@ fast path within benchmark noise.
 **M5 — SWMR and growth (later, separate design).** `Storage::len()` may grow;
 add `File::refresh()` that re-reads the superblock/EOF and invalidates cached
 blocks past the old end. Needs libhdf5 SWMR semantics research first.
+- *Status 2026-09-27:* done on branch `feat/p3-m5-swmr-reader`; design and
+  libhdf5 research in [`swmr.md`](swmr.md). Differences from the sketch
+  above: the refresh is per dataset (`Dataset::refresh`, as libhdf5's
+  `H5Drefresh`), not per file — a SWMR writer only grows datasets, and the
+  superblock's EOF is not kept up to date by it, so there is nothing to
+  re-read there (`File::swmr_writer_active` re-reads its flags). A live
+  file (`File::open_swmr`) reads through `FileStorage` (positioned reads,
+  `len()` the current length) with no block or chunk cache, rather than
+  invalidating cached blocks: `BlockCache`/`HttpStorage` stay snapshot
+  readers. Operations that fail with an error a racing write can cause are
+  retried up to 100 times (libhdf5's default for SWMR readers).
+  Tested against a live h5py writer and h5py's SWMR reader
+  (`crates/clawhdf5/tests/swmr_interop.rs`).
 
 Total: roughly 6–10 engineer-weeks for M0–M4 (estimate, not measured).
 

docs/design/swmr.md

@@ -86,16 +86,25 @@ writer keeps appending. What the format and the library guarantee:
    elements inside that extent.
 4. **Bounded retries.** In a live file, an operation (open, dataset lookup,
    refresh, every read) that fails with an error a concurrent write can
-   cause — a checksum or Fletcher-32 mismatch, a short read, a bad
-   signature, an undecodable chunk or chunk index — is run again from the
-   start, up to `File::swmr_read_attempts()` times (default 100, libhdf5's
-   default; `set_swmr_read_attempts` changes it), sleeping 1 µs, 2 µs, …
-   up to 10 ms between attempts (under a second in all). libhdf5 retries
-   the one structure whose checksum failed; we retry the whole operation,
-   because the parsers are pure functions of the bytes they read. Results
-   are only returned from a run where every structure verified, so an
-   error is returned instead of torn data. Other errors are returned at
-   once, and non-live files never retry.
+   cause is run again from the start, up to `File::swmr_read_attempts()`
+   times (default 100, libhdf5's default; `set_swmr_read_attempts` changes
+   it), sleeping 1 µs, 2 µs, … up to 10 ms between attempts (under a second
+   in all). libhdf5 retries the one structure whose checksum failed; we
+   retry the whole operation, because the parsers are pure functions of the
+   bytes they read. Which errors: a garbled read can fail whichever check
+   the parser makes first — a checksum, a signature, a version byte, a
+   size (a test that garbles every byte of a header got
+   `InvalidObjectHeaderVersion`, not a checksum error) — so every format
+   error counts except those later bytes cannot change: a path or selection
+   the caller got wrong, an unsupported filter or external storage, a bad
+   argument. Those, and a live file's permanent errors of the other kinds
+   after the last attempt, are returned; non-live files never retry.
+   Results are only returned from a run where every structure verified, so
+   a torn metadata read is an error, never data. `File::swmr_retries()`
+   counts the retries (libhdf5: `H5Fget_metadata_read_retry_info`).
+   Raw data has no checksum in HDF5 (unless Fletcher-32 is on), in libhdf5
+   as here: correctness rests on the writer's ordering (chunk data before
+   the index entry, only appends), as for libhdf5's reader.
 5. **Writer state.** `File::swmr_writer_active()` reads the superblock
    flags again, so a reader can tell when the writer has closed the file.
 
@@ -120,5 +129,15 @@ writer cannot add them), and `MmapFile`/`LazyFile`.
 
 ## Status
 
-See the end of this file's history in `CHANGELOG.md` ("Range reads,
-milestone M5").
+Implemented 2026-09-27 on branch `feat/p3-m5-swmr-reader` as designed
+above (`CHANGELOG.md`, "Range reads, milestone M5"). Observed on tank the
+same day (h5py 3.16 / HDF5 2.0, `cargo test -p clawhdf5 --test
+swmr_interop`, and once with `CLAWHDF5_SWMR_STEPS=20000` in a release
+build): no read returned a value the writer had not written at that
+position, h5py's reader agreed, and retries were needed but rare (the
+failures seen were checksum mismatches, each cured by one retry). A
+variant of the test with the chunk cache left on in live mode fails it
+(stale chunk index / edge chunk), which is why live files do not use it.
+
+Also found: `File::open` of such a file had been failing since the
+end-of-file check of 2026-09-26 (item 1; `docs/known-issues.md`).

docs/known-issues.md

@@ -7,6 +7,29 @@ deleting it.
 
 ---
 
+## Files a SWMR writer had open could not be read past a stale end of file
+
+**Status:** fixed 2026-09-27 (branch `feat/p3-m5-swmr-reader`), before any
+release: reads have been bounded by the recorded end of file since
+`7d7a7e7` (2026-09-26), which no release contains.
+
+A libhdf5 writer in SWMR mode (h5py `f.swmr_mode = True`) sets the
+superblock's SWMR-write flag and does not keep its end-of-file address up to
+date: a copy h5py made of its own file mid-write records 715 in a
+6 030-byte file (h5py 3.16 / HDF5 2.0, tank). `Superblock::data_end` only
+ignored the recorded end when it lay *past* the end of the file, so every
+reader bounded such a file at 715 bytes: it listed, but every chunked read
+failed ("unexpected EOF: need 787 bytes, have 715") and `h5rs check`
+reported the chunk indexes past the end of the file. Never wrong data.
+
+**Fix:** for a v3 superblock with the SWMR-write flag, the data ends at the
+end of the file, as libhdf5's SWMR reader reads it. **Test:**
+`crates/clawhdf5/tests/swmr_interop.rs` (the mid-write copy,
+`tests/fixtures/swmr_mid_write.h5`, through every open path and against
+h5py's SWMR reader). A file still being written is read with
+`File::open_swmr` (see `docs/design/swmr.md`); `File::open` maps the file
+at its length at open and is not meant for files that change while open.
+
 ## Fletcher-32 checksums disagreed with libhdf5 on about 1 chunk in 32768
 
 **Status:** fixed 2026-09-26, after v2.7.0. **Every release (v2.1.0 to