facade: retry every read path of a live file (strings, vlen, attributes)
read_string, read_string_bytes, read_string_selection, read_vlen and read_vlen_selection now retry as a whole, the global-heap decoding after the read included; so do File::decode_strings / decode_string_bytes / decode_vlen, Group::datasets / groups / attrs / attr, Dataset::attrs / attr, the typed full reads (read_f64 ...), the header messages behind shape() and dtype() (a shared message is read from another header), and verify_provenance. Before, a transient failure on those reached the caller. Attribute reads leave out an attribute they cannot read (or return a variable-length string one as AttrValue::Raw) instead of failing, which hid a transient error as a missing or raw attribute: on a live file such an error of a retried kind now runs the read again too, and after the last attempt the last result is returned as before. The format crate gains find_attribute_reporting_in, which returns the errors find_attribute_in skips (dense name-index lookups dropped them). The zero-copy reads need the file in memory, which a live file never is, so they have nothing to retry. Test: a storage that fails one read with a checksum mismatch; for 14 read paths over a new fixture (tests/fixtures/swmr_strings_attrs.h5, an h5py copy with the SWMR-write flag: vlen strings, vlen int32, dense attributes), each read the path makes fails once in turn and the path must return the same result with exactly one retry. It fails on the previous commit (root.attrs, read 0). Dataset::attr on dense attributes returned None before find_attribute_reporting_in. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
+20
-2
@@ -96,8 +96,11 @@ writer keeps appending. What the format and the library guarantee:
|
||||
read use the new extent. Like h5py, a handle that is not refreshed keeps
|
||||
its extent; its reads still read the index as it is now, and only return
|
||||
elements inside that extent.
|
||||
4. **Bounded retries.** In a live file, an operation (open, dataset lookup,
|
||||
refresh, every read) that fails with an error a concurrent write can
|
||||
4. **Bounded retries.** In a live file, an operation (open, lookups and
|
||||
listings, refresh, every dataset read — typed, raw, selections, strings
|
||||
and variable-length data with their global-heap decoding — attributes,
|
||||
`File::decode_*`, `verify_provenance`) that fails with an error a
|
||||
concurrent write can
|
||||
cause is run again from the start, up to `File::swmr_read_attempts()`
|
||||
times (default 100, libhdf5's default; `set_swmr_read_attempts` changes
|
||||
it), sleeping 1 µs, 2 µs, … up to 10 ms between attempts (under a second
|
||||
@@ -116,6 +119,16 @@ writer keeps appending. What the format and the library guarantee:
|
||||
Results are only returned from a run where every structure verified, so
|
||||
a torn metadata read is an error, never data. `File::swmr_retries()`
|
||||
counts the retries (libhdf5: `H5Fget_metadata_read_retry_info`).
|
||||
Attribute reads leave out an attribute they cannot read (or return a
|
||||
variable-length string one as `AttrValue::Raw`) instead of failing;
|
||||
on a live file such an error of a retried kind runs the read again too,
|
||||
and after the last attempt the last result is returned as before. The
|
||||
zero-copy reads (`read_raw_ref`, `read_as_slice`, `read_*_zerocopy`)
|
||||
need the file in memory, which a live file never is: they report
|
||||
`None` / `ContiguousStorageRequired` without reading data.
|
||||
Global heap collections (variable-length data) have no checksum in
|
||||
HDF5, like raw data, so a torn read of one is only caught when it fails
|
||||
a check (its signature, a bound).
|
||||
Raw data has no checksum in HDF5 (unless Fletcher-32 is on), in libhdf5
|
||||
as here: correctness rests on the writer's ordering (chunk data before
|
||||
the index entry, only appends), as for libhdf5's reader.
|
||||
@@ -135,6 +148,11 @@ writer cannot add them), and `MmapFile`/`LazyFile`.
|
||||
EOF is below the file length.
|
||||
- `crates/clawhdf5/tests/swmr_interop.rs`:
|
||||
- a copy of a file taken mid-write (fixture) reads like h5py's SWMR reader;
|
||||
- the same bytes with the flags cleared read as `File::open` reads them
|
||||
(and how h5py's two readers read them);
|
||||
- a permanent error returns at once; every read path (listings,
|
||||
attributes, strings, variable-length data) with each of its reads
|
||||
failing once in turn returns the same result;
|
||||
- a live test: an h5py writer (`swmr_mode = True`) appends to a 1-D and a
|
||||
2-D dataset with one unlimited dimension (Extensible Array, one of them
|
||||
gzip) and a 2-D dataset with two (v2 B-tree), flushing after every step,
|
||||
|
||||
Reference in New Issue
Block a user