Merge branch 'feat/p3-storage-trait' into feat/p3-range-zfp-edit
# Conflicts: # CHANGELOG.md # crates/clawhdf5-format/src/attribute.rs # crates/clawhdf5-format/src/btree_v1.rs # crates/clawhdf5-format/src/data_layout.rs # crates/clawhdf5-format/src/extensible_array.rs # crates/clawhdf5-format/src/fixed_array.rs # crates/clawhdf5-format/src/fractal_heap.rs # crates/clawhdf5-format/src/local_heap.rs # crates/clawhdf5-format/src/shared_message.rs
This commit is contained in:
@@ -1,7 +1,9 @@
|
||||
# Design: range reads (reading HDF5 without holding the whole file)
|
||||
|
||||
Status: proposal, 2026-09-26. No library code has changed; this document is
|
||||
the plan for Phase 3's largest architectural change. Every count below was
|
||||
Status: proposal, 2026-09-26; the plan for Phase 3's largest architectural
|
||||
change. Progress: M1, first part (the `Storage` trait and the metadata
|
||||
parsers listed in `CHANGELOG.md` under "Range reads, milestone M1") is done;
|
||||
group B-tree v2 lookups, dense groups and the facade are not converted yet. Every count below was
|
||||
taken on `tank` on 2026-09-26 at commit `de2a53f`, with the commands given
|
||||
next to it. No timing numbers appear here on purpose: the machine was shared
|
||||
with other build jobs when this was written.
|
||||
@@ -259,6 +261,11 @@ every `file_data[a..b]` becomes `file.read_at(a, b - a)?`.
|
||||
(binary size matters for wasm); `&dyn Storage` costs one indirect call per
|
||||
structure read, negligible next to parsing. Hot raw-data loops keep their
|
||||
speed through `as_contiguous()`.
|
||||
*M1 outcome:* `&dyn` was not negligible for small structures — with it,
|
||||
`ObjectHeader::parse` was ~25% and a 400-group facade listing ~14% slower
|
||||
than the slice code (provisional, shared machine). The cores are now
|
||||
generic (`S: Storage + ?Sized`), so the `&[u8]` wrappers get a `[u8]`
|
||||
instance and `dyn Storage` is one more instance, not one per backend.
|
||||
|
||||
### (b) A page-cache "virtual slice"
|
||||
|
||||
@@ -400,7 +407,7 @@ fast path within benchmark noise.
|
||||
B-tree v1/v2, fractal heap, fixed/extensible array, symbol table, group
|
||||
v1/v2, shared messages, attributes, fill value, data layout — one commit
|
||||
each. The old `&[u8]` signature stays as a thin wrapper over the new one
|
||||
(`fn parse(data: &[u8], ..) { parse_in(data as &dyn Storage, ..) }`), so
|
||||
(`fn parse(data: &[u8], ..) { parse_in(data, ..) }`, generic core), so
|
||||
callers and the other crates don't move yet.
|
||||
- Replace the 5 open-ended slices and 38 `len()` checks with bounded reads.
|
||||
|
||||
@@ -441,8 +448,13 @@ Total: roughly 6–10 engineer-weeks for M0–M4 (estimate, not measured).
|
||||
- `impl Storage for [u8]` returns `Cow::Borrowed` — no copy, no allocation.
|
||||
- Hot loops (raw-data copies, contiguous typed reads, `read_selection_native`)
|
||||
branch once on `as_contiguous()` and then run today's code.
|
||||
- `&dyn` dispatch is per structure, not per byte; parse code keeps working on
|
||||
- The parser cores are generic over `S: Storage + ?Sized`, so the in-memory
|
||||
instance has no dispatch at all; a remote backend behind `&dyn Storage`
|
||||
pays one indirect call per structure read. Parse code keeps working on
|
||||
the returned slice.
|
||||
- Reads sized by untrusted fields cover what the parser uses (see
|
||||
`CHANGELOG.md`, M1), so a hostile size costs no read of the rest of the
|
||||
file.
|
||||
- Gate: `crates/clawhdf5/benches/mmap_bench.rs`, the concurrent-read benches in
|
||||
`clawhdf5-bench`, and the conformance run time, before and after each M1/M2
|
||||
commit, on an otherwise idle machine. Anything outside noise blocks the
|
||||
|
||||
Reference in New Issue
Block a user