docs: changelog and design status for range-read milestone M1 (part 1)
Lists the parsers now reading through Storage, what still needs the
whole file (v2 B-tree-indexed structures: a clean error; raw data: M2),
the equivalence harness, and the evidence that nothing changed: existing
tests, a byte-identical conformance results.json and per-file probe
output against f2ff2c4, and identical slice-API transcripts over 748
files between the two builds.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
@@ -15,6 +15,49 @@
|
|||||||
fast path; implemented for `[u8]`, `Vec<u8>`, and references, `Box`es
|
fast path; implemented for `[u8]`, `Vec<u8>`, and references, `Box`es
|
||||||
and (with `std`) `Arc`s of a `Storage`. Slices and `Vec`s serve borrowed
|
and (with `std`) `Arc`s of a `Storage`. Slices and `Vec`s serve borrowed
|
||||||
bytes, so parsing an in-memory file costs no copy.
|
bytes, so parsing an in-memory file costs no copy.
|
||||||
|
- **The metadata parsers read through `Storage`.** Each converted parser has
|
||||||
|
an `*_in(&dyn Storage, ..)` core, and its `&[u8]` function is now a thin
|
||||||
|
wrapper over it, so no caller changes: the superblock
|
||||||
|
(`Superblock::parse_in`), its extension and cache image
|
||||||
|
(`read_superblock_extension_in`, `cache_image_state_in`), object headers
|
||||||
|
with their continuation chunks (`ObjectHeader::parse_in`), local and
|
||||||
|
global heaps, symbol-table nodes and the group B-tree (v1), fractal heaps,
|
||||||
|
fixed and extensible array chunk indexes, shared messages and the SOHM
|
||||||
|
table (`message_data_in`, `message_data_with_sohm_in`,
|
||||||
|
`load_sohm_table_in`, …), attributes (`extract_attributes_full_in`,
|
||||||
|
`extract_attributes_tolerant_in`, `AttributeMessage::parse_in_storage`),
|
||||||
|
fill values (`dataset_fill_value_from_storage`) and virtual-dataset
|
||||||
|
mappings (`DataLayout::resolve_vds_mappings_in`); also
|
||||||
|
`signature::find_signature_in`. Each structure is read with bounded reads
|
||||||
|
(a prefix, then the structure) instead of slicing the whole file; the
|
||||||
|
open-ended `&file_data[addr..]` slices in these modules are gone. Bounds
|
||||||
|
errors keep their values (absolute position, file length).
|
||||||
|
- Structures still indexed by a v2 B-tree — dense attribute storage, a SOHM
|
||||||
|
B-tree index and huge fractal-heap objects found through their B-tree —
|
||||||
|
are not converted yet (the
|
||||||
|
v2 B-tree and dense groups come with milestone M3); over a backend without
|
||||||
|
the whole file in memory they are the clean `ContiguousStorageRequired`
|
||||||
|
error, never a partial result. Raw data, chunk B-tree (v1) indexes and VL
|
||||||
|
data are milestone M2.
|
||||||
|
- **No behaviour change**, checked three ways (2026-09-26, tank): every
|
||||||
|
existing test passes unchanged; the conformance sweep
|
||||||
|
(`conformance/run.sh --no-fetch`) gives a byte-identical `results.json`
|
||||||
|
and identical per-file probe output for all 697 files at `f2ff2c4` and on
|
||||||
|
this branch; and a transcript of every converted `&[u8]` function's
|
||||||
|
result over the fixtures, the conformance corpus and the h5py-written
|
||||||
|
files below (748 files, 7 603 object headers) is byte-identical between
|
||||||
|
the two builds.
|
||||||
|
- New equivalence harness `clawhdf5-format/tests/storage_equivalence.rs`:
|
||||||
|
every converted parser runs over the file as a slice and over
|
||||||
|
`storage::CountingStorage` — a `Storage` that serves an in-memory buffer
|
||||||
|
through `read_at` only (`as_contiguous()` is `None`), copying what it
|
||||||
|
serves and counting reads — and must give identical results. It walks
|
||||||
|
the fixtures, files h5py writes for it (extensible arrays with super
|
||||||
|
blocks and paged data blocks, paged fixed arrays, large v1 and dense
|
||||||
|
groups, a user block, SOHM list and B-tree indexes, dense, shared and
|
||||||
|
committed-type attributes), and with `CLAWHDF5_STORAGE_CORPUS=<dir>` a
|
||||||
|
corpus (all 653 HDF5 files of the conformance corpus pass). Milestones M2
|
||||||
|
and M3 extend it.
|
||||||
|
|
||||||
### Chunked full reads (2026-09-26)
|
### Chunked full reads (2026-09-26)
|
||||||
- **Chunks are decoded straight into the output, into reused buffers.** A
|
- **Chunks are decoded straight into the output, into reused buffers.** A
|
||||||
|
|||||||
@@ -1,7 +1,9 @@
|
|||||||
# Design: range reads (reading HDF5 without holding the whole file)
|
# Design: range reads (reading HDF5 without holding the whole file)
|
||||||
|
|
||||||
Status: proposal, 2026-09-26. No library code has changed; this document is
|
Status: proposal, 2026-09-26; the plan for Phase 3's largest architectural
|
||||||
the plan for Phase 3's largest architectural change. Every count below was
|
change. Progress: M1, first part (the `Storage` trait and the metadata
|
||||||
|
parsers listed in `CHANGELOG.md` under "Range reads, milestone M1") is done;
|
||||||
|
group B-tree v2 lookups, dense groups and the facade are not converted yet. Every count below was
|
||||||
taken on `tank` on 2026-09-26 at commit `de2a53f`, with the commands given
|
taken on `tank` on 2026-09-26 at commit `de2a53f`, with the commands given
|
||||||
next to it. No timing numbers appear here on purpose: the machine was shared
|
next to it. No timing numbers appear here on purpose: the machine was shared
|
||||||
with other build jobs when this was written.
|
with other build jobs when this was written.
|
||||||
|
|||||||
Reference in New Issue
Block a user