docs: changelog and design status for range-read milestone M1 (part 1)
Lists the parsers now reading through Storage, what still needs the
whole file (v2 B-tree-indexed structures: a clean error; raw data: M2),
the equivalence harness, and the evidence that nothing changed: existing
tests, a byte-identical conformance results.json and per-file probe
output against f2ff2c4, and identical slice-API transcripts over 748
files between the two builds.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
@@ -15,6 +15,49 @@
|
||||
fast path; implemented for `[u8]`, `Vec<u8>`, and references, `Box`es
|
||||
and (with `std`) `Arc`s of a `Storage`. Slices and `Vec`s serve borrowed
|
||||
bytes, so parsing an in-memory file costs no copy.
|
||||
- **The metadata parsers read through `Storage`.** Each converted parser has
|
||||
an `*_in(&dyn Storage, ..)` core, and its `&[u8]` function is now a thin
|
||||
wrapper over it, so no caller changes: the superblock
|
||||
(`Superblock::parse_in`), its extension and cache image
|
||||
(`read_superblock_extension_in`, `cache_image_state_in`), object headers
|
||||
with their continuation chunks (`ObjectHeader::parse_in`), local and
|
||||
global heaps, symbol-table nodes and the group B-tree (v1), fractal heaps,
|
||||
fixed and extensible array chunk indexes, shared messages and the SOHM
|
||||
table (`message_data_in`, `message_data_with_sohm_in`,
|
||||
`load_sohm_table_in`, …), attributes (`extract_attributes_full_in`,
|
||||
`extract_attributes_tolerant_in`, `AttributeMessage::parse_in_storage`),
|
||||
fill values (`dataset_fill_value_from_storage`) and virtual-dataset
|
||||
mappings (`DataLayout::resolve_vds_mappings_in`); also
|
||||
`signature::find_signature_in`. Each structure is read with bounded reads
|
||||
(a prefix, then the structure) instead of slicing the whole file; the
|
||||
open-ended `&file_data[addr..]` slices in these modules are gone. Bounds
|
||||
errors keep their values (absolute position, file length).
|
||||
- Structures still indexed by a v2 B-tree — dense attribute storage, a SOHM
|
||||
B-tree index and huge fractal-heap objects found through their B-tree —
|
||||
are not converted yet (the
|
||||
v2 B-tree and dense groups come with milestone M3); over a backend without
|
||||
the whole file in memory they are the clean `ContiguousStorageRequired`
|
||||
error, never a partial result. Raw data, chunk B-tree (v1) indexes and VL
|
||||
data are milestone M2.
|
||||
- **No behaviour change**, checked three ways (2026-09-26, tank): every
|
||||
existing test passes unchanged; the conformance sweep
|
||||
(`conformance/run.sh --no-fetch`) gives a byte-identical `results.json`
|
||||
and identical per-file probe output for all 697 files at `f2ff2c4` and on
|
||||
this branch; and a transcript of every converted `&[u8]` function's
|
||||
result over the fixtures, the conformance corpus and the h5py-written
|
||||
files below (748 files, 7 603 object headers) is byte-identical between
|
||||
the two builds.
|
||||
- New equivalence harness `clawhdf5-format/tests/storage_equivalence.rs`:
|
||||
every converted parser runs over the file as a slice and over
|
||||
`storage::CountingStorage` — a `Storage` that serves an in-memory buffer
|
||||
through `read_at` only (`as_contiguous()` is `None`), copying what it
|
||||
serves and counting reads — and must give identical results. It walks
|
||||
the fixtures, files h5py writes for it (extensible arrays with super
|
||||
blocks and paged data blocks, paged fixed arrays, large v1 and dense
|
||||
groups, a user block, SOHM list and B-tree indexes, dense, shared and
|
||||
committed-type attributes), and with `CLAWHDF5_STORAGE_CORPUS=<dir>` a
|
||||
corpus (all 653 HDF5 files of the conformance corpus pass). Milestones M2
|
||||
and M3 extend it.
|
||||
|
||||
### Chunked full reads (2026-09-26)
|
||||
- **Chunks are decoded straight into the output, into reused buffers.** A
|
||||
|
||||
@@ -1,7 +1,9 @@
|
||||
# Design: range reads (reading HDF5 without holding the whole file)
|
||||
|
||||
Status: proposal, 2026-09-26. No library code has changed; this document is
|
||||
the plan for Phase 3's largest architectural change. Every count below was
|
||||
Status: proposal, 2026-09-26; the plan for Phase 3's largest architectural
|
||||
change. Progress: M1, first part (the `Storage` trait and the metadata
|
||||
parsers listed in `CHANGELOG.md` under "Range reads, milestone M1") is done;
|
||||
group B-tree v2 lookups, dense groups and the facade are not converted yet. Every count below was
|
||||
taken on `tank` on 2026-09-26 at commit `de2a53f`, with the commands given
|
||||
next to it. No timing numbers appear here on purpose: the machine was shared
|
||||
with other build jobs when this was written.
|
||||
|
||||
Reference in New Issue
Block a user