diff --git a/CHANGELOG.md b/CHANGELOG.md index 4c6d6e6..f33cba0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,6 +15,49 @@ fast path; implemented for `[u8]`, `Vec`, and references, `Box`es and (with `std`) `Arc`s of a `Storage`. Slices and `Vec`s serve borrowed bytes, so parsing an in-memory file costs no copy. +- **The metadata parsers read through `Storage`.** Each converted parser has + an `*_in(&dyn Storage, ..)` core, and its `&[u8]` function is now a thin + wrapper over it, so no caller changes: the superblock + (`Superblock::parse_in`), its extension and cache image + (`read_superblock_extension_in`, `cache_image_state_in`), object headers + with their continuation chunks (`ObjectHeader::parse_in`), local and + global heaps, symbol-table nodes and the group B-tree (v1), fractal heaps, + fixed and extensible array chunk indexes, shared messages and the SOHM + table (`message_data_in`, `message_data_with_sohm_in`, + `load_sohm_table_in`, …), attributes (`extract_attributes_full_in`, + `extract_attributes_tolerant_in`, `AttributeMessage::parse_in_storage`), + fill values (`dataset_fill_value_from_storage`) and virtual-dataset + mappings (`DataLayout::resolve_vds_mappings_in`); also + `signature::find_signature_in`. Each structure is read with bounded reads + (a prefix, then the structure) instead of slicing the whole file; the + open-ended `&file_data[addr..]` slices in these modules are gone. Bounds + errors keep their values (absolute position, file length). +- Structures still indexed by a v2 B-tree — dense attribute storage, a SOHM + B-tree index and huge fractal-heap objects found through their B-tree — + are not converted yet (the + v2 B-tree and dense groups come with milestone M3); over a backend without + the whole file in memory they are the clean `ContiguousStorageRequired` + error, never a partial result. Raw data, chunk B-tree (v1) indexes and VL + data are milestone M2. +- **No behaviour change**, checked three ways (2026-09-26, tank): every + existing test passes unchanged; the conformance sweep + (`conformance/run.sh --no-fetch`) gives a byte-identical `results.json` + and identical per-file probe output for all 697 files at `f2ff2c4` and on + this branch; and a transcript of every converted `&[u8]` function's + result over the fixtures, the conformance corpus and the h5py-written + files below (748 files, 7 603 object headers) is byte-identical between + the two builds. +- New equivalence harness `clawhdf5-format/tests/storage_equivalence.rs`: + every converted parser runs over the file as a slice and over + `storage::CountingStorage` — a `Storage` that serves an in-memory buffer + through `read_at` only (`as_contiguous()` is `None`), copying what it + serves and counting reads — and must give identical results. It walks + the fixtures, files h5py writes for it (extensible arrays with super + blocks and paged data blocks, paged fixed arrays, large v1 and dense + groups, a user block, SOHM list and B-tree indexes, dense, shared and + committed-type attributes), and with `CLAWHDF5_STORAGE_CORPUS=` a + corpus (all 653 HDF5 files of the conformance corpus pass). Milestones M2 + and M3 extend it. ### Chunked full reads (2026-09-26) - **Chunks are decoded straight into the output, into reused buffers.** A diff --git a/docs/design/range-reads.md b/docs/design/range-reads.md index 2e82a6c..488db01 100644 --- a/docs/design/range-reads.md +++ b/docs/design/range-reads.md @@ -1,7 +1,9 @@ # Design: range reads (reading HDF5 without holding the whole file) -Status: proposal, 2026-09-26. No library code has changed; this document is -the plan for Phase 3's largest architectural change. Every count below was +Status: proposal, 2026-09-26; the plan for Phase 3's largest architectural +change. Progress: M1, first part (the `Storage` trait and the metadata +parsers listed in `CHANGELOG.md` under "Range reads, milestone M1") is done; +group B-tree v2 lookups, dense groups and the facade are not converted yet. Every count below was taken on `tank` on 2026-09-26 at commit `de2a53f`, with the commands given next to it. No timing numbers appear here on purpose: the machine was shared with other build jobs when this was written.