From 5d17712adb3a8f2a656223f30e4aa24bb350f788 Mon Sep 17 00:00:00 2001 From: osobh Date: Sat, 26 Sep 2026 13:29:11 -0500 Subject: [PATCH] docs: changelog and design status for range-read milestone M1 (part 1) Lists the parsers now reading through Storage, what still needs the whole file (v2 B-tree-indexed structures: a clean error; raw data: M2), the equivalence harness, and the evidence that nothing changed: existing tests, a byte-identical conformance results.json and per-file probe output against f2ff2c4, and identical slice-API transcripts over 748 files between the two builds. Co-Authored-By: Claude Opus 5.5 (1M context) --- CHANGELOG.md | 43 ++++++++++++++++++++++++++++++++++++++ docs/design/range-reads.md | 6 ++++-- 2 files changed, 47 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4c6d6e6..f33cba0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,6 +15,49 @@ fast path; implemented for `[u8]`, `Vec`, and references, `Box`es and (with `std`) `Arc`s of a `Storage`. Slices and `Vec`s serve borrowed bytes, so parsing an in-memory file costs no copy. +- **The metadata parsers read through `Storage`.** Each converted parser has + an `*_in(&dyn Storage, ..)` core, and its `&[u8]` function is now a thin + wrapper over it, so no caller changes: the superblock + (`Superblock::parse_in`), its extension and cache image + (`read_superblock_extension_in`, `cache_image_state_in`), object headers + with their continuation chunks (`ObjectHeader::parse_in`), local and + global heaps, symbol-table nodes and the group B-tree (v1), fractal heaps, + fixed and extensible array chunk indexes, shared messages and the SOHM + table (`message_data_in`, `message_data_with_sohm_in`, + `load_sohm_table_in`, …), attributes (`extract_attributes_full_in`, + `extract_attributes_tolerant_in`, `AttributeMessage::parse_in_storage`), + fill values (`dataset_fill_value_from_storage`) and virtual-dataset + mappings (`DataLayout::resolve_vds_mappings_in`); also + `signature::find_signature_in`. Each structure is read with bounded reads + (a prefix, then the structure) instead of slicing the whole file; the + open-ended `&file_data[addr..]` slices in these modules are gone. Bounds + errors keep their values (absolute position, file length). +- Structures still indexed by a v2 B-tree — dense attribute storage, a SOHM + B-tree index and huge fractal-heap objects found through their B-tree — + are not converted yet (the + v2 B-tree and dense groups come with milestone M3); over a backend without + the whole file in memory they are the clean `ContiguousStorageRequired` + error, never a partial result. Raw data, chunk B-tree (v1) indexes and VL + data are milestone M2. +- **No behaviour change**, checked three ways (2026-09-26, tank): every + existing test passes unchanged; the conformance sweep + (`conformance/run.sh --no-fetch`) gives a byte-identical `results.json` + and identical per-file probe output for all 697 files at `f2ff2c4` and on + this branch; and a transcript of every converted `&[u8]` function's + result over the fixtures, the conformance corpus and the h5py-written + files below (748 files, 7 603 object headers) is byte-identical between + the two builds. +- New equivalence harness `clawhdf5-format/tests/storage_equivalence.rs`: + every converted parser runs over the file as a slice and over + `storage::CountingStorage` — a `Storage` that serves an in-memory buffer + through `read_at` only (`as_contiguous()` is `None`), copying what it + serves and counting reads — and must give identical results. It walks + the fixtures, files h5py writes for it (extensible arrays with super + blocks and paged data blocks, paged fixed arrays, large v1 and dense + groups, a user block, SOHM list and B-tree indexes, dense, shared and + committed-type attributes), and with `CLAWHDF5_STORAGE_CORPUS=` a + corpus (all 653 HDF5 files of the conformance corpus pass). Milestones M2 + and M3 extend it. ### Chunked full reads (2026-09-26) - **Chunks are decoded straight into the output, into reused buffers.** A diff --git a/docs/design/range-reads.md b/docs/design/range-reads.md index 2e82a6c..488db01 100644 --- a/docs/design/range-reads.md +++ b/docs/design/range-reads.md @@ -1,7 +1,9 @@ # Design: range reads (reading HDF5 without holding the whole file) -Status: proposal, 2026-09-26. No library code has changed; this document is -the plan for Phase 3's largest architectural change. Every count below was +Status: proposal, 2026-09-26; the plan for Phase 3's largest architectural +change. Progress: M1, first part (the `Storage` trait and the metadata +parsers listed in `CHANGELOG.md` under "Range reads, milestone M1") is done; +group B-tree v2 lookups, dense groups and the facade are not converted yet. Every count below was taken on `tank` on 2026-09-26 at commit `de2a53f`, with the commands given next to it. No timing numbers appear here on purpose: the machine was shared with other build jobs when this was written.