Range reads M0/M1 (indexed lookups, Storage trait), ZFP, in-place editing #17

Merged
osobh merged 52 commits from feat/p3-range-zfp-edit into main 2026-09-26 20:42:22 +00:00
2 changed files with 47 additions and 2 deletions
Showing only changes of commit 5d17712adb - Show all commits
+43
View File
@@ -15,6 +15,49 @@
fast path; implemented for `[u8]`, `Vec<u8>`, and references, `Box`es
and (with `std`) `Arc`s of a `Storage`. Slices and `Vec`s serve borrowed
bytes, so parsing an in-memory file costs no copy.
- **The metadata parsers read through `Storage`.** Each converted parser has
an `*_in(&dyn Storage, ..)` core, and its `&[u8]` function is now a thin
wrapper over it, so no caller changes: the superblock
(`Superblock::parse_in`), its extension and cache image
(`read_superblock_extension_in`, `cache_image_state_in`), object headers
with their continuation chunks (`ObjectHeader::parse_in`), local and
global heaps, symbol-table nodes and the group B-tree (v1), fractal heaps,
fixed and extensible array chunk indexes, shared messages and the SOHM
table (`message_data_in`, `message_data_with_sohm_in`,
`load_sohm_table_in`, …), attributes (`extract_attributes_full_in`,
`extract_attributes_tolerant_in`, `AttributeMessage::parse_in_storage`),
fill values (`dataset_fill_value_from_storage`) and virtual-dataset
mappings (`DataLayout::resolve_vds_mappings_in`); also
`signature::find_signature_in`. Each structure is read with bounded reads
(a prefix, then the structure) instead of slicing the whole file; the
open-ended `&file_data[addr..]` slices in these modules are gone. Bounds
errors keep their values (absolute position, file length).
- Structures still indexed by a v2 B-tree — dense attribute storage, a SOHM
B-tree index and huge fractal-heap objects found through their B-tree —
are not converted yet (the
v2 B-tree and dense groups come with milestone M3); over a backend without
the whole file in memory they are the clean `ContiguousStorageRequired`
error, never a partial result. Raw data, chunk B-tree (v1) indexes and VL
data are milestone M2.
- **No behaviour change**, checked three ways (2026-09-26, tank): every
existing test passes unchanged; the conformance sweep
(`conformance/run.sh --no-fetch`) gives a byte-identical `results.json`
and identical per-file probe output for all 697 files at `f2ff2c4` and on
this branch; and a transcript of every converted `&[u8]` function's
result over the fixtures, the conformance corpus and the h5py-written
files below (748 files, 7 603 object headers) is byte-identical between
the two builds.
- New equivalence harness `clawhdf5-format/tests/storage_equivalence.rs`:
every converted parser runs over the file as a slice and over
`storage::CountingStorage` — a `Storage` that serves an in-memory buffer
through `read_at` only (`as_contiguous()` is `None`), copying what it
serves and counting reads — and must give identical results. It walks
the fixtures, files h5py writes for it (extensible arrays with super
blocks and paged data blocks, paged fixed arrays, large v1 and dense
groups, a user block, SOHM list and B-tree indexes, dense, shared and
committed-type attributes), and with `CLAWHDF5_STORAGE_CORPUS=<dir>` a
corpus (all 653 HDF5 files of the conformance corpus pass). Milestones M2
and M3 extend it.
### Chunked full reads (2026-09-26)
- **Chunks are decoded straight into the output, into reused buffers.** A
+4 -2
View File
@@ -1,7 +1,9 @@
# Design: range reads (reading HDF5 without holding the whole file)
Status: proposal, 2026-09-26. No library code has changed; this document is
the plan for Phase 3's largest architectural change. Every count below was
Status: proposal, 2026-09-26; the plan for Phase 3's largest architectural
change. Progress: M1, first part (the `Storage` trait and the metadata
parsers listed in `CHANGELOG.md` under "Range reads, milestone M1") is done;
group B-tree v2 lookups, dense groups and the facade are not converted yet. Every count below was
taken on `tank` on 2026-09-26 at commit `de2a53f`, with the commands given
next to it. No timing numbers appear here on purpose: the machine was shared
with other build jobs when this was written.