Range reads M0/M1 (indexed lookups, Storage trait), ZFP, in-place editing #17

Merged
osobh merged 52 commits from feat/p3-range-zfp-edit into main 2026-09-26 20:42:22 +00:00
2 changed files with 44 additions and 0 deletions
Showing only changes of commit 1b4a93f65a - Show all commits
+35
View File
@@ -2,6 +2,41 @@
## Unreleased
### Name lookups through the name index (2026-09-26)
- **Finding one link or attribute by name reads the name index, not every
entry.** In a dense group (links in a fractal heap) the v2 B-tree name
index (record type 5, lookup3 hash of the name) is descended to the
records with the name's hash and only their links are read — O(log n)
instead of all n. Path resolution (`File::dataset`, `resolve_path_any`,
soft-link targets) and `Group::dataset`/`Group::group` (on `File`,
`MmapFile` and `LazyFile`, which listed the whole group per call) use it;
names whose hashes collide are all compared, so the order libhdf5 gives
them does not matter. New `clawhdf5_format::group_v2::resolve_child`,
`btree_v2::find_btree_v2_records` (records in one key range), and a
`lookup-stats` feature counting heap objects read, for tests. Huge heap
objects are found through their index the same way.
- **`attr(name)`** on the facade's groups and datasets (all three file
types): one attribute, found in dense storage through its name index
(record type 8) instead of reading every attribute
(`clawhdf5_format::attribute::find_attribute_in_file`).
- **`Group::entries()` and `File::group_at(address)`**: a listing's
`(name, address)` pairs, to open children without looking names up again.
- Test: `crates/clawhdf5/tests/indexed_lookup_interop.rs` — every child of
an h5py-written 35 001-link group (with colliding hashes) opened by name
reads at most two links per lookup (before: 35 001), matches h5py, and
every link kind (soft, relative, dangling, external) resolves as h5py
resolves it in dense and compact groups.
### Checked address conversion (2026-09-26)
- **No 64-bit file value is truncated on a 32-bit target.** Every
`u64 as usize` cast in `clawhdf5-format` (115) is gone: file addresses,
lengths and counts go through `addr::to_usize`, which fails with
`FormatError::Overflow` where the value does not fit (wasm32 and other
32-bit targets; it used to wrap onto another part of the file), and
in-memory counts through `addr::saturating_usize`. On 64-bit targets
nothing changes. `scripts/check-32bit-casts.sh` (run by `ci-test.sh`)
lints the wasm32 build and fails on any new truncating cast.
### Chunked full reads (2026-09-26)
- **Chunks are decoded straight into the output, into reused buffers.** A
full read of a chunked dataset faulted in about three times its size in
+9
View File
@@ -379,6 +379,15 @@ fast path within benchmark noise.
n children decodes its links O(n) times. Look names up through the index
(above) and let a listing hand out its entries, so the cache has less to
absorb.
- *Status 2026-09-26:* done on branch `perf/p3-indexed-lookups` — link and
attribute names through the name indexes (`group_v2::resolve_child`,
`attribute::find_attribute_in_file`; creation-order lookups by name do
not exist in the API, so the creation-order index is still only listed),
`addr::to_usize`/`saturating_usize` for all 115 `u64 as usize` casts in
`clawhdf5-format` (the 133 above counted any `*addr*/*offset* as usize`,
mostly widening `u8`/`u32` casts; `scripts/check-32bit-casts.sh` lints
wasm32 for the truncating ones), and `Group::entries`/`File::group_at`.
The facade, io and ann casts are not converted.
**M1 — metadata over the trait, in-memory impl identical to today (2–3 weeks).**
- Add `Storage` (above) to `clawhdf5-format`, `no_std`-compatible, with