Commit Graph
7 Commits
Author SHA1 Message Date
osobh 437e81cfff Merge branch 'feat/p3-storage-trait' into feat/p3-range-zfp-edit
# Conflicts:
#	CHANGELOG.md
#	crates/clawhdf5-format/src/attribute.rs
#	crates/clawhdf5-format/src/btree_v1.rs
#	crates/clawhdf5-format/src/data_layout.rs
#	crates/clawhdf5-format/src/extensible_array.rs
#	crates/clawhdf5-format/src/fixed_array.rs
#	crates/clawhdf5-format/src/fractal_heap.rs
#	crates/clawhdf5-format/src/local_heap.rs
#	crates/clawhdf5-format/src/shared_message.rs
2026-09-26 14:51:51 -05:00
osobhandClaude Opus 5.5 895c79a2fe docs: M1 changelog after review: probe identity, speed, bounded reads
- Per-file probe output is identical for 696 of 697 files, not all:
  cve-2025-2310.h5's error string depends on which parallel chunk decode
  fails first, at f2ff2c4 as on this branch.
- The parser cores are generic (S: Storage + ?Sized); provisional A/B
  numbers against f2ff2c4, including the one bench that still shows
  ObjectHeader::parse slower when old and new are separate binaries.
- Reads sized by untrusted fields are bounded; the harness only accepts
  the known whole-file fallbacks.
- range-reads.md records why M1 went generic rather than &dyn.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
2026-09-26 14:34:35 -05:00
osobhandClaude Opus 5.5 b6cbd2319f format: checked chunk addresses on the parallel read path
Three `chunk_info.address as usize` casts behind the `parallel` feature
survived the conversion, because check-32bit-casts.sh linted only default
features plus plugin-filters. On a 32-bit target with rayon a chunk address
past 4 GiB still wrapped onto another part of the file. They go through
addr::to_usize now, and the lane index (h % n, always < n) through
saturating_usize.

The script now lints no default features, default features, and every
optional feature but szip (wasm32; the set with zstd, which does not build
for wasm32, on the host, where the lint reports the same casts). With the
old parallel_read.rs/lane_partition.rs it fails listing the four casts; the
old script passed them. CHANGELOG and the design note give the exact count
(119) and what is not covered.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
2026-09-26 14:08:57 -05:00
osobhandClaude Opus 5.5 1b4a93f65a docs: indexed name lookups and checked address conversion (M0)
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
2026-09-26 13:33:24 -05:00
osobhandClaude Opus 5.5 5d17712adb docs: changelog and design status for range-read milestone M1 (part 1)
Lists the parsers now reading through Storage, what still needs the
whole file (v2 B-tree-indexed structures: a clean error; raw data: M2),
the equivalence harness, and the evidence that nothing changed: existing
tests, a byte-identical conformance results.json and per-file probe
output against f2ff2c4, and identical slice-API transcripts over 748
files between the two builds.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
2026-09-26 13:29:11 -05:00
osobhandClaude Opus 5.5 1c1af460b6 docs: design for range reads (S3/HTTP, wasm lazy loading, SWMR, 32-bit)
Inventory of the 109 whole-file parser functions, a measurement of the
metadata ranges clawhdf5 touches on three corpus files against libhdf5,
options (storage trait, virtual slice, metadata prefetch, userfaultfd)
with how ros3, h5py+fsspec, pyfive, h5wasm and object_store do it, and
an incremental plan: M0 name-index lookups, M1 metadata over a Storage
trait, M2 raw data with batched ranges, M3 object_store backend,
M4 wasm fetch-driven lazy loading.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
2026-09-26 10:17:02 -05:00
osobhandClaude Opus 5.5 c5cd14c2b2 docs: tools to measure how a range reader would read HDF5
inventory.py counts the functions and call sites that take the whole
file as &[u8]; range-trace (standalone crate, x86-64 Linux) records every
load clawhdf5 makes from a file by mprotect + single-step, unchanged
library code; libhdf5_reads.py counts libhdf5's reads through h5py's
fileobj driver and prints a dataset's chunk extents.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
2026-09-26 10:17:02 -05:00