Every crate under crates/ now has a README (android, bench, cli, napi and
wasm had none), each saying what the crate is, its main types and
functions (names checked against the code), its cargo features with
defaults and which ones build C (checked with `cargo tree`), and links to
the top-level docs.
Corrections to the old stubs:
- clawhdf5-derive: the derive is `H5Type`, not `HDF5Type`, and it needs
clawhdf5-format as a dependency.
- clawhdf5-filters: deflate backends only, and no library crate depends
on it; the filter pipeline and every other codec are in -format.
- clawhdf5-gpu: vector distance compute, not I/O; not used by
HDF5Memory::search.
- clawhdf5-io: MpiVol is root-read + broadcast, not collective MPI-IO.
- clawhdf5-ann: from_hdf5/search(q, k) did not exist; load_from_hdf5 and
search(q, k, ef).
- clawhdf5-accel: checksum::crc32_simd did not exist; the SSE4 and wasm
backends are reported but run the scalar kernels.
- clawhdf5-gpu: the old example called l2_distances, which does not
exist (l2_search).
- clawhdf5-agent: it described a "vector store" with "GPU acceleration";
it now covers HDF5Memory, search options, WAL, signing, the graph.
- crates.io/docs.rs badges removed and `cargo install <crate>` replaced:
nothing is published; depend on git.
- fuzz: the opt-in CLAWHDF5_FUZZ_SECONDS smoke run in ci-test.sh.
- tools: the FileEditor interop tests that live in this crate.
- remote, py: license, other front ends, limits, File.mode/flush/chunks.
The Rust examples of the facade, format, filters, accel, ann, derive and
agent READMEs were compiled and run as tests (netcdf4, gpu and remote
compiled only) in a scratch crate; the CLI example was run.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
- Add ensure_len(data, offset, needed) helper to chunked_read.rs,
data_read.rs, and local_heap.rs (matching the existing btree_v1.rs/
object_header.rs convention) and use it at every plain-arithmetic
offset+size bounds check found in these files, closing usize-overflow
panics reachable from crafted near-usize::MAX offsets/addresses.
- collect_chunk_info: add a depth-limited internal wrapper
(collect_chunk_info_inner, MAX_CHUNK_BTREE_DEPTH=64) to reject a
crafted self-referencing/cyclic B-tree v1 chunk index instead of
recursing unboundedly (stack-overflow DoS).
- read_compound_fields: validate byte_offset+field_size against the
compound's declared element size before slicing, instead of an
unguarded out-of-bounds panic on a crafted member offset.
- read_chunked_data/_cached/_sweep/_indexed: guard `ndims - 1` against
underflow for a degenerate zero-dimension chunked layout.
- copy_chunk_to_output: rewrite all offset/stride arithmetic (both the
1-D fast path and the general N-D path) to use checked_add/checked_mul,
skipping an out-of-range row/chunk instead of panicking on overflow.
Add a new cargo-fuzz target, fuzz_dataset_read, that walks every dataset
in a parsed file via the clawhdf5 facade and exercises the contiguous/
chunked/compact raw-data read paths that the existing fuzz_full_file
target doesn't reach. Seeded with the chunked/VDS/compound-relevant test
fixtures plus two crash regressions found during this pass (the
copy_chunk_to_output overflow and the ndims-1 underflow, both fixed
above — this target found real bugs within the first couple of runs).
Not wired into CI (nightly-only, multi-minute runs); documented in
fuzz/README.md as a manual/scheduled check instead. Also fixed the
README's stale rustyhdf5-format naming while touching this file.
Added regression tests for every fix (near-usize::MAX offsets, the
self-referencing B-tree case, the compound byte_offset overrun, the
zero-dim layout, and both copy_chunk_to_output overflow paths) so these
are caught by `cargo test`, not just the fuzz corpus.
