Omar Sobh
297ee5ec17
CI / test (push) Failing after 4s
- Add ensure_len(data, offset, needed) helper to chunked_read.rs, data_read.rs, and local_heap.rs (matching the existing btree_v1.rs/ object_header.rs convention) and use it at every plain-arithmetic offset+size bounds check found in these files, closing usize-overflow panics reachable from crafted near-usize::MAX offsets/addresses. - collect_chunk_info: add a depth-limited internal wrapper (collect_chunk_info_inner, MAX_CHUNK_BTREE_DEPTH=64) to reject a crafted self-referencing/cyclic B-tree v1 chunk index instead of recursing unboundedly (stack-overflow DoS). - read_compound_fields: validate byte_offset+field_size against the compound's declared element size before slicing, instead of an unguarded out-of-bounds panic on a crafted member offset. - read_chunked_data/_cached/_sweep/_indexed: guard `ndims - 1` against underflow for a degenerate zero-dimension chunked layout. - copy_chunk_to_output: rewrite all offset/stride arithmetic (both the 1-D fast path and the general N-D path) to use checked_add/checked_mul, skipping an out-of-range row/chunk instead of panicking on overflow. Add a new cargo-fuzz target, fuzz_dataset_read, that walks every dataset in a parsed file via the clawhdf5 facade and exercises the contiguous/ chunked/compact raw-data read paths that the existing fuzz_full_file target doesn't reach. Seeded with the chunked/VDS/compound-relevant test fixtures plus two crash regressions found during this pass (the copy_chunk_to_output overflow and the ndims-1 underflow, both fixed above — this target found real bugs within the first couple of runs). Not wired into CI (nightly-only, multi-minute runs); documented in fuzz/README.md as a manual/scheduled check instead. Also fixed the README's stale rustyhdf5-format naming while touching this file. Added regression tests for every fix (near-usize::MAX offsets, the self-referencing B-tree case, the compound byte_offset overrun, the zero-dim layout, and both copy_chunk_to_output overflow paths) so these are caught by `cargo test`, not just the fuzz corpus.
76 lines
2.7 KiB
Markdown
76 lines
2.7 KiB
Markdown
# Fuzz Testing for clawhdf5-format
|
|
|
|
Uses [cargo-fuzz](https://github.com/rust-fuzz/cargo-fuzz) (libFuzzer) to test parser robustness against malformed inputs.
|
|
|
|
## Prerequisites
|
|
|
|
```bash
|
|
cargo install cargo-fuzz
|
|
rustup toolchain install nightly
|
|
```
|
|
|
|
## Fuzz Targets
|
|
|
|
| Target | Parser | Description |
|
|
|--------|--------|-------------|
|
|
| `fuzz_superblock` | `Superblock::parse` | Superblock parsing (v0-v3) with signature search |
|
|
| `fuzz_object_header` | `ObjectHeader::parse` | Object header v1/v2 with various offset/length sizes |
|
|
| `fuzz_datatype` | `Datatype::parse` | All 12 HDF5 datatype classes (recursive) |
|
|
| `fuzz_dataspace` | `Dataspace::parse` | Dataspace messages with various length sizes |
|
|
| `fuzz_fractal_heap` | `FractalHeapHeader::parse` | Fractal heap header parsing |
|
|
| `fuzz_btree_v2` | `BTreeV2Header::parse` | B-tree v2 header parsing |
|
|
| `fuzz_filter_pipeline` | `FilterPipeline::parse` | Filter pipeline messages (v1/v2) |
|
|
| `fuzz_full_file` | signature + superblock + root group | End-to-end file parsing chain |
|
|
| `fuzz_dataset_read` | `Dataset::read_*` (via `clawhdf5`) | Walks every dataset in the parsed file and exercises the contiguous/chunked/compact raw-data read paths (`chunked_read.rs`, `data_read.rs`) that `fuzz_full_file` doesn't reach |
|
|
|
|
## Running
|
|
|
|
Run a single target (runs indefinitely until stopped or a crash is found):
|
|
|
|
```bash
|
|
cd crates/clawhdf5-format
|
|
cargo +nightly fuzz run fuzz_datatype
|
|
```
|
|
|
|
Run with a time limit (seconds):
|
|
|
|
```bash
|
|
cargo +nightly fuzz run fuzz_datatype -- -max_total_time=60
|
|
```
|
|
|
|
Run all targets for 30 seconds each:
|
|
|
|
```bash
|
|
for target in fuzz_superblock fuzz_object_header fuzz_datatype fuzz_dataspace \
|
|
fuzz_fractal_heap fuzz_btree_v2 fuzz_filter_pipeline fuzz_full_file \
|
|
fuzz_dataset_read; do
|
|
echo "=== $target ==="
|
|
cargo +nightly fuzz run "$target" -- -max_total_time=30 -max_len=4096
|
|
done
|
|
```
|
|
|
|
## CI
|
|
|
|
These targets are **not** run in CI (`.gitea/workflows/ci.yml`) — cargo-fuzz
|
|
requires nightly and each meaningful run takes minutes, which doesn't fit a
|
|
per-PR gate. Run them manually on a schedule (e.g. before a release, or after
|
|
touching parser code) instead.
|
|
|
|
## Reproducing Crashes
|
|
|
|
If a crash is found, the input is saved to `fuzz/artifacts/<target>/`. Reproduce with:
|
|
|
|
```bash
|
|
cargo +nightly fuzz run fuzz_datatype fuzz/artifacts/fuzz_datatype/crash-<hash>
|
|
```
|
|
|
|
Minimize the crashing input:
|
|
|
|
```bash
|
|
cargo +nightly fuzz tmin fuzz_datatype fuzz/artifacts/fuzz_datatype/crash-<hash>
|
|
```
|
|
|
|
## Design
|
|
|
|
Each fuzz target feeds arbitrary bytes directly to a parser entry point. The parsers must never panic on any input -- they should return `Err(FormatError)` for malformed data. Any panic found by fuzzing is a bug that should be fixed with proper bounds checks and error returns.
|