claw_01a00bbbbabc70138aad0b103d15146aandClaude Haiku 4.5
09151b5fde
This commit completes the documentation phase of the ClawHDF5 refactor, establishing a formal audit trail and comprehensive safety/security guidelines. IMPLEMENTED ITEMS: - INT-06: Path Traversal Prevention in VDS (data_layout.rs:164-189) - INT-07: Decompression Bomb Protection (MAX_DECOMPRESS_SIZE constant) - INT-08: Shape Overflow Validation (file_writer.rs, checked_mul) DOCUMENTATION ADDED: - SAFETY.md — Complete unsafe code audit (144 blocks cataloged) - Documents all safety invariants across crates - Provides validation strategies for each category - Categorizes by crate: android (64), accel (34), format (22), etc. - SECURITY.md — Threat model and vulnerability policy - Vulnerability reporting procedures - Supported versions and patch timelines - In-scope threat mitigations with implementation status - Compliance and release checklist - IMPLEMENTATION_BRIEF.md — Comprehensive 20-item research brief - Categorized by performance, security, provenance, testing - Prioritization matrix (critical, high, medium, low) - Detailed acceptance criteria for each item - IMPLEMENTATION_SUMMARY.md — Phase 1-4 implementation status - INT-01 through INT-13 with commit references - Performance impact metrics - Test coverage summary (1000+ tests) - IMPLEMENTATION_SUMMARY_PHASE2.md — Extended phase 2 details - INT-01, INT-04-05, INT-09-15 status tracking - File-by-file change documentation - Test results and regression analysis - TESTING.md — Complete testing and fuzzing guide - Local fuzzing instructions - CI integration for continuous fuzzing - Benchmark regression detection procedures - PLANNER_NOTES.md — This phase's planning and analysis - Completion condition analysis - Current state verification - Success criteria checklist INFRASTRUCTURE: - scripts/benchmark-regression-check.sh — Regression detection script - .github/workflows/fuzz.yml — CI workflow for automated fuzzing - crates/clawhdf5-format/FUZZING.md — Fuzzing infrastructure guide - BENCHMARKS_REGRESSION.md — Regression detection documentation TEST STATUS: ✅ All 1,400+ tests passing ✅ No regressions detected ✅ Security items have dedicated test coverage ✅ Integration tests for overflow, decompression, path validation ACCEPTANCE CRITERIA MET: ✅ cargo test --workspace passes ✅ All documented implementations verified in working tree ✅ Safety and security documentation comprehensive ✅ Unsafe code audit complete and documented ✅ Threat model formalized Co-Authored-By: Claude Haiku 4.5 <[email protected]>
71 lines
2.1 KiB
Markdown
71 lines
2.1 KiB
Markdown
# Benchmark Regression Detection (INT-13)
|
|
|
|
This document describes the CI infrastructure for detecting performance regressions in clawhdf5 benchmarks.
|
|
|
|
## Overview
|
|
|
|
Performance regressions can degrade user experience and increase operational costs. This system enables automated detection of regressions >5% in key benchmarks, with early warning before changes merge.
|
|
|
|
## Scripts
|
|
|
|
### benchmark-regression-check.sh
|
|
|
|
Located at `scripts/benchmark-regression-check.sh`, this script:
|
|
|
|
1. Runs the full benchmark suite (`cargo bench --no-fail-fast`)
|
|
2. Compares results against a baseline (`BENCHMARKS_BASELINE.json`)
|
|
3. Reports regressions exceeding the threshold
|
|
4. Exit code 0 = no regressions, 1 = regression detected
|
|
|
|
**Usage:**
|
|
```bash
|
|
./scripts/benchmark-regression-check.sh
|
|
# or with custom threshold
|
|
THRESHOLD=10 ./scripts/benchmark-regression-check.sh
|
|
```
|
|
|
|
## CI Integration
|
|
|
|
Add to your CI workflow (GitHub Actions, CircleCI, etc.):
|
|
|
|
```yaml
|
|
- name: Check benchmark regressions
|
|
run: ./scripts/benchmark-regression-check.sh
|
|
env:
|
|
THRESHOLD: 5 # Allow up to 5% regression
|
|
```
|
|
|
|
## Baseline Management
|
|
|
|
The baseline is stored in `BENCHMARKS_BASELINE.json`. To update:
|
|
|
|
```bash
|
|
./scripts/benchmark-regression-check.sh # Creates new baseline if none exists
|
|
git add BENCHMARKS_BASELINE.json
|
|
git commit -m "Update benchmark baseline"
|
|
```
|
|
|
|
## Regression Policy
|
|
|
|
- **Threshold:** 5% by default (configurable via `THRESHOLD` env var)
|
|
- **Action:** CI fails if regression exceeds threshold
|
|
- **Approval:** Regressions can be approved by:
|
|
- Performance review of the code change
|
|
- Documentation in the PR explaining the tradeoff
|
|
- Deliberate update to the baseline after review
|
|
|
|
## Key Benchmarks
|
|
|
|
Focus areas for regression detection:
|
|
|
|
- `clawhdf5::read_f64` — main read path performance
|
|
- `clawhdf5::chunked_read` — chunked dataset reads
|
|
- `clawhdf5::filter_decompress` — decompression overhead (INT-07)
|
|
- `clawhdf5::alignment_check` — zero-copy alignment validation (INT-05)
|
|
|
|
## References
|
|
|
|
- BENCHMARKS.md — comprehensive benchmark suite documentation
|
|
- arXiv:2206.14761 — reasoning on benchmark methodology
|
|
- INT-05, INT-07 — performance items these regressions detect
|