build: pure-Rust zlib-rs as the default deflate backend
The core crates (clawhdf5, -agent, -format, -io, -filters, -ann, -accel, -netcdf4, -cli) now build no C by default: deflate defaults to zlib-rs, a pure-Rust port of zlib-ng, and zlib-ng becomes the opt-in `fast-deflate`, which overrides zlib-rs wherever it is enabled. A default build no longer needs cmake or a C compiler. Measured on tank, both builds run alternately, three rounds, medians: zlib-rs is within 6% of zlib-ng on every HDF5 read and write (512x512 deflate-6 chunked write 1.458 vs 1.484 ms; 64 MB compressed read 64.4 vs 65.2 ms), and compressed output is byte-identical. Details in BENCHMARKS.md, "Deflate backend". Getting there took two fixes the first measurement exposed: - zlib-rs needs `std` to detect SIMD at runtime. flate2 enables it via its default `runtime_detection`, which `default-features = false` had switched off, leaving zlib-rs 3.5x slower on inflate. The `zlib-rs` features now enable it. - Both deflate paths streamed through flate2's 32 KiB read/write wrappers. They now hand the codec the whole chunk in one call, into a buffer sized up front (~5% on chunked writes). This also fixes a silent short read: the streaming reader returned a truncated stream's bytes without an error; a truncated chunk is now DecompressionError. In clawhdf5-filters, output longer than the stated size is now an error rather than silently cut off. CI: ci-test.sh lints and tests the zlib-ng path, and fails if a C-building crate (*-sys, cc, cmake) enters a core crate's default dependency tree. The arm64 job no longer installs cmake. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
@@ -27,7 +27,11 @@ Cargo workspace with 16 crates under `crates/` (plus `libaec-sys`, an internal F
|
||||
| `clawhdf5-bench` | Benchmark suite |
|
||||
|
||||
## Key Features
|
||||
- Zero-dependency HDF5 read/write (no libhdf5 C library required)
|
||||
- Zero-C-dependency HDF5 read/write: no libhdf5, and deflate defaults to
|
||||
pure-Rust zlib-rs (`fast-deflate` opts into zlib-ng, which needs cmake).
|
||||
`ci-test.sh` fails if a C-building crate enters the core crates' default
|
||||
tree. flate2 must keep `runtime_detection` with zlib-rs — without it zlib-rs
|
||||
loses SIMD and inflates 3.5x slower.
|
||||
- HNSW vector index for semantic similarity search over agent memories — the
|
||||
`clawhdf5-agent` `hnsw` feature is **on by default**, so `hybrid_search` uses
|
||||
the approximate `clawhdf5-ann` index for the vector stage (the index mirrors
|
||||
@@ -128,8 +132,9 @@ cargo test --workspace
|
||||
|
||||
Keep workflows free of JavaScript actions (`actions/checkout`, `actions/cache`,
|
||||
…): `rust:latest` has no `node`, and not every runner reaches GitHub, where
|
||||
they are fetched from. Check out with plain `git` instead. Both jobs need
|
||||
`cmake` for `libz-ng-sys` (from `clawhdf5-format`'s default `fast-deflate`).
|
||||
they are fetched from. Check out with plain `git` instead. The `test` job
|
||||
installs `cmake` for the opt-in `fast-deflate` (zlib-ng) steps; the default
|
||||
build needs no C toolchain, so `test-arm64` does not.
|
||||
All runners are on `gitea-runner` 3.5.0, from `docker.gitea.com/act_runner`
|
||||
— `gitea/act_runner:latest` on Docker Hub is frozen at 0.6.1.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user