build: pure-Rust zlib-rs as the default deflate backend
The core crates (clawhdf5, -agent, -format, -io, -filters, -ann, -accel, -netcdf4, -cli) now build no C by default: deflate defaults to zlib-rs, a pure-Rust port of zlib-ng, and zlib-ng becomes the opt-in `fast-deflate`, which overrides zlib-rs wherever it is enabled. A default build no longer needs cmake or a C compiler. Measured on tank, both builds run alternately, three rounds, medians: zlib-rs is within 6% of zlib-ng on every HDF5 read and write (512x512 deflate-6 chunked write 1.458 vs 1.484 ms; 64 MB compressed read 64.4 vs 65.2 ms), and compressed output is byte-identical. Details in BENCHMARKS.md, "Deflate backend". Getting there took two fixes the first measurement exposed: - zlib-rs needs `std` to detect SIMD at runtime. flate2 enables it via its default `runtime_detection`, which `default-features = false` had switched off, leaving zlib-rs 3.5x slower on inflate. The `zlib-rs` features now enable it. - Both deflate paths streamed through flate2's 32 KiB read/write wrappers. They now hand the codec the whole chunk in one call, into a buffer sized up front (~5% on chunked writes). This also fixes a silent short read: the streaming reader returned a truncated stream's bytes without an error; a truncated chunk is now DecompressionError. In clawhdf5-filters, output longer than the stated size is now an error rather than silently cut off. CI: ci-test.sh lints and tests the zlib-ng path, and fails if a C-building crate (*-sys, cc, cmake) enters a core crate's default dependency tree. The arm64 job no longer installs cmake. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
@@ -3,6 +3,14 @@
|
||||
## Unreleased
|
||||
|
||||
### Upgrade Notes
|
||||
- **The default build no longer compiles any C.** Deflate now defaults to the
|
||||
pure-Rust zlib-rs instead of zlib-ng, so building the core crates needs
|
||||
neither cmake nor a C compiler. Speed on HDF5 reads and writes is within 6%
|
||||
of zlib-ng, and compressed output is byte-identical. To keep zlib-ng, enable
|
||||
`fast-deflate` (on `clawhdf5`, `clawhdf5-format` or `clawhdf5-filters`); it
|
||||
overrides zlib-rs wherever it is on.
|
||||
- **A truncated deflate chunk is now an error.** It used to read back short,
|
||||
with no error.
|
||||
- **New stores use the int8 vector index by default.**
|
||||
`MemoryConfig::quantized_index` now defaults to `true`: a quarter of the
|
||||
index memory, builds 1.8x (x86-64) and 2.3x (Raspberry Pi 5) faster, and
|
||||
@@ -13,6 +21,28 @@
|
||||
`quantized_index = false`, or pass `create --f32-index` to the CLI, to opt
|
||||
out. The CLI's `--quantized-index` is still accepted but is now a no-op.
|
||||
|
||||
### Build
|
||||
- **Pure-Rust default.** `clawhdf5-format`, `clawhdf5-filters` and the
|
||||
`clawhdf5` facade default to the `zlib-rs` deflate backend; `fast-deflate`
|
||||
(zlib-ng) is opt-in. No crate in the default dependency tree of the core
|
||||
crates compiles C, and `ci-test.sh` now fails if one appears. The facade's
|
||||
`fast-deflate` was on by default and is now off. See `BENCHMARKS.md`,
|
||||
"Deflate backend".
|
||||
- `zlib-rs` also enables flate2's `runtime_detection`. Without it zlib-rs has
|
||||
no `std`, cannot detect SIMD at runtime, and inflates 3.5x slower; the
|
||||
workspace builds flate2 with `default-features = false`, which had been
|
||||
switching it off.
|
||||
- CI keeps zlib-ng building and tested; the arm64 job no longer needs cmake.
|
||||
|
||||
### Correctness
|
||||
- `clawhdf5-format`: **a truncated deflate chunk read back short, with no
|
||||
error.** The deflate filter used flate2's streaming reader, which returns the
|
||||
bytes it has when the input runs out before the end-of-stream marker. It now
|
||||
decodes in one pass into a buffer sized to the chunk and reports a
|
||||
truncated stream as `DecompressionError`. Same fix in `clawhdf5-filters`,
|
||||
where output longer than the stated size was also silently cut off; it is
|
||||
now an error.
|
||||
|
||||
### Defaults
|
||||
- `clawhdf5-agent`: `MemoryConfig::quantized_index` defaults to `true` for new
|
||||
stores. The reason it had been off — that int8 search was slower on ARM —
|
||||
@@ -26,6 +56,10 @@
|
||||
knew to ask; it now only ever switches the default off.
|
||||
|
||||
### Performance
|
||||
- `clawhdf5-format`, `clawhdf5-filters`: both deflate paths hand the codec the
|
||||
whole chunk in one call, into a buffer allocated once, instead of streaming
|
||||
it through a 32 KiB buffer: about 5% on chunked writes and 10% on zlib-ng's
|
||||
1 MB inflate.
|
||||
- `clawhdf5-accel`: **`dot_i8` has aarch64 kernels** — `SDOT` for CPUs with
|
||||
the ARMv8.2 dot-product extension (Cortex-A76 and later, Neoverse-N1, every
|
||||
Apple Silicon generation) and plain NEON (`vmull_s8` + `vpadalq_s16`) for
|
||||
|
||||
Reference in New Issue
Block a user