build: pure-Rust zlib-rs as the default deflate backend

The core crates (clawhdf5, -agent, -format, -io, -filters, -ann, -accel,
-netcdf4, -cli) now build no C by default: deflate defaults to zlib-rs,
a pure-Rust port of zlib-ng, and zlib-ng becomes the opt-in
`fast-deflate`, which overrides zlib-rs wherever it is enabled. A default
build no longer needs cmake or a C compiler.

Measured on tank, both builds run alternately, three rounds, medians:
zlib-rs is within 6% of zlib-ng on every HDF5 read and write (512x512
deflate-6 chunked write 1.458 vs 1.484 ms; 64 MB compressed read 64.4
vs 65.2 ms), and compressed output is byte-identical. Details in
BENCHMARKS.md, "Deflate backend".

Getting there took two fixes the first measurement exposed:

- zlib-rs needs `std` to detect SIMD at runtime. flate2 enables it via
  its default `runtime_detection`, which `default-features = false` had
  switched off, leaving zlib-rs 3.5x slower on inflate. The `zlib-rs`
  features now enable it.
- Both deflate paths streamed through flate2's 32 KiB read/write
  wrappers. They now hand the codec the whole chunk in one call, into a
  buffer sized up front (~5% on chunked writes). This also fixes a
  silent short read: the streaming reader returned a truncated stream's
  bytes without an error; a truncated chunk is now DecompressionError.
  In clawhdf5-filters, output longer than the stated size is now an
  error rather than silently cut off.

CI: ci-test.sh lints and tests the zlib-ng path, and fails if a
C-building crate (*-sys, cc, cmake) enters a core crate's default
dependency tree. The arm64 job no longer installs cmake.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-23 11:05:22 -05:00
co-authored by Claude Opus 5.5
parent e9b37a9602
commit 735db117a7
13 changed files with 471 additions and 113 deletions
+34
View File
@@ -3,6 +3,14 @@
## Unreleased
### Upgrade Notes
- **The default build no longer compiles any C.** Deflate now defaults to the
pure-Rust zlib-rs instead of zlib-ng, so building the core crates needs
neither cmake nor a C compiler. Speed on HDF5 reads and writes is within 6%
of zlib-ng, and compressed output is byte-identical. To keep zlib-ng, enable
`fast-deflate` (on `clawhdf5`, `clawhdf5-format` or `clawhdf5-filters`); it
overrides zlib-rs wherever it is on.
- **A truncated deflate chunk is now an error.** It used to read back short,
with no error.
- **New stores use the int8 vector index by default.**
`MemoryConfig::quantized_index` now defaults to `true`: a quarter of the
index memory, builds 1.8x (x86-64) and 2.3x (Raspberry Pi 5) faster, and
@@ -13,6 +21,28 @@
`quantized_index = false`, or pass `create --f32-index` to the CLI, to opt
out. The CLI's `--quantized-index` is still accepted but is now a no-op.
### Build
- **Pure-Rust default.** `clawhdf5-format`, `clawhdf5-filters` and the
`clawhdf5` facade default to the `zlib-rs` deflate backend; `fast-deflate`
(zlib-ng) is opt-in. No crate in the default dependency tree of the core
crates compiles C, and `ci-test.sh` now fails if one appears. The facade's
`fast-deflate` was on by default and is now off. See `BENCHMARKS.md`,
"Deflate backend".
- `zlib-rs` also enables flate2's `runtime_detection`. Without it zlib-rs has
no `std`, cannot detect SIMD at runtime, and inflates 3.5x slower; the
workspace builds flate2 with `default-features = false`, which had been
switching it off.
- CI keeps zlib-ng building and tested; the arm64 job no longer needs cmake.
### Correctness
- `clawhdf5-format`: **a truncated deflate chunk read back short, with no
error.** The deflate filter used flate2's streaming reader, which returns the
bytes it has when the input runs out before the end-of-stream marker. It now
decodes in one pass into a buffer sized to the chunk and reports a
truncated stream as `DecompressionError`. Same fix in `clawhdf5-filters`,
where output longer than the stated size was also silently cut off; it is
now an error.
### Defaults
- `clawhdf5-agent`: `MemoryConfig::quantized_index` defaults to `true` for new
stores. The reason it had been off — that int8 search was slower on ARM —
@@ -26,6 +56,10 @@
knew to ask; it now only ever switches the default off.
### Performance
- `clawhdf5-format`, `clawhdf5-filters`: both deflate paths hand the codec the
whole chunk in one call, into a buffer allocated once, instead of streaming
it through a 32 KiB buffer: about 5% on chunked writes and 10% on zlib-ng's
1 MB inflate.
- `clawhdf5-accel`: **`dot_i8` has aarch64 kernels** — `SDOT` for CPUs with
the ARMv8.2 dot-product extension (Cortex-A76 and later, Neoverse-N1, every
Apple Silicon generation) and plain NEON (`vmull_s8` + `vpadalq_s16`) for