build: pure-Rust zlib-rs as the default deflate backend
The core crates (clawhdf5, -agent, -format, -io, -filters, -ann, -accel, -netcdf4, -cli) now build no C by default: deflate defaults to zlib-rs, a pure-Rust port of zlib-ng, and zlib-ng becomes the opt-in `fast-deflate`, which overrides zlib-rs wherever it is enabled. A default build no longer needs cmake or a C compiler. Measured on tank, both builds run alternately, three rounds, medians: zlib-rs is within 6% of zlib-ng on every HDF5 read and write (512x512 deflate-6 chunked write 1.458 vs 1.484 ms; 64 MB compressed read 64.4 vs 65.2 ms), and compressed output is byte-identical. Details in BENCHMARKS.md, "Deflate backend". Getting there took two fixes the first measurement exposed: - zlib-rs needs `std` to detect SIMD at runtime. flate2 enables it via its default `runtime_detection`, which `default-features = false` had switched off, leaving zlib-rs 3.5x slower on inflate. The `zlib-rs` features now enable it. - Both deflate paths streamed through flate2's 32 KiB read/write wrappers. They now hand the codec the whole chunk in one call, into a buffer sized up front (~5% on chunked writes). This also fixes a silent short read: the streaming reader returned a truncated stream's bytes without an error; a truncated chunk is now DecompressionError. In clawhdf5-filters, output longer than the stated size is now an error rather than silently cut off. CI: ci-test.sh lints and tests the zlib-ng path, and fails if a C-building crate (*-sys, cc, cmake) enters a core crate's default dependency tree. The arm64 job no longer installs cmake. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
@@ -1,6 +1,6 @@
|
||||
# ClawhDF5
|
||||
|
||||
**The memory layer AI agents deserve. One file. Pure Rust. No libhdf5.**
|
||||
**The memory layer AI agents deserve. One file. Pure Rust. Zero C dependencies.**
|
||||
|
||||
[](LICENSE)
|
||||
[](https://www.rust-lang.org)
|
||||
@@ -11,7 +11,7 @@
|
||||
ClawHDF5 is a pure-Rust HDF5 implementation combined with a research-grade agent memory engine. It gives AI agents persistent, searchable, integrity-checked memory — all stored in a single portable file.
|
||||
|
||||
> **Two things live here:**
|
||||
> - **A general-purpose, pure-Rust HDF5 library** — no libhdf5, NetCDF-4 support, SIMD/GPU acceleration. See the **[Crate Map](#crate-map)** and **[BENCHMARKS.md](BENCHMARKS.md)** for the libhdf5 head-to-head numbers.
|
||||
> - **A general-purpose, pure-Rust HDF5 library** — zero C dependencies, NetCDF-4 support, SIMD/GPU acceleration. See the **[Crate Map](#crate-map)** and **[BENCHMARKS.md](BENCHMARKS.md)** for the libhdf5 head-to-head numbers.
|
||||
> - **An agent memory layer built on top of it** — vector search, knowledge graph, hippocampal-style consolidation, in `clawhdf5-agent`.
|
||||
|
||||
The crates are not on crates.io yet, so depend on them from git:
|
||||
@@ -22,10 +22,16 @@ clawhdf5 = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" } # cor
|
||||
clawhdf5-agent = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" } # + agent memory layer
|
||||
```
|
||||
|
||||
> **C dependencies, precisely:** the HDF5 format code is pure Rust and never
|
||||
> links libhdf5. The default deflate backend is zlib-ng (`fast-deflate`), a C
|
||||
> library built from source, so a default build needs `cmake` and a C
|
||||
> compiler. Opt-in codecs (`zstd`, `szip`) and BLAS backends link C too.
|
||||
> **C dependencies, precisely:** the core crates (`clawhdf5`, `clawhdf5-agent`,
|
||||
> `-format`, `-io`, `-filters`, `-ann`, `-accel`, `-netcdf4`, `-cli`) build no C
|
||||
> code by default — no libhdf5, and deflate is the pure-Rust
|
||||
> [zlib-rs](https://github.com/trifectatechfoundation/zlib-rs), which matches
|
||||
> zlib-ng on HDF5 reads and writes and produces byte-identical output
|
||||
> ([BENCHMARKS.md § Deflate backend](BENCHMARKS.md#deflate-backend-zlib-rs-vs-zlib-ng)).
|
||||
> CI fails if a C-building crate enters their default dependency tree. C comes
|
||||
> in only when you ask for it: `fast-deflate` (zlib-ng, needs cmake), `zstd`,
|
||||
> `szip`, the BLAS backends, `clawhdf5-migrate` (bundled SQLite) and the
|
||||
> Node.js bindings.
|
||||
|
||||
> **New here?** Start with the **[Quickstart Guide](docs/QUICKSTART.md)** · See **[Use Cases](docs/USE_CASES.md)** · Read **[Benchmarks](BENCHMARKS.md)**
|
||||
|
||||
@@ -117,6 +123,12 @@ Figures below are from an independent reproduction run on a second machine (AMD
|
||||
| Sequential read (100K f32) | 23.3 µs | 63.6 µs | **2.7×** |
|
||||
| Sequential write (100K f32) | 210 µs | 189 µs | **≈ tie** |
|
||||
|
||||
The chunked-write row was re-measured on the same machine on 2026-09-23, after
|
||||
the default deflate backend became pure-Rust zlib-rs: 1.46 ms against
|
||||
libhdf5's 51.4 ms (**35×**), and 1.48 ms with zlib-ng. libhdf5's own time on
|
||||
that machine moved from 65.0 to 51.4 ms between the two dates, which is most
|
||||
of the difference from 45×; compare same-day numbers only.
|
||||
|
||||
### Vector Search
|
||||
|
||||
**HNSW (the default backend for `hybrid_search`)** — `search_harness`, clustered
|
||||
@@ -564,14 +576,15 @@ setting existed keep their `f32` index; opt out for new stores with
|
||||
| `deflate` | yes | Deflate compression |
|
||||
| `checksum` | yes | Jenkins lookup3 verification |
|
||||
| `provenance` | yes | SHA-256 provenance attributes |
|
||||
| `fast-deflate` | **yes** | zlib-ng backend for faster deflate (C; needs `cmake`) |
|
||||
| `zlib-rs` | **yes** | Pure-Rust deflate backend ([zlib-rs](https://github.com/trifectatechfoundation/zlib-rs)) |
|
||||
| `fast-deflate` | no | zlib-ng deflate backend instead (C; needs `cmake`). Overrides `zlib-rs` when both are on |
|
||||
| `system-zlib-decompress` | **yes** | Use Apple's system libz for decompression (macOS only; no effect elsewhere) |
|
||||
| `parallel` | no | Parallel chunk encoding + compression (rayon) |
|
||||
| `fast-checksum` | no | crc32fast-accelerated checksums |
|
||||
| `lz4` | no | LZ4 block compression filter (id 32004) |
|
||||
| `zstd` | no | Zstandard compression filter (id 32015) |
|
||||
| `pcodec` | no | Pcodec lossless numerical codec (id 32023, via `pco` crate) |
|
||||
| `system-zlib` / `zlib-rs` | no | Alternative zlib backends for deflate |
|
||||
| `system-zlib` | no | System zlib backend for deflate (C) |
|
||||
| `blake3_hash` | no | BLAKE3 content hashing for provenance |
|
||||
| `szip` | no | SZIP filter (id 4) via libaec (C, through the internal `libaec-sys` crate) |
|
||||
|
||||
@@ -601,7 +614,7 @@ setting existed keep their `f32` index; opt out for new stores with
|
||||
## Building
|
||||
|
||||
```bash
|
||||
# Default (needs cmake + a C compiler for zlib-ng)
|
||||
# Default (pure Rust: no cmake or C compiler needed)
|
||||
cargo build --workspace
|
||||
|
||||
# Agent memory with all accelerations (Linux)
|
||||
@@ -723,5 +736,5 @@ MIT
|
||||
|
||||
<p align="center">
|
||||
<em>Built by <a href="https://git.redclaw.dev/quantumclaw">RedClaw Systems</a></em><br>
|
||||
<em>~86,000 lines of Rust. No libhdf5. One file to remember everything.</em>
|
||||
<em>~86,000 lines of Rust. Zero C dependencies. One file to remember everything.</em>
|
||||
</p>
|
||||
|
||||
Reference in New Issue
Block a user