osobhandClaude Opus 5.5
735db117a7
The core crates (clawhdf5, -agent, -format, -io, -filters, -ann, -accel, -netcdf4, -cli) now build no C by default: deflate defaults to zlib-rs, a pure-Rust port of zlib-ng, and zlib-ng becomes the opt-in `fast-deflate`, which overrides zlib-rs wherever it is enabled. A default build no longer needs cmake or a C compiler. Measured on tank, both builds run alternately, three rounds, medians: zlib-rs is within 6% of zlib-ng on every HDF5 read and write (512x512 deflate-6 chunked write 1.458 vs 1.484 ms; 64 MB compressed read 64.4 vs 65.2 ms), and compressed output is byte-identical. Details in BENCHMARKS.md, "Deflate backend". Getting there took two fixes the first measurement exposed: - zlib-rs needs `std` to detect SIMD at runtime. flate2 enables it via its default `runtime_detection`, which `default-features = false` had switched off, leaving zlib-rs 3.5x slower on inflate. The `zlib-rs` features now enable it. - Both deflate paths streamed through flate2's 32 KiB read/write wrappers. They now hand the codec the whole chunk in one call, into a buffer sized up front (~5% on chunked writes). This also fixes a silent short read: the streaming reader returned a truncated stream's bytes without an error; a truncated chunk is now DecompressionError. In clawhdf5-filters, output longer than the stated size is now an error rather than silently cut off. CI: ci-test.sh lints and tests the zlib-ng path, and fails if a C-building crate (*-sys, cc, cmake) enters a core crate's default dependency tree. The arm64 job no longer installs cmake. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
87 lines
4.2 KiB
YAML
87 lines
4.2 KiB
YAML
name: CI
|
|
on:
|
|
push:
|
|
branches: [main]
|
|
pull_request:
|
|
branches: [main]
|
|
jobs:
|
|
test:
|
|
runs-on: ubuntu-latest
|
|
container: rust:latest
|
|
steps:
|
|
# Plain git rather than actions/checkout: that is a JavaScript action,
|
|
# and rust:latest has no `node`, so it failed with exit 127 before any
|
|
# code was built — on every push. actions/cache went for the same reason.
|
|
- name: Check out
|
|
run: |
|
|
git init -q .
|
|
git remote add origin "${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}.git"
|
|
for i in 1 2 3; do git fetch -q --depth 1 origin "${GITHUB_SHA}" && break; sleep 5; done
|
|
git checkout -q FETCH_HEAD
|
|
- name: Install rustfmt & clippy components
|
|
run: rustup component add rustfmt clippy
|
|
- name: Install thumbv7em-none-eabihf target
|
|
run: rustup target add thumbv7em-none-eabihf
|
|
- name: Install Python interop dependencies
|
|
# The interop suites used to skip silently when python3/h5py were
|
|
# missing, so they never ran in CI. Install them and make a missing
|
|
# dependency a failure (CLAWHDF5_REQUIRE_INTEROP below).
|
|
run: |
|
|
apt-get update
|
|
# cmake builds libz-ng-sys for the opt-in `fast-deflate` (zlib-ng)
|
|
# steps in ci-test.sh; rust:latest does not ship it. The default
|
|
# build (pure-Rust zlib-rs) does not need it.
|
|
apt-get install -y --no-install-recommends python3 python3-venv cmake
|
|
python3 -m venv /opt/interop
|
|
/opt/interop/bin/pip install --no-cache-dir h5py numpy netCDF4 xarray
|
|
echo "/opt/interop/bin" >> "$GITHUB_PATH"
|
|
- name: Show interop library versions
|
|
run: /opt/interop/bin/python -c "import h5py, netCDF4; print('h5py', h5py.__version__, 'HDF5', h5py.version.hdf5_version, 'netCDF4', netCDF4.__version__)"
|
|
- name: Run CI script
|
|
env:
|
|
# Name the interpreter outright rather than relying on $GITHUB_PATH
|
|
# reaching the test processes: if `python3` resolved to the system
|
|
# one instead of the venv, every interop suite would skip.
|
|
# CLAWHDF5_REQUIRE_INTEROP turns that skip into a failure, so the
|
|
# two together mean the suites either run or the build goes red.
|
|
CLAWHDF5_PYTHON: /opt/interop/bin/python
|
|
CLAWHDF5_REQUIRE_INTEROP: "1"
|
|
run: bash scripts/ci-test.sh
|
|
|
|
test-arm64:
|
|
# The aarch64 kernels in clawhdf5-accel — NEON `dot_i8`, including the
|
|
# SDOT path, and the f32 NEON kernels — are cfg'd out on x86, so the job
|
|
# above never compiles, lints or tests them.
|
|
#
|
|
# `linux_arm64` is served by two runners that execute differently:
|
|
# vision-01 runs steps on the host (Rust already installed) and vision-02
|
|
# runs them in docker.gitea.com/runner-images. So the steps work in both:
|
|
# no `container:`, no JavaScript actions (they are fetched from GitHub,
|
|
# which not every runner reliably reaches), and an explicit `+stable`
|
|
# toolchain rather than whatever a host happens to default to.
|
|
runs-on: linux_arm64
|
|
env:
|
|
CARGO_NET_RETRY: "10"
|
|
CARGO_TERM_COLOR: always
|
|
steps:
|
|
- name: Check out
|
|
run: |
|
|
git init -q .
|
|
git remote add origin "${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}.git"
|
|
for i in 1 2 3; do git fetch -q --depth 1 origin "${GITHUB_SHA}" && break; sleep 5; done
|
|
git checkout -q FETCH_HEAD
|
|
- name: Rust stable
|
|
run: |
|
|
export PATH="$HOME/.cargo/bin:$PATH"
|
|
command -v rustup >/dev/null || curl -sSf --retry 5 https://sh.rustup.rs | sh -s -- -y --profile minimal --default-toolchain none
|
|
rustup toolchain install stable --profile minimal --component clippy
|
|
echo "$HOME/.cargo/bin" >> "$GITHUB_PATH"
|
|
- name: Confirm aarch64
|
|
run: |
|
|
test "$(uname -m)" = aarch64
|
|
if grep -q asimddp /proc/cpuinfo; then echo "dot-product extension present: SDOT kernel runs"; else echo "no dot-product extension: plain NEON kernel runs"; fi
|
|
- name: Clippy (aarch64 kernels)
|
|
run: cargo +stable clippy -p clawhdf5-accel --all-targets -- -D warnings
|
|
- name: Test
|
|
run: cargo +stable test -p clawhdf5-accel -p clawhdf5-ann -p clawhdf5-format
|