With the `remote` feature (`remote-https` for https://), ls, dump, stat
and diff take an http(s):// (or s3://, gs://, az:// with those
clawhdf5-remote features) URL wherever they take a file, and read it by
range requests through clawhdf5-remote's block cache. check validates
every byte, so it downloads a remote file whole and checks it as before.
Without the feature a URL is a clean error naming it.
The tools read the file through File::storage instead of as_bytes: object
headers, shared messages, attributes, v1 and v2 group links, dense
storage (fractal heaps and v2 B-trees), path resolution, chunk listings
and variable-length values go through the format crate's *_in functions,
and the fractal-heap block verifier reads each block through the storage
(a read failure of a remote file is reported as a problem, not as "past
the end of the file"). A local file's storage is its mapped bytes, so its
reads are still slices. stat's file size comes from the opened file, so
it is right for a URL.
Tests: tests/remote.rs serves fixtures (old and new formats, a paged
file, a metadata cache image, a multi-block fractal heap, compounds, v1
groups) with the clawhdf5-remote test server and requires every
subcommand's output and exit status for the URL to equal the local
file's, and diff of the two to be clean; 404s, non-HDF5 bodies and
https without its feature are clean errors. Local output is unchanged:
the old and new h5rs print the same for ls -r -v, dump, stat and check
--data on the 747 conformance and CVE corpus files (tank, 2026-09-26;
the dumps of h5diff_hyper1/2.h5 were too large for the comparison
script, their ls, stat and check agree), except cve-2025-2310.h5, whose
dump error messages differ between runs of the old binary too (which
failing chunk is reported first).
ci-test.sh lints h5rs with remote-https, runs the URL tests and checks
h5rs with remote for C.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
ObjectStoreStorage (feature `object-store`, pure Rust) reads one object
of any object_store store by ranged get_opts, pinned at open by a head
request: If-Match with its ETag (and the ETag and size of every response
compared), else its version or modification time. A change is
RemoteError::FileChanged. object_store is async and Storage is not, so
the storage owns a small multi-threaded tokio runtime (two workers) and
blocks the calling thread on it; the ranges of one read_ranges call are
fetched concurrently (up to 8). From inside another tokio runtime it
refuses with RemoteError::Usage instead of blocking a worker, and it
shuts its runtime down in the background on drop so dropping it in async
code does not panic.
open_object(store, path, options) opens a file through a block cache
(first block prefetched); open_url accepts s3://, gs:// and az:// with
the `s3`, `gcs` and `azure` features, configured from the environment by
object_store's from_env builders. Those pull object_store's cloud clients
and aws-lc-rs (C), so they are opt-in; without them the URL is a clean
UnsupportedScheme error naming the feature.
Tests against object_store's in-memory and local-file stores (no cloud):
every fixture's transcript equals File::open's, a multi-block object is
fetched in coalesced block runs, an object replaced while open is an
error, and a missing object or a read from inside a runtime is a clean
error. ci-test.sh lints all backends, runs these tests (with s3 for its
URL parsing test) and checks object-store for C in the no-C step.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Range-read milestone M3, first half: a new crate with the block cache the
design makes mandatory for remote files and an HTTP backend, so
open_url("http://...") gives a clawhdf5::File over File::open_storage.
BlockCache wraps any Storage: aligned blocks (1 MiB by default, the size
docs/design/range-reads.md section 2 measured), LRU with a byte budget,
the missing blocks of one read_at/read_ranges fetched with one backend
read_ranges call as runs of consecutive blocks (a one-block gap filled to
merge runs, each request at most 8 MiB), and reads that miss more than
half the budget not kept. Thread-safe without holding the lock across a
fetch: a block being fetched is in flight, a second reader waits for it
instead of fetching it again, and a failed fetch fails its waiters and is
not cached. A backend holding the file in memory passes through.
HttpStorage (ureq, no TLS by default; `https` adds rustls with ring):
opening is one ranged GET of the first block, whose Content-Range gives
the length (the cache keeps the bytes). The file is pinned by a strong
ETag (If-Match), else Last-Modified (If-Unmodified-Since), and its length,
checked on every response: a change is RemoteError::FileChanged, never
mixed data. A server that ignores Range is refused without reading the
body unless a full download is allowed. Connection errors, timeouts,
408/429/5xx and short bodies are retried with exponential backoff;
Accept-Encoding: identity, and an encoded body is refused. read_ranges
fetches its ranges in parallel.
Tests (a std-only HTTP/1.1 server in tests/common/server.rs, also the
range_server example): every fixture read over HTTP gives File::open's
transcript (CLAWHDF5_REMOTE_CORPUS adds the conformance corpus), with
request counts per file with and without the cache; an h5py-written file
against libhdf5's values; a multi-block file fetched in whole blocks, each
once; a server ignoring Range; a file replaced mid-read (ETag,
Last-Modified, length only); truncated bodies and 503s (retried, then an
error, never cached); a slow server with 8 concurrent readers (no block
fetched twice); bad URLs, 404, encoded bodies, non-HDF5 data. The cache
has unit tests for coalescing, splitting, LRU order, large reads,
failures and concurrent in-flight dedup.
ci-test.sh: clawhdf5-remote joins the no-C default-build check, and its
https feature is linted.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Every `u64 as usize` cast in clawhdf5-format (115 on wasm32) now goes
through addr::to_usize for values read from the file — addresses, lengths,
counts, dimensions: FormatError::Overflow where the value does not fit
instead of wrapping onto another part of the file on a 32-bit target — or
addr::saturating_usize for counts bounded by something in memory (codec
progress counters, writer sizes), which fail a bounds check or allocation
rather than wrap. A chunk whose offset does not fit lies outside the
dataset and is skipped; partial reads treat such an offset as out of the
buffers. On 64-bit targets nothing changes.
scripts/check-32bit-casts.sh (run by ci-test.sh) lints the wasm32 build
with clippy's cast_possible_truncation and fails on any u64 -> usize
finding; before this commit it listed 115.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The facade forwards `zfp` and adds it to `plugin-filters`.
tests/zfp_interop.rs: h5py + hdf5plugin (H5Z-ZFP 1.1.1, zfp 1.0.1)
write 2205 datasets over 16 modes (rate, precision, accuracy,
reversible, expert settings at their edges) x int32/int64/float/double
x 1-4-D shapes with partial edge chunks, partial blocks and unit chunk
dimensions x smooth/noisy/wide-range/zero/inf-NaN data; clawhdf5 must
read each byte for byte as h5py does (read back after closing the file:
h5py returns a chunk still in libhdf5's cache without decoding it). A
second test swaps a file's header words to what a big-endian writer
stores and checks the byte-swapped values match h5py's.
The left-out-filter test now expects ZFP only in builds without it; CI
lints `zfp` alone and runs the new test with the plugin-filter step.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
hdf5plugin's Blosc2 was a clear "not implemented" error. It stores each
HDF5 chunk as a Blosc2 contiguous frame; for chunks of 2+ dimensions the
frame holds a B2ND array whose data are cut into (padded) blocks stored
one after another.
filters_blosc2 (feature `blosc2`, in `plugin-filters`; the facade
forwards both) decodes, following c-blosc2's decoder and hdf5-blosc2's
blosc2_filter.c:
- the frame: the msgpack header's fixed fields, the metalayer index, the
offsets chunk (with the special zero/NaN/uninitialised offsets) and
chunk lookup;
- Blosc2 chunks: 16- and 32-byte headers, special chunks (zeros, NaN,
uninitialised, one repeated value), split and unsplit streams, zero and
run-length streams, and the filter pipeline run backwards (shuffle,
shuffle with a byte-group size, bit shuffle including the version-2 and
later handling of a partial group of 8, delta against the first block,
truncated precision);
- the codecs, shared with Blosc 1: BloscLZ, LZ4/LZ4HC, Zlib, Zstandard;
- B2ND arrays: blocks gathered into C order, padding dropped, several
chunks per array, and the array shape checked against the chunk shape
in cd_values as the HDF5 filter does.
Dictionaries, lazy chunks, variable-length blocks, user-defined codecs
and registered filters (e.g. bytedelta) are errors. Uninitialised chunks
read as zeros. No encoder.
Tests: h5py + hdf5plugin write every codec x filter (none, shuffle,
bitshuffle, delta) and levels 0-9 over the plugin-filter cases, then
i1..u8/f4/f8 in 1-D to 5-D chunks with partial edge chunks, datasets of
zeros, one value and NaN, and Fletcher32 before Blosc2 (plain frames for
n-D chunks); clawhdf5 reads each exactly as its unfiltered twin, and
truncated precision exactly as h5py reads it. Fixture frames from
python-blosc2 (tests/fixtures/blosc2/generate.py) cover what hdf5plugin
never writes: special chunks, delta over many blocks and odd type sizes,
odd bit-shuffle blocks, forced splitting, multi-chunk B2ND arrays with a
zero chunk, and the refused features. The decoder is fuzzed (random and
mutated frames and chunks: no panic, output within the limit).
Conformance: h5ex_d_blosc2.h5 now reads (576 of 697 ok, baseline 575).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
ci-test.sh gains a step that lints clawhdf5-py, builds its wheel with
maturin, unpacks it under target/ (the interpreter's environment is not
touched) and runs the pytest suite, which compares reads with h5py. It
skips without maturin/pytest, and fails instead under
CLAWHDF5_REQUIRE_INTEROP=1. The CI interop venv installs maturin and
pytest, so CI runs it.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
parallel_integration.rs declared `_sequential` and used `sequential`
under the parallel feature, which nothing in CI enabled for the facade.
ci-test.sh now lints and tests the facade with parallel on.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The clawhdf5-tools interop tests compare h5rs with h5ls, h5stat, h5dump and
h5diff, and CLAWHDF5_REQUIRE_INTEROP=1 turns a missing tool into a failure.
Also hold clawhdf5-tools to the no-C-in-the-default-build check.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The check matches any *-sys crate, and clawhdf5-wasm pulls in js-sys,
wasm-bindgen's bindings to JavaScript, which compiles no C. Exempt it
by name so the check keeps catching real C for the wasm crate.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
scripts/ci-test.sh: the format feature matrix (clippy and tests) adds
plugin-filters; bitshuffle, bzip2 and blosc are each linted alone (blosc
and bitshuffle share code); the facade is linted with plugin-filters; and
the interop section runs tests/plugin_filters_interop.rs with it, against
h5py + hdf5plugin (CI's venv already installs hdf5plugin).
conformance/probe enables plugin-filters. Sweep (tank, 2026-09-26,
conformance/run.sh --no-fetch against the cached corpus): 573 of 697 ok
(baseline 569), no regressions; newly ok: h5ex_d_blosc.h5,
h5ex_d_bshuf.h5, h5ex_d_bzip2.h5, h5ex_d_lzf.h5. h5ex_d_blosc2.h5 and
h5ex_d_zfp.h5 remain UnsupportedFilter.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Drop a file (or pass ?file=<url>&path=<object>), browse the tree lazily,
see a dataset's type, shape, max shape and attributes, and page through
its values as 50x12 hyperslab windows (leading dims of 3-D+ data held
at chosen indices). build.sh produces pkg/ (not committed) with
wasm-bindgen --target web and checks the CLI matches the crate version.
test/run.sh builds it and runs test.mjs under Node against the h5py/
netCDF4 fixture (250 checks: every dataset whole and as a strided
hyperslab, listings, attributes, error paths, the page's DOM-free
helpers), then browser.sh renders the page in headless Chromium for
eight objects and checks the DOM. The fixture gains LZ4 (read) and Zstd
(refused: links C) datasets and a compound attribute (value null plus
its type). ci-test.sh runs it when node and wasm-bindgen exist; the CI
container has neither, so CI relies on the native h5py_interop test.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
open(bytes) -> H5File with kind/list/info/attrs/attrErrors/read/
readHyperslab. Numeric data comes back in the typed array of the
stored width (Int16Array for i16, BigInt64Array for i64, Float32Array
for f32/f16, ...), strings and enum names as string arrays, array
datatypes flattened with their dims appended to the shape. Compound,
reference, opaque and VL-sequence datasets are refused with an error
naming the type; nothing is returned as reinterpreted bytes.
The logic is in a plain-Rust core module, tested natively: unit tests,
and h5py_interop, which compares every dataset, hyperslab, listing and
attribute of an h5py- and a netCDF4-written file with what libhdf5
reads back (generator shared with the Node test of the built package).
No mmap, no threads; lz4 is on, zstd/szip (C) are not. A
wasm-release profile (opt-level s, LTO) serves the browser build.
ci-test.sh lints the crate for wasm32 and checks it builds no C.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The facade already builds for the browser target without mmap (and
with it: memmap2 compiles there and File::open just fails, as std::fs
does). Nothing needed gating; keep it that way with a ci-test.sh step,
and install the target in the CI container.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The interop step built writer_h5py_tests without the lz4/zstd features,
so the hdf5plugin round-trips added with the registered LZ4 framing and
the Zstd content-size fix never compiled in CI, and CI never installed
hdf5plugin.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
rust-version = "1.92" in [workspace.package], inherited by every crate.
1.92 is the floor: wgpu (clawhdf5-gpu) requires it, and the whole
workspace, Python bindings included, checks cleanly on it. ci-test.sh
reads the version from Cargo.toml and checks the workspace on exactly
that toolchain, so the manifests and the README badge cannot drift from
what actually builds. The badge said 1.75, below edition 2024's own
floor.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The core crates (clawhdf5, -agent, -format, -io, -filters, -ann, -accel,
-netcdf4, -cli) now build no C by default: deflate defaults to zlib-rs,
a pure-Rust port of zlib-ng, and zlib-ng becomes the opt-in
`fast-deflate`, which overrides zlib-rs wherever it is enabled. A default
build no longer needs cmake or a C compiler.
Measured on tank, both builds run alternately, three rounds, medians:
zlib-rs is within 6% of zlib-ng on every HDF5 read and write (512x512
deflate-6 chunked write 1.458 vs 1.484 ms; 64 MB compressed read 64.4
vs 65.2 ms), and compressed output is byte-identical. Details in
BENCHMARKS.md, "Deflate backend".
Getting there took two fixes the first measurement exposed:
- zlib-rs needs `std` to detect SIMD at runtime. flate2 enables it via
its default `runtime_detection`, which `default-features = false` had
switched off, leaving zlib-rs 3.5x slower on inflate. The `zlib-rs`
features now enable it.
- Both deflate paths streamed through flate2's 32 KiB read/write
wrappers. They now hand the codec the whole chunk in one call, into a
buffer sized up front (~5% on chunked writes). This also fixes a
silent short read: the streaming reader returned a truncated stream's
bytes without an error; a truncated chunk is now DecompressionError.
In clawhdf5-filters, output longer than the stated size is now an
error rather than silently cut off.
CI: ci-test.sh lints and tests the zlib-ng path, and fails if a
C-building crate (*-sys, cc, cmake) enters a core crate's default
dependency tree. The arm64 job no longer installs cmake.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Every Python interop suite had stopped running on this machine: the h5py
writer round-trips, the facade suite, netCDF4 and the reference files.
`python3` is 3.14, nothing on the box has h5py, and PEP 668 refuses to
install it into a system interpreter at all — so the availability probes
all returned false and each suite skipped without failing.
A silent skip here is exactly how the v5 compound-datatype bug reached a
release, so the probes now read `CLAWHDF5_PYTHON` and `ci-test.sh` picks
up `.venv/bin/python` on its own. The detection sits at the top of the
script rather than beside the interop step, because the non-ignored
suites run in the earlier `cargo test` step and would otherwise still
miss it. `CLAWHDF5_REQUIRE_INTEROP=1` continues to turn a skip into a
failure.
Verified against a venv with h5py 3.16 / HDF5 2.0.0: 94 interop tests
across the four suites, all passing.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
- tests/wal_properties.rs — deterministic generator, reproducible by seed:
everything appended is read back intact (300 cases), and after ANY damage
to the file (bit flips, truncation, inserted/deleted bytes, duplicated or
rotated regions, overwritten ranges; 1500 cases) reading never panics and
yields an exact prefix of what was written — the guarantee the chained CRC
exists to give. Opening for append then repairs the tail and a new entry
lands right behind the surviving prefix.
- tests/crash_recovery.rs — builds the on-disk images a process crash can
leave and reopens each against a model of what was acknowledged: an image
after every operation (random saves, in-place updates, checkpoints, small
wal_max_entries), the checkpoint window (new .h5 + not-yet-truncated WAL)
over several rounds, and the WAL torn at every byte length, which must
recover the checkpoint plus a prefix of the operations logged since.
- fuzz/fuzz_wal_replay — arbitrary bytes as a WAL: read and open-for-append
must not panic, and open() must not change what is replayable. Based on the
target from the clawmates mission branch (4aee2fa), with the repair
property added. The CI fuzz step now covers both fuzz crates.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
- clippy --all-targets plus a clawhdf5-format feature matrix (parallel, lz4,
zstd, pcodec, fast-checksum); fix the accumulated lint backlog in test,
bench and feature-gated code (no behaviour changes).
- Install python3 + h5py/numpy/netCDF4/xarray in the CI container and set
CLAWHDF5_REQUIRE_INTEROP=1, which makes a missing interop dependency a test
failure. Every h5py/netCDF4 interop test used to skip silently in CI. Run
the #[ignore]d writer_h5py_tests suite explicitly.
- cargo bench --no-run so benches can't rot; fix bench.rs and memory_bench.rs,
which no longer compiled against the current strategy/consolidation APIs.
- Optional fuzz smoke run via CLAWHDF5_FUZZ_SECONDS.
- CHANGELOG and docs/known-issues.md updated.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
- Add .gitea/workflows/ci.yml running scripts/ci-test.sh (fmt, clippy,
test, no_std check) on push/PR to main.
- Fix stale rustyhdf5-py/rustyhdf5-format package names in
ci-test.sh/check-nostd.sh, which had been silently no-op'ing those
checks (cargo warns but doesn't fail on an unknown --exclude/-p
target).
- With those checks actually running, fix the real issues they surface:
- clippy: useless_conversion in chunked_write.rs, byte_char_slices in
global_heap.rs/object_header.rs.
- cargo fmt: apply formatting across the workspace (whitespace only).
- no_std (thumbv7em-none-eabihf) build errors in clawhdf5-format:
core::sync::atomic::AtomicU64 doesn't exist on that target (no
native 64-bit atomics) — switch profiling.rs's counters to
portable-atomic, which falls back to a CAS-based emulation there
and is a no-op wrapper elsewhere. Add missing alloc imports for
Box (filters.rs), Vec (filters_szip.rs), and format! (dict_encoding.rs)
on no_std paths. Replace f64::powi (std/libm-only) with a small
local exponentiation-by-squaring helper in the scale-offset filter.
