HDF5 stores every chunk at the full chunk size (edge chunks are padded
before filtering, and with "don't filter partial edge chunks" they are
stored raw at full size), so a filter pipeline that decodes to fewer bytes
means a corrupt chunk. Every chunk reader padded it with zeros and
returned it as data. libhdf5 returns the rest uninitialised, or fails when
the filter checks (Blosc with nbytes = 0).
New filters::decompress_chunk_exact decodes and then requires exactly the
chunk size, with the chunk's coordinates in the error
(ChunkedReadError "chunk at [16] decoded to 16 bytes, expected 32"). It
replaces decompress_chunk_masked at every chunk read path: the full read
(sequential and lane-partitioned), the cached read, the sweep read, the
planned-selection read, parallel_read's three decoders and partial_read's
box read. decompress_chunk_masked is unchanged (fractal-heap huge objects
already checked their own size). Blosc also rejects a frame declaring no
data where the chunk size is known.
Tests, each failing with the check disabled: filters and parallel_read
unit tests; h5py_short_decoded_chunk_is_an_error (gzip chunks rewritten
short with write_direct_chunk: 1-D, a 2-D edge chunk, and 40 chunks with
shuffle, read through File full/cached/selection reads, a selection that
avoids the chunk still reads, MmapFile and LazyFile, with and without the
parallel feature); plugin_filters_interop short_decoding_chunks_are_errors
(Blosc nbytes=0 and short, LZF and bzip2 short; the Blosc nbytes=0 case
read as 16 zeros before). The existing don't-filter-partial-edge-chunks
tests still pass. Conformance (tank, 2026-09-26): 573 of 697 ok, and no
file changed class, reader result or first issue against the pre-fix run.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Audited LZF, bitshuffle, bzip2 and Blosc/BloscLZ for arithmetic on
header fields and unchecked slicing. The only live bug was the Blosc
frame-size underflow fixed in the previous commit; bzip2's output-growth
step now uses a saturating subtraction as well (the allocator may hand
back more capacity than asked for).
src/test_fuzz.rs (tests only) feeds each decoder random bytes, truncated
seeds and one-to-four-edit mutations of valid frames, biased towards
edge-case u32 values in size and offset fields, and asserts no panic and
no output over the limit (tests build with overflow checks and debug
assertions). Per decoder: LZF, bzip2, bitshuffle in all six mode/block
settings plus hostile cd_values, Blosc across four codecs, three shuffles,
stored frames and a hand-built BloscLZ frame, and BloscLZ streams alone.
With the previous commit's check removed, fuzzed_frames_never_panic panics
at the same subtraction. A 100x-iteration soak (different seed) found no
other panic.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A hostile chunk whose header gave a compressed size below 16 bytes, not
stored raw, made the block-table check subtract past zero: a panic in any
build with overflow checks (cargo test, maturin develop, debug CLI). The
frame size is now checked against the header size, and the stream-length
read no longer adds to an untrusted offset.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
CHANGELOG (Unreleased): LZF, bitshuffle, bzip2 and Blosc read and write
in pure Rust, their features, the ChunkOptions::plugin field (breaking for
struct-literal construction), the filter registry, the named
UnsupportedFilter message, and Blosc2/ZFP still unimplemented.
README: the clawhdf5-format feature table gains lzf (default),
bitshuffle, bzip2, blosc and plugin-filters, with how to write them and
what is not implemented; no speed claims. docs/known-issues.md: the audit's
filter gap is marked fixed 2026-09-26 for LZF/bitshuffle/bzip2/Blosc,
Blosc2 and ZFP still open. CLAUDE.md: the clawhdf5-filters row no longer
says "No Blosc". clawhdf5-format's crate docs list the new features.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
scripts/ci-test.sh: the format feature matrix (clippy and tests) adds
plugin-filters; bitshuffle, bzip2 and blosc are each linted alone (blosc
and bitshuffle share code); the facade is linted with plugin-filters; and
the interop section runs tests/plugin_filters_interop.rs with it, against
h5py + hdf5plugin (CI's venv already installs hdf5plugin).
conformance/probe enables plugin-filters. Sweep (tank, 2026-09-26,
conformance/run.sh --no-fetch against the cached corpus): 573 of 697 ok
(baseline 569), no regressions; newly ok: h5ex_d_blosc.h5,
h5ex_d_bshuf.h5, h5ex_d_bzip2.h5, h5ex_d_lzf.h5. h5ex_d_blosc2.h5 and
h5ex_d_zfp.h5 remain UnsupportedFilter.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Blosc2 (32026) is out of reach for now: hdf5plugin's Blosc2 filter stores
each HDF5 chunk as a Blosc2 super-chunk frame (msgpack header, a compressed
chunk-offset index, trailer metalayers) and, for 2-D and larger chunks, as
a B2ND array whose n-D blocks have to be reassembled - on top of the Blosc2
chunk format itself (extended header, filter pipeline, special-value
chunks). ZFP (32013) is out of scope. Both keep failing with
UnsupportedFilter, and the message now says what the ID is:
"unsupported filter: 32026 (Blosc2, not implemented by clawhdf5)", or,
for a filter this build left out, "... (Blosc; this build lacks the
`blosc` feature)". filter_registry::known_filter exposes the table.
tests/plugin_filters_interop.rs: hdf5plugin's Blosc2 and ZFP datasets
read as an error naming the filter, never as data.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
hdf5plugin's Blosc (hdf5-blosc) failed with UnsupportedFilter(32001). The
new `blosc` feature decodes the Blosc 1 frame c-blosc 1.x writes: the
16-byte header, raw ("memcpyed") frames, the block table, blocks split
into one stream per byte plane (and the "do not split" flag), streams
stored raw, the byte shuffle and bit shuffle (whole 8-element groups, the
rest copied) - and every codec hdf5plugin offers: BloscLZ (implemented
here from c-blosc 1.21's blosclz_decompress, including its rejection of
malformed and truncated streams), LZ4/LZ4HC (lz4_flex), Snappy (snap),
Zlib (flate2) and Zstandard (ruzstd). Every stream must decode to exactly
its size and the frame to at most the chunk size; a frame of another
format version (Blosc 2) is a clear error.
It also encodes (DatasetBuilder::with_blosc(codec, level, shuffle)):
LZ4, Snappy, Zlib or Zstandard, with c-blosc's split rule, raw streams
where compression does not pay, and a stored frame for level 0 or
incompressible data. It cannot write BloscLZ (asking for it is an
error). `plugin-filters` enables LZF, bitshuffle, bzip2 and Blosc.
Interop: hdf5plugin writes all six codecs x {no, byte, bit} shuffle at
levels 5/9/1, plus level 0, over the 12-case matrix, read byte for byte;
our four codecs x four shuffle/level settings read back through
hdf5plugin. Both fail with the decoder removed. `cargo tree` with
`plugin-filters` has no -sys crate other than libbz2-rs-sys (pure Rust),
no cc and no cmake.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
hdf5plugin's BZip2 failed with UnsupportedFilter(307). The new `bzip2`
feature decodes the single bzip2 stream H5Zbzip2.c stores, bounded by the
chunk size (a truncated stream is an error, not short data), and encodes
at block size cd_values[0] (DatasetBuilder::with_bzip2(level)). It uses the
bzip2 crate's default backend, libbz2-rs-sys, a pure-Rust port of
libbzip2: `cargo tree` shows no cc/cmake, and nothing is compiled from C.
Interop: hdf5plugin writes block sizes 9, 1 and 5+shuffle over the
12-case matrix, read byte for byte; ours at 9 (shuffled) and 1 (not)
reads back through hdf5plugin. Both fail with the decoder removed.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
hdf5plugin's Bitshuffle failed with UnsupportedFilter(32008). The new
`bitshuffle` feature (pure Rust: lz4_flex, and ruzstd for Zstandard — the
`zstd` feature's libzstd is not needed) decodes all three modes of
bshuf_h5filter.c — transpose only, LZ4 and Zstandard blocks behind the
12-byte header — including the default and explicit block sizes, the
shorter last block rounded down to a multiple of 8 elements, and the
untransposed trailing elements. Sizes read from the chunk are bounded by
the chunk size.
It also encodes: DatasetBuilder::with_bitshuffle(BitshuffleCompression)
or PluginFilter::Bitshuffle { block_size, compression } writes the filter
with hdf5plugin's cd_values and no automatic byte shuffle. ruzstd has one
compression level (about zstd's 1); the requested level is recorded.
The bit transpose is checked bit for bit against a one-bit-at-a-time model
(which matched hdf5plugin's output) and is shared with blosc next.
Interop: hdf5plugin writes none/LZ4/Zstandard at default and explicit
block sizes and levels over the 12-case matrix, read byte for byte; our
three modes at two block sizes read back through hdf5plugin. Both fail
with the decoder removed.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
h5py's built-in compression="lzf" failed with UnsupportedFilter(32000). The
new `lzf` feature (no dependencies, on by default in clawhdf5-format and
the facade) decodes the raw liblzf stream h5py's filter stores, bounded by
the chunk size, and encodes it: DatasetBuilder::with_lzf() (or
with_plugin_filter(PluginFilter::Lzf)) writes the filter with h5py's
cd_values (filter version 4, liblzf 0x0105, chunk size in bytes), flagged
optional as h5py does. ChunkOptions gains a `plugin` field for the plugin
filters; build_pipeline_for_chunk passes the chunk size to filters that
record it.
tests/plugin_filters_interop.rs: h5py writes LZF (alone, with shuffle, with
shuffle+fletcher32) over 12 dtype/shape/chunk/data cases with partial edge
chunks and incompressible data, and every dataset reads byte for byte equal
to its unfiltered twin; our LZF output (1-D and 2-D, edge chunks, with and
without shuffle) reads back in h5py. Both fail with the decoder removed.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
decompress_chunk_masked and compress_chunk matched on the filter ID. They
now look the ID up in filter_registry: a static table of the built-in
filters compiled into this build (a filter whose cargo feature is off is
simply absent), then the codecs an application registered at run time with
register_filter (a FilterCodec, or a plain decoding closure). Registered
codecs cannot shadow a built-in one, and their output is held to the same
per-stage bound as the built-in decoders. An ID in neither tier still fails
with UnsupportedFilter(id).
The "feature off" stub functions that returned UnsupportedFilter are gone:
the table leaves those filters out instead.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Regenerated on tank at 10d1029: ok 467 -> 569, our-error 123 -> 14,
mismatch 15 -> 22 (six user-defined-link files moved from our-error to a
listing difference), no panics, hangs, crashes or OOM. Baseline raised.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Superblock::parse now refuses a user-block offset, and the raw read path
no longer guesses a VDS fill value. The probe looks at the file from the
superblock on and reads virtual datasets with vds::read_virtual_dataset,
the dataset's fill value and its source-derived extent.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The test compared h5py against its own expected table, so it passed with
the fix reverted. Found by the adversarial review.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Group::attrs now leaves out an attribute it cannot decode. The agent read
its settings through it, so a store whose float16 (or compression,
quantized_index, WAL mark, signature...) attribute could not be decoded
opened with the default in its place, and no error. /meta is now read
with attrs_with_errors and any unreadable attribute is a Schema error,
as it was before attrs became tolerant. Found by the adversarial review.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Their elements are global-heap IDs and object addresses in the source
file. The VDS reader copied them raw, so anything decoding them against
the virtual dataset's file got another object's data with no error.
Same-file sources are unaffected. Found by the adversarial review.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Unlimited VDS mappings were refused, and printf-style source names
("f-%b.h5") were not expanded, so those regions read as fill (read-matrix
case 0470: 29 of 30 values wrong). All 7 virtual datasets in the libhdf5
test set use such mappings.
Implement H5Dvirtual.c's semantics in the vds module:
- %b is the block number, %% a literal %, other specifiers are an error;
block j of the virtual selection comes from the source named with j,
probing from 0 to the first missing source (printf gap 0);
- unlimited source/virtual selections are clipped to what the source's
current extent fills (H5S_hyper_get_clip_extent_match, partial last
block included);
- the extent is recomputed as H5Dget_space does (view "last available":
the largest clip, never below what limited mappings need), exposed as
vds::virtual_dataset_extent and used by Dataset::shape();
- a source in the other byte order is byte-swapped; other conversions stay
an error.
Tests: vds_interop::vds_printf_source_names,
vds_unlimited_mappings_follow_source_extents (h5py low-level API, earliest
and latest format) and vds_libhdf5_test_files (vds-eiger, 4_vds and
vds-percival-unlim-maxmin from HDF5's tools/test/testfiles/vds, committed
as fixtures) all compare shape and values with h5py; unit tests for the
clip arithmetic, name parsing and mapping rules.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A message shared through the file's SOHM heap (H5Pset_shared_mesg_index)
is referenced by heap ID, which needs the SOHM table from the superblock
extension. Only message_data_with_sohm (used for fill values) loaded it;
resolve_shared_message passed no table, so a SOHM-shared datatype,
dataspace, filter pipeline or attribute failed with "invalid shared
message version: 2" and the dataset or attribute could not be read.
resolve_shared_message now loads the table when the reference carries a
heap ID.
Found while making attrs() tolerant: SOHM attributes turned from an
error into missing keys in the audit read matrix. With this fix all 36
SOHM cases there match h5py (datasets, fill values and attributes, every
shareable message type, libver earliest and latest).
Regression test: sohm_shared_messages_resolve (h5py writes files sharing
each message type on its own and all of them; values and attributes
checked).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
libhdf5 walks a local heap's free list when it loads the heap's data and
refuses the heap ("bad heap free list") when a free block starts or ends
outside the data segment, or links to offset 0. We never looked at the
free list, so a damaged old-style group listed names read from the broken
heap: once the user block of cve-2021-36977.h5 was applied, its root
listed eight garbage names where libhdf5 fails.
LocalHeap::validate_free_list (new) mirrors H5HL__fl_deserialize, with a
cycle bound, and accepts H5HL_FREE_NULL (1) or an all-ones head as the
end of the list. Like libhdf5 it runs when the first name is needed, not
on parse, so an empty group with a damaged heap still lists as empty
(cve-2018-13871.h5, cve-2024-29166.h5, gh-4431-poc-03.h5 keep matching
h5py).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A file may start with a user block (h5py userblock_size, h5jam), putting
the superblock at 512, 1024, ...; every address in the file is then
relative to the superblock. The signature search found it, but every
reader passed the whole file to the parsers, so addresses landed
userblock bytes early and the root group failed with
InvalidObjectHeaderVersion (twithub.h5, twithub513.h5,
h5clear_fsm_persist_user_*.h5).
Readers now view the file from the superblock on, taking the signature's
position as the base address as libhdf5 does: File (mmap, buffered,
from_bytes), MmapFile, LazyFile, AsyncHDF5File, the VOL and MPI VOL
readers, the HNSW loader and external VDS source files. File, MmapFile
and LazyFile gain user_block_size(). The new signature::split_user_block
returns the two parts, and Superblock::parse refuses a non-zero offset
(UserBlockNotStripped) so a format-level caller cannot silently apply
superblock-relative addresses to the whole file.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A version-1 shared message (HDF5 1.6) embeds the target as a symbol-table
entry: after six reserved bytes comes a length-sized local-heap offset,
then the object header address. We read the heap offset as the address,
so datasets using a committed datatype in 1.6-era files (tcompound.h5,
tcompound2.h5) failed with InvalidObjectHeaderVersion. parse_shared_ref
now takes length_size and skips the offset, as libhdf5 does.
Resolving a reference also no longer falls back to the first message of
any type in the target header: a missing target message is
SharedMessageTargetMissing instead of garbage.
Fixture: tcompound.h5 from libhdf5's tools/test/testfiles (8 KiB).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
HDF5 1.6 encoded a compound member that is a fixed-size array through
legacy per-member fields (dimensionality, permutation, four dimension
sizes) that the v1 decoder skipped, so a [4] i32 member read as one i32
with the wrong size. Build the array type from those fields as libhdf5
does (ignoring the permutation) and refuse more than four dimensions.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Runs conformance/run.sh in rust:latest on a schedule and on demand, with its
own venv (pinned h5py/numpy/hdf5plugin/netCDF4) and hdf5-tools. Fails on any
panic, hang, crash or OOM in clawhdf5 and on a drop against
conformance/baseline.json; prints CONFORMANCE.md into the job log and
uploads nothing. Plain git checkout, no JavaScript actions.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
conformance/run.sh fetches eight public HDF5 corpora pinned by commit
(conformance/corpus.txt) into a gitignored cache, reads every file with
clawhdf5 (conformance/probe, a crate outside the workspace) and with
h5py/libhdf5 (ref.py), and the CVE files with h5dump, each under a timeout
and an address-space limit; compare.py classifies the files, report.py
writes CONFORMANCE.md and check.py gates on panics/hangs/crashes/OOM and on
regressions against conformance/baseline.json. ~25 s once cached.
Changes from the ad-hoc audit harness:
- the probe compares non-IEEE-layout floats (N-Bit) and integers with a bit
offset or reduced precision as the values libhdf5 converts them to, not
raw file bytes: 8 files that showed as mismatches now read identically;
- ref.py exits without tearing down h5py objects: libhdf5 2.0 aborts while
freeing them for two files about half the time, which flipped them
between ok and h5py-cannot-read from run to run;
- the file list is defined (list_files.py): netCDF classic files are left
out, 11 HDF5 files the ad-hoc sweep missed are in.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Elements of a virtual dataset that no mapping supplies (unmapped regions,
a missing source file, a missing source dataset) read as 0 instead of the
fill value libhdf5 returns — silent wrong data for any VDS created with a
non-zero fillvalue (read-matrix cases 0471/0472: -1 and 7 read as 0). A
missing source dataset was an error; libhdf5 reads it as fill.
Move VDS assembly into a new vds module following H5Dvirtual.c:
vds::read_virtual_dataset takes the dataset's fill value and a
VdsFileResolver that can refuse a name, and reports how many elements were
unmapped. Sources are read with their own fill value, and a source whose
datatype differs from the virtual dataset's is an error (libhdf5 converts).
File passes the dataset's fill value, resolves source names against the
virtual file's directory, and refuses names that leave it with an error
instead of reading them as fill. read_selection on a VDS goes through the
same fill-aware path.
The raw-read API (read_raw_data_full*) has no fill value, so it now errors
for a VDS with unmapped elements instead of guessing zeros.
Tests: vds_interop::vds_unmapped_regions_read_as_fill_value (external,
same-file, missing file/dataset, sparse source with its own fill, int
fill; earliest and latest format) and
vds_source_outside_directory_is_an_error_not_fill, both against h5py;
integration_test::v4_virtual_dataset_raw_api_refuses_to_guess_the_fill_value.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
attrs() read every attribute of an object through extract_attributes_full,
so one attribute it could not read (a corrupt or unsupported attribute
message, or a heap object it could not locate) failed all of them — the
same shape as the huge-object bug, where one 8 KiB attribute hid every
attribute on a NetCDF file's root group.
- clawhdf5-format: new attribute::extract_attributes_tolerant returns the
attributes it could read plus one error per attribute it could not.
Errors in the attribute index itself (Attribute Info message, dense
heap header, B-tree) still fail, since then it is unknown which
attributes exist. extract_attributes_full is unchanged (strict); both
share one implementation.
- clawhdf5: attrs() on Group/Dataset, MmapGroup/MmapDataset and
LazyGroup/LazyDataset leaves an unreadable attribute out (documented),
and the new attrs_with_errors() returns the map with the per-attribute
errors. A value is either returned complete or not at all.
Regression test: one_unreadable_attribute_does_not_hide_the_others (h5py
writes 11 dense attributes; one message's version byte is corrupted;
before: attrs() failed with InvalidAttributeVersion(127), after: the 10
others come back with their values and one error is reported).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Group::datasets()/groups() (and the Mmap/Lazy handles) listed only hard
links, so a soft link to a dataset or group was missing, and dataset(name)
/ group(name) on a group handle could not open one. In old-style (symbol
table) groups a soft link's entry has no object header address, and the
listing failed outright trying to parse one.
The three facade handles each had their own copy of the child-listing
code; they now share group_v2::resolve_group_children, which returns hard
links plus soft links resolved to their targets (relative targets from
the group holding the link, via the new resolve_path_from). A dangling or
cyclic soft link, an external link and a user-defined link are left out —
h5py lists their names but cannot open them. Any other error met while
resolving is returned, not hidden.
Path resolution now walks a relative soft link's target from the group
holding it instead of rebuilding the path from the root (same result,
one less re-walk), and ignores "." components.
Regression test: soft_links_are_listed_as_their_targets (h5py writes
absolute, relative, group, dangling, cyclic and external links with
libver latest and earliest; listings compared with h5py for File,
MmapFile and LazyFile).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
With a 2.0 low version bound, libhdf5 stores the VDS mapping list as heap
block version 1: every entry starts with a flags byte (0x04 same file, no
file name; 0x01/0x02 file/dataset name shared with an earlier entry, whose
index is stored in place of the name). The parser treated only a leading
0x04 byte as special, so a 0x00 flags byte read as an empty (same-file)
name and shared names were read as garbage.
Decode it as H5D__virtual_load_layout does, refusing unknown flags,
forward references and block versions above 1.
Test: vds_interop::vds_mapping_block_version1_shared_names (h5py
libver=("v200","v200") with repeated long names; failed before with
"unknown dataspace selection type") plus the exact heap block as a unit
test.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
libhdf5 serializes a VDS hyperslab as version 1 (irregular, 4-byte block
corners) for the default format bounds, and as version 2 (regular, 8-byte)
for unlimited selections in the 1.10 format. Only version 3 was accepted,
so every h5py VDS written with default libver failed with "only version-3
hyperslab selections are supported" (5 libhdf5 test files in the sweep).
Decode all three versions following H5S__hyper_deserialize, including
irregular hyperslabs (a union of blocks, enumerated in row-major order as
libhdf5 iterates them) and the all-ones "unlimited" count/block marker.
SerializedSelection exposes the raw form for unlimited-mapping support.
Test: vds_interop::vds_version1_irregular_hyperslab_selections compares
default-libver h5py VDS reads (contiguous, strided and 2-D block mappings)
with libhdf5's values.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Link types 65-255 are user-defined: their target is only meaningful to
the application that registered the link class. LinkMessage::parse
rejects them with InvalidLinkType, and group traversal propagated that,
so one such link made the whole group unlistable and every path through
it unresolvable (libhdf5's tall.h5 and tudlink.h5, class 187).
Group traversal (compact and dense) now leaves user-defined links out,
the way h5py leaves out links it cannot open; reserved types (2-63) are
still an error.
Regression test: user_defined_links_do_not_break_the_listing, on
libhdf5's own tools/test/testfiles tall.h5 and tudlink.h5 (BSD-style
HDF5 licence, 10 KB and 1 KB), committed as fixtures.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A heap ID's type is in bits 4-5 of its first byte (H5HF_ID_TYPE_MASK
0x30); bits 6-7 are the ID version. The reader took the type from bits
6-7, so every huge object ID (0x10) was decoded as a managed one and
failed — and since dense attributes are read all at once, one attribute
over the heap's 4 KiB managed limit made every attribute on its object
unreadable (netcdf4-python's issue671.nc / issue672.nc).
- Huge objects (type 1): located directly from the ID when address and
length fit in it, otherwise through the huge-object v2 B-tree (record
types 1 and 2); filtered huge objects are decoded with the heap's
pipeline and their filter mask.
- Tiny objects (type 2): read from the ID itself.
- Filtered heaps: the header's pipeline is parsed (it was skipped short,
so the header checksum was read from the wrong place), indirect-block
entries for direct blocks carry their filtered size and mask, and
direct blocks are decoded before objects are read from them.
- An unknown ID version is an error.
FractalHeapHeader gains huge_btree_address, filter_pipeline,
root_direct_block_filtered_size, root_direct_block_filter_mask,
offset_size and length_size; read_managed_object now accepts any ID type.
Regression tests (h5py-written, compared with h5py):
dense_attribute_stored_as_a_huge_heap_object, dense_group_with_a_huge_link,
dense_group_with_a_filtered_link_heap; unit tests
tiny_object_is_read_from_the_id, huge_object_with_a_direct_id,
unknown_heap_id_version_is_refused.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A version-1 shared message reference is version, type, six reserved bytes
and then an old-style symbol table entry: link-name offset (length size),
object header address, cache type, reserved, scratch. We read the address
straight after the reserved bytes, i.e. the link-name offset, and the
committed datatype lookup failed with InvalidObjectHeaderVersion (the bytes
checked in tcompound.h5: name offset 0x10, then 0x590 = /type1). Datasets
of 1.4/1.6-era files that use a committed datatype were unreadable.
Skip the name offset. parse_shared_ref has no length size, so add
parse_shared_ref_sized and use it in every internal caller;
parse_shared_ref keeps its signature and assumes length size == offset
size. The old parse_v1_ref unit test encoded the wrong layout and now uses
the real bytes.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Compound datatype version 1 carries, per member, a dimensionality and four
dimension sizes (HDF5 before 1.4 had no array class). The parser skipped
those 28 bytes, so a member such as `f: f32[4]` came back as a single f32
at the member's offset: the compound's size was right but its members were
wrong. libhdf5 wraps such a member in an array type of the first
`dimensionality` sizes and ignores the permutation; do the same, and
reject a dimensionality above 4 as libhdf5 does.
Only files old enough to also use layout message v1 have these, so this
became reachable with the previous commit (tarrold.h5, tcompound.h5).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
HDF5 1.4/1.6-era files store the layout as version 1 or 2: version,
dimensionality, class, 5 reserved bytes, an address (contiguous and chunked
only), dimensionality 32-bit sizes (with the trailing element-size
dimension) and, for compact storage, a 32-bit size and the raw data. They
failed with InvalidLayoutVersion — 84 of the 686 files in the audit sweep,
205 datasets.
Map them onto the existing variants: chunked uses the same version-1
B-tree chunk index as version 3 and is reported as version 3, so every
chunked read path (filters, selections, caches) applies unchanged.
Contiguous size is the product of the stored dimensions, which is what
libhdf5 computes from the dataspace; a disagreement fails the reader's size
check instead of returning wrong data.
Fixtures are HDF5's own deflate.h5 (v1, chunked + deflate) and
h5ex_g_iterate.h5 (v2, contiguous, one unallocated dataset); the new
interop test compares every dataset byte for byte against h5py.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
An internal node's child pointer is an address, the child's record count
and (below the first internal level) the child subtree's total record
count. libhdf5 (H5B2__hdr_init) encodes the record count in the width of
a leaf's maximum and the subtree total in the width of cum_max_nrec for
that depth, computed level by level from the node size. The reader
guessed 2 * leaf_max and leaf_max^depth, which agree at depth 2 but not
at depth 3: a 24 000-link group's name index has depth 3, its root's
pointers were read 3 bytes wide instead of 2, and listing failed with a
garbage heap offset.
Regression tests: dense_group_with_a_three_level_name_index (h5py writes
24 000 links; listing compared with h5py) and
subtree_capacity_matches_libhdf5.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A child indirect block in row r of a fractal heap's doubling table spans
that row's block size of heap space, so it has
log2(size) - log2(start_block_size * width) + 1 rows (libhdf5's
H5HF__dtable_size_to_rows). The reader used row - first_indirect_row + 1,
which undercounts, so every object stored past the root block's direct
rows (512 KiB with libhdf5's defaults) was unreachable: dense groups with
a few thousand long link names, or ~20 000 short ones, could not be listed.
Regression test: dense_group_whose_heap_outgrows_the_root_direct_rows
(h5py writes 2 500 links with 248-byte names; listing compared with h5py).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
CHANGELOG: upgrade notes (changed read results for max-shape files,
saturating conversions, new writer errors, format-crate API changes) and
the reader/writer correctness fixes. known-issues: the silent-wrong-data
table with before/after sweep numbers, the gaps still open, and a
correction to the Extensible Array entry, which said files we wrote were
unaffected. CLAUDE.md: clawhdf5-gpu is vector distance computation, not
I/O, and clawhdf5-filters holds only deflate backends (no Blosc).
Also a facade test that libhdf5's 20-bit N-Bit float test data reads as
libhdf5's values.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The interop step built writer_h5py_tests without the lz4/zstd features,
so the hdf5plugin round-trips added with the registered LZ4 framing and
the Zstd content-size fix never compiled in CI, and CI never installed
hdf5plugin.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The maxshape checks added to chunked_write use format!, which a no_std
build has to import from alloc (scripts/check-nostd.sh).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
dataset_fill_value treated a shared Fill Value message as "no fill
value", so unwritten storage of a dataset whose fill value lives in the
file's shared-message (SOHM) heap read as zeros rather than its fill
value. libhdf5 shares fill values whenever the file has a SOHM index for
them.
- fill_value::dataset_fill_value_in follows the reference (another object
header, or the SOHM heap); read_full_with_fill and the facade's
selection read use it.
- dataset_fill_value, which has no file to follow a reference into, now
returns UnresolvedSharedMessage for a shared message instead of None.
- shared_message::load_sohm_table / message_data_with_sohm load the SOHM
table from the superblock extension on demand.
- parse_sohm_table skipped each index's leading version byte, reading
every field one byte off; SOHM references could never resolve.
Fixture shared_fill_value.h5 (HDF5 2.0, gen_shared_fill.py): sohm_b read
[0,1,2,3,0,0,0,0] and now reads [0,1,2,3,-7,-7,-7,-7], as h5py does.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
libhdf5 applies filters in pipeline order, so with Fletcher32 before
deflate (h5repack_filters.h5 /dset_all: shuffle, fletcher32, deflate; or
h5py's set_fletcher32() then set_deflate()) the compressor holds the
chunk plus a 4-byte checksum. decompress_chunk bounded every stage by the
chunk size and rejected it: "deflate: output exceeds size limit". Bound
each stage by the chunk size plus 4 bytes per Fletcher32 that precedes
it in the pipeline.
Test: fletcher32_before_deflate_decodes (h5py-written chunk, and our own
shuffle + fletcher32 + deflate round trip); failed before.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Any maxshape forced chunked storage, even one equal to the shape, which
cannot grow. h5py and the library store such a dataset contiguously; we
now do too unless chunks (or a filter) are requested.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
D-scale floats were rebuilt as `minval + code / 10^D` in f64 and then
rounded to f32 once, but libhdf5 (H5Z_scaleoffset_modify_3/4 with
`float`/`powf`) computes `(float)(int)code / powf(10, D) + min` in single
precision. The two differ by 1 ULP for some values: le_data.h5
/Scale_offset_float_data_{le,be} gave 1.6663332 (0x3fd54a69) where
libhdf5 gives 1.6663333 (0x3fd54a6a). Use f32 arithmetic for 4-byte
floats and `(double)(long)code / pow(10, D) + min` for 8-byte ones.
Test: scaleoffset_float_dscale_matches_libhdf5_bits (le_data.h5 float
LE/BE and double chunks, bit-exact against h5py); failed before.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A dataset with more than one unlimited dimension got an Extensible Array
index, which libhdf5 refuses ("already found unlimited dimension"), so
the whole file failed to open in h5py and h5dump. The previous commit
turned that into a write error; this one writes what the library itself
uses there: a version-2 B-tree chunk index (record type 10/11), as a
single leaf of the library's 2048-byte node size, or a larger leaf when
the records do not fit. The root's record count is 16-bit, so more than
65535 chunks is still refused rather than written wrong.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The object header parser failed on an unknown message with flag bit 3
set and ignored bit 7. Per the spec, bit 3 means "fail if unknown and
the file is opened for writing" and bit 7 "fail if unknown, always".
The parser only reads, so it now ignores bit 3 (as libhdf5 does for a
read-only open) and refuses bit 7, in v1 headers, v2 headers and their
continuation chunks.
On libhdf5's conformance file tbogus.h5 (added as a fixture) we used to
refuse Dataset2 and open Dataset3; we now match libhdf5: Dataset1, 2, 4
and 5 open, Dataset3 is refused.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
- libhdf5 sets cd_values[1] ("need not compress") when every field is
already full width and then stores the chunk unchanged
(H5Z__filter_nbit: `if (cd_values[1]) HGOTO_DONE`). We ignored it and
tried to unpack, so tfilters.h5 / h5stat_filters.h5 `/all` (shuffle +
szip + deflate + fletcher32 + N-Bit) failed with "nbit: packed data too
short". A type with no N-Bit parameters (cd = [3, 1, nelmts]) is now
accepted the same way.
- Class 4 (H5Z_NBIT_NOOPTYPE: enum, string, opaque, ... members) is
stored whole, 8 bits per byte; it was UnsupportedFilter(5)
(h5repack_nested_8bit_enum_deflated.h5).
N-Bit on floats was not wrong in the filter: for le_data.h5 /
Nbit_float_data_* our output equals libhdf5's decoded bytes in the file
datatype (a 20-bit float, offset 7, bias 31). h5py's values differ
because libhdf5 then converts that custom float layout to IEEE, which
our datatype reader does not do; nbit_float_matches_libhdf5_file_type_bytes
pins the filter output and the doc comment says where conversion belongs.
Tests: nbit_need_not_compress_is_passthrough,
nbit_in_multi_filter_pipeline_matches_libhdf5 (tfilters.h5 chunk, szip
feature), nbit_compound_with_enum_member_matches_libhdf5 all failed
before; nbit_float_matches_libhdf5_file_type_bytes (guard).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
FileWriter writes the root group plus one level of groups; it has no way
to create intermediate groups. create_group("a/b") therefore stored a
single link literally named "a/b", which no HDF5 reader can resolve
(h5py: "component not found"). Nesting would mean restructuring the
writer's layout around a group tree, so for now finish() rejects any
group, dataset or external-link name that is empty, "." or contains '/'.
Attribute names may still contain '/'.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A File is Send + Sync and keeps one ChunkCache for all its datasets.
The cached readers bound that cache to "the current dataset" with
ensure_dataset(addr), then checked, built and read its index and its
decompressed chunks in separate lock acquisitions. Two threads reading
two chunked datasets interleaved those steps, so one could store its
chunk index under the other's binding, or get the other's decompressed
chunk for the same coordinate: wrong data, or an index-out-of-bounds
panic when the ranks differed (16 threads x 40 reads over 24 datasets
panicked on every run).
The cache now keeps per-dataset state keyed by chunk-index address:
the chunk index, ChunkIndex and ChunkLayout per dataset (held as Arcs,
built outside the lock, first writer wins), and decompressed chunks
keyed by (address, coordinate). The chunked readers use the new
addr-taking methods (chunks_for, chunk_layout_for, get/put_decompressed_in,
prefetch_hint_in) exclusively. Memory stays bounded: decompressed data by
the existing byte/slot budget across datasets, indexes by at most 64
datasets and 2^20 index entries in total, dropping the least recently
used dataset's index first. Switching datasets no longer throws away the
other datasets' cached chunks.
The address-less methods remain and act on the dataset last bound with
ensure_dataset; they are documented as not for concurrent readers.
Regression: threads_reading_different_datasets_get_their_own_chunks
(crates/clawhdf5/tests/concurrent_chunk_cache.rs), plus cache unit tests
datasets_sharing_coordinates_stay_separate, dataset_indexes_are_bounded
and concurrent_readers_of_different_datasets_see_their_own_chunks.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The writer indexed chunks by their position in the current shape, the
same mistake the reader had. With a finite maxshape larger than the shape
the Fixed Array was sized for the shape, so libhdf5 looked up chunks past
its end ("addr overflow"); with the unlimited dimension anywhere but first,
e.g. maxshape (20, None), libhdf5 swizzles that dimension to the slowest
position and read our Extensible Array scrambled. Two unlimited dimensions
produced a file libhdf5 refused to open ("already found unlimited
dimension").
Chunks are now placed with the shared chunk_grid linearisation: Fixed
Array slots cover every chunk of the maximum extent (unwritten ones
undefined), Extensible Array indexes are swizzled, Single Chunk is only
used when the maximum extent is one chunk, and a maxshape that is smaller
than the shape, has more than one unlimited dimension, or would need an
absurd Fixed Array is an error instead of a bad file.
build_chunked_data_from_precompressed now returns a Result.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
An empty AttrValue::String (or a StringArray of empty strings) was
written with a size-0 fixed-length string type. libhdf5 rejects that
("invalid datatype size"), and the failure takes every attribute on the
object with it. Strings are now at least 1 byte, NUL-padded, which is
how h5py stores "" and reads back as "" in both h5py and our reader.
check_encodable also refuses a size-0 string type passed in directly.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
SZIP-filtered datasets from libhdf5 came back as garbage or zeros with no
error (ref_szip.h5, h5repack_szip.h5, noencoder.h5, le_data/be_data
Szip_float_data_*), and 64-bit ones failed with "invalid bits per
sample" (h5wasm compressed.h5). The decoder called aec_buffer_decode
directly, but libhdf5 goes through szlib's SZ_BufftoBuffDecompress
(H5Zszip.c), which libaec implements with reshaping (sz_compat.c).
Differences, all fixed:
- H5Zszip.c prefixes the stream with the 4-byte LE uncompressed size; it
was fed to libaec as data.
- 32- and 64-bit samples are coded as byte planes of 8-bit samples and
must be de-interleaved.
- The reference sample interval is ceil(pixels_per_scanline /
pixels_per_block), not a fixed 128.
- Scanlines that are not a whole number of blocks are padded and must be
unpadded.
- Byte order comes from the MSB option bit; LE data was decoded as MSB.
Test: szip_decodes_libhdf5_chunks_exactly compares chunks from HDF Group
test files (noencoder.h5, le_data.h5) and an h5py-written file (64-bit,
16-bit, padded scanlines, NN and EC) byte for byte with h5py's values;
it failed before on the first case.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
FillTime::to_byte had the fill-time field rotated against libhdf5
(H5D_FILL_TIME_ALLOC = 0, NEVER = 1, IFSET = 2): Never was written as
ALLOC, Alloc as IFSET and IfSet as NEVER, as h5py reported. The flags
byte is now late allocation plus the right code, and FillTime::from_byte
decodes it.
The default becomes IfSet, which is libhdf5's default and exactly the
byte (0x0a) every dataset was already written with, so default output
does not change; `Alloc` was documented as the C library's default but
never was. DatasetCreateProps follows.
DatasetBuilder::with_fill_value sets a user-defined fill value (one
element's stored bytes, checked against the datatype size), written as a
defined value in the fill value message. h5py reports it, and extending
the dataset in h5py fills the new elements with it.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
FileWriter::with_page_size wrote a "version 4" superblock with an extra
page-size field. HDF5 has no superblock version 4, so libhdf5 refused
every such file ("bad superblock version number").
A paged file is now what libhdf5 itself writes for fs_strategy="page":
a v3 superblock whose extension object header holds a File Space Info
message (strategy PAGE, the page size, free space not persisted; same
bytes and flags as HDF5 2.0), with the file padded to a whole page.
h5py opens it, reports the strategy and page size, and can modify it in
r+ mode. Page sizes outside libhdf5's 512 B..1 GiB are an error.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The Extensible Array writer only filled the index block's 4 inline
elements and the 6 data blocks it addresses directly (240 elements); its
super block addresses were always undefined. Chunks from index 244 on were
written to the file but never indexed, so they read back as fill values in
our reader and in libhdf5, without an error.
The writer now lays out data blocks and super blocks for any element
count as H5EA__hdr_init sizes them, pages data blocks larger than 1024
elements (page-init bits in the owning super block), leaves blocks with no
defined element unallocated, and records real header statistics
(max_idx_set is one past the highest defined index).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
read_vl_bytes cut each element to the reference's length field, which
counts sequence elements, not bytes: a VL int32 [1, 2, 3] came back as
3 bytes. Return the whole global-heap object, which is element count x
base size bytes. No in-tree caller depended on the old behaviour.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
read_i64/read_u64/read_i32/read_f64/read_f32 refused enumeration
datatypes, including h5py's bool (an enum of int8), with a type
mismatch. Read them as their base type's integer values, the way array
datatypes already read through theirs.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Every 2-byte float was decoded as IEEE half, so bfloat16 (HDF5 2.0's
H5T_FLOAT_BFLOAT16*, or any custom 8-bit-exponent type) read wrong:
1.5 as 1.9375, +inf as NaN. 1-byte FP8 floats were refused.
Read the exponent/mantissa location and size and the bias from the
datatype message: IEEE half/single/double keep their existing paths
(half still through clawhdf5_format::float16), any other IEEE-style
layout up to 64 bits whose values fit f64 (bfloat16, FP8 E4M3/E5M2, ...)
is decoded generically, and the bulk-copy and zero-copy fast paths now
require the IEEE layout rather than just the size. Datatypes with fields
that describe no float still fall back to IEEE by size; layouts that
cannot be represented in f64 (x87 80-bit, binary128) remain an error.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Reading wider or differently-signed integers kept the low bits: i64
2^40+5 read as i32 was 5, u64::MAX read as i64 was -1, and -1 read as
u64 was 4294967295. u32 data read as i32 also took the bulk-copy fast
path meant for i32. Saturate at the target range like libhdf5's hard
conversions (a negative value read as unsigned is 0), and keep the i32
fast path to signed data.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
read_i32/read_i64/read_u64 on a floating-point dataset reinterpreted the
IEEE bits (1.5 read as i64 was 4609434218613702656). Convert like
libhdf5's hard conversions instead: truncate toward zero and saturate at
the target range; NaN reads as 0.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Layout message v4 flag bit 0 (H5D_CHUNK_DONT_FILTER_PARTIAL_CHUNKS, set
with H5Pset_chunk_opts) makes libhdf5 store every chunk that extends past
the dataset's extent without the filter pipeline, while its filter mask
still reads 0. The parser ignored the flag, so readers tried to inflate
raw bytes: libhdf5's own h5fc_edge_v3.h5 failed with "deflate: ...
unknown compression method".
DataLayout::Chunked gains dont_filter_partial_edge_chunks (always false
for v3), and list_chunks — the one place every read path gets its chunk
list from — marks such partial chunks as having skipped every filter, so
the full, cached, indexed, parallel and selection readers all copy them
as-is. chunked_write.rs gets `..` in one exhaustive test pattern for the
new field.
Regression: libhdf5_edge_chunk_fixture_reads (h5fc_edge_v3.h5 from the
HDF5 tools test files, committed as a 2.5 KB fixture), and
h5py_unfiltered_partial_edge_chunks_read (the flag set through h5py's
bundled libhdf5 via ctypes, as h5py has no binding for it: fixed array,
extensible array and B-tree v2 indexes, 1-D and 2-D, plus a hyperslab
of the last chunk), and v4_chunked_dont_filter_partial_edge_chunks_flag.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The Fixed Array writer always packed every element into one data block
behind one checksum. Past 2^10 elements libhdf5 (and our reader) expect a
paged block: a page-init bitmap after the prefix, then one checksummed page
per 1024 elements. Any dataset with more than 1024 chunks and no unlimited
dimension failed with "incorrect metadata checksum" in h5py, h5dump and
our own reader.
build_fixed_array_at now takes one Option<WrittenChunk> per array slot so
later fixes can leave unallocated slots.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Datatype::serialize returned an empty message for these four classes, so
any dataset or attribute of them (including a Raw attribute copied from
another file) was unreadable by libhdf5 ("ran off end of input buffer
while decoding"). They now encode exactly as libhdf5 does: legacy object
and region references as datatype version 1, H5T_STD_REF kinds as version
4 with their encoding version, opaque tags NUL-padded to 8 bytes.
Parsing an opaque tag now stops at its first NUL, so libhdf5's padding
no longer becomes part of the tag. Datatype::check_encodable rejects
what has no encoding (an opaque tag over 248 bytes); FileWriter::finish
calls it for every dataset and attribute type.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Every decode stage was capped at the chunk's decoded size. That holds
only when every filter ahead of the codec preserves size; Fletcher32
does not (it appends a 4-byte checksum), so a pipeline with Fletcher32
before deflate (NetCDF-4's fletcher32 -> shuffle -> deflate ordering,
h5repack's "all filters") failed with "deflate: output exceeds size
limit" on every chunk.
decompress_chunk_masked now computes each stage's bound by running the
chunk size forward through the filters that precede it in write order
(and that the chunk's mask did not skip): shuffle keeps the size,
Fletcher32 adds 4, any codec adds at most n/8 + 64. The cap is still a
small constant factor of the chunk, so a decompression bomb is rejected
as before (tested).
Shuffle also had to learn libhdf5's handling of a length that is not a
whole number of elements (chunk + checksum): shuffle the whole elements
and leave the trailing bytes in place, in both directions. It used to
refuse such data.
Regression: h5py_fletcher32_before_deflate_reads (fletcher->shuffle->
gzip, fletcher->gzip, shuffle->fletcher->gzip, and a 2-D i32 grid),
fletcher32_ahead_of_deflate_stays_bounded and
shuffle_leaves_a_partial_trailing_element_in_place.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Pcodec chunks were written as filter 32023, which the HDF Group registry
assigns to Granular BitRound (GBR). Pcodec has no registered ID (checked
2026-09-25 against hdf5_plugins/docs/RegisteredFilterPlugins.md, which
ends at 32033 with no pcodec entry). GBR's decode is a pass-through, so
libhdf5 with that plugin loaded would have returned the compressed bytes
as the dataset's values.
Write pcodec as 480, from the registry's testing/private range (256-511),
named "pcodec (clawhdf5 private)", and document it as non-interoperable:
only clawhdf5 with the `pcodec` feature reads it. Chunks under 32023 are
still read as pcodec when the filter is named exactly "pcodec" (what
clawhdf5 <= 2.7.0 wrote); any other 32023 is UnsupportedFilter.
Test: pcodec_uses_private_id_and_reads_legacy_32023.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A chunk's filter mask has one bit per pipeline filter; bit i set means
filter i was not applied to that chunk (an optional filter that
declined, or a direct chunk write). Every read path treated any nonzero
mask as "no filters applied" and returned the stored bytes, so a chunk
that skipped only gzip in a shuffle+gzip pipeline came back still
shuffled (h5py write_direct_chunk with filter_mask=0b10: 8 of 32 values
wrong).
decompress_chunk_masked undoes the filters the mask leaves set and skips
the rest; an unsupported filter is no longer an error when the chunk
skipped it. The full, cached, sweep, indexed, parallel and selection
(partial_read) paths all use it, and a chunk is copied straight from the
file only when every filter was skipped. decompress_chunk is the mask-0
case.
Regression: h5py_partial_filter_mask_skips_only_masked_filters (1-D
shuffle+gzip with masks 0, 0b10 and 0b11; 2-D with 0b01; full and
hyperslab reads) and filter_mask_skips_only_the_masked_filters.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Filter 32015 chunks were written with the streaming encoder
(zstd::encode_all), whose frames carry no content size. The registered
HDF5 Zstandard filter (H5Zzstd.c, libhdf5 + hdf5plugin) sizes its output
from ZSTD_getFrameContentSize and fails on such frames, so h5py could not
read our zstd datasets ("filter returned failure during read"). Compress
with the one-shot API, which records the size.
Tests: zstd_frames_record_content_size (content size was None before),
hdf5plugin_reads_our_zstd (ignored interop test; failed before).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Both indexes place each chunk at a linear index computed from the
dataset's maximum dimensions (libhdf5's max_down_chunks), and the
Extensible Array first swizzles its unlimited dimension to the slowest
position. We linearised by the current dimensions, so any dataset whose
shape was smaller than its maxshape, or whose unlimited dimension was not
the first, read back scrambled without an error: h5py libver="latest"
files with maxshape (10, None) or (20, 10), and the libhdf5 test files
h5fc_ext*.h5 and test_ld.h5.
The linearisation now lives in chunk_grid (shared with the writers), and
slots beyond the current extent are ignored as the library does.
read_fixed_array_chunks / read_extensible_array_chunks take the
dataspace's max dimensions.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A v2 object header message has a 2-byte size field. The writer truncated
larger sizes to 16 bits, so an attribute over ~64 KiB (or a compact
dataset of 65532-65535 bytes, whose layout message adds 4 bytes) produced
a file libhdf5 rejects ("message of unshareable class flagged as
shareable", "bad flag combination").
ObjectHeaderWriter::serialize now returns a Result and fails on any message
over MAX_MESSAGE_SIZE; FileWriter::finish propagates it. Compact storage
falls back to contiguous above 65531 bytes, the real limit. Dense storage
for large attributes remains future work.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Filter 32004 chunks were framed as a 4-byte little-endian size plus one
LZ4 block. That is not the registered HDF5 LZ4 format (H5Zlz4.c: 8-byte
big-endian total size, 4-byte big-endian block size, then per block a
4-byte big-endian compressed length and the block, stored raw when the
length equals the block size), so libhdf5 + hdf5plugin could not read
our LZ4 datasets and we could not read theirs (h5ex_d_lz4.h5:
"lz4: 0 is not a valid match offset").
Write the registered format (cd_values[0] is honoured as the block size,
default 1 GiB like the plugin) and read it, multi-block and raw blocks
included. Chunks in the old framing stay readable: an HDF5 chunk is under
4 GiB, so a registered chunk always starts with four zero bytes and is at
least 12 bytes long, while an old one starts with four zero bytes only
when empty (5 bytes).
Tests: lz4_reads_registered_hdf5_format (chunk of the HDF Group's
h5ex_d_lz4.h5, block size 3), lz4_writes_registered_hdf5_format,
lz4_reads_legacy_clawhdf5_format, and hdf5plugin_reads_our_lz4 (ignored
interop test; failed before with "filter returned failure during read").
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A type-1 (raw data chunk) B-tree key holds the chunk size, the filter
mask and one offset per dimension, and those offsets are always 8 bytes:
they are dataset coordinates, not file addresses. The reader used the
superblock's size-of-offsets for them, so in a file with 4-byte offsets
every key was misparsed. Unfiltered chunked datasets read as zeros (with
stray bytes where a misread address landed on data) and filtered ones
failed with "deflate: truncated stream".
Only the sibling and child addresses follow size-of-offsets now. The
unit-test B-tree builder wrote keys the same wrong way, which is why its
tests passed; it now matches the format.
Regression: h5py_four_byte_offsets_chunked_reads (h5py, set_sizes(4, 4)
and (4, 8); 1-D and 2-D, unfiltered and gzip) and the unit test
collect_chunks_with_four_byte_addresses.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
An 82-byte fuzz_btree_v2 crash input from 2026-09-20 was left untracked
in fuzz/artifacts. Replayed today it runs cleanly: the depth cap and
record budget added to B-tree v2 traversal that day fixed it. It is now
in the committed fuzz corpus, and a robustness test replays the fuzz
target's exact code path on it so a regression fails CI rather than
waiting for someone to run the fuzzer.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The previous commit said clawhdf5 has no integration at all. ClawBrainHub
(clawverse/clawbrainhub) does use it: cbh-core reads and writes .brain
files through the facade, cbh-scanner uses the facade, and cbh-cli uses
clawhdf5_agent::bm25::BM25Index, all via path dependencies on this repo.
Checked on 2026-09-25 against main: it builds on its pinned toolchain and
its 204 tests pass. CLAUDE.md now records that, and that path
dependencies mean API changes here reach it directly.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
CLAUDE.md said ZeroClaw "imports this as a Cargo feature (clawhdf5
feature flag)" and uses clawhdf5 as its memory backend; the agent crate
called itself the "ZeroClaw agent memory HDF5 backend"; the migrator
claimed to read "the ZeroClaw layout". Checked on 2026-09-25 against
ZeroClaw v0.8.5 (its latest release), the osobh/zeroclaw fork (on
v0.8.5) and both histories back to February 2026:
- no `clawhdf5` feature, dependency or memory backend has ever existed
in ZeroClaw; its backends are sqlite, lucid, postgres, qdrant,
markdown and none, behind its own `Memory` trait;
- ZeroClaw's SQLite schema is a single `memories` table (id, key,
content, category, embedding, created_at, updated_at); the
migrator's memory_chunks/sessions/entities/relations layout never
existed in ZeroClaw, so it cannot read a ZeroClaw database.
Decision: withdraw the claims (as with OpenClaw); clawhdf5 is a
standalone library with no framework integration. The migrator's
default layout is documented as its own. ZEROCLAW_VERSION keeps its name
and value (it is the persisted `edgehdf5_version` writer tag) with a
doc comment saying it is unrelated to ZeroClaw.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The docs described a "drop-in" OpenClaw memory backend enabled with
`memory.backend = "clawhdf5"`. Checked against OpenClaw's source and
docs (v2026.2.26 through v2026.9.6): that config was never valid —
v2026.2-v2026.7 accepted only "builtin"/"qmd" and rejected unknown
keys, so a Gateway given it refuses to start, and v2026.8.1 (OpenClaw
2.0) removed the key. No plugin was ever built (no manifest, no
registration, no tools), nothing was tested against OpenClaw, the
linked github.com/redclawsystems/openclaw is a 404, and
@redclaw/clawhdf5 was never published.
Decision (2026-09-25): not pursuing an OpenClaw plugin for now; ZeroClaw
is the integration target.
- Remove openclaw-integration.md, openclaw-config.md and
migration-guide.md; add docs/openclaw.md: the status, what a memory
plugin needs against v2026.9.6 (plugins.slots.memory, manifest with
kind "memory", registerMemoryCapability / MemorySearchManager,
prebuilt native packages), and what this repo has as building blocks.
- README, QUICKSTART, USE_CASES, ROADMAP (Track 7 withdrawn), CLAUDE.md
and the `openclaw` module docs describe ClawhdfBackend as what it is:
a Markdown-oriented library backend, not an OpenClaw plugin. The
QUICKSTART example is corrected (the old one called a three-argument
create that does not exist) and states its limits.
- packages/clawhdf5-node: marked unpublished and broken, "private": true
so it cannot be published by accident; its bugs (snake_case vs
camelCase fields, wrong addon path, no way to store an embedding,
wrong WAL name) are recorded in docs/known-issues.md.
- Two broken rustdoc links fixed along the way.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Makes the README's "cryptographically verifiable memory" true.
With HDF5Memory::set_signing_key(key), every checkpoint stores a signed
manifest of the store: a SHA-256 per memory record (text, embedding as
stored, channel, timestamp, session, tags, deleted flag, activation) in
a Merkle tree, plus hashes of the settings (and WAL mark), sessions and
knowledge graph. The signature, public key and manifest hashes go in
/meta; the per-record hashes in /integrity/record_hashes, so
HDF5Memory::verify(path, &public_key) can say which records changed, not
just that something did. A forged manifest fails the signature.
Decisions, as agreed:
- the key is set on the open store and never persisted;
- a signed store refuses to checkpoint without its key
(MemoryError::SigningKeyRequired); remove_signature() is the
deliberate way back to unsigned;
- checkpoints only: saves still in the WAL are not covered, and verify
reports how many there are.
The hashes cover exactly what the file persists, in the form the loader
returns it (strings lose trailing NULs; an empty WAL mark is not
written), so untouched stores verify across any number of reopen and
checkpoint cycles. MemoryError becomes #[non_exhaustive] (it already
gains variants in this unreleased version).
CLI: keygen (owner-only key file), --signing-key / CLAWHDF5_SIGNING_KEY
on writing commands (create signs immediately), verify --public-key
(JSON; exit 2 if not valid), `signed` in create/stats output.
Tests: reopen/checkpoint cycles with awkward strings (f16 and f32),
refusal without the key, wrong and rotated keys, eight kinds of edit
each detected and located, a forged manifest, unsigned stores, NULs in
text, and an edit made in place with h5py that verify pinpoints.
Cost on tank (search_harness --signing-study --full, 3 runs): ~20% of a
checkpoint (+9 ms at 10K, +89-112 ms at 100K), verify 18.6 ms / 247 ms,
32 bytes per record in the file. New deps ed25519-dalek, sha2,
rand_core: pure Rust, the no-C check passes.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
It enabled nothing — the agent layer is always built — yet the README,
QUICKSTART and USE_CASES told people to pass it. Removed, with those
snippets fixed: they now depend on the git repository (nothing is on
crates.io, so `version = "2.0"` never resolved) and USE_CASES no longer
presents the `float16` feature as half-precision storage (that is
MemoryConfig::float16, on by default for new stores).
Breaking for anyone passing `features = ["agent"]`: drop it.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
consolidation_efficiency never finished: stopped after 19 minutes on one
core while building its 100K case. Not the consolidation cycle (linear:
17 us at 100 records, 2.16 ms at 10K) but the setup — every add_memory
scores the new record's novelty against the whole working tier, the
benchmark lets that tier reach 50K, and each comparison recomputed both
norms: ~5e9 comparisons of three passes each.
ImportanceScorer::score_surprise now computes the new record's norm
once, takes each comparison in one fused, 8-lane pass (dot product and
the other norm together), and splits a working tier of 4096+ records
across threads with the `parallel` feature. Same results: tested against
the old cosine formula, including shorter, empty and zero vectors and
the parallel path. The work stays quadratic in the working-tier size by
design; with regular consolidation the tier stays near
working_capacity (100) and inserts are cheap.
The complete run takes 8 min 10 s on tank and fills in the 100K cycle
row (46.66 ms) and the memory-reduction table, which had never been
published. The binary no longer prints a record-count ratio as a
"BM25 Speedup" (never measured; Part 1 measures search latency) or
claims sub-linear cycle scaling (its own numbers grow slightly faster
than linearly).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Every undated or pre-September section re-run on one machine on one day
(tank, AMD Ryzen 7 7800X3D, 2026-09-24, commit 5c8323c), 24 commands run
serially with the load average checked before each, with the command
recorded for each section. A separate check traced every changed number
back to the raw output; its corrections are applied (e.g. the on-disk
~820 B/record is float16 plus always-deflated text on a synthetic corpus
of 40 distinct texts, not float16 alone).
Two apparent regressions were isolated rather than published:
- knowledge-graph traversal: a real bug, fixed in the previous commit;
- the write path: v2.3.0 built and run on the same machine measures the
same as today, so the old 18 us / 6.17 ms figures (undated, other
hardware) are not reproducible; float16 adds ~2 us per save and the
int8 index nothing (both isolated by switching the bench's config).
Also:
- new multimodal_bench: cross-modal search at 1K/10K records, which the
README claimed but nothing measured;
- footprint_bench reports whether it built float16 or f32 stores and
takes --f32 (it kept printing "f32" after the default changed);
- README: performance tables, the "Why" table figures and the SQLite
migration section (from the previous migrate commit);
- CHANGELOG for this branch.
Not re-run: consolidation_efficiency's 100K row and its memory-reduction
part (stopped for time), and cross_platform.sh.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
bfs_neighbors and spreading_activation built an adjacency index over the
whole graph on every call (1efd82c), so a 2-hop BFS over 1K entities
paid to index every entity and relation first: 155 us, 6.5x the 24 us
the README quoted. Found by the dated benchmark re-run.
The index is now cached on KnowledgeCache and checked against a
fingerprint of the graph on each use — one pass over entity ids and
relation endpoints, no allocation — so any change, including direct
edits of the public entities/relations Vecs (schema.rs's load path
pushes to them), still triggers a rebuild. A test edits the graph
directly in every way (push, in-place rewire, pop + push at equal
length) between traversals.
tank, 2026-09-24: BFS 1K entities 155.1 -> 23.1 us, 100 entities
17.5 -> 5.23 us, spreading activation 100 22.8 -> 10.1 us.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
clawhdf5-migrate wrote a layout of its own (/chunks, /sessions,
/entities, /relations, root attributes, no /meta or schema_version) that
HDF5Memory::open rejects, so a "migrated" SQLite database could not be
used as agent memory — contrary to the README.
It now writes through the agent's own API (HDF5Memory::create/open,
save_batch, the session cache and the knowledge graph), so there is no
second copy of the schema:
- sessions and entities/relations carry over; deleted rows become
deleted records (or are left out with --skip-deleted);
- embeddings follow the library default (float16), --f32 opts out and
--float16 is a hidden no-op, as in clawhdf5-cli; the `half`-based
conversion is gone;
- every source row is checked before the output is created: a wrong
embedding length, an empty embedding, a dimension that differs from
an existing store's, or a float16 value beyond +-65504 is an error
naming the chunk id, and an existing store is left untouched;
- --incremental opens the existing store, adds only rows it does not
hold (matched by content) and follows the source's deleted flags;
- a source with no memory rows needs --embedding-dim;
- validation reads the result back with HDF5Memory::open_read_only,
compares every field (embeddings bit for bit, round_to_f16 of the
source for float16) and checks a migrated record is found by search.
clawhdf5-agent gains HDF5Memory::sessions()/sessions_mut(),
HDF5Memory::delete_batch (one save, all-or-nothing, no auto-compact),
SessionCache::add_at, and re-exports SessionCache/SessionEntry.
The old layout's per-dataset SHA-256 provenance attributes have no place
in the agent schema and are gone. An adversarial review found two
blockers (silent truncation of long embeddings; an --incremental
dimension check that could never fire) and four majors (a failed run
wiping the existing store, dim-0 stores, deleted-flag drift); all are
fixed with regression tests. 42 migrate tests, incl. h5py opening a
migrated store.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
MemoryConfig::float16 now defaults to true for new stores, on
measurement: on the full LongMemEval haystack with real MiniLM
embeddings every retrieval metric matched f32 (previous commit), and at
100K the file is 48% smaller with faster checkpoints and opens.
Existing stores are unaffected: every agent store has recorded
`float16 = false` in /meta and keeps it. A test opens the v2.5.0
fixture, saves and checkpoints, and checks the embeddings are still f32
with the old rows bit-identical; another checks a new store is float16.
CLI: `create --f32` opts out; like `--f32-index` it only ever switches
the default off. `--float16` is still accepted and now a no-op.
Values beyond +-65504 are refused, so f32 remains the choice for
unnormalised vectors — the upgrade note says so.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
`longmemeval_bench --float16` builds every per-question store with
MemoryConfig::float16, so the vector stage searches half-rounded
embeddings exactly as such a store holds them.
Full longmemeval_s (500 questions, ~494 turns each) with real
all-MiniLM-L6-v2 embeddings, f32 vs float16, on tank (CUDA): identical
at every Hit@k and MRR, turn and session level, in all eight modes —
bar RRF session MRR 0.9253 vs 0.9254 and one or two flips out of ~320
in which gold session ranks first. The f32 run reproduces the published
hybrid numbers exactly. The earlier float16 evidence was synthetic
clustered data only; this is the real-embedding check.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
`HDF5Memory::search(query_embedding, query_text, &SearchOptions)` is the
store's full search path. `SearchOptions::new(k)` is plain hybrid search
with the tuned default fusion; each further stage is opt-in:
- `with_sources([..])`: only records from these source channels. The
filter applies before ranking, so a filtered search still returns up
to k results, normalised over what it can return. The HNSW pool is
over-fetched in proportion to what the filter removes, and the allowed
records are scanned exactly whenever that costs fewer distance
evaluations than the index would (~pool x M) — and as the fallback if
the pool comes back short. Keyword matches are filtered too.
- `with_rerank(ReRankConfig)` re-ranks a max(3k, 10) candidate pool by
relevance, recency, source authority and activation;
`with_confidence(ConfidenceConfig)` drops low-confidence results;
`at_time(now)` pins the recency clock.
These were reachable only through the OpenClaw backend, which is now
`search` with both on. Its Hebbian boost now goes to the k results it
returns rather than the whole 3k candidate pool. `hybrid_search` and
`hybrid_search_with` are wrappers and unchanged (tested bit for bit).
Measured on tank (search_harness --options-study --full, 3 runs): at
100K every filter — 50%, 10%, 1% of the store, and records far from the
query — returns the exact filtered top 10, and none is slower than an
unfiltered search (1%: 2.3 ms vs 4.6 ms). Re-rank + confidence costs
about 3%. A first version decided between index and exact scan by pool
size vs store size; it measured 0.976 recall at 12.3 ms on the
far-from-query filter, which is why the rule compares costs instead.
Tests: tests/search_options.rs (filter correctness and full pages via
both paths, far-from-query fallback, edge cases, equality with
hybrid_search_with, re-rank recency, confidence, boost scope).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The setting was persisted in /meta and otherwise ignored: embeddings
were always written as f32. It now does what it says.
clawhdf5-format:
- `DatasetBuilder::with_f16_data` writes IEEE binary16 (numpy float16),
rounding to nearest-even, and `make_f16_type`.
- `clawhdf5_format::float16` holds the f32 <-> f16 conversions, the one
implementation the writer, the reader and the agent all use. Checked
against the `half` crate on 16.7M f32 values and round-trips all 65536
half values; the h5py interop tests confirm the rounding matches
numpy's bit for bit (4020 values incl. ties, subnormals, overflow).
- Reading little-endian float16 as f32 has a fast path.
clawhdf5-agent:
- A float16 store writes /memory/embeddings as half precision, and
`MemoryCache::half_precision` rounds each embedding as it enters the
cache (save, update, WAL replay, and on load of a store still f32 on
disk), so memory and file agree bit for bit and a store searches the
same before and after a reopen (tested).
- Values beyond +-65504 are refused with the new
`MemoryError::InvalidEntry` rather than stored as infinity, on every
save path; batches are all or nothing, and a rejected ephemeral entry
stays in the ephemeral tier. Breaking for exhaustive matches.
- CLI: `create --float16`. Off by default.
Measured on tank, 384-dim, six runs alternating order, medians
(search_harness --float16-study --full): at 100K the file goes from
154.0 to 80.8 MiB (-48%), checkpoint 752 -> 512 ms, open 300 -> 252 ms;
vector recall@10 against an exact scan and hybrid_search latency do not
change. At 10K open is 3 ms slower. Also a test that h5py opens a whole
agent store, f32 and float16, and decodes every dataset.
Docs: README, BENCHMARKS.md ("float16 embedding storage"), CHANGELOG
(including the h5py interop fixes in the previous commit), CLAUDE.md.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Two write-side bugs, both present in every release (the first at least
since v2.1.0), made libhdf5 refuse files written by clawhdf5. Our own
reader ignores both fields, and the interop suites only ever wrote f64
from our side, so nothing here caught them.
- Every f32 dataset: "sign bit position out of bounds". The float
datatype encoder hard-coded the sign bit's position (bits 8-15 of the
class bit field) to 63, which is right only for f64. It is now derived
from the type: bit_offset + bit_precision - 1. This covered every
agent store's embeddings, norms and activation weights.
- Every empty dataset: "invalid dataset size, likely file corruption".
It was written with a real address and size 0, which trips libhdf5's
`addr + size <= addr` overflow check. An empty contiguous dataset now
gets the undefined address, as libhdf5 writes it. This covered every
agent store without sessions or a knowledge graph.
Agent stores are rewritten in full at each checkpoint, so they become
readable at their next checkpoint on a fixed build; other files with f32
or empty datasets need rewriting. Both are recorded in
docs/known-issues.md.
Tests: the sign position byte for f32/f64, and h5py reading our f32
datasets (plain and chunked + deflate) bit for bit.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
rust-version = "1.92" in [workspace.package], inherited by every crate.
1.92 is the floor: wgpu (clawhdf5-gpu) requires it, and the whole
workspace, Python bindings included, checks cleanly on it. ci-test.sh
reads the version from Cargo.toml and checks the workspace on exactly
that toolchain, so the manifests and the README badge cannot drift from
what actually builds. The badge said 1.75, below edition 2024's own
floor.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The core crates (clawhdf5, -agent, -format, -io, -filters, -ann, -accel,
-netcdf4, -cli) now build no C by default: deflate defaults to zlib-rs,
a pure-Rust port of zlib-ng, and zlib-ng becomes the opt-in
`fast-deflate`, which overrides zlib-rs wherever it is enabled. A default
build no longer needs cmake or a C compiler.
Measured on tank, both builds run alternately, three rounds, medians:
zlib-rs is within 6% of zlib-ng on every HDF5 read and write (512x512
deflate-6 chunked write 1.458 vs 1.484 ms; 64 MB compressed read 64.4
vs 65.2 ms), and compressed output is byte-identical. Details in
BENCHMARKS.md, "Deflate backend".
Getting there took two fixes the first measurement exposed:
- zlib-rs needs `std` to detect SIMD at runtime. flate2 enables it via
its default `runtime_detection`, which `default-features = false` had
switched off, leaving zlib-rs 3.5x slower on inflate. The `zlib-rs`
features now enable it.
- Both deflate paths streamed through flate2's 32 KiB read/write
wrappers. They now hand the codec the whole chunk in one call, into a
buffer sized up front (~5% on chunked writes). This also fixes a
silent short read: the streaming reader returned a truncated stream's
bytes without an error; a truncated chunk is now DecompressionError.
In clawhdf5-filters, output longer than the stated size is now an
error rather than silently cut off.
CI: ci-test.sh lints and tests the zlib-ng path, and fails if a
C-building crate (*-sys, cc, cmake) enters a core crate's default
dependency tree. The arm64 job no longer installs cmake.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The README had fallen behind v2.3.0-v2.7.0, and parts of it were not
true. Checked every claim against the code and BENCHMARKS.md:
- Three of the six Quick Start snippets no longer compiled (Agent
Memory, Consolidation, OpenClaw); all six now do.
- Hybrid search was described as RRF throughout. The default has been
weighted 0.4/0.6 fusion since v2.5.0; re-ranking and confidence
rejection run only in the OpenClaw backend.
- The `float16` feature does not halve embedding storage (the store
always writes f32), `--features agent` enables nothing, "Source
Isolation" is not wired in, and nothing backs "billion-scale" IVF-PQ.
- "Cryptographically verifiable" overstated an unkeyed, session-scoped
FNV-1a ledger; "Zero C dependencies" was false while zlib-ng was the
default deflate backend.
- Stale numbers: tests (1,650 -> 1,868), Rust badge (1.75 is below
edition 2024's floor), 6.5 KB/record on disk (BENCHMARKS.md: 1.7 KB),
consolidation and hybrid-search latency, and a feature-flag table
broken by a paragraph pasted into it.
- The file schema, module table and crate map now match the code.
Adds a "What's new (v2.2 -> v2.7)" section for collaborators, leading
with the silent Extensible Array read bug fixed in v2.7.0. Footer links
point at git.redclaw.dev. CLAUDE.md: clawhdf5-migrate is the SQLite
migration tool, and MemoryConfig::compression is off by default.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
