Requesting a filter without chunk dimensions made the whole dataset a single
chunk. Any read, even one row, then decompresses everything, and a large
dataset cannot be decoded in parallel — which also made the new partial reads
pointless for such files.
auto_chunk_dims keeps datasets up to 1 MiB as one chunk (unchanged behaviour)
and splits larger ones by halving the dimensions in turn, so chunks keep
roughly the dataset's proportions, until a chunk is at most 1 MiB — h5py's
approach. An empty (unlimited, unwritten) dimension is treated as 1024. The
writer passes the element size through resolve_chunk_dims_for; the old
resolve_chunk_dims assumes 8-byte elements. Explicit with_chunks always wins.
Interop test: h5py reads an auto-chunked 13 MB deflate dataset, sees chunks
between 128 KiB and 1 MiB, and a small dataset still has one chunk.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
- Add .gitea/workflows/ci.yml running scripts/ci-test.sh (fmt, clippy,
test, no_std check) on push/PR to main.
- Fix stale rustyhdf5-py/rustyhdf5-format package names in
ci-test.sh/check-nostd.sh, which had been silently no-op'ing those
checks (cargo warns but doesn't fail on an unknown --exclude/-p
target).
- With those checks actually running, fix the real issues they surface:
- clippy: useless_conversion in chunked_write.rs, byte_char_slices in
global_heap.rs/object_header.rs.
- cargo fmt: apply formatting across the workspace (whitespace only).
- no_std (thumbv7em-none-eabihf) build errors in clawhdf5-format:
core::sync::atomic::AtomicU64 doesn't exist on that target (no
native 64-bit atomics) — switch profiling.rs's counters to
portable-atomic, which falls back to a CAS-based emulation there
and is a no-op wrapper elsewhere. Add missing alloc imports for
Box (filters.rs), Vec (filters_szip.rs), and format! (dict_encoding.rs)
on no_std paths. Replace f64::powi (std/libm-only) with a small
local exponentiation-by-squaring helper in the scale-offset filter.
- Remove clawhdf5-types (empty 1-line stub crate; type defs already live in
clawhdf5-format). Update workspace Cargo.toml and CLAUDE.md accordingly.
- Implement HDF5 superblock v4 (page-buffer mode) read and write support in
clawhdf5-format: Superblock::parse_v4, page_size field, v4 serialize
branch, and FileWriter::with_page_size. This was the one task left
unimplemented from docs/superpowers/plans/2026-06-29-format-write-extensions.md.
- Reconcile the three docs/superpowers/plans/*.md docs (filter codecs,
format write extensions, MPI-IO VOL) against actual shipped code: they
were pre-work plans for d6c4d4f (2026-06-30) committed to git late on
2026-08-03 with all checkboxes still unchecked. Mark completed tasks done
and add a status note so they read as historical records, not open work.
- Refresh ROADMAP.md's "What's Next" section against current repo state.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Four independent write-path improvements:
1. Cache compressed chunks between Pass 1 and Pass 2 (chunked_write.rs,
file_writer.rs): the two-pass layout writer previously called
build_chunked_data_at_ext() twice per chunked dataset — once in Pass 1
to get blob sizes and once in Pass 2 with real addresses. Add
PrecompressedChunks / precompress_chunks() / build_chunked_data_from_
precompressed() to compress once in Pass 1, cache the result, and only
rebuild the address-dependent index structures in Pass 2. Expected
~2× speedup for chunked+deflate writes (512×512 deflate: 3.33ms → ~1.7ms).
2. SIMD-vectorisable shuffle filter (filters.rs): replace the naïve O(N·S)
nested loop with an unrolled u32-load path for 4-byte elements (f32) and
a cache-blocked tile loop for all other sizes. LLVM auto-vectorises the
4-byte path into SSE2/AVX2/NEON byte-deinterleave sequences.
3. Zstd benchmark variant (h5bench_write.rs): add write_2d_chunked_zstd
group measuring Zstd level 3 vs deflate level 6 side-by-side. Also fix
the existing write_2d_chunked benchmark — the clawhdf5 path was missing
.with_deflate(6), making the comparison apples-to-oranges. Add arXiv-
backed doc recommendation on DatasetBuilder::with_zstd().
4. Zero-copy HNSW save (hnsw.rs, clawhdf5-io/lib.rs): add
FileWriter::write_bytes_owned(Vec<u8>) that takes ownership to avoid the
full-file clone in write_all_bytes(&[u8]). HNSW::save_to_hdf5 uses it.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Critical fixes from whole-branch code review:
- libaec-sys: fix flag constants to match <libaec.h> exactly
(PREPROCESS=8, MSB=4, RESTRICTED=16; drop non-existent AEC_ALLOW_K13)
and add aec_buffer_encode FFI declaration
- filters_szip: fix cd index for bits_per_sample (cd[2] per H5Z_SZIP_PARM_BPP,
not cd[4]); fix option-mask mapping (NN=0x20, MSB unconditional); add two
real encode→decode roundtrip tests (no-NN and NN) that exercise libaec end-to-end
- file_writer: fix serialize_vds_mappings to delegate to data_layout_write
(eliminates the buggy duplicate that always emitted version=1 even for
external-file mappings); retains trailing Jenkins checksum
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Add `virtual_sources: Option<Vec<VdsMapping>>` field and `with_virtual_sources()` method to `DatasetBuilder`. In `FileWriter::finish()`, VDS datasets skip raw-data storage and instead serialize their source mappings into a global heap collection (version-1 same-file encoding) referenced by an HDF5 v4 layout-class-3 message. The two-pass address-computation loop handles VDS in both passes: pass 1 computes the fixed-size OH and pre-builds the heap blob; pass 2 places the blob at the correct file offset and rebuilds the OH with the real global heap address. Three new tests verify: (a) same-file two-source round-trip with mapping verification, (b) external-file source encoding, and (c) empty mapping list.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Dense attribute and dense link storage capped at a single fractal-heap direct
block (~64 KiB of heap data — a few thousand objects); beyond that the writer
produced an invalid oversized block. Lift the cap with a root indirect block.
When the serialized objects don't fit in one direct block, build a root
indirect block (FHIB) over multiple direct blocks sized by the doubling table
(start 512, width 4, doubling per row up to 64 KiB). Objects are packed
row-major across blocks, each block carries its logical block offset, and heap
IDs encode each object's heap offset (block offset + position). The FRHP points
root -> FHIB with the row count; unused slots in the current rows are undefined.
The fractal-heap builder is unified: FractalHeapBlock now carries the full heap
blob, and a shared write_frhp helper serializes the header for both the
single-block and multi-block paths. The single-block path is unchanged
(byte-identical), so existing dense attrs/links stay valid.
Validated end-to-end: a 2,500-attribute object and a 2,500-link group round-trip
through our reader and are read correctly by h5py. Objects still may not span a
block (no huge-object path).
Tests: facade round-trips for multi-block dense attrs and dense links, plus an
h5py-gated interop test (verified against the real h5py environment).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
A group with more than 8 links (libhdf5's compact max_compact default) is now
written densely instead of as inline Link messages: the links live in a
single-direct-block fractal heap indexed by a v2 B-tree of type 5 (link-name
index), referenced from the group's LinkInfo message. This matches libhdf5's
compact->dense switchover and keeps large groups out of the object header.
- Extract the byte-identical fractal-heap builder shared by dense attributes
and dense links, parameterized by heap_id_length / max_heap_size. Attributes
keep 8 / 40; links use 7 / 32 to match libhdf5 (reverse-engineered: an
h5py-written dense group uses heap_id_length 7, max_heap_size 32, type-5
record = hash(4) + heap_id(7) = 11 bytes). This was the cause of an initial
"object overruns end of direct block" error from h5py.
- build_group_oh gained an optional dense LinkInfo (omitting inline Link
messages); the two-pass file assembly allocates each group's link blob after
its object header and rebuilds it with real target addresses in the final
pass (link-message size is address-independent, so layout is stable).
Validated end-to-end: our reader round-trips dense groups, h5py reads the
groups we write, and dense attributes remain byte-identical (still h5py-valid).
The agent's 9-dataset memory group now writes densely and round-trips. Single
direct block only (~a couple thousand links); indirect blocks remain a TODO.
Tests: facade round-trip (20-link dense group + compact sibling) and an
h5py-gated interop test confirming libhdf5 reads our dense groups.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
