quantumclaw/clawhdf5
3
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity
Files
908af4028287ba7ce62d947c1775e039a497fae8
clawhdf5/CHANGELOG.md
T
osobhandClaude Opus 4.8 908af40282 feat: assemble 1-D same-file Virtual Datasets (VDS) 
A virtual layout previously returned UnsupportedVersion. Implement reading
for the common 1-D, same-file case, reverse-engineered and validated against
HDF5 2.0.

- Rewrite parse_vds_mappings to the real global-heap block format
  (version(1) · nused(length_size) · entries · checksum(4)), where each
  entry is source-file(null) · source-dataset(null) · source-selection ·
  virtual-selection. Block version 1 encodes a same-file source as a single
  0x04 marker in place of the file name; version 0 stores an explicit file
  name. The selections are H5S-serialized and self-describing in length, so
  they are decoded to find entry boundaries. The previous parser used a
  guessed layout that did not match real files.

- Extend Selection with decode_serialized() (H5S_select_serialize: ALL,
  NONE, and version-3 regular hyperslabs) and iter_linear_1d().

- Add read_virtual_data: resolve the mapping block from the global heap,
  read each same-file source dataset, and scatter its selected elements into
  the virtual buffer; unmapped regions stay at the zero fill value.
  External-file sources and N-D selections return a clean unsupported error.

Tests: real-file integration test (vds_same_file.h5: partial source slice +
fill gap), selection decoder unit tests built from the fixture bytes, and
same-file/external mapping-parser unit tests.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-06-03 20:01:51 +00:00

6.5 KiB
Raw Blame History

Changelog

Unreleased

New Features

  • clawhdf5-format: decode the scale-offset filter (id 6) — both the integer variant (H5Z_SO_INT) and the floating-point D-scale variant (H5Z_SO_FLOAT_DSCALE). Handles signed/unsigned int sizes, f32/f64, negative minima, decimal scale factors and fill values; reverse-engineered against HDF5 2.0 and validated end-to-end. The float E-scale variant remains unsupported.
  • clawhdf5-format: decode the N-Bit filter (id 5) — atomic, compound and array layouts (the full recursive type tree, nestable to any depth), previously unsupported. Signed and unsigned reduced-precision integers and float members all read end-to-end, validated against HDF5 2.0.

New Features

  • clawhdf5-format: assemble 1-D, same-file Virtual Datasets (VDS). Previously a virtual layout returned UnsupportedVersion. The reader now decodes the global-heap mapping block (reverse-engineered against HDF5 2.0: version · nused · [source-file · source-dataset · source-selection · virtual-selection]* · checksum, including the block-version-1 same-file marker), decodes the H5S source/virtual dataspace selections (ALL, NONE, and version-3 regular hyperslabs), reads each same-file source dataset, and scatters its selected elements into the virtual buffer; unmapped regions are left at the zero fill value. External-file sources and N-dimensional selections return a clean unsupported error. The previous parse_vds_mappings used a guessed layout that did not match real files and is replaced.

Bug Fixes

  • clawhdf5-format: read paged Fixed Array chunk indexes. A filtered, fixed-dimension dataset with more than one data-block page (>1024 chunks by default) previously failed with "paged Fixed Array data blocks not yet supported". The reader now walks the page-init bitmap (MSB-first), skips uninitialized pages, and resolves each page's fixed full-size slot (including the short final page). Reverse-engineered and validated end-to-end against an HDF5 2.0 file.
  • clawhdf5-format: read array-typed datatypes (e.g. an array-typed compound member) via read_as_i32/i64/u64/f32/f64 — previously a TypeMismatch. The array is read as a flat sequence of its base elements (recursing for nested arrays), applying base-type precision rules.
  • clawhdf5-format: sign-extend reduced-precision fixed-point integers on read. A signed integer whose datatype precision is smaller than its storage size is stored zero-filled, so e.g. a 16-bit-precision -1 previously read as 65535. The integer read paths now extract the precision field and sign-extend (full-width types are unchanged). Completes signed N-Bit reads and also fixes un-filtered reduced-precision integer datasets.
  • clawhdf5-format: read datasets written by modern HDF5 (1.14+/2.0, i.e. libver=latest). Compound (class 6) and array (class 10) datatype version 5 messages and data layout version 5 messages were rejected as invalid; they reuse the v3/v4 binary structure, so they are now accepted. This unblocks reading compound types and — critically — every chunked/compressed dataset written by HDF5 2.0. Found by running the h5py interop tests against h5py 3.16 / HDF5 2.0.

Performance

  • clawhdf5-format: chunked writes now compress all chunks up front via compress_all_chunks, running across rayon threads under the parallel feature when there are more than 4 filtered chunks. On-disk layout is unchanged. Speeds up compressed embedding writes in clawhdf5-agent (which enables parallel).

Documentation

  • Fix stale package names across all 13 per-crate READMEs (rustyhdf5-* / edgehdf5-* → clawhdf5-*, usage versions → 2.1.0).
  • Correct README workspace/test/crate stats and the CLAUDE.md CLI subcommand list; document the hnsw and format compression/checksum feature flags and the entity_extract / async_memory modules.

v2.1.0 (2026-06-03)

New Features

  • clawhdf5-agent: HNSW now backs the vector stage of hybrid_search. The hnsw feature is on by default, so semantic search uses the approximate clawhdf5-ann index instead of a linear cosine scan. The index mirrors the memory cache (node id == cache index) and self-heals — it rebuilds whenever it drifts from the cache length, so no mutation path can desync it. Non-indexable stores (no/zero-dim/mixed embeddings) and dimension-mismatched queries fall back to the exact linear scan. Disable with --no-default-features --features float16 for exact search.
  • clawhdf5-ann: HNSW is now a live, mutable index — added insert, mark_deleted (soft-delete bitset; deleted nodes are traversed for connectivity but never returned), compact (drops deleted vectors and renumbers survivors), and new (empty index). Serialization gains a format version tag (HNSW_FORMAT_VERSION = 2) and persists the deleted bitset; pre-existing v1 files still load.
  • clawhdf5-agent: hybrid::merge_vector_keyword exposes the shared normalize-and-fuse step used by both the linear and HNSW vector paths.
  • Expose max_dimensions() API on Dataset, MmapDataset, and LazyDataset
  • NetCDF-4 unlimited dimension detection now works correctly
  • Python bindings (clawhdf5-py) build and link on macOS with system Python

Bug Fixes

  • clawhdf5-py: upgrade PyO3 and numpy 0.23 → 0.28 so the bindings build on Python 3.14 (PyO3 0.23 capped at 3.13 and hard-failed cargo build --workspace). Updated for the removed PyObject alias (Py<PyAny>) and the Python::allow_threads → Python::detach rename.
  • Fix GPU L2 distance test (squared vs actual L2 mismatch in test helper)
  • Mark Android JNI functions as unsafe for Rust 2024 edition compliance
  • Add # Safety documentation to all public unsafe extern functions
  • Fix all clippy warnings: needless_range_loop, manual_strip, ptr_arg, etc.
  • Rename RelationType::from_str to from_label to avoid trait confusion
  • Isolate h5py interop tests with #[ignore] when h5py unavailable

Code Quality

  • Full rustfmt pass across workspace (61 files)
  • Refine inner unsafe blocks for Rust 2024 edition style
  • Zero clippy warnings, zero clippy errors across entire workspace
  • 1,546 tests passing, 0 failures

v2.0.0 (2026-03-19)

  • Unified rustyhdf5 (11 crates) and edgehdf5 (4 crates) into a single workspace
  • All crates renamed to clawhdf5-* prefix
  • Version bumped to 2.0.0 across all crates
  • Git dependencies replaced with in-workspace path dependencies
  • Added agent feature flag to clawhdf5-agent
Reference in New Issue View Git Blame Copy Permalink