Commit Graph
98 Commits
Author SHA1 Message Date
Omar SobhandClaude Sonnet 4.6 3a1fcc5cb3 docs: add standalone benchmark.md with clawhdf5 vs libhdf5 comparison
Full head-to-head results from Criterion suite (100 samples each):
sequential read/write, chunked write + deflate, metadata ops, group
traversal. Includes interpretation section explaining the structural
reasons for each gap.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-06-30 19:26:16 +00:00
Omar SobhandClaude Sonnet 4.6 bf197b70e3 docs+fix: add h5bench benchmark results and repair libhdf5-compare feature
Add h5bench-equivalent Criterion benchmark results to BENCHMARKS.md
(sequential read/write, chunked read/write, metadata throughput).

Fix libhdf5-compare feature for HDF5 1.14.x:
- Switch to hdf5-metno 0.12 (aliased as 'hdf5') in clawhdf5-bench
- Fix h5bench_meta.rs: AttributeBuilderEmpty::create takes &str not &String;
  shape=[1] dataset uses write(&[val]) not write_scalar
- Fix h5bench_read.rs: libhdf5-compare variant now writes its own reference
  file via hdf5-metno instead of dumping clawhdf5 bytes (avoids float
  datatype message incompatibility)

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-06-30 16:39:52 +00:00
Omar SobhandClaude Sonnet 4.6 cb0b0e9df2 fix: correct libaec constants, HDF5→libaec option mapping, and VDS serialization
Critical fixes from whole-branch code review:
- libaec-sys: fix flag constants to match <libaec.h> exactly
  (PREPROCESS=8, MSB=4, RESTRICTED=16; drop non-existent AEC_ALLOW_K13)
  and add aec_buffer_encode FFI declaration
- filters_szip: fix cd index for bits_per_sample (cd[2] per H5Z_SZIP_PARM_BPP,
  not cd[4]); fix option-mask mapping (NN=0x20, MSB unconditional); add two
  real encode→decode roundtrip tests (no-NN and NN) that exercise libaec end-to-end
- file_writer: fix serialize_vds_mappings to delegate to data_layout_write
  (eliminates the buggy duplicate that always emitted version=1 even for
  external-file mappings); retains trailing Jenkins checksum

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-06-30 11:41:41 +00:00
Omar SobhandClaude Sonnet 4.6 e91f7fc539 fix: correct libaec FFI to use aec_stream struct (fixes SIGSEGV)
The previous aec_buffer_decode declaration used flat parameters which
don't match the actual libaec C API; this caused a SIGSEGV at runtime.
Replace with the correct aec_stream struct (mirroring <libaec.h>) and
update filters_szip.rs to populate and pass &mut AecStream.
Also add empty-input guard in szip_decode_impl and fallback library
path search in build.rs for distros that omit the .pc file.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-06-30 11:41:41 +00:00
Omar SobhandClaude Sonnet 4.6 d6c4d4f111 feat: implement filter codecs, format write extensions, and MPI-IO VOL
All three SDD plans fully wired and committed to main:

Filter Codecs (FC):
- FC-1: Implement float E-scale in scaleoffset_decompress (value = minval +
  code * 2^E, negative exponents via cast to i32); add two round-trip tests.
- FC-2: filters_szip.rs — feature-gated SZIP decode via libaec FFI; SZIP
  dispatch arm added to decompress_chunk.
- FC-3: libaec-sys workspace crate with pkg-config probe and aec_buffer_decode
  FFI binding; added to workspace members.

Format Write Extensions (FWE):
- FWE-1: GroupBuilder::add_external_link() API; wired through FinishedGroup
  → GrpFlat → file_writer pass 1/2/3 (OH size, layout cursor, final write);
  external_link_write_roundtrip test.
- FWE-2: data_layout_write.rs — serialize_vds_mappings with length_size param
  and version 0/1 (external vs same-file) selection; declared as pub mod.
- FWE-3: with_virtual_sources empty-mapping guard (Important #9) — empty vec
  is silently ignored; vds_empty_mapping_list test updated to assert non-VDS
  layout results.

MPI-IO VOL Backend (MPI):
- MPI-1/2/3: mpi_vol.rs — MpiVol implementing VirtualObjectLayer; root-read
  + broadcast collective read; gather + root-write collective write; feature-
  gated mpi-io feature; wired into clawhdf5-io lib.rs.
- MPI-4: mpi_io_bench binary (h5bench-equivalent MPI-IO throughput bench).

mpi_vol.rs reviewer fixes:
- Doc-comment updated to accurately describe root-read+broadcast pattern
  (not MPI_File_read_at); MpiVol::expected_capabilities() associated fn
  added so tests can verify capabilities without a live MPI universe;
  rank_and_size_stub_values renamed to no_feature_error_contains_feature_name.

Workspace check: zero warnings, 20 test suites pass.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-06-30 11:41:41 +00:00
Omar SobhandClaude Sonnet 4.6 90bdd7cd13 feat: add h5bench-equivalent Criterion benchmarks to clawhdf5-bench
Adds three Criterion benchmark suites mirroring the h5bench HPC I/O
benchmark workloads in pure Rust — no C libhdf5 required for the default
path, with an optional `libhdf5-compare` feature for side-by-side numbers.

  - benches/h5bench_write.rs: write_1d_contiguous, write_2d_chunked,
    write_f64_batch, write_multi_dataset, write_with_attrs
  - benches/h5bench_read.rs: read_sequential, read_f64_sequential,
    read_chunked_2d, read_from_disk, read_hyperslab
  - benches/h5bench_meta.rs: metadata_attrs_write, metadata_attrs_read,
    metadata_groups_create, metadata_groups_traverse, metadata_string_attrs

All benchmarks pass `cargo bench --bench <name> -- --test` and clippy
reports zero warnings.  Run with `cargo bench -p clawhdf5-bench`.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-06-30 11:41:41 +00:00
Omar SobhandClaude Sonnet 4.6 28a0dc3384 feat: add Virtual Dataset (VDS) write support and round-trip tests
Add `virtual_sources: Option<Vec<VdsMapping>>` field and `with_virtual_sources()` method to `DatasetBuilder`. In `FileWriter::finish()`, VDS datasets skip raw-data storage and instead serialize their source mappings into a global heap collection (version-1 same-file encoding) referenced by an HDF5 v4 layout-class-3 message. The two-pass address-computation loop handles VDS in both passes: pass 1 computes the fixed-size OH and pre-builds the heap blob; pass 2 places the blob at the correct file offset and rebuilds the OH with the real global heap address. Three new tests verify: (a) same-file two-source round-trip with mapping verification, (b) external-file source encoding, and (c) empty mapping list.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-06-30 11:41:41 +00:00
osobh bae80d030b Update README.md 2026-06-29 23:58:43 +00:00
osobhandClaude Opus 4.8 8534c7d204 feat: migrate-engine improvements (content validation, schema, streaming, incremental)
clawhdf5-migrate:
- Real content validation: the post-migration check reads the written HDF5
  back (new hdf5_reader) and compares actual content — chunk text, embeddings,
  and every session/entity/relation field — to the source, not just row counts.
  A representative sample of chunk rows is verified by default; --validate-full
  checks every row. A count-preserving corruption no longer passes.
- Configurable schema: SQL is built from a SchemaConfig (table + ordered column
  names, defaulting to the ZeroClaw layout) instead of hardcoded queries, with
  --chunks-table / --sessions-table / --entities-table / --relations-table.
- Streaming count pass: --dry-run does a COUNT(*)-only pass per table instead
  of loading every row.
- Incremental migration: --incremental reads the existing output, reads only
  source chunks with id greater than the last migrated id, and appends them
  (metadata groups refreshed from source) rather than re-migrating everything.

clawhdf5-format:
- read_as_f32 / read_as_f64 now decode IEEE-754 half-precision (2-byte) floats
  via a no_std-safe bit conversion — needed to read float16-stored embeddings
  back (e.g. for migrate's content validation), previously a TypeMismatch.

Tests: f16 read unit test; migrate tests for content-corruption detection,
custom table names, and incremental append; CLI smoke-tested end-to-end and the
dense/incremental output verified with h5py.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
pre-rewrite-2026-06-05
2026-06-04 02:19:31 +00:00
osobhandClaude Opus 4.8 0754afb7f2 feat: write multi-block fractal heaps (root indirect block)
Dense attribute and dense link storage capped at a single fractal-heap direct
block (~64 KiB of heap data — a few thousand objects); beyond that the writer
produced an invalid oversized block. Lift the cap with a root indirect block.

When the serialized objects don't fit in one direct block, build a root
indirect block (FHIB) over multiple direct blocks sized by the doubling table
(start 512, width 4, doubling per row up to 64 KiB). Objects are packed
row-major across blocks, each block carries its logical block offset, and heap
IDs encode each object's heap offset (block offset + position). The FRHP points
root -> FHIB with the row count; unused slots in the current rows are undefined.

The fractal-heap builder is unified: FractalHeapBlock now carries the full heap
blob, and a shared write_frhp helper serializes the header for both the
single-block and multi-block paths. The single-block path is unchanged
(byte-identical), so existing dense attrs/links stay valid.

Validated end-to-end: a 2,500-attribute object and a 2,500-link group round-trip
through our reader and are read correctly by h5py. Objects still may not span a
block (no huge-object path).

Tests: facade round-trips for multi-block dense attrs and dense links, plus an
h5py-gated interop test (verified against the real h5py environment).

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-06-04 01:37:42 +00:00
osobhandClaude Opus 4.8 0aab49f2f0 fix: read multi-direct-block fractal heaps (root indirect block)
The fractal-heap reader split direct vs indirect block rows using the FRHP
"Starting # of Rows in Root Indirect Block" field (a constant, typically 1),
mislabeled as starting_row_of_indirect_blocks. For any heap whose data spans
more than one direct block — common in libhdf5 files with a large group or
many dense attributes — this treated direct blocks as indirect and walked into
garbage, failing with InvalidFractalHeapSignature.

Derive the split from the heap geometry instead: max_direct_rows =
log2(max_direct_block_size / starting_block_size) + 2. Rows below it hold
direct blocks; rows at/above hold child indirect blocks.

Validated against an h5py-written group with 400 dense attributes (root
indirect block, 4 rows, 13 direct blocks): all values now read correctly.
Regression fixture covers an 80-attribute multi-block heap.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-06-04 01:27:45 +00:00
osobhandClaude Opus 4.8 20ad16ab69 feat: write dense group link storage (fractal heap + v2 B-tree)
A group with more than 8 links (libhdf5's compact max_compact default) is now
written densely instead of as inline Link messages: the links live in a
single-direct-block fractal heap indexed by a v2 B-tree of type 5 (link-name
index), referenced from the group's LinkInfo message. This matches libhdf5's
compact->dense switchover and keeps large groups out of the object header.

- Extract the byte-identical fractal-heap builder shared by dense attributes
  and dense links, parameterized by heap_id_length / max_heap_size. Attributes
  keep 8 / 40; links use 7 / 32 to match libhdf5 (reverse-engineered: an
  h5py-written dense group uses heap_id_length 7, max_heap_size 32, type-5
  record = hash(4) + heap_id(7) = 11 bytes). This was the cause of an initial
  "object overruns end of direct block" error from h5py.
- build_group_oh gained an optional dense LinkInfo (omitting inline Link
  messages); the two-pass file assembly allocates each group's link blob after
  its object header and rebuilds it with real target addresses in the final
  pass (link-message size is address-independent, so layout is stable).

Validated end-to-end: our reader round-trips dense groups, h5py reads the
groups we write, and dense attributes remain byte-identical (still h5py-valid).
The agent's 9-dataset memory group now writes densely and round-trips. Single
direct block only (~a couple thousand links); indirect blocks remain a TODO.

Tests: facade round-trip (20-link dense group + compact sibling) and an
h5py-gated interop test confirming libhdf5 reads our dense groups.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-06-04 01:12:23 +00:00
osobhandClaude Opus 4.8 e0189cd5c4 harden: make the new readers panic-free on malformed input
The readers added this cycle parse untrusted bytes, so malformed/hostile
input must produce errors — never a panic, OOM, or unbounded recursion.
Audited each new surface and fixed the concrete vectors, each covered by an
adversarial regression test:

- Paged Fixed Array: `1 << max_nelmts_bits` shift overflow (u8 up to 255);
  element-count bounded by file size; element/page offset multiplies checked.
- H5S selection decoder: ALL/NONE validate they have the 16 bytes they claim
  to consume; hyperslab rank capped at 32 (H5S_MAX_RANK); iter_linear
  coordinate/stride/product arithmetic uses checked ops.
- VDS mapping parser: drop pre-allocation from the untrusted `nused`;
  bounds-check all selection slicing.
- scale-offset / N-Bit filters: `1 << minbits` overflow at minbits==64; N-Bit
  `bit_offset + precision` overflow; N-Bit type-tree recursion depth capped to
  stop a crafted nested tree from overflowing the stack; element counts bounded
  by the chunk's expected decompressed size (threaded the previously-unused
  chunk_size into both decoders) so a bogus count can't over-allocate.
- VDS assembly: a virtual dataset whose source is itself virtual (a cycle) now
  errors instead of recursing into a stack overflow.

16 new adversarial tests; full format suite (482 lib) + agent + facade green;
clippy clean.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-06-04 00:28:33 +00:00
osobhandClaude Opus 4.8 4fa7e89a46 feat: compress fixed-length string datasets (+ fix shared chunk-cache bug)
clawhdf5-agent: fixed-length string datasets (memory text chunks, session
summaries, ids, tags, entity/relation names) were stored uncompressed behind
a stale "chunked compound not yet supported" comment. Chunked writes work for
fixed-size string/compound datatypes like any other, so write_string_dataset
now chunks + deflates once a dataset's payload reaches 4 KiB — large,
redundant NullPad content compresses well while tiny metadata stays
contiguous (no chunk-overhead bloat). The dead `compress` parameter is
removed in favor of this size heuristic.

clawhdf5-format: enabling string compression exposed a latent bug — the
per-file ChunkCache built its chunk index once and reused it for every
chunked dataset in the file, keyed only by chunk coordinate with no dataset
discrimination. With one chunked dataset per file this never surfaced; with
two of different rank (a 1-D compressed string array and the 2-D embeddings
matrix) the first dataset's rank-1 index was reused for the second, panicking
with an out-of-bounds chunk coordinate. The cache now binds to a dataset by
its chunk-index address and rebinds — dropping the index, chunk-index map,
layout, and decompressed slots — whenever the dataset being read changes,
while still caching repeated/sequential access to the same dataset.

Tests: facade regression reading a 1-D compressed string dataset and a 2-D
compressed f32 dataset through one shared File cache (verified to panic
without the fix); existing agent e2e tests (large text chunks, migration
round-trip) now pass with compression on.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-06-03 22:48:39 +00:00
osobhandClaude Opus 4.8 57adc88320 test: regression for scale-offset float E-scale (raw + masked filter)
The HDF5 library does not implement the scale-offset filter's floating-point
E-scale mode. When asked for it (cd_values[0] = 1) it stores the chunk raw
(no minbits/minval header) and sets the chunk filter mask to skip the filter,
so such datasets read back verbatim purely by honoring the per-chunk filter
mask — no E-scale decoder is required.

Add a fixture written via the HDF5 low-level API (exact-representable values)
and a test asserting it reads back verbatim, locking in the filter-mask path.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-06-03 22:14:01 +00:00
osobhandClaude Opus 4.8 e6f0d8f161 feat: read external-file Virtual Datasets via a source resolver
VDS sources living in other files were previously unsupported because the
pure-byte read API has no filesystem. Add a resolver seam and wire a default.

clawhdf5-format:
- Add VdsSourceResolver (Fn(&str) -> Option<Vec<u8>>) and
  read_raw_data_full_with_resolver. read_virtual_data uses the resolver to
  fetch an external source file's bytes by its stored name, then reads the
  named source dataset from those bytes and scatters as usual. A resolver
  returning None leaves the region at fill (HDF5's missing-source behavior);
  an external source with no resolver at all is a clean error. read_raw_data_full
  is unchanged (delegates with no resolver).

clawhdf5:
- File now records the directory it was opened from and, for virtual layouts,
  reads through a default resolver that loads sibling source files relative to
  that directory. So File::open(virt).dataset(d).read_*() transparently
  assembles cross-file VDS. In-memory files (from_bytes) have no directory, so
  only same-file VDS resolves there.

Tests: format-layer external read with an injected resolver (and the
no-resolver error path), plus facade tests that drop both files in a temp dir
and read through File::open — covering successful resolution and the
missing-source-is-fill case.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-06-03 21:19:36 +00:00
osobhandClaude Opus 4.8 98ccc69411 feat: extend same-file VDS assembly to N dimensions
Generalize Selection iteration from 1-D to arbitrary rank: iter_linear(dims)
enumerates a selection's row-major linear indices over a dataspace of the
given shape (ALL, NONE, regular hyperslabs, points), which is the order HDF5
uses to pair virtual and source selections.

read_virtual_data now passes the full virtual/source dimensions instead of a
single extent, so multi-dimensional block mappings scatter to the correct
non-contiguous linear positions. read_named_dataset_raw returns the source
dataset's dimensions. The rank-1 restriction is removed; only external-file
sources remain unsupported.

Tests: 2-D integration fixture (vds_2d_same_file.h5: two 2x2 sources placed
as non-contiguous blocks in a 4x4 virtual) plus N-D iter_linear unit tests
(block, strided, ALL, rank-mismatch). The 1-D path is unchanged.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-06-03 20:57:12 +00:00
osobhandClaude Opus 4.8 908af40282 feat: assemble 1-D same-file Virtual Datasets (VDS)
A virtual layout previously returned UnsupportedVersion. Implement reading
for the common 1-D, same-file case, reverse-engineered and validated against
HDF5 2.0.

- Rewrite parse_vds_mappings to the real global-heap block format
  (version(1) · nused(length_size) · entries · checksum(4)), where each
  entry is source-file(null) · source-dataset(null) · source-selection ·
  virtual-selection. Block version 1 encodes a same-file source as a single
  0x04 marker in place of the file name; version 0 stores an explicit file
  name. The selections are H5S-serialized and self-describing in length, so
  they are decoded to find entry boundaries. The previous parser used a
  guessed layout that did not match real files.

- Extend Selection with decode_serialized() (H5S_select_serialize: ALL,
  NONE, and version-3 regular hyperslabs) and iter_linear_1d().

- Add read_virtual_data: resolve the mapping block from the global heap,
  read each same-file source dataset, and scatter its selected elements into
  the virtual buffer; unmapped regions stay at the zero fill value.
  External-file sources and N-D selections return a clean unsupported error.

Tests: real-file integration test (vds_same_file.h5: partial source slice +
fill gap), selection decoder unit tests built from the fixture bytes, and
same-file/external mapping-parser unit tests.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-06-03 20:01:51 +00:00
osobhandClaude Opus 4.8 a24fcb8be4 feat: read paged Fixed Array chunk indexes
A filtered, fixed-dimension dataset with more than one Fixed Array
data-block page (>1024 chunks by default) previously failed with
"paged Fixed Array data blocks not yet supported".

Implement the paged data-block layout, reverse-engineered and validated
against an HDF5 2.0 file:
- after the FADB prefix: a page-init bitmap (one bit per page, MSB-first
  within each byte), a 4-byte checksum, then the pages;
- each page is a fixed full-size slot of page_nelmts elements plus a
  4-byte checksum, with only the final page shorter;
- uninitialized pages still occupy their slot (zero-filled), so the
  bitmap — not a 0xFF sentinel — marks a whole page unallocated.

Element parsing is factored into parse_fa_element, shared by the
non-paged and paged paths.

Tests: real-file integration test against a minimal 2-page gzip fixture
(v4_fixed_array_paged.h5) plus a synthetic unit test covering a
multi-byte/MSB-first bitmap, a skipped uninitialized page, and a short
final page.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-06-03 17:41:31 +00:00
osobhandClaude Opus 4.8 4b1f4e369a docs: changelog for array-typed datatype reads
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-06-03 20:12:33 +00:00
osobhandClaude Opus 4.8 c99fb39ffd feat: read array-typed datatypes (incl. array compound members)
The typed read paths (read_as_i32/i64/u64/f32/f64) rejected Array datatypes
with a TypeMismatch, so an array-typed compound member (common with N-Bit /
reduced-precision data) could not be read. They now unwrap an Array to its base
type and read the flat sequence of base elements, recursing for nested arrays.
Base-type precision rules (e.g. reduced-precision sign extension) apply to the
elements.

Validated end-to-end against an HDF5 2.0 compound with an array member: the
array field reads [-1, 100, 1000, -32768] with correct 16-bit sign extension.
Adds a regression test for flat and nested array reads.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-06-03 20:12:21 +00:00
osobhandClaude Opus 4.8 dbd683dcaf docs: changelog for compound/array N-Bit support
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-06-03 13:04:22 +00:00
osobhandClaude Opus 4.8 06a1ef5285 feat: decode compound and array N-Bit layouts
Generalizes the N-Bit filter (id 5) decoder from atomic-only to the full
recursive type tree carried in the filter client data: atomic
([1, size, order, precision, offset]), array ([2, total_size, <base>]) and
compound ([3, total_size, nmembers, (offset, <node>)*]), nestable to any depth.

The decoder parses the tree once, then walks it per element with an MSB-first
bit reader, placing each leaf field's significant bits at its byte/bit offset in
a zero-filled element — HDF5's canonical layout. Float members are encoded as
full-precision atomics and handled transparently. Validated end-to-end against
HDF5 2.0 / h5py: compound int+int, compound with an array member, and compound
with a float member all decode to the exact canonical bytes. Adds h5py-free unit
tests from real captured chunks. (Reading array-typed compound *fields* into a
flat buffer is a separate datatype-reader concern.)

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-06-03 13:04:08 +00:00
osobhandClaude Opus 4.8 8c68b5de33 docs: changelog for reduced-precision integer sign-extension
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-06-03 12:48:23 +00:00
osobhandClaude Opus 4.8 249841e232 fix: sign-extend reduced-precision fixed-point integers on read
HDF5 stores a fixed-point value whose datatype precision is smaller than its
storage size zero-filled above the precision; the sign of a reduced-precision
signed integer lives in the precision field, not the storage word, and is
applied during datatype conversion. clawhdf5 previously read the full storage
word, so e.g. a 16-bit-precision -1 (stored 0x0000ffff) read as 65535.

The integer read paths (read_as_i32/i64/u64/f32/f64) now extract the
[bit_offset, bit_offset+bit_precision) field and sign-extend (signed) or mask
(unsigned). Full-width types are unchanged — the bulk-copy fast paths are gated
to full width, so the common case keeps its memcpy and behaviour.

This completes signed N-Bit reads (now exact end-to-end) and also fixes
un-filtered reduced-precision signed/unsigned integer datasets. Validated
against HDF5 2.0 / h5py; adds h5py-free regression tests.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-06-03 12:48:08 +00:00
osobhandClaude Opus 4.8 bff039fa29 docs: changelog for float D-scale and N-Bit filter support
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-06-03 12:36:50 +00:00
osobhandClaude Opus 4.8 4a5ab1c584 feat: decode the HDF5 N-Bit filter (atomic variant)
Implements decompression for the N-Bit filter (id 5), atomic integer/float
variant — previously returned UnsupportedFilter(5). N-Bit packs each element's
significant `precision` bits MSB-first with no header; decode reads those bits
per element and places them at the datatype's bit offset in a zero-filled
`size`-byte element, reproducing HDF5's canonical reduced-precision layout
(verified byte-for-byte against the equivalent un-filtered dataset).

Reverse-engineered and validated against HDF5 2.0 / h5py: unsigned
reduced-precision datasets now read end-to-end with exact values. Signed
reduced-precision values are restored to their canonical (zero-filled) bytes;
sign-extending them to the application width is the datatype reader's job — a
pre-existing concern shared with un-filtered reduced-precision data. Recursive
compound/array N-Bit layouts remain unsupported. Adds h5py-free unit tests from
real captured chunks.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-06-03 12:36:13 +00:00
osobhandClaude Opus 4.8 9062b3fb53 feat: decode the float D-scale scale-offset variant
Extends the scale-offset filter (id 6) decoder to the floating-point D-scale
variant (H5Z_SO_FLOAT_DSCALE) alongside the integer variant. Shares the header
parsing and MSB-first code unpacking; reconstruction is
`value = minval + code / 10^scale_factor`, where `minval` is the minimum float
stored in the header and the all-ones code is the (defined) fill value.

Reverse-engineered and validated against HDF5 2.0 / h5py across f32 and f64,
negatives, decimal scale factors D=1..5 and multi-chunk datasets (decoded values
match h5py to full precision). The float E-scale variant remains unsupported.
Adds h5py-free unit tests from real captured chunks.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-06-03 12:28:20 +00:00
osobhandClaude Opus 4.8 ec7357de45 docs: changelog entry for scale-offset filter support
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-06-03 12:18:55 +00:00
osobhandClaude Opus 4.8 6ab42c2f07 feat: decode the HDF5 scale-offset filter (integer variant)
Implements decompression for the scale-offset filter (id 6), integer mode
(H5Z_SO_INT) — previously returned UnsupportedFilter(6). The on-disk format was
reverse-engineered against HDF5 2.0 / h5py and verified across signed/unsigned
element sizes, negative minima, multi-chunk datasets and fill-value handling:

  minbits (u32 LE) | 0x08 | minval (8 bytes LE) | 8 reserved bytes |
  MSB-first packed codes (nelmts * minbits bits)

Each code is `value - minval`; the all-ones code is reserved for the (defined)
fill value. The floating-point variants (D-scale/E-scale) use a different
algorithm and remain reported as unsupported.

Validated end-to-end (a 200-element chunked scale-offset dataset, plus negative
and unsigned datasets, now decode to the exact h5py values). Adds h5py-free unit
tests using real captured compressed chunks.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-06-03 12:18:42 +00:00
osobhandClaude Opus 4.8 19ca662975 docs: changelog entry for HDF5 2.0 (version-5) read fixes
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-06-03 12:01:58 +00:00
osobhandClaude Opus 4.8 bc3a3a977a fix: read array datatypes and chunked layouts from HDF5 2.0 (version 5)
Follow-up to the v5 compound fix, found by an interop sweep over diverse
h5py/HDF5 2.0 (libver=latest) datasets:

- Array datatype (class 10) version 5 was rejected. v3/v4/v5 share the same
  array encoding, so the parser now accepts 3-5.
- Data Layout message version 5 was rejected, which broke EVERY chunked/
  compressed dataset written by modern HDF5. v5 reuses the v4 message
  structure, so it now routes through parse_v4.

Validated end-to-end: a gzip-compressed, Fixed-Array-indexed v5 chunked dataset
now decodes to the correct values. Adds h5py-free regression tests using the
real v5 array-datatype and chunked-layout bytes, and updates the layout
invalid-version test to use v6.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-06-03 12:00:38 +00:00
osobhandClaude Opus 4.8 a13ff51918 fix: read compound datatypes from HDF5 1.14+/2.0 (datatype version 5)
clawhdf5 rejected datatype message version 5 for the compound class with
"invalid datatype version 5 for class 6", so it could not read compound
datasets written by modern HDF5 / h5py with libver=latest. v5 reuses the same
compact member encoding as v3/v4 (name, variable-width offset, member type), so
the parser now accepts versions 3-5 for compound.

Found by running the previously-ignored h5py interop tests against h5py 3.16 /
HDF5 2.0. Adds an h5py-free regression test using the real v5 datatype bytes.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-06-03 11:55:15 +00:00
osobhandClaude Opus 4.8 3ff501c8ef docs: add Unreleased changelog section for post-2.1.0 changes
Records the parallel chunk-compression perf change and the doc sweep that
landed after the v2.1.0 tag.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-06-03 11:42:43 +00:00
osobhandClaude Opus 4.8 49a99a9a40 docs: document hnsw/format feature flags and missing agent modules
- Add the `hnsw` flag (default-on) to the agent feature table and the
  fast-deflate/system-zlib/fast-checksum/lz4/zstd/blake3 flags to the format
  table.
- Add entity_extract and async_memory to the agent module overview.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-06-03 11:40:41 +00:00
osobhandClaude Opus 4.8 b9fac46ea5 perf: wire parallel chunk compression into the write path
build_chunked_data_at_ext now compresses all chunks via compress_all_chunks
(previously dead code) before laying them out, so compression runs across
rayon threads under the `parallel` feature when there are >4 filtered chunks.
Layout is unchanged — compression preserves chunk order, so on-disk bytes are
identical to the sequential path. The agent crate enables `parallel`, so this
speeds up compressed embedding writes.

Removes the #[allow(dead_code)] on compress_all_chunks and gates
PARALLEL_COMPRESS_THRESHOLD behind the `parallel` feature.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-06-03 11:40:40 +00:00
osobhandClaude Opus 4.8 f1762f82a7 docs: fix stale package names, counts, and CLI subcommands
Sweep of the docs after the v2.0.0 rename and recent changes:
- Per-crate READMEs (13 files): rename leftover rustyhdf5-*/edgehdf5-*
  package names and badges to clawhdf5-*, bump usage versions to 2.1.0.
- README: update stale test badge (417 -> 1500+), workspace stats
  (15 crates/72K -> 17 crates/84K), agent crate stats (20.7K/32 modules),
  and add the missing clawhdf5-napi and clawhdf5-bench crates to the tree.
- CLAUDE.md: correct the CLI subcommand list (inspect/dump/index/search ->
  the actual create/save/search/recall/stats/flush-wal/agents-md/export/snapshot).

No code changes. Verified there are zero todo!()/unimplemented!() macros and
no TODO/FIXME comments in the tree.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-06-03 11:17:04 +00:00
osobhandClaude Opus 4.8 41a8fb7734 docs: finalize CHANGELOG for v2.1.0 release
Fold the Unreleased HNSW / Python 3.14 entries into a single v2.1.0
(2026-06-03) section being cut and tagged now.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
v2.1.0
2026-06-03 10:39:34 +00:00
osobhandClaude Opus 4.8 a16e9ab8cb chore: bump workspace to 2.1.0
Bumps [workspace.package] and all 17 crate package versions (and internal
path-dependency requirements) from 2.0.0 to 2.1.0 for a coordinated release.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-06-03 10:36:17 +00:00
osobhandClaude Opus 4.8 c304c7e299 chore: bump clawhdf5-agent to 2.1.0 for HNSW integration
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-06-03 10:26:29 +00:00
osobhandClaude Opus 4.8 4e2f4c3d2a docs: changelog entry for HNSW integration and Python 3.14 build fix
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-06-03 10:10:31 +00:00
osobhandClaude Opus 4.8 48a0bd23aa docs: note HNSW is the default vector-search backend in README
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-06-03 10:04:06 +00:00
osobhandClaude Opus 4.8 5f5626e47f docs: note HNSW is the default agent search backend in CLAUDE.md
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-06-03 10:03:10 +00:00
osobhandClaude Opus 4.8 a2bf1ab27e Merge HNSW agent integration + Python 3.14 fix
Lands the changes from PR #1, whose server-side squash merge landed as an
empty commit (Gitea merge working-tree error). See branch
feat/hnsw-agent-integration-py314 for the full history.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-06-03 08:48:04 +00:00
osobhandClaude Opus 4.8 8f9dbd812c feat: integrate HNSW into agent search, fix Python 3.14 build
Resolves two gaps found in a project-state review:

1. Python build was broken: PyO3/numpy 0.23 caps at Python 3.13 but the
   environment has 3.14. Bumped to 0.28 and updated the two breaking APIs
   (PyObject -> Py<PyAny>, allow_threads -> detach). The extension module now
   imports and round-trips under Python 3.14, unblocking cargo build --workspace.

2. The "HNSW vector search over agent memories" headline was unwired:
   clawhdf5-ann had zero dependents and the agent used a linear cosine+BM25 scan.
   - clawhdf5-ann is now a live index: insert, mark_deleted (soft delete with a
     deleted bitset, traversed but never returned), compact, and a format
     version tag (v2) with backward-compatible load of v1 files.
   - clawhdf5-agent wires HNSW behind the `hnsw` feature (ON by default). The
     index mirrors the cache (node id == cache index) and self-heals: it rebuilds
     whenever hnsw_synced_len drifts from cache.len(), so unhooked pushes can't
     desync it. Non-indexable stores (no/zero-dim/mixed embeddings) and queries
     whose dim doesn't match fall back to the exact linear scan.
   - hybrid.rs gains merge_vector_keyword, shared by the linear and HNSW paths.
   - tests/hnsw_integration.rs validates recall vs a brute-force oracle plus
     insert/delete/batch behaviour.

Disable HNSW for exact search with `--no-default-features --features float16`.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-06-03 07:10:48 +00:00
osobh ce842fca05 feat: integrate HNSW into agent search, fix Python 3.14 build (#1) 2026-06-03 00:46:09 +00:00
Omar Sobh d1f74702c0 refactor: use std::io::Error::other() instead of Error::new(ErrorKind::Other, e) 2026-05-21 09:40:53 -07:00
redclawsystems 3f222f6956 Merge pull request 'docs(clawhdf5): document DType variants, fix unresolved doc links' (#17) from sdlc-docs/clawhdf5-types-20260514-165210 into main 2026-05-14 23:54:48 +00:00