Mission: 01a00c41-bac0-7eb3-a8c8-8b7044f3086d
Phase: 01a00c41-bac2-71e3-a58b-c473421200ee
Committed by the ClawMates delivery pipeline from the agents' working tree. Authored by agents, not by the named committer.
clawhdf5-migrate:
- Real content validation: the post-migration check reads the written HDF5
back (new hdf5_reader) and compares actual content — chunk text, embeddings,
and every session/entity/relation field — to the source, not just row counts.
A representative sample of chunk rows is verified by default; --validate-full
checks every row. A count-preserving corruption no longer passes.
- Configurable schema: SQL is built from a SchemaConfig (table + ordered column
names, defaulting to the ZeroClaw layout) instead of hardcoded queries, with
--chunks-table / --sessions-table / --entities-table / --relations-table.
- Streaming count pass: --dry-run does a COUNT(*)-only pass per table instead
of loading every row.
- Incremental migration: --incremental reads the existing output, reads only
source chunks with id greater than the last migrated id, and appends them
(metadata groups refreshed from source) rather than re-migrating everything.
clawhdf5-format:
- read_as_f32 / read_as_f64 now decode IEEE-754 half-precision (2-byte) floats
via a no_std-safe bit conversion — needed to read float16-stored embeddings
back (e.g. for migrate's content validation), previously a TypeMismatch.
Tests: f16 read unit test; migrate tests for content-corruption detection,
custom table names, and incremental append; CLI smoke-tested end-to-end and the
dense/incremental output verified with h5py.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Omar Sobh