Files

T

osobhandClaude Opus 5.5 a8fb758489 fix(migrate): write a real clawhdf5-agent store

clawhdf5-migrate wrote a layout of its own (/chunks, /sessions,
/entities, /relations, root attributes, no /meta or schema_version) that
HDF5Memory::open rejects, so a "migrated" SQLite database could not be
used as agent memory — contrary to the README.

It now writes through the agent's own API (HDF5Memory::create/open,
save_batch, the session cache and the knowledge graph), so there is no
second copy of the schema:

- sessions and entities/relations carry over; deleted rows become
  deleted records (or are left out with --skip-deleted);
- embeddings follow the library default (float16), --f32 opts out and
  --float16 is a hidden no-op, as in clawhdf5-cli; the `half`-based
  conversion is gone;
- every source row is checked before the output is created: a wrong
  embedding length, an empty embedding, a dimension that differs from
  an existing store's, or a float16 value beyond +-65504 is an error
  naming the chunk id, and an existing store is left untouched;
- --incremental opens the existing store, adds only rows it does not
  hold (matched by content) and follows the source's deleted flags;
- a source with no memory rows needs --embedding-dim;
- validation reads the result back with HDF5Memory::open_read_only,
  compares every field (embeddings bit for bit, round_to_f16 of the
  source for float16) and checks a migrated record is found by search.

clawhdf5-agent gains HDF5Memory::sessions()/sessions_mut(),
HDF5Memory::delete_batch (one save, all-or-nothing, no auto-compact),
SessionCache::add_at, and re-exports SessionCache/SessionEntry.

The old layout's per-dataset SHA-256 provenance attributes have no place
in the agent schema and are gone. An adversarial review found two
blockers (silent truncation of long embeddings; an --incremental
dimension check that could never fire) and four majors (a failed run
wiping the existing store, dim-0 stores, deleted-flag drift); all are
fixed with regression tests. 42 migrate tests, incl. h5py opening a
migrated store.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

2026-09-24 23:45:15 -05:00

src

fix(migrate): write a real clawhdf5-agent store

2026-09-24 23:45:15 -05:00

Cargo.toml

fix(migrate): write a real clawhdf5-agent store

2026-09-24 23:45:15 -05:00

README.md

fix(migrate): write a real clawhdf5-agent store

2026-09-24 23:45:15 -05:00

README.md

clawhdf5-migrate

CLI tool to migrate SQLite agent memory databases (the ZeroClaw layout) to a clawhdf5-agent store.

The output is written through clawhdf5-agent's own API, so it opens with HDF5Memory::open and is searchable immediately: memory records, sessions and the knowledge graph (entities and relations) are carried over.

Installation

cargo install clawhdf5-migrate

Usage

clawhdf5-migrate --sqlite agent.db --hdf5 agent.h5 --agent-id my-agent

Embeddings are stored as float16 (the library default for new stores); pass --f32 for full precision. Every embedding must have the same dimension (the first row's, or --embedding-dim, which a source with no memory records requires); rows are never truncated, and the whole source is checked before an existing output store is replaced. --incremental adds only new rows to an existing store of the same dimension and carries over changes to rows' deleted flags, --skip-deleted leaves out tombstoned rows, and --dry-run only counts. See clawhdf5-migrate --help for every option.

License

MIT