fix(migrate): write a real clawhdf5-agent store
clawhdf5-migrate wrote a layout of its own (/chunks, /sessions, /entities, /relations, root attributes, no /meta or schema_version) that HDF5Memory::open rejects, so a "migrated" SQLite database could not be used as agent memory — contrary to the README. It now writes through the agent's own API (HDF5Memory::create/open, save_batch, the session cache and the knowledge graph), so there is no second copy of the schema: - sessions and entities/relations carry over; deleted rows become deleted records (or are left out with --skip-deleted); - embeddings follow the library default (float16), --f32 opts out and --float16 is a hidden no-op, as in clawhdf5-cli; the `half`-based conversion is gone; - every source row is checked before the output is created: a wrong embedding length, an empty embedding, a dimension that differs from an existing store's, or a float16 value beyond +-65504 is an error naming the chunk id, and an existing store is left untouched; - --incremental opens the existing store, adds only rows it does not hold (matched by content) and follows the source's deleted flags; - a source with no memory rows needs --embedding-dim; - validation reads the result back with HDF5Memory::open_read_only, compares every field (embeddings bit for bit, round_to_f16 of the source for float16) and checks a migrated record is found by search. clawhdf5-agent gains HDF5Memory::sessions()/sessions_mut(), HDF5Memory::delete_batch (one save, all-or-nothing, no auto-compact), SessionCache::add_at, and re-exports SessionCache/SessionEntry. The old layout's per-dataset SHA-256 provenance attributes have no place in the agent schema and are gone. An adversarial review found two blockers (silent truncation of long embeddings; an --incremental dimension check that could never fire) and four majors (a failed run wiping the existing store, dim-0 stores, deleted-flag drift); all are fixed with regression tests. 42 migrate tests, incl. h5py opening a migrated store. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
@@ -3,9 +3,12 @@
|
||||
[](https://crates.io/crates/clawhdf5-migrate)
|
||||
[](https://docs.rs/clawhdf5-migrate)
|
||||
|
||||
CLI tool to migrate SQLite agent memory databases to HDF5 format.
|
||||
CLI tool to migrate SQLite agent memory databases (the ZeroClaw layout) to a
|
||||
[clawhdf5-agent](https://crates.io/crates/clawhdf5-agent) store.
|
||||
|
||||
Converts existing SQLite-based agent memory stores (embeddings, text chunks, metadata) into the HDF5 format used by [clawhdf5-agent](https://crates.io/crates/clawhdf5-agent).
|
||||
The output is written through `clawhdf5-agent`'s own API, so it opens with
|
||||
`HDF5Memory::open` and is searchable immediately: memory records, sessions and
|
||||
the knowledge graph (entities and relations) are carried over.
|
||||
|
||||
## Installation
|
||||
|
||||
@@ -16,9 +19,19 @@ cargo install clawhdf5-migrate
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
clawhdf5-migrate --input agent.db --output agent.h5
|
||||
clawhdf5-migrate --sqlite agent.db --hdf5 agent.h5 --agent-id my-agent
|
||||
```
|
||||
|
||||
Embeddings are stored as float16 (the library default for new stores); pass
|
||||
`--f32` for full precision. Every embedding must have the same dimension
|
||||
(the first row's, or `--embedding-dim`, which a source with no memory records
|
||||
requires); rows are never truncated, and the whole source is checked before an
|
||||
existing output store is replaced. `--incremental` adds only new rows to an
|
||||
existing store of the same dimension and carries over changes to rows'
|
||||
deleted flags, `--skip-deleted` leaves out tombstoned rows, and `--dry-run`
|
||||
only counts.
|
||||
See `clawhdf5-migrate --help` for every option.
|
||||
|
||||
## License
|
||||
|
||||
MIT
|
||||
|
||||
Reference in New Issue
Block a user