clawhdf5

87 Commits 1 Branch 12 Tags

3 Commits

Include renames

Author	SHA1	Message	Date
Omar Sobh	55959b4920	ci: wire up CI, fix no_std build, fix stale package names in scripts CI / test (push) Failing after 15s Details - Add .gitea/workflows/ci.yml running scripts/ci-test.sh (fmt, clippy, test, no_std check) on push/PR to main. - Fix stale rustyhdf5-py/rustyhdf5-format package names in ci-test.sh/check-nostd.sh, which had been silently no-op'ing those checks (cargo warns but doesn't fail on an unknown --exclude/-p target). - With those checks actually running, fix the real issues they surface: - clippy: useless_conversion in chunked_write.rs, byte_char_slices in global_heap.rs/object_header.rs. - cargo fmt: apply formatting across the workspace (whitespace only). - no_std (thumbv7em-none-eabihf) build errors in clawhdf5-format: core::sync::atomic::AtomicU64 doesn't exist on that target (no native 64-bit atomics) — switch profiling.rs's counters to portable-atomic, which falls back to a CAS-based emulation there and is a no-op wrapper elsewhere. Add missing alloc imports for Box (filters.rs), Vec (filters_szip.rs), and format! (dict_encoding.rs) on no_std paths. Replace f64::powi (std/libm-only) with a small local exponentiation-by-squaring helper in the scale-offset filter.	2026-08-05 10:50:13 -07:00
osobhandClaude Opus 4.8	8534c7d204	feat: migrate-engine improvements (content validation, schema, streaming, incremental) clawhdf5-migrate: - Real content validation: the post-migration check reads the written HDF5 back (new hdf5_reader) and compares actual content — chunk text, embeddings, and every session/entity/relation field — to the source, not just row counts. A representative sample of chunk rows is verified by default; --validate-full checks every row. A count-preserving corruption no longer passes. - Configurable schema: SQL is built from a SchemaConfig (table + ordered column names, defaulting to the ZeroClaw layout) instead of hardcoded queries, with --chunks-table / --sessions-table / --entities-table / --relations-table. - Streaming count pass: --dry-run does a COUNT(*)-only pass per table instead of loading every row. - Incremental migration: --incremental reads the existing output, reads only source chunks with id greater than the last migrated id, and appends them (metadata groups refreshed from source) rather than re-migrating everything. clawhdf5-format: - read_as_f32 / read_as_f64 now decode IEEE-754 half-precision (2-byte) floats via a no_std-safe bit conversion — needed to read float16-stored embeddings back (e.g. for migrate's content validation), previously a TypeMismatch. Tests: f16 read unit test; migrate tests for content-corruption detection, custom table names, and incremental append; CLI smoke-tested end-to-end and the dense/incremental output verified with h5py. Co-Authored-By: Claude Opus 4.8 <[email protected]>	2026-06-04 02:19:31 +00:00
redclawsystems	3f222f6956	Merge pull request 'docs(clawhdf5): document DType variants, fix unresolved doc links' (#17 ) from sdlc-docs/clawhdf5-types-20260514-165210 into main	2026-05-14 23:54:48 +00:00

Author

SHA1

Message

Date

Omar Sobh

55959b4920

ci: wire up CI, fix no_std build, fix stale package names in scripts

CI / test (push) Failing after 15s

Details

- Add .gitea/workflows/ci.yml running scripts/ci-test.sh (fmt, clippy,
  test, no_std check) on push/PR to main.
- Fix stale rustyhdf5-py/rustyhdf5-format package names in
  ci-test.sh/check-nostd.sh, which had been silently no-op'ing those
  checks (cargo warns but doesn't fail on an unknown --exclude/-p
  target).
- With those checks actually running, fix the real issues they surface:
  - clippy: useless_conversion in chunked_write.rs, byte_char_slices in
    global_heap.rs/object_header.rs.
  - cargo fmt: apply formatting across the workspace (whitespace only).
  - no_std (thumbv7em-none-eabihf) build errors in clawhdf5-format:
    core::sync::atomic::AtomicU64 doesn't exist on that target (no
    native 64-bit atomics) — switch profiling.rs's counters to
    portable-atomic, which falls back to a CAS-based emulation there
    and is a no-op wrapper elsewhere. Add missing alloc imports for
    Box (filters.rs), Vec (filters_szip.rs), and format! (dict_encoding.rs)
    on no_std paths. Replace f64::powi (std/libm-only) with a small
    local exponentiation-by-squaring helper in the scale-offset filter.

2026-08-05 10:50:13 -07:00

osobhandClaude Opus 4.8

8534c7d204

feat: migrate-engine improvements (content validation, schema, streaming, incremental)

clawhdf5-migrate:
- Real content validation: the post-migration check reads the written HDF5
  back (new hdf5_reader) and compares actual content — chunk text, embeddings,
  and every session/entity/relation field — to the source, not just row counts.
  A representative sample of chunk rows is verified by default; --validate-full
  checks every row. A count-preserving corruption no longer passes.
- Configurable schema: SQL is built from a SchemaConfig (table + ordered column
  names, defaulting to the ZeroClaw layout) instead of hardcoded queries, with
  --chunks-table / --sessions-table / --entities-table / --relations-table.
- Streaming count pass: --dry-run does a COUNT(*)-only pass per table instead
  of loading every row.
- Incremental migration: --incremental reads the existing output, reads only
  source chunks with id greater than the last migrated id, and appends them
  (metadata groups refreshed from source) rather than re-migrating everything.

clawhdf5-format:
- read_as_f32 / read_as_f64 now decode IEEE-754 half-precision (2-byte) floats
  via a no_std-safe bit conversion — needed to read float16-stored embeddings
  back (e.g. for migrate's content validation), previously a TypeMismatch.

Tests: f16 read unit test; migrate tests for content-corruption detection,
custom table names, and incremental append; CLI smoke-tested end-to-end and the
dense/incremental output verified with h5py.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

2026-06-04 02:19:31 +00:00

redclawsystems

3f222f6956

Merge pull request 'docs(clawhdf5): document DType variants, fix unresolved doc links' (#17 ) from sdlc-docs/clawhdf5-types-20260514-165210 into main

2026-05-14 23:54:48 +00:00