quantumclaw/clawhdf5
3
1
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases 1 Wiki Activity

Files open in h5py again; float16 embedding storage #4

Merged
osobh merged 2 commits from feat/float16-embeddings into main 2026-09-25 03:18:07 +00:00
Conversation 0 Commits 2 Files Changed 20
+1230 -45
osobh commented 2026-09-24 17:00:55 +00:00
Owner
Copy Link
Copy Source

Summary

Two things, one of them urgent for anyone reading our files with h5py.

1. Files written by clawhdf5 now open in h5py and libhdf5 (5e4aa1c)

Two write-side bugs, present in every release (at least since v2.1.0):

  • Every f32 dataset was refused with "sign bit position out of bounds": the float datatype encoder hard-coded the sign-bit position to 63, which is only right for f64. This covers every agent store's embeddings.
  • Every empty dataset was refused with "invalid dataset size, likely file corruption": it had a real address and 0 bytes, which trips libhdf5's addr + size <= addr check. This covers every agent store without sessions or a knowledge graph.

Our own reader ignores both fields, and the interop suites only wrote f64 from our side, which is how they went unnoticed. Agent stores fix themselves at their next checkpoint on this build; other files with f32 or empty datasets need rewriting. Details in docs/known-issues.md.

2. MemoryConfig::float16 stores half-precision embeddings (d0db838)

It was persisted and otherwise ignored. Now it writes /memory/embeddings as numpy float16, off by default (CLI create --float16).

100K × 384, tank, median of 6 f32 float16
File 154.0 MiB 80.8 MiB (−48%)
Checkpoint 752 ms 512 ms
Open 300 ms 252 ms
Vector recall@10 vs exact 0.994 0.999 (no loss)
Hybrid p50 4.68 ms 4.65 ms
  • Embeddings are rounded as they're saved (and on WAL replay and load), so memory and file agree bit for bit, and a store searches the same before and after a reopen.
  • Rounding matches numpy's float16 bit for bit, and the half crate on 16.7M values.
  • Values beyond ±65504 return the new MemoryError::InvalidEntry. Breaking for exhaustive matches.
  • New clawhdf5_format::float16 module and DatasetBuilder::with_f16_data.

Test plan

  • scripts/ci-test.sh locally on tank: fmt, the clippy matrix (default, format features, ann parallel, zlib-ng), MSRV 1.92, all tests, the fast-deflate tests, h5py interop, no_std, bench compile
  • New interop tests: f32 and float16 in both directions; h5py opens a whole agent store (f32 and float16) and decodes every dataset
  • float16_store tests: on-disk dtype and values, 2× size, identical search after reopen, out-of-range refusal (incl. batch atomicity and WAL), WAL replay rounding
  • Commit 1 builds and passes on its own
  • Gitea CI on this PR

🤖 Generated with Claude Code

## Summary Two things, one of them urgent for anyone reading our files with h5py. ### 1. Files written by clawhdf5 now open in h5py and libhdf5 (`5e4aa1c`) Two write-side bugs, present in every release (at least since v2.1.0): - **Every `f32` dataset** was refused with *"sign bit position out of bounds"*: the float datatype encoder hard-coded the sign-bit position to 63, which is only right for `f64`. This covers every agent store's embeddings. - **Every empty dataset** was refused with *"invalid dataset size, likely file corruption"*: it had a real address and 0 bytes, which trips libhdf5's `addr + size <= addr` check. This covers every agent store without sessions or a knowledge graph. Our own reader ignores both fields, and the interop suites only wrote `f64` from our side, which is how they went unnoticed. Agent stores fix themselves at their next checkpoint on this build; other files with `f32` or empty datasets need rewriting. Details in `docs/known-issues.md`. ### 2. `MemoryConfig::float16` stores half-precision embeddings (`d0db838`) It was persisted and otherwise ignored. Now it writes `/memory/embeddings` as numpy `float16`, off by default (CLI `create --float16`). | 100K × 384, tank, median of 6 | f32 | float16 | |---|---:|---:| | File | 154.0 MiB | **80.8 MiB (−48%)** | | Checkpoint | 752 ms | 512 ms | | Open | 300 ms | 252 ms | | Vector recall@10 vs exact | 0.994 | 0.999 (no loss) | | Hybrid p50 | 4.68 ms | 4.65 ms | - Embeddings are rounded as they're saved (and on WAL replay and load), so memory and file agree bit for bit, and a store searches the same before and after a reopen. - Rounding matches numpy's `float16` bit for bit, and the `half` crate on 16.7M values. - Values beyond ±65504 return the new `MemoryError::InvalidEntry`. **Breaking** for exhaustive matches. - New `clawhdf5_format::float16` module and `DatasetBuilder::with_f16_data`. ## Test plan - [x] `scripts/ci-test.sh` locally on tank: fmt, the clippy matrix (default, format features, ann parallel, zlib-ng), MSRV 1.92, all tests, the fast-deflate tests, h5py interop, no_std, bench compile - [x] New interop tests: `f32` and `float16` in both directions; h5py opens a whole agent store (`f32` and `float16`) and decodes every dataset - [x] `float16_store` tests: on-disk dtype and values, 2× size, identical search after reopen, out-of-range refusal (incl. batch atomicity and WAL), WAL replay rounding - [x] Commit 1 builds and passes on its own - [ ] Gitea CI on this PR 🤖 Generated with [Claude Code](https://claude.com/claude-code)
osobh added 2 commits 2026-09-24 17:00:55 +00:00
fix(format): files we write now open in h5py and libhdf5 5e4aa1c6bf 
Two write-side bugs, both present in every release (the first at least
since v2.1.0), made libhdf5 refuse files written by clawhdf5. Our own
reader ignores both fields, and the interop suites only ever wrote f64
from our side, so nothing here caught them.

- Every f32 dataset: "sign bit position out of bounds". The float
  datatype encoder hard-coded the sign bit's position (bits 8-15 of the
  class bit field) to 63, which is right only for f64. It is now derived
  from the type: bit_offset + bit_precision - 1. This covered every
  agent store's embeddings, norms and activation weights.
- Every empty dataset: "invalid dataset size, likely file corruption".
  It was written with a real address and size 0, which trips libhdf5's
  `addr + size <= addr` overflow check. An empty contiguous dataset now
  gets the undefined address, as libhdf5 writes it. This covered every
  agent store without sessions or a knowledge graph.

Agent stores are rewritten in full at each checkpoint, so they become
readable at their next checkpoint on a fixed build; other files with f32
or empty datasets need rewriting. Both are recorded in
docs/known-issues.md.

Tests: the sign position byte for f32/f64, and h5py reading our f32
datasets (plain and chunked + deflate) bit for bit.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
feat(agent): MemoryConfig::float16 stores half-precision embeddings
CI / test-arm64 (pull_request) Successful in 1m19s
Details
CI / test (pull_request) Successful in 4m58s
Details
d0db83812b 
The setting was persisted in /meta and otherwise ignored: embeddings
were always written as f32. It now does what it says.

clawhdf5-format:
- `DatasetBuilder::with_f16_data` writes IEEE binary16 (numpy float16),
  rounding to nearest-even, and `make_f16_type`.
- `clawhdf5_format::float16` holds the f32 <-> f16 conversions, the one
  implementation the writer, the reader and the agent all use. Checked
  against the `half` crate on 16.7M f32 values and round-trips all 65536
  half values; the h5py interop tests confirm the rounding matches
  numpy's bit for bit (4020 values incl. ties, subnormals, overflow).
- Reading little-endian float16 as f32 has a fast path.

clawhdf5-agent:
- A float16 store writes /memory/embeddings as half precision, and
  `MemoryCache::half_precision` rounds each embedding as it enters the
  cache (save, update, WAL replay, and on load of a store still f32 on
  disk), so memory and file agree bit for bit and a store searches the
  same before and after a reopen (tested).
- Values beyond +-65504 are refused with the new
  `MemoryError::InvalidEntry` rather than stored as infinity, on every
  save path; batches are all or nothing, and a rejected ephemeral entry
  stays in the ephemeral tier. Breaking for exhaustive matches.
- CLI: `create --float16`. Off by default.

Measured on tank, 384-dim, six runs alternating order, medians
(search_harness --float16-study --full): at 100K the file goes from
154.0 to 80.8 MiB (-48%), checkpoint 752 -> 512 ms, open 300 -> 252 ms;
vector recall@10 against an exact scan and hybrid_search latency do not
change. At 10K open is 3 ms slower. Also a test that h5py opens a whole
agent store, f32 and float16, and decodes every dataset.

Docs: README, BENCHMARKS.md ("float16 embedding storage"), CHANGELOG
(including the h5py interop fixes in the previous commit), CLAUDE.md.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
osobh merged commit 73bb068264 into main 2026-09-25 03:18:07 +00:00
osobh deleted branch feat/float16-embeddings 2026-09-25 03:18:11 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
nsobh osobh scottb (Scott B.)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: quantumclaw/clawhdf5#4