feat(agent): MemoryConfig::float16 stores half-precision embeddings
The setting was persisted in /meta and otherwise ignored: embeddings
were always written as f32. It now does what it says.
clawhdf5-format:
- `DatasetBuilder::with_f16_data` writes IEEE binary16 (numpy float16),
rounding to nearest-even, and `make_f16_type`.
- `clawhdf5_format::float16` holds the f32 <-> f16 conversions, the one
implementation the writer, the reader and the agent all use. Checked
against the `half` crate on 16.7M f32 values and round-trips all 65536
half values; the h5py interop tests confirm the rounding matches
numpy's bit for bit (4020 values incl. ties, subnormals, overflow).
- Reading little-endian float16 as f32 has a fast path.
clawhdf5-agent:
- A float16 store writes /memory/embeddings as half precision, and
`MemoryCache::half_precision` rounds each embedding as it enters the
cache (save, update, WAL replay, and on load of a store still f32 on
disk), so memory and file agree bit for bit and a store searches the
same before and after a reopen (tested).
- Values beyond +-65504 are refused with the new
`MemoryError::InvalidEntry` rather than stored as infinity, on every
save path; batches are all or nothing, and a rejected ephemeral entry
stays in the ephemeral tier. Breaking for exhaustive matches.
- CLI: `create --float16`. Off by default.
Measured on tank, 384-dim, six runs alternating order, medians
(search_harness --float16-study --full): at 100K the file goes from
154.0 to 80.8 MiB (-48%), checkpoint 752 -> 512 ms, open 300 -> 252 ms;
vector recall@10 against an exact scan and hybrid_search latency do not
change. At 10K open is 3 ms slower. Also a test that h5py opens a whole
agent store, f32 and float16, and decodes every dataset.
Docs: README, BENCHMARKS.md ("float16 embedding storage"), CHANGELOG
(including the h5py interop fixes in the previous commit), CLAUDE.md.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
@@ -82,6 +82,15 @@ breaking change, are in [CHANGELOG.md](CHANGELOG.md).
|
||||
100K × 384 store to 1.74× the raw vectors. At equal recall it is also faster
|
||||
than `f32`: 1.63× QPS on AVX2, 1.18× on a Raspberry Pi 5 (NEON `SDOT`).
|
||||
|
||||
**Interop (unreleased)**
|
||||
- **Files we write now open in h5py and libhdf5.** Every `f32` dataset —
|
||||
including every agent store's embeddings — and every empty dataset was
|
||||
refused by libhdf5. Both were write-side bugs in every release; agent stores
|
||||
fix themselves at their next checkpoint. See
|
||||
[docs/known-issues.md](docs/known-issues.md).
|
||||
- `MemoryConfig::float16` now stores half-precision embeddings (it was
|
||||
ignored): 48% smaller files at the same recall.
|
||||
|
||||
**Tooling**
|
||||
- CI now runs the h5py/netCDF4 interop suites for real (they had been skipping
|
||||
silently) and runs an aarch64 job for the NEON kernels.
|
||||
@@ -244,6 +253,9 @@ retrieval recall reported as QA accuracy typically overstates by 20–30 points.
|
||||
| 10K | 17.0 MB | 1.7 KB | 2.7 MB (6.2x) |
|
||||
| 100K | 169.8 MB | 1.7 KB | 26.9 MB (6.2x) |
|
||||
|
||||
With `MemoryConfig::float16` the embeddings take half the space: an agent
|
||||
store of 100K × 384 records is 80.8 MiB instead of 154.0.
|
||||
|
||||
**In memory** — a store reopened from disk, 384-dim `f32`, measured with a
|
||||
counting allocator ([BENCHMARKS.md § Memory footprint](BENCHMARKS.md#memory-footprint)):
|
||||
|
||||
@@ -538,7 +550,7 @@ ClawhDF5's agent memory design draws from 15+ recent papers:
|
||||
|
||||
| Flag | Default | Description |
|
||||
|------|---------|-------------|
|
||||
| `float16` | **yes** | Half-precision cosine kernel (`cosine_similarity_f16`). The store itself always writes `f32` embeddings; `MemoryConfig::float16` is recorded in `/meta` but not yet applied |
|
||||
| `float16` | **yes** | Half-precision cosine kernel (`cosine_similarity_f16`). Half-precision *storage* is the `MemoryConfig::float16` setting below, and needs no feature |
|
||||
| `hnsw` | **yes** | HNSW approximate vector index for `hybrid_search` (via `clawhdf5-ann`); disable for an exact linear scan |
|
||||
| `parallel` | **yes** | Parallel HNSW bulk build (same graph, ~3× faster on 16 cores) and Rayon brute-force search strategies |
|
||||
| `zstd` | no | Compress embeddings with Zstd instead of deflate when `MemoryConfig::compression` is on (links libzstd) |
|
||||
@@ -568,6 +580,14 @@ setting existed keep their `f32` index; opt out for new stores with
|
||||
`quantized_index = false` or `clawhdf5-cli create --f32-index`. See
|
||||
[BENCHMARKS.md § Quantising the index copy](BENCHMARKS.md#quantising-the-index-copy-quantized_index).
|
||||
|
||||
`MemoryConfig::float16` (off by default; CLI `create --float16`) stores the
|
||||
embeddings on disk as IEEE half precision (numpy `float16`): at 100K × 384 the
|
||||
file drops from 154 to 81 MiB, checkpoints and opens get faster, and vector
|
||||
recall and search latency do not change. Embeddings are rounded as they are
|
||||
saved, so the store searches the same before and after a reopen; values must
|
||||
lie within ±65504. See
|
||||
[BENCHMARKS.md § float16 embedding storage](BENCHMARKS.md#float16-embedding-storage-memoryconfigfloat16).
|
||||
|
||||
### `clawhdf5-format`
|
||||
|
||||
| Flag | Default | Description |
|
||||
@@ -656,8 +676,9 @@ agent_memory.h5
|
||||
│ └── ann_generation (ties the .ann sidecar to this checkpoint)
|
||||
├── /memory
|
||||
│ ├── chunks: string[N]
|
||||
│ ├── embeddings: f32[N × D] (chunked; deflate, or Zstd with the
|
||||
│ │ `zstd` feature, when compression is on)
|
||||
│ ├── embeddings: f32[N × D], or f16 for a `float16` store
|
||||
│ │ (chunked; deflate, or Zstd with the `zstd`
|
||||
│ │ feature, when compression is on)
|
||||
│ ├── source_channel: string[N]
|
||||
│ ├── timestamps: f64[N]
|
||||
│ ├── session_ids: string[N]
|
||||
|
||||
Reference in New Issue
Block a user