fix(format): files we write now open in h5py and libhdf5

Two write-side bugs, both present in every release (the first at least
since v2.1.0), made libhdf5 refuse files written by clawhdf5. Our own
reader ignores both fields, and the interop suites only ever wrote f64
from our side, so nothing here caught them.

- Every f32 dataset: "sign bit position out of bounds". The float
  datatype encoder hard-coded the sign bit's position (bits 8-15 of the
  class bit field) to 63, which is right only for f64. It is now derived
  from the type: bit_offset + bit_precision - 1. This covered every
  agent store's embeddings, norms and activation weights.
- Every empty dataset: "invalid dataset size, likely file corruption".
  It was written with a real address and size 0, which trips libhdf5's
  `addr + size <= addr` overflow check. An empty contiguous dataset now
  gets the undefined address, as libhdf5 writes it. This covered every
  agent store without sessions or a knowledge graph.

Agent stores are rewritten in full at each checkpoint, so they become
readable at their next checkpoint on a fixed build; other files with f32
or empty datasets need rewriting. Both are recorded in
docs/known-issues.md.

Tests: the sign position byte for f32/f64, and h5py reading our f32
datasets (plain and chunked + deflate) bit for bit.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-24 12:00:26 -05:00
co-authored by Claude Opus 5.5
parent 1cceb930b2
commit 5e4aa1c6bf
4 changed files with 121 additions and 4 deletions
@@ -86,6 +86,12 @@ pub(crate) fn build_dataset_oh(
let mut dl = Vec::new();
dl.push(4); // version
dl.push(1); // class = contiguous
// An empty dataset has no storage: its address must be the undefined
// address, as libhdf5 writes it. A real address with size 0 trips
// libhdf5's `addr + size <= addr` overflow check, and it refuses the
// dataset as "invalid dataset size, likely file corruption" — which made
// every store with no sessions or knowledge graph unreadable by h5py.
let data_addr = if data_size == 0 { u64::MAX } else { data_addr };
dl.extend_from_slice(&data_addr.to_le_bytes());
dl.extend_from_slice(&data_size.to_le_bytes());
w.add_message(MessageType::DataLayout, dl);