quantumclaw/clawhdf5
3
1
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases 1 Wiki Activity

docs: record the 2026-09-25 HDF5 audit fixes and open gaps
CI / test-arm64 (pull_request) Successful in 1m21s
Details
CI / test (pull_request) Successful in 5m59s
Details

Browse Source 
CHANGELOG: upgrade notes (changed read results for max-shape files,
saturating conversions, new writer errors, format-crate API changes) and
the reader/writer correctness fixes. known-issues: the silent-wrong-data
table with before/after sweep numbers, the gaps still open, and a
correction to the Extensible Array entry, which said files we wrote were
unaffected. CLAUDE.md: clawhdf5-gpu is vector distance computation, not
I/O, and clawhdf5-filters holds only deflate backends (no Blosc).

Also a facade test that libhdf5's 20-bit N-Bit float test data reads as
libhdf5's values.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-25 21:26:56 -05:00
co-authored by Claude Opus 5.5
parent 650f355219
commit 72b9cfb1e1
4 changed files with 206 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CLAUDE.md
+4 -4
View File
@@ -1,7 +1,7 @@
# clawhdf5
## Purpose
Pure-Rust HDF5 format implementation with HNSW vector search, WAL-backed persistence, agent memory storage, and GPU-accelerated I/O. A standalone library. Its one verified consumer is ClawBrainHub (`.brain` files); no agent framework integrates it (OpenClaw and ZeroClaw claims were withdrawn on 2026-09-25 — neither was ever true).
Pure-Rust HDF5 format implementation with HNSW vector search, WAL-backed persistence, agent memory storage, and GPU-accelerated vector search. A standalone library. Its one verified consumer is ClawBrainHub (`.brain` files); no agent framework integrates it (OpenClaw and ZeroClaw claims were withdrawn on 2026-09-25 — neither was ever true).
## Architecture
@@ -11,13 +11,13 @@ Cargo workspace with 16 crates under `crates/` (plus `libaec-sys`, an internal F
|-------|------|
| `clawhdf5-format` | HDF5 binary spec parser (superblock, B-tree, heap) — also holds shared type definitions and physical constants |
| `clawhdf5-io` | Read/write implementation |
| `clawhdf5-filters` | Compression filters (gzip, LZ4, Zstd, Blosc) |
| `clawhdf5-filters` | Deflate backends (zlib-rs, zlib-ng, Apple Compression); the HDF5 filter pipeline and the other codecs (LZ4, Zstd, SZIP, N-Bit, scale-offset, pcodec) live in `clawhdf5-format`. No Blosc. |
| `clawhdf5-derive` | Proc-macro derive for HDF5-serializable structs |
| `clawhdf5` | Main facade crate |
| `clawhdf5-netcdf4` | NetCDF-4 compatibility layer |
| `clawhdf5-ann` | HNSW approximate nearest-neighbor vector index |
| `clawhdf5-agent` | Agent memory, session history, knowledge graph storage |
| `clawhdf5-gpu` | GPU-accelerated I/O via wgpu (hand-written WGSL compute shaders) |
| `clawhdf5-gpu` | GPU vector distance computation via wgpu (hand-written WGSL compute shaders) — not dataset I/O |
| `clawhdf5-accel` | CPU SIMD acceleration path |
| `clawhdf5-migrate` | SQLite → HDF5 agent-memory migration |
| `clawhdf5-android` | Android JNI bindings |
@@ -148,7 +148,7 @@ Cargo workspace with 16 crates under `crates/` (plus `libaec-sys`, an internal F
Alerts never block a save — drain them with `HDF5Memory::take_anomaly_alerts`.
`MemorySource` for this bookkeeping is inferred from the caller-supplied
`source_channel` string (a heuristic, not an authenticated trust boundary).
- GPU-accelerated batch I/O for large dataset processing
- GPU-accelerated vector distance computation (`clawhdf5-gpu`, wgpu); HDF5 I/O itself is CPU-only
- Python and Node.js bindings for cross-language use
- NetCDF-4 compatibility for scientific data interop