ClawHDF5 PlannerandClaude Sonnet 4.6
14db35aa74
Seven research briefs covering the full mission scope: 01 — Architecture overview (crate map, format coverage, agent modules) 02 — Roadmap status and strategic gaps (distribution, MPI-IO, encryption) 03 — HDF5 ecosystem and cutting-edge developments (HDF5 2.0, Blosc2, ANN trends) 04 — Performance optimizations (10 opportunities, prioritized) 05 — Robustness enhancements (fuzzing gaps, bounds audit, WAL, KG cycle guard) 06 — Security hardening (encryption, signing, embedding poisoning, JNI safety) 07 — Synthesis and 15 actionable next steps with INT-NN task markers Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
126 lines
8.2 KiB
Markdown
126 lines
8.2 KiB
Markdown
# HDF5 Ecosystem & Cutting-Edge Developments
|
||
|
||
*Research brief — generated 2026-08-12*
|
||
|
||
---
|
||
|
||
## 1. HDF5 Format Evolution
|
||
|
||
### 1.1 HDF5 2.0 (released ~2025–2026)
|
||
The HDF Group has shipped HDF5 2.0. Key changes relevant to ClawHDF5:
|
||
|
||
- **Compound/array datatype version 5** and **data layout version 5** are now emitted by `libhdf5 --with-libver=latest`. ClawHDF5 HEAD already handles these (v3/v4 and v5 share the same binary structure; the version fields were previously rejected as invalid — fixed in the unreleased changelog).
|
||
- **Paged Fixed Array** chunk index is now the default for filtered, fixed-dimension datasets beyond a threshold. ClawHDF5 added full paged-Fixed-Array support in the unreleased work.
|
||
- **HDF5 2.0 removes deprecated APIs** (H5Oopen_by_idx, H5Gopen, etc.). Not directly relevant to a pure-Rust implementation but worth noting for interop test suites.
|
||
|
||
### 1.2 VOL (Virtual Object Layer) Plugins
|
||
HDF5 1.12+ introduced the Virtual Object Layer, allowing backend substitution (e.g. HDF5 API calls routed to object stores, databases, or in-memory formats). The ClawHDF5 roadmap has a `docs/superpowers/plans/2026-06-29-mpi-io-vol-backend.md` plan but this is not a VOL backend in the HDF5 sense — it is an internal I/O abstraction.
|
||
|
||
Opportunity: Implementing an HDF5 VOL plugin (C-facing) that routes to ClawHDF5's Rust backend would allow existing Python/C++ codebases to use ClawHDF5 transparently without changing their HDF5 API calls. High effort; high ecosystem value.
|
||
|
||
### 1.3 HDF5 REST VOL / HSDS
|
||
The HDF Group's HSDS (Highly Scalable Data Service) exposes HDF5 via REST, enabling cloud-native HDF5 access. An HTTP-backed `clawhdf5-io` backend would make ClawHDF5 a drop-in client for HSDS-hosted datasets.
|
||
|
||
---
|
||
|
||
## 2. Compression Codec Landscape
|
||
|
||
### 2.1 Currently Supported
|
||
| Filter | ID | Feature Flag |
|
||
|--------|----|-------------|
|
||
| Deflate (zlib-ng) | 1 | Default |
|
||
| Shuffle | 2 | Default |
|
||
| Fletcher32 | 3 | Default |
|
||
| SZIP (libaec) | 4 | `szip` |
|
||
| N-Bit | 5 | Default |
|
||
| Scale-offset | 6 | Default |
|
||
| LZ4 | 32004 | `lz4` |
|
||
| Zstandard | 32015 | `zstd` |
|
||
| Pcodec | 32023 | `pcodec` |
|
||
|
||
### 2.2 Missing / Emerging Codecs
|
||
|
||
**Blosc2** (filter id 32001): The most widely used third-party HDF5 filter in scientific computing. Blosc2 is a meta-compressor supporting multiple internal codecs (zstd, lz4, blosclz) with multithreaded compression and an internal shuffle transform. The HDF5 filter plugin is widely deployed in `h5py` workflows. ClawHDF5 has a `clawhdf5-filters` crate that is positioned for this — adding Blosc2 would dramatically expand file compatibility.
|
||
|
||
**ZFP** (filter id 32013): Lossy compression for floating-point arrays. Widely used in scientific HDF5 files (climate, simulation output). Not yet supported.
|
||
|
||
**Bitshuffle + LZ4** (filter id 32008): Popular in synchrotron/X-ray detector workflows. Different from plain shuffle.
|
||
|
||
**ZLIB-RS**: A pure-Rust zlib implementation. ClawHDF5 already has a `zlib-rs` feature flag stub but it is not the default (zlib-ng C wrapper is). Switching to zlib-rs would eliminate the last C dep path in the default build.
|
||
|
||
---
|
||
|
||
## 3. Vector Search / ANN Index Developments
|
||
|
||
### 3.1 State of HNSW
|
||
HNSW remains the dominant ANN algorithm for in-memory exact-approximate tradeoffs. Key research frontiers (2025–2026):
|
||
|
||
- **DiskANN / SPANN**: Graph-based ANN designed for SSD storage at billion scale. Relevant if ClawHDF5 targets graphs > 10M vectors. DiskANN's key insight is keeping the graph on disk and using a small in-memory cache for hot edges.
|
||
- **HNSW with quantization (ScaNN, FAISS)**: Product quantization inside HNSW edges (not just leaf vectors) cuts memory 4–8× with <5% recall loss. ClawHDF5 has IVF-PQ but not PQ-within-HNSW.
|
||
- **Filtered ANN**: Combining vector search with metadata predicates (e.g. "find top-5 nearest neighbors where source_channel='user'"). ClawHDF5 currently filters post-retrieval; pre-filtering at the index level would be faster and more accurate for high-selectivity filters.
|
||
|
||
### 3.2 Embedding Model Trends
|
||
- **Matryoshka embeddings** (MRL — Matryoshka Representation Learning): models trained to produce embeddings that can be truncated to smaller dimensions without re-training. OpenAI's `text-embedding-3-small` supports this. ClawHDF5 stores a fixed `embedding_dim`; support for variable-dimension storage (or separate dim-reduced index) would align with this trend.
|
||
- **Binary embeddings**: 1-bit quantization of embeddings. Hamming distance search is ~32× faster than cosine on CPU SIMD. Used in retrieval pre-filtering stages.
|
||
|
||
---
|
||
|
||
## 4. Agent Memory Research Landscape (2025–2026)
|
||
|
||
### 4.1 Papers Already Incorporated
|
||
ClawHDF5 cites 15+ papers in its research foundation (MemX, CraniMem, D-MEM, SYNAPSE, MemoryGraft, etc.). These are all implemented.
|
||
|
||
### 4.2 Emerging Research Not Yet Incorporated
|
||
|
||
**MemoryBank / MemoryStream** (2025): Streaming memory consolidation where new memories trigger re-evaluation of existing ones. The current ClawHDF5 consolidation model is periodic (explicit `consolidate()` call) rather than streaming.
|
||
|
||
**Chain-of-Thought Memory** (2026): Storing the reasoning chain alongside the conclusion, enabling future queries to retrieve not just "what was decided" but "why". ClawHDF5 stores `chunk` (text) + `embedding`; no structured reasoning field exists.
|
||
|
||
**Forgetting curves (Leitner / Ebbinghaus)**: Spaced-repetition scheduling for memory decay. The current time-decay is a fixed exponential half-life. A Leitner-style scheduler would adjust decay rate based on retrieval history.
|
||
|
||
**Episodic memory replay** (inspired by neuroscience): Replay important memories during idle periods to strengthen their embeddings without adding new information. Related to ClawHDF5's `consolidation` tier but not yet implemented.
|
||
|
||
**Cross-agent memory sharing** (MemoryArena 2026): Standardized protocols for agents to share verified memories. ClawHDF5's knowledge graph export/import is a step in this direction but lacks a standardized protocol.
|
||
|
||
---
|
||
|
||
## 5. Rust Ecosystem Dependencies
|
||
|
||
| Dependency Area | Current | Opportunity |
|
||
|-----------------|---------|-------------|
|
||
| Async runtime | `tokio` (`async` feature) | Consider `smol` or `async-std` for embedded targets |
|
||
| Serialization | `serde` | Already in `[workspace.dependencies]` |
|
||
| Parallelism | `rayon` (optional) | Rayon is well-established; no change needed |
|
||
| GPU | `wgpu` + WGSL shaders | `wgpu` 0.20+ has better Metal/Vulkan support; worth tracking |
|
||
| Compression | Mixed C/Rust | `zlib-rs` for deflate; `lz4_flex` for LZ4 — both pure Rust |
|
||
| Crypto | FNV-1a (unkeyed), SHA-256 | `blake3` (`blake3_hash` feature already exists) for high-speed content hashing; `aes-gcm` for encryption |
|
||
| FFI | `libaec-sys` (SZIP) | Only remaining non-optional C dep path |
|
||
|
||
---
|
||
|
||
## 6. NetCDF-4 and Scientific Computing Context
|
||
|
||
NetCDF-4 is built on HDF5 (it IS HDF5 with specific conventions). ClawHDF5's `clawhdf5-netcdf4` crate provides compatibility. Scientific domains that use HDF5/NetCDF-4:
|
||
|
||
- **Climate science**: CMIP6 datasets, ERA5 reanalysis (petabytes of NetCDF-4)
|
||
- **Genomics**: HDF5-backed formats (AnnData/h5ad for single-cell RNA-seq)
|
||
- **Particle physics**: CERN ROOT/HDF5 format
|
||
- **Astronomy**: FITS and HDF5 hybrid formats; SKA telescope data
|
||
|
||
For ClawHDF5 to serve these domains, the key gaps are:
|
||
1. Parallel collective I/O (MPI) — required for multi-node HPC ingestion
|
||
2. Blosc2 filter support — de-facto standard in h5py scientific workflows
|
||
3. ZFP lossy compression — common in simulation output
|
||
|
||
---
|
||
|
||
## 7. Security Research Context
|
||
|
||
### 7.1 Memory Poisoning
|
||
The MemoryGraft (2025) and SSGM (2026) papers that ClawHDF5 cites are the current frontier. New attack vectors emerging:
|
||
- **Gradient-based poisoning**: Adversarially crafting embeddings that are near arbitrary queries in vector space. ClawHDF5's anomaly detection checks text patterns but not embedding-space manipulation.
|
||
- **Temporal poisoning**: Injecting memories with falsified timestamps to manipulate temporal reasoning. ClawHDF5's WAL has CRC32 integrity but timestamps are not signed.
|
||
|
||
### 7.2 Supply Chain
|
||
The `szip` feature introduces a C FFI dependency (`libaec`). If not compiled in, there is no C dependency. The `system-zlib-decompress` feature also links against the system zlib. Both paths should be audited in deployments that require supply-chain provenance.
|