# HDF5 Ecosystem & Cutting-Edge Developments

*Research brief — generated 2026-08-12*

---

## 1. HDF5 Format Evolution

### 1.1 HDF5 2.0 (released ~2025–2026)
The HDF Group has shipped HDF5 2.0. Key changes relevant to ClawHDF5:

- **Compound/array datatype version 5** and **data layout version 5** are now emitted by `libhdf5 --with-libver=latest`. ClawHDF5 HEAD already handles these (v3/v4 and v5 share the same binary structure; the version fields were previously rejected as invalid — fixed in the unreleased changelog).
- **Paged Fixed Array** chunk index is now the default for filtered, fixed-dimension datasets beyond a threshold. ClawHDF5 added full paged-Fixed-Array support in the unreleased work.
- **HDF5 2.0 removes deprecated APIs** (H5Oopen_by_idx, H5Gopen, etc.). Not directly relevant to a pure-Rust implementation but worth noting for interop test suites.

### 1.2 VOL (Virtual Object Layer) Plugins
HDF5 1.12+ introduced the Virtual Object Layer, allowing backend substitution (e.g. HDF5 API calls routed to object stores, databases, or in-memory formats). The ClawHDF5 roadmap has a `docs/superpowers/plans/2026-06-29-mpi-io-vol-backend.md` plan but this is not a VOL backend in the HDF5 sense — it is an internal I/O abstraction.

Opportunity: Implementing an HDF5 VOL plugin (C-facing) that routes to ClawHDF5's Rust backend would allow existing Python/C++ codebases to use ClawHDF5 transparently without changing their HDF5 API calls. High effort; high ecosystem value.

### 1.3 HDF5 REST VOL / HSDS
The HDF Group's HSDS (Highly Scalable Data Service) exposes HDF5 via REST, enabling cloud-native HDF5 access. An HTTP-backed `clawhdf5-io` backend would make ClawHDF5 a drop-in client for HSDS-hosted datasets.

---

## 2. Compression Codec Landscape

### 2.1 Currently Supported
| Filter | ID | Feature Flag |
|--------|----|-------------|
| Deflate (zlib-ng) | 1 | Default |
| Shuffle | 2 | Default |
| Fletcher32 | 3 | Default |
| SZIP (libaec) | 4 | `szip` |
| N-Bit | 5 | Default |
| Scale-offset | 6 | Default |
| LZ4 | 32004 | `lz4` |
| Zstandard | 32015 | `zstd` |
| Pcodec | 32023 | `pcodec` |

### 2.2 Missing / Emerging Codecs

**Blosc2** (filter id 32001): The most widely used third-party HDF5 filter in scientific computing. Blosc2 is a meta-compressor supporting multiple internal codecs (zstd, lz4, blosclz) with multithreaded compression and an internal shuffle transform. The HDF5 filter plugin is widely deployed in `h5py` workflows. ClawHDF5 has a `clawhdf5-filters` crate that is positioned for this — adding Blosc2 would dramatically expand file compatibility.

**ZFP** (filter id 32013): Lossy compression for floating-point arrays. Widely used in scientific HDF5 files (climate, simulation output). Not yet supported.

**Bitshuffle + LZ4** (filter id 32008): Popular in synchrotron/X-ray detector workflows. Different from plain shuffle.

**ZLIB-RS**: A pure-Rust zlib implementation. ClawHDF5 already has a `zlib-rs` feature flag stub but it is not the default (zlib-ng C wrapper is). Switching to zlib-rs would eliminate the last C dep path in the default build.

---

## 3. Vector Search / ANN Index Developments

### 3.1 State of HNSW
HNSW remains the dominant ANN algorithm for in-memory exact-approximate tradeoffs. Key research frontiers (2025–2026):

- **DiskANN / SPANN**: Graph-based ANN designed for SSD storage at billion scale. Relevant if ClawHDF5 targets graphs > 10M vectors. DiskANN's key insight is keeping the graph on disk and using a small in-memory cache for hot edges.
- **HNSW with quantization (ScaNN, FAISS)**: Product quantization inside HNSW edges (not just leaf vectors) cuts memory 4–8× with <5% recall loss. ClawHDF5 has IVF-PQ but not PQ-within-HNSW.
- **Filtered ANN**: Combining vector search with metadata predicates (e.g. "find top-5 nearest neighbors where source_channel='user'"). ClawHDF5 currently filters post-retrieval; pre-filtering at the index level would be faster and more accurate for high-selectivity filters.

### 3.2 Embedding Model Trends
- **Matryoshka embeddings** (MRL — Matryoshka Representation Learning): models trained to produce embeddings that can be truncated to smaller dimensions without re-training. OpenAI's `text-embedding-3-small` supports this. ClawHDF5 stores a fixed `embedding_dim`; support for variable-dimension storage (or separate dim-reduced index) would align with this trend.
- **Binary embeddings**: 1-bit quantization of embeddings. Hamming distance search is ~32× faster than cosine on CPU SIMD. Used in retrieval pre-filtering stages.

---

## 4. Agent Memory Research Landscape (2025–2026)

### 4.1 Papers Already Incorporated
ClawHDF5 cites 15+ papers in its research foundation (MemX, CraniMem, D-MEM, SYNAPSE, MemoryGraft, etc.). These are all implemented.

### 4.2 Emerging Research Not Yet Incorporated

**MemoryBank / MemoryStream** (2025): Streaming memory consolidation where new memories trigger re-evaluation of existing ones. The current ClawHDF5 consolidation model is periodic (explicit `consolidate()` call) rather than streaming.

**Chain-of-Thought Memory** (2026): Storing the reasoning chain alongside the conclusion, enabling future queries to retrieve not just "what was decided" but "why". ClawHDF5 stores `chunk` (text) + `embedding`; no structured reasoning field exists.

**Forgetting curves (Leitner / Ebbinghaus)**: Spaced-repetition scheduling for memory decay. The current time-decay is a fixed exponential half-life. A Leitner-style scheduler would adjust decay rate based on retrieval history.

**Episodic memory replay** (inspired by neuroscience): Replay important memories during idle periods to strengthen their embeddings without adding new information. Related to ClawHDF5's `consolidation` tier but not yet implemented.

**Cross-agent memory sharing** (MemoryArena 2026): Standardized protocols for agents to share verified memories. ClawHDF5's knowledge graph export/import is a step in this direction but lacks a standardized protocol.

---

## 5. Rust Ecosystem Dependencies

| Dependency Area | Current | Opportunity |
|-----------------|---------|-------------|
| Async runtime | `tokio` (`async` feature) | Consider `smol` or `async-std` for embedded targets |
| Serialization | `serde` | Already in `[workspace.dependencies]` |
| Parallelism | `rayon` (optional) | Rayon is well-established; no change needed |
| GPU | `wgpu` + WGSL shaders | `wgpu` 0.20+ has better Metal/Vulkan support; worth tracking |
| Compression | Mixed C/Rust | `zlib-rs` for deflate; `lz4_flex` for LZ4 — both pure Rust |
| Crypto | FNV-1a (unkeyed), SHA-256 | `blake3` (`blake3_hash` feature already exists) for high-speed content hashing; `aes-gcm` for encryption |
| FFI | `libaec-sys` (SZIP) | Only remaining non-optional C dep path |

---

## 6. NetCDF-4 and Scientific Computing Context

NetCDF-4 is built on HDF5 (it IS HDF5 with specific conventions). ClawHDF5's `clawhdf5-netcdf4` crate provides compatibility. Scientific domains that use HDF5/NetCDF-4:

- **Climate science**: CMIP6 datasets, ERA5 reanalysis (petabytes of NetCDF-4)
- **Genomics**: HDF5-backed formats (AnnData/h5ad for single-cell RNA-seq)
- **Particle physics**: CERN ROOT/HDF5 format
- **Astronomy**: FITS and HDF5 hybrid formats; SKA telescope data

For ClawHDF5 to serve these domains, the key gaps are:
1. Parallel collective I/O (MPI) — required for multi-node HPC ingestion
2. Blosc2 filter support — de-facto standard in h5py scientific workflows
3. ZFP lossy compression — common in simulation output

---

## 7. Security Research Context

### 7.1 Memory Poisoning
The MemoryGraft (2025) and SSGM (2026) papers that ClawHDF5 cites are the current frontier. New attack vectors emerging:
- **Gradient-based poisoning**: Adversarially crafting embeddings that are near arbitrary queries in vector space. ClawHDF5's anomaly detection checks text patterns but not embedding-space manipulation.
- **Temporal poisoning**: Injecting memories with falsified timestamps to manipulate temporal reasoning. ClawHDF5's WAL has CRC32 integrity but timestamps are not signed.

### 7.2 Supply Chain
The `szip` feature introduces a C FFI dependency (`libaec`). If not compiled in, there is no C dependency. The `system-zlib-decompress` feature also links against the system zlib. Both paths should be audited in deployments that require supply-chain provenance.