clawhdf5/research/03-hdf5-ecosystem.md

HDF5 Ecosystem & Cutting-Edge Developments

Research brief — generated 2026-08-12

---

1. HDF5 Format Evolution

1.1 HDF5 2.0 (released ~2025–2026)

The HDF Group has shipped HDF5 2.0. Key changes relevant to ClawHDF5:

• Compound/array datatype version 5 and data layout version 5 are now emitted by libhdf5 --with-libver=latest. ClawHDF5 HEAD already handles these (v3/v4 and v5 share the same binary structure; the version fields were previously rejected as invalid — fixed in the unreleased changelog).
• Paged Fixed Array chunk index is now the default for filtered, fixed-dimension datasets beyond a threshold. ClawHDF5 added full paged-Fixed-Array support in the unreleased work.
• HDF5 2.0 removes deprecated APIs (H5Oopen_by_idx, H5Gopen, etc.). Not directly relevant to a pure-Rust implementation but worth noting for interop test suites.

1.2 VOL (Virtual Object Layer) Plugins

HDF5 1.12+ introduced the Virtual Object Layer, allowing backend substitution (e.g. HDF5 API calls routed to object stores, databases, or in-memory formats). The ClawHDF5 roadmap has a docs/superpowers/plans/2026-06-29-mpi-io-vol-backend.md plan but this is not a VOL backend in the HDF5 sense — it is an internal I/O abstraction.

Opportunity: Implementing an HDF5 VOL plugin (C-facing) that routes to ClawHDF5's Rust backend would allow existing Python/C++ codebases to use ClawHDF5 transparently without changing their HDF5 API calls. High effort; high ecosystem value.

1.3 HDF5 REST VOL / HSDS

The HDF Group's HSDS (Highly Scalable Data Service) exposes HDF5 via REST, enabling cloud-native HDF5 access. An HTTP-backed clawhdf5-io backend would make ClawHDF5 a drop-in client for HSDS-hosted datasets.

---

2. Compression Codec Landscape

2.1 Currently Supported

Filter	ID	Feature Flag
Deflate (zlib-ng)	1	Default
Shuffle	2	Default
Fletcher32	3	Default
SZIP (libaec)	4	szip
N-Bit	5	Default
Scale-offset	6	Default
LZ4	32004	lz4
Zstandard	32015	zstd
Pcodec	32023	pcodec

2.2 Missing / Emerging Codecs

Blosc2 (filter id 32001): The most widely used third-party HDF5 filter in scientific computing. Blosc2 is a meta-compressor supporting multiple internal codecs (zstd, lz4, blosclz) with multithreaded compression and an internal shuffle transform. The HDF5 filter plugin is widely deployed in h5py workflows. ClawHDF5 has a clawhdf5-filters crate that is positioned for this — adding Blosc2 would dramatically expand file compatibility.

ZFP (filter id 32013): Lossy compression for floating-point arrays. Widely used in scientific HDF5 files (climate, simulation output). Not yet supported.

Bitshuffle + LZ4 (filter id 32008): Popular in synchrotron/X-ray detector workflows. Different from plain shuffle.

ZLIB-RS: A pure-Rust zlib implementation. ClawHDF5 already has a zlib-rs feature flag stub but it is not the default (zlib-ng C wrapper is). Switching to zlib-rs would eliminate the last C dep path in the default build.

---

3. Vector Search / ANN Index Developments

3.1 State of HNSW

HNSW remains the dominant ANN algorithm for in-memory exact-approximate tradeoffs. Key research frontiers (2025–2026):

• DiskANN / SPANN: Graph-based ANN designed for SSD storage at billion scale. Relevant if ClawHDF5 targets graphs > 10M vectors. DiskANN's key insight is keeping the graph on disk and using a small in-memory cache for hot edges.
• HNSW with quantization (ScaNN, FAISS): Product quantization inside HNSW edges (not just leaf vectors) cuts memory 4–8× with <5% recall loss. ClawHDF5 has IVF-PQ but not PQ-within-HNSW.
• Filtered ANN: Combining vector search with metadata predicates (e.g. "find top-5 nearest neighbors where source_channel='user'"). ClawHDF5 currently filters post-retrieval; pre-filtering at the index level would be faster and more accurate for high-selectivity filters.

3.2 Embedding Model Trends

• Matryoshka embeddings (MRL — Matryoshka Representation Learning): models trained to produce embeddings that can be truncated to smaller dimensions without re-training. OpenAI's text-embedding-3-small supports this. ClawHDF5 stores a fixed embedding_dim; support for variable-dimension storage (or separate dim-reduced index) would align with this trend.
• Binary embeddings: 1-bit quantization of embeddings. Hamming distance search is ~32× faster than cosine on CPU SIMD. Used in retrieval pre-filtering stages.

---

4. Agent Memory Research Landscape (2025–2026)

4.1 Papers Already Incorporated

ClawHDF5 cites 15+ papers in its research foundation (MemX, CraniMem, D-MEM, SYNAPSE, MemoryGraft, etc.). These are all implemented.

4.2 Emerging Research Not Yet Incorporated

MemoryBank / MemoryStream (2025): Streaming memory consolidation where new memories trigger re-evaluation of existing ones. The current ClawHDF5 consolidation model is periodic (explicit consolidate() call) rather than streaming.

Chain-of-Thought Memory (2026): Storing the reasoning chain alongside the conclusion, enabling future queries to retrieve not just "what was decided" but "why". ClawHDF5 stores chunk (text) + embedding; no structured reasoning field exists.

Forgetting curves (Leitner / Ebbinghaus): Spaced-repetition scheduling for memory decay. The current time-decay is a fixed exponential half-life. A Leitner-style scheduler would adjust decay rate based on retrieval history.

Episodic memory replay (inspired by neuroscience): Replay important memories during idle periods to strengthen their embeddings without adding new information. Related to ClawHDF5's consolidation tier but not yet implemented.

Cross-agent memory sharing (MemoryArena 2026): Standardized protocols for agents to share verified memories. ClawHDF5's knowledge graph export/import is a step in this direction but lacks a standardized protocol.

---

5. Rust Ecosystem Dependencies

Dependency Area	Current	Opportunity
Async runtime	tokio (async feature)	Consider smol or async-std for embedded targets
Serialization	serde	Already in [workspace.dependencies]
Parallelism	rayon (optional)	Rayon is well-established; no change needed
GPU	wgpu + WGSL shaders	wgpu 0.20+ has better Metal/Vulkan support; worth tracking
Compression	Mixed C/Rust	zlib-rs for deflate; lz4_flex for LZ4 — both pure Rust
Crypto	FNV-1a (unkeyed), SHA-256	blake3 (blake3_hash feature already exists) for high-speed content hashing; aes-gcm for encryption
FFI	libaec-sys (SZIP)	Only remaining non-optional C dep path

---

6. NetCDF-4 and Scientific Computing Context

NetCDF-4 is built on HDF5 (it IS HDF5 with specific conventions). ClawHDF5's clawhdf5-netcdf4 crate provides compatibility. Scientific domains that use HDF5/NetCDF-4:

• Climate science: CMIP6 datasets, ERA5 reanalysis (petabytes of NetCDF-4)
• Genomics: HDF5-backed formats (AnnData/h5ad for single-cell RNA-seq)
• Particle physics: CERN ROOT/HDF5 format
• Astronomy: FITS and HDF5 hybrid formats; SKA telescope data

For ClawHDF5 to serve these domains, the key gaps are:

1. Parallel collective I/O (MPI) — required for multi-node HPC ingestion
2. Blosc2 filter support — de-facto standard in h5py scientific workflows
3. ZFP lossy compression — common in simulation output

---

7. Security Research Context

7.1 Memory Poisoning

The MemoryGraft (2025) and SSGM (2026) papers that ClawHDF5 cites are the current frontier. New attack vectors emerging:

• Gradient-based poisoning: Adversarially crafting embeddings that are near arbitrary queries in vector space. ClawHDF5's anomaly detection checks text patterns but not embedding-space manipulation.
• Temporal poisoning: Injecting memories with falsified timestamps to manipulate temporal reasoning. ClawHDF5's WAL has CRC32 integrity but timestamps are not signed.

7.2 Supply Chain

The szip feature introduces a C FFI dependency (libaec). If not compiled in, there is no C dependency. The system-zlib-decompress feature also links against the system zlib. Both paths should be audited in deployments that require supply-chain provenance.