feat: migrate-engine improvements (content validation, schema, streaming, incremental)

clawhdf5-migrate:
- Real content validation: the post-migration check reads the written HDF5
  back (new hdf5_reader) and compares actual content — chunk text, embeddings,
  and every session/entity/relation field — to the source, not just row counts.
  A representative sample of chunk rows is verified by default; --validate-full
  checks every row. A count-preserving corruption no longer passes.
- Configurable schema: SQL is built from a SchemaConfig (table + ordered column
  names, defaulting to the ZeroClaw layout) instead of hardcoded queries, with
  --chunks-table / --sessions-table / --entities-table / --relations-table.
- Streaming count pass: --dry-run does a COUNT(*)-only pass per table instead
  of loading every row.
- Incremental migration: --incremental reads the existing output, reads only
  source chunks with id greater than the last migrated id, and appends them
  (metadata groups refreshed from source) rather than re-migrating everything.

clawhdf5-format:
- read_as_f32 / read_as_f64 now decode IEEE-754 half-precision (2-byte) floats
  via a no_std-safe bit conversion — needed to read float16-stored embeddings
  back (e.g. for migrate's content validation), previously a TypeMismatch.

Tests: f16 read unit test; migrate tests for content-corruption detection,
custom table names, and incremental append; CLI smoke-tested end-to-end and the
dense/incremental output verified with h5py.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

CHANGELOG.md

@@ -1,5 +1,181 @@
 # Changelog
 
+## Unreleased
+
+### New Features
+- `clawhdf5-migrate`: substantial engine improvements:
+  - **Real content validation** — the post-migration check now reads the written
+    HDF5 back and compares actual content (chunk text, embeddings, and every
+    session/entity/relation field) against the source, not just row counts. A
+    representative sample of chunk rows is verified by default; `--validate-full`
+    checks every row. A corrupt migration that preserves counts no longer passes.
+  - **Configurable schema** — table names are no longer hardcoded; queries are
+    built from a `SchemaConfig` (table + ordered column names, defaulting to the
+    ZeroClaw layout) with `--chunks-table` / `--sessions-table` /
+    `--entities-table` / `--relations-table` overrides.
+  - **Streaming count pass** — `--dry-run` now does a `COUNT(*)`-only pass per
+    table instead of loading every row into memory.
+  - **Incremental migration** — `--incremental` reads the existing output, reads
+    only source chunks newer than the last migrated id, and appends them
+    (refreshing the metadata groups), instead of re-migrating everything.
+- `clawhdf5-format`: read **IEEE-754 half-precision (f16)** floats. `read_as_f32`
+  / `read_as_f64` previously only handled 4- and 8-byte floats; 2-byte floats
+  (e.g. float16-stored embeddings) now decode via a no_std-safe bit conversion.
+- `clawhdf5-format`: **write multi-block fractal heaps** (root indirect block).
+  Dense attribute and dense link storage previously capped at a single direct
+  block (~64 KiB of heap data — a few thousand attributes/links). When the
+  objects exceed one direct block, the heap now lays out a root indirect block
+  (FHIB) over multiple direct blocks sized by the doubling table, distributing
+  objects across blocks with correct per-block heap offsets. Validated
+  end-to-end: a 2,500-attribute object and a 2,500-link group round-trip
+  through our reader and are read correctly by h5py. (Objects still may not
+  span a block — no huge-object path.)
+- `clawhdf5-format`: **write dense group link storage** (fractal heap + v2
+  B-tree). A group with more than 8 links (libhdf5's compact `max_compact`
+  default) is now written densely — its links live in a fractal heap indexed by
+  a v2 B-tree of type 5 (link-name index) referenced from the group's LinkInfo
+  message — instead of as inline Link messages. This matches libhdf5's
+  compact→dense switchover and keeps large groups out of the object header.
+  Reverse-engineered against libhdf5: link heaps use `heap_id_length` 7 /
+  `max_heap_size` 32 (vs 8 / 40 for attributes). The shared single-direct-block
+  fractal-heap builder is now parameterized and used by both dense attributes
+  and dense links. Validated end-to-end: our reader round-trips, and h5py reads
+  the dense groups we write. (Single direct block — up to ~a couple thousand
+  links per group; beyond that needs indirect blocks, still unsupported.)
+
+### Robustness
+- `clawhdf5-format`: harden the readers added this cycle against malformed /
+  hostile input — they parse untrusted bytes and must return errors, never
+  panic, OOM, or recurse without bound. Fixed concrete vectors found by audit
+  and locked in with adversarial tests:
+  - **Paged Fixed Array**: `1 << max_nelmts_bits` shift overflow (a `u8` ≥ 64);
+    element/page offset multiplications now checked; element count bounded by
+    file size.
+  - **H5S selection decoder**: `ALL`/`NONE` no longer claim 16 bytes they don't
+    have; hyperslab `rank` capped at 32 (`H5S_MAX_RANK`) to stop a giant
+    allocation; `iter_linear` coordinate/stride/product arithmetic is checked.
+  - **VDS mapping parser**: no pre-allocation from the untrusted `nused`; all
+    selection slicing is bounds-checked.
+  - **scale-offset / N-Bit filters**: `1 << minbits` overflow at `minbits == 64`;
+    N-Bit `bit_offset + precision` overflow; N-Bit type-tree recursion depth
+    capped (no stack overflow from a crafted nested tree); element counts
+    bounded by the chunk's expected decompressed size so a bogus count can't
+    drive a huge allocation.
+  - **Virtual Dataset assembly**: a virtual dataset whose source is itself
+    virtual (a cycle) now errors instead of recursing into a stack overflow.
+
+### New Features
+- `clawhdf5-agent`: **compress fixed-length string datasets** (memory text
+  chunks, session summaries, ids, tags, entity/relation names, …). These were
+  always stored uncompressed with a "chunked compound not yet supported" note
+  that was simply stale — chunked writes work for fixed-size string/compound
+  datatypes like any other. `write_string_dataset` now chunks + deflates a
+  string dataset once its payload reaches 4 KiB, so large, highly-redundant
+  NullPad content shrinks substantially while tiny metadata stays contiguous
+  (no chunk-overhead bloat).
+- `clawhdf5-format`: decode the **scale-offset filter** (id 6) — both the
+  integer variant (`H5Z_SO_INT`) and the floating-point **D-scale** variant
+  (`H5Z_SO_FLOAT_DSCALE`). Handles signed/unsigned int sizes, f32/f64, negative
+  minima, decimal scale factors and fill values; reverse-engineered against
+  HDF5 2.0 and validated end-to-end. The float E-scale variant remains
+  unsupported.
+- `clawhdf5-format`: decode the **N-Bit filter** (id 5) — atomic, **compound**
+  and **array** layouts (the full recursive type tree, nestable to any depth),
+  previously unsupported. Signed and unsigned reduced-precision integers and
+  float members all read end-to-end, validated against HDF5 2.0.
+
+### New Features
+- `clawhdf5` / `clawhdf5-format`: read **external-file Virtual Datasets (VDS)**.
+  The format layer gains `read_raw_data_full_with_resolver` and a
+  `VdsSourceResolver` callback (`Fn(&str) -> Option<Vec<u8>>`) that maps a
+  stored source file name to its bytes, so the pure-byte reader can pull in
+  external sources without a filesystem of its own. The `clawhdf5` `File` API
+  wires a default resolver that reads sibling source files relative to the
+  opened file's directory, so `File::open(...).dataset(...).read_*()` now
+  transparently assembles cross-file VDS. A source file the resolver cannot
+  supply leaves its region at the fill value (matching HDF5); an external
+  source with no resolver at all is a clean error. In-memory files
+  (`File::from_bytes`) have no directory, so only same-file VDS resolves there.
+- `clawhdf5-format`: assemble **same-file Virtual Datasets (VDS)** of any rank.
+  Previously a virtual layout returned `UnsupportedVersion`. The reader now
+  decodes the global-heap mapping block (reverse-engineered against HDF5 2.0:
+  `version · nused · [source-file · source-dataset · source-selection ·
+  virtual-selection]* · checksum`, including the block-version-1 same-file
+  marker), decodes the `H5S` source/virtual dataspace **selections** (ALL,
+  NONE, and version-3 regular hyperslabs), reads each same-file source dataset,
+  and scatters its selected elements into the virtual buffer in row-major order
+  (so multi-dimensional block mappings land correctly); unmapped regions are
+  left at the zero fill value. External-file sources return a clean unsupported
+  error. The previous `parse_vds_mappings` used a guessed layout that did not
+  match real files and is replaced.
+
+### Tests
+- `clawhdf5-format`: regression test for **scale-offset float E-scale**
+  datasets. The HDF5 library does not implement E-scale encoding — when asked
+  for it (`cd_values[0] = 1`) it stores the chunk raw and sets the chunk filter
+  mask to skip the filter — so these files read back verbatim purely by
+  honoring the per-chunk filter mask. The test locks in that behavior against a
+  fixture produced via the HDF5 low-level API; no E-scale decoder is needed.
+
+### Bug Fixes
+- `clawhdf5-format`: **read multi-direct-block fractal heaps**. The reader split
+  direct vs indirect block rows using the FRHP "Starting # of Rows in Root
+  Indirect Block" field (a constant, typically 1), so any heap whose data spans
+  more than one direct block — common in libhdf5 files with a large group or
+  many dense attributes — was misread as having indirect blocks and failed with
+  `InvalidFractalHeapSignature`. The split is now derived from the heap geometry
+  (`max_direct_rows = log2(max_direct / start) + 2`). Validated against an
+  h5py-written 400-dense-attribute group (root indirect block, 4 rows, 13 direct
+  blocks).
+- `clawhdf5-format`: scope the per-file **chunk cache by dataset**. The shared
+  `ChunkCache` built its chunk index once and reused it for every chunked
+  dataset in the file, keyed only by chunk coordinate with no dataset
+  discrimination. With a single chunked dataset per file this was latent; once a
+  file holds two chunked datasets of different rank (e.g. a 1-D compressed
+  string array and the 2-D embeddings matrix), the first dataset's index was
+  reused for the second, panicking with an out-of-bounds chunk coordinate. The
+  cache now rebinds (dropping its index, chunk-index map, layout, and
+  decompressed slots) whenever the dataset being read changes, while still
+  caching repeated/sequential access to the same dataset.
+- `clawhdf5-format`: read **paged Fixed Array** chunk indexes. A filtered,
+  fixed-dimension dataset with more than one data-block page (>1024 chunks by
+  default) previously failed with "paged Fixed Array data blocks not yet
+  supported". The reader now walks the page-init bitmap (MSB-first), skips
+  uninitialized pages, and resolves each page's fixed full-size slot (including
+  the short final page). Reverse-engineered and validated end-to-end against an
+  HDF5 2.0 file.
+- `clawhdf5-format`: read **array-typed datatypes** (e.g. an array-typed
+  compound member) via `read_as_i32/i64/u64/f32/f64` — previously a
+  `TypeMismatch`. The array is read as a flat sequence of its base elements
+  (recursing for nested arrays), applying base-type precision rules.
+- `clawhdf5-format`: **sign-extend reduced-precision fixed-point integers** on
+  read. A signed integer whose datatype precision is smaller than its storage
+  size is stored zero-filled, so e.g. a 16-bit-precision `-1` previously read as
+  `65535`. The integer read paths now extract the precision field and
+  sign-extend (full-width types are unchanged). Completes signed N-Bit reads and
+  also fixes un-filtered reduced-precision integer datasets.
+- `clawhdf5-format`: read datasets written by modern HDF5 (1.14+/2.0, i.e.
+  `libver=latest`). Compound (class 6) and array (class 10) datatype **version 5**
+  messages and data layout **version 5** messages were rejected as invalid; they
+  reuse the v3/v4 binary structure, so they are now accepted. This unblocks
+  reading compound types and — critically — every chunked/compressed dataset
+  written by HDF5 2.0. Found by running the h5py interop tests against
+  h5py 3.16 / HDF5 2.0.
+
+### Performance
+- `clawhdf5-format`: chunked writes now compress all chunks up front via
+  `compress_all_chunks`, running across rayon threads under the `parallel`
+  feature when there are more than 4 filtered chunks. On-disk layout is
+  unchanged. Speeds up compressed embedding writes in `clawhdf5-agent` (which
+  enables `parallel`).
+
+### Documentation
+- Fix stale package names across all 13 per-crate READMEs (`rustyhdf5-*` /
+  `edgehdf5-*` → `clawhdf5-*`, usage versions → 2.1.0).
+- Correct README workspace/test/crate stats and the CLAUDE.md CLI subcommand
+  list; document the `hnsw` and format compression/checksum feature flags and
+  the `entity_extract` / `async_memory` modules.
+
 ## v2.1.0 (2026-06-03)
 
 ### New Features

CLAUDE.md

@@ -54,7 +54,7 @@ cargo test --workspace
 ### CLI
 ```bash
 cargo run -p clawhdf5-cli -- --help
-# inspect, dump, index, search subcommands
+# create, save, search, recall, stats, flush-wal, agents-md, export, snapshot subcommands
 ```
 
 ### Python bindings

README.md

@@ -4,7 +4,7 @@
 
 [![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
 [![Rust](https://img.shields.io/badge/rust-1.75%2B-orange.svg)](https://www.rust-lang.org)
-[![Tests](https://img.shields.io/badge/tests-417%20passing-brightgreen.svg)](#benchmarks)
+[![Tests](https://img.shields.io/badge/tests-1500%2B%20passing-brightgreen.svg)](#benchmarks)
 [![LongMemEval](https://img.shields.io/badge/LongMemEval-Hit@5%2046%25%20BM25--only-blue.svg)](BENCHMARKS.md#longmemeval-results)
 [![Footprint](https://img.shields.io/badge/footprint-6.5%20KB%2Frecord-lightgrey.svg)](BENCHMARKS.md#memory-footprint)
 
@@ -170,9 +170,11 @@ ClawhDF5's agent memory engine implements research from 15+ recent papers on age
 | **`vector_search`** | Flat cosine, pre-normed, SIMD, BLAS, GPU, parallel search paths |
 | **`ivf` / `pq`** | IVF-PQ approximate nearest neighbor for billion-scale search |
 | **`bm25`** | BM25 keyword index with TF-IDF scoring |
+| **`entity_extract`** | Rule-based entity extraction from text chunks into the knowledge graph |
 | **`wal`** | Write-ahead log for crash-safe persistence |
 | **`memory_strategy`** | Pluggable strategies: save-every, semantic-shift, user-correction detection |
 | **`decision_gate`** | Sub-microsecond trivial/substantive classification |
+| **`async_memory`** | Tokio-based async wrapper over the memory store (`async` feature) |
 
 ---
 
@@ -313,7 +315,7 @@ let exported = backend.export_markdown("MEMORY.md")?;
 ## Crate Map
 
 ```
-clawhdf5 workspace (15 crates, 72K lines of Rust)
+clawhdf5 workspace (17 crates, 84K lines of Rust)
 │
 ├── Core HDF5
 │   ├── clawhdf5-types      — Type system definitions
@@ -327,14 +329,18 @@ clawhdf5 workspace (15 crates, 72K lines of Rust)
 │   └── clawhdf5-gpu         — GPU compute (wgpu)
 │
 ├── Agent Memory
-│   ├── clawhdf5-agent       — Memory engine (16.8K lines, 29 modules)
+│   ├── clawhdf5-agent       — Memory engine (20.7K lines, 32 modules)
-│   ├── clawhdf5-ann         — HNSW approximate nearest neighbor
+│   ├── clawhdf5-ann         — HNSW approximate nearest neighbor (default backend)
 │   ├── clawhdf5-migrate     — SQLite → HDF5 migration
 │   ├── clawhdf5-android     — Android JNI bridge
 │   └── clawhdf5-cli         — CLI tool
 │
-└── Bindings
+├── Bindings
-    └── clawhdf5-py          — Python (PyO3)
+│   ├── clawhdf5-py          — Python (PyO3)
+│   └── clawhdf5-napi        — Node.js (napi-rs)
+│
+└── Tooling
+    └── clawhdf5-bench       — Benchmark suite
 ```
 
 ---
@@ -365,6 +371,7 @@ ClawhDF5's agent memory design draws from 15+ recent papers:
 |------|---------|-------------|
 | `agent` | no | Full agent memory layer |
 | `float16` | **yes** | Half-precision embedding storage (2× compression) |
+| `hnsw` | **yes** | HNSW approximate vector index for `hybrid_search` (via `clawhdf5-ann`); disable for an exact linear scan |
 | `parallel` | no | Rayon parallel search |
 | `fast-math` | no | BLAS matrix-vector multiply |
 | `accelerate` | no | Apple Accelerate / AMX (macOS) |
@@ -380,7 +387,14 @@ ClawhDF5's agent memory design draws from 15+ recent papers:
 | `deflate` | yes | Deflate compression |
 | `checksum` | yes | Jenkins lookup3 verification |
 | `provenance` | yes | SHA-256 provenance attributes |
-| `parallel` | no | Parallel chunk encoding (rayon) |
+| `fast-deflate` | **yes** | zlib-ng backend for faster deflate |
+| `system-zlib-decompress` | **yes** | Use the system zlib for decompression where available |
+| `parallel` | no | Parallel chunk encoding + compression (rayon) |
+| `fast-checksum` | no | crc32fast-accelerated checksums |
+| `lz4` | no | LZ4 block compression filter (id 32004) |
+| `zstd` | no | Zstandard compression filter (id 32015) |
+| `system-zlib` / `zlib-rs` | no | Alternative zlib backends for deflate |
+| `blake3_hash` | no | BLAKE3 content hashing for provenance |
 
 ---
 

crates/clawhdf5-accel/README.md

@@ -1,9 +1,9 @@
-# rustyhdf5-accel
+# clawhdf5-accel
 
-[![crates.io](https://img.shields.io/crates/v/rustyhdf5-accel.svg)](https://crates.io/crates/rustyhdf5-accel)
+[![crates.io](https://img.shields.io/crates/v/clawhdf5-accel.svg)](https://crates.io/crates/clawhdf5-accel)
-[![docs.rs](https://docs.rs/rustyhdf5-accel/badge.svg)](https://docs.rs/rustyhdf5-accel)
+[![docs.rs](https://docs.rs/clawhdf5-accel/badge.svg)](https://docs.rs/clawhdf5-accel)
 
-SIMD-accelerated operations for rustyhdf5.
+SIMD-accelerated operations for clawhdf5.
 
 ## Features
 
@@ -15,7 +15,7 @@ SIMD-accelerated operations for rustyhdf5.
 ## Usage
 
 ```rust
-use rustyhdf5_accel::checksum::crc32_simd;
+use clawhdf5_accel::checksum::crc32_simd;
 
 let crc = crc32_simd(&data);
 ```

crates/clawhdf5-agent/README.md

@@ -1,18 +1,18 @@
-# edgehdf5-memory
+# clawhdf5-agent
 
-[![crates.io](https://img.shields.io/crates/v/edgehdf5-memory.svg)](https://crates.io/crates/edgehdf5-memory)
+[![crates.io](https://img.shields.io/crates/v/clawhdf5-agent.svg)](https://crates.io/crates/clawhdf5-agent)
-[![docs.rs](https://img.shields.io/docsrs/edgehdf5-memory)](https://docs.rs/edgehdf5-memory)
+[![docs.rs](https://img.shields.io/docsrs/clawhdf5-agent)](https://docs.rs/clawhdf5-agent)
 
 HDF5-backed persistent memory store for on-device AI agents.
 
-Built on [rustyhdf5](https://crates.io/crates/rustyhdf5), edgehdf5-memory provides a vector-searchable memory backend optimized for edge AI workloads. Store embeddings, text chunks, and metadata in a single HDF5 file with SIMD-accelerated similarity search.
+Built on [clawhdf5](https://crates.io/crates/clawhdf5), clawhdf5-agent provides a vector-searchable memory backend optimized for edge AI workloads. Store embeddings, text chunks, and metadata in a single HDF5 file with SIMD-accelerated similarity search.
 
 ## Features
 
 - Persistent vector store in HDF5 format
 - Cosine similarity and L2 distance search
-- SIMD-accelerated via rustyhdf5-accel (AVX2, NEON)
+- SIMD-accelerated via clawhdf5-accel (AVX2, NEON)
-- Optional GPU acceleration via rustyhdf5-gpu
+- Optional GPU acceleration via clawhdf5-gpu
 - Memory-mapped access for large stores
 - f16 storage support for compact embeddings
 
@@ -20,7 +20,7 @@ Built on [rustyhdf5](https://crates.io/crates/rustyhdf5), edgehdf5-memory provid
 
 ```toml
 [dependencies]
-edgehdf5-memory = "1.93"
+clawhdf5-agent = "2.1.0"
 ```
 
 ## License

crates/clawhdf5-agent/src/schema.rs

@@ -65,7 +65,7 @@ fn build_memory_group(
     let mut group = builder.create_group("memory");
 
     // chunks: fixed-length string array
-    write_string_dataset(&mut group, "chunks", &cache.chunks, false);
+    write_string_dataset(&mut group, "chunks", &cache.chunks);
 
     // embeddings: f32 [N x D]
     let n = cache.embeddings.len() as u64;
@@ -101,7 +101,7 @@ fn build_memory_group(
     }
 
     // source_channel: fixed-length string array
-    write_string_dataset(&mut group, "source_channel", &cache.source_channels, false);
+    write_string_dataset(&mut group, "source_channel", &cache.source_channels);
 
     // timestamps: f64 array
     group
@@ -109,11 +109,11 @@ fn build_memory_group(
         .with_f64_data(&cache.timestamps)
         .fill_time(FillTime::Never);
 
-    // session_ids: fixed-length string array (no compression — chunked compound not yet supported)
+    // session_ids: fixed-length string array (auto-compressed when large)
-    write_string_dataset(&mut group, "session_ids", &cache.session_ids, false);
+    write_string_dataset(&mut group, "session_ids", &cache.session_ids);
 
-    // tags: fixed-length string array (no compression — chunked compound not yet supported)
+    // tags: fixed-length string array (auto-compressed when large)
-    write_string_dataset(&mut group, "tags", &cache.tags, false);
+    write_string_dataset(&mut group, "tags", &cache.tags);
 
     // tombstones: u8 array — use compact if small
     {
@@ -150,7 +150,7 @@ fn build_sessions_group(
     let mut group = builder.create_group("sessions");
 
     let ids: Vec<String> = sessions.entries.iter().map(|e| e.id.clone()).collect();
-    write_string_dataset(&mut group, "ids", &ids, false);
+    write_string_dataset(&mut group, "ids", &ids);
 
     let start_idxs: Vec<i64> = sessions
         .entries
@@ -165,14 +165,14 @@ fn build_sessions_group(
     group.create_dataset("end_idxs").with_i64_data(&end_idxs);
 
     let channels: Vec<String> = sessions.entries.iter().map(|e| e.channel.clone()).collect();
-    write_string_dataset(&mut group, "channels", &channels, false);
+    write_string_dataset(&mut group, "channels", &channels);
 
     let timestamps: Vec<f64> = sessions.entries.iter().map(|e| e.ts).collect();
     group
         .create_dataset("timestamps")
         .with_f64_data(&timestamps);
 
-    write_string_dataset(&mut group, "summaries", &sessions.summaries, false);
+    write_string_dataset(&mut group, "summaries", &sessions.summaries);
 
     let finished = group.finish();
     builder.add_group(finished);
@@ -192,14 +192,14 @@ fn build_knowledge_group(
         .with_i64_data(&entity_ids);
 
     let entity_names: Vec<String> = knowledge.entities.iter().map(|e| e.name.clone()).collect();
-    write_string_dataset(&mut group, "entity_names", &entity_names, false);
+    write_string_dataset(&mut group, "entity_names", &entity_names);
 
     let entity_types: Vec<String> = knowledge
         .entities
         .iter()
         .map(|e| e.entity_type.clone())
         .collect();
-    write_string_dataset(&mut group, "entity_types", &entity_types, false);
+    write_string_dataset(&mut group, "entity_types", &entity_types);
 
     let emb_idxs: Vec<i64> = knowledge.entities.iter().map(|e| e.embedding_idx).collect();
     group
@@ -222,7 +222,7 @@ fn build_knowledge_group(
         .iter()
         .map(|r| r.relation.clone())
         .collect();
-    write_string_dataset(&mut group, "relation_types", &rel_types, false);
+    write_string_dataset(&mut group, "relation_types", &rel_types);
 
     let rel_weights: Vec<f32> = knowledge.relations.iter().map(|r| r.weight).collect();
     group
@@ -234,7 +234,7 @@ fn build_knowledge_group(
 
     // Aliases
     if !knowledge.alias_strings.is_empty() {
-        write_string_dataset(&mut group, "alias_strings", &knowledge.alias_strings, false);
+        write_string_dataset(&mut group, "alias_strings", &knowledge.alias_strings);
         group
             .create_dataset("alias_entity_ids")
             .with_i64_data(&knowledge.alias_entity_ids);
@@ -252,11 +252,15 @@ fn build_knowledge_group(
 ///
 /// When `compress` is true, uses chunked storage with deflate(6) —
 /// NullPad strings have high redundancy and compress very well.
+/// Payload size (bytes) at or above which a fixed-length string dataset is
+/// stored chunked + deflate-compressed. Below this, the chunk B-tree/heap
+/// overhead outweighs the savings, so the data is left contiguous.
+const STRING_COMPRESS_THRESHOLD: usize = 4096;
+
 fn write_string_dataset(
     group: &mut clawhdf5_format::type_builders::GroupBuilder,
     name: &str,
     strings: &[String],
-    compress: bool,
 ) {
     if strings.is_empty() {
         // Empty dataset: use 1-byte string type with no data
@@ -278,6 +282,7 @@ fn write_string_dataset(
         bytes.resize(max_len, 0);
         raw.extend_from_slice(&bytes);
     }
+    let raw_len = raw.len();
 
     let dtype = Datatype::String {
         size: max_len as u32,
@@ -288,9 +293,12 @@ fn write_string_dataset(
         .create_dataset(name)
         .with_compound_data(dtype, raw, strings.len() as u64);
 
-    // Deflate compression for string datasets — NullPad has high redundancy
+    // Fixed-length NullPad strings have high redundancy (padding + repeated
-    if compress && strings.len() > 1 {
+    // content), so deflate pays off once the payload is large enough to absorb
-        // Chunk size: target ~64KB chunks for string data
+    // the chunking overhead. Fixed-length string datasets are chunkable like
+    // any other fixed-size datatype.
+    if strings.len() > 1 && raw_len >= STRING_COMPRESS_THRESHOLD {
+        // Target ~64KB chunks for string data.
         let elem_size = max_len as u64;
         let target_chunk = 64 * 1024;
         let rows_per_chunk = (target_chunk / elem_size).max(1).min(strings.len() as u64);

crates/clawhdf5-ann/README.md

@@ -1,7 +1,7 @@
-# rustyhdf5-ann
+# clawhdf5-ann
 
-[![crates.io](https://img.shields.io/crates/v/rustyhdf5-ann.svg)](https://crates.io/crates/rustyhdf5-ann)
+[![crates.io](https://img.shields.io/crates/v/clawhdf5-ann.svg)](https://crates.io/crates/clawhdf5-ann)
-[![docs.rs](https://docs.rs/rustyhdf5-ann/badge.svg)](https://docs.rs/rustyhdf5-ann)
+[![docs.rs](https://docs.rs/clawhdf5-ann/badge.svg)](https://docs.rs/clawhdf5-ann)
 
 HNSW approximate nearest neighbor index stored as HDF5.
 
@@ -14,7 +14,7 @@ HNSW approximate nearest neighbor index stored as HDF5.
 ## Usage
 
 ```rust
-use rustyhdf5_ann::HnswIndex;
+use clawhdf5_ann::HnswIndex;
 
 let index = HnswIndex::from_hdf5("vectors.h5").unwrap();
 let neighbors = index.search(&query, 10);

crates/clawhdf5-derive/README.md

@@ -1,9 +1,9 @@
-# rustyhdf5-derive
+# clawhdf5-derive
 
-[![crates.io](https://img.shields.io/crates/v/rustyhdf5-derive.svg)](https://crates.io/crates/rustyhdf5-derive)
+[![crates.io](https://img.shields.io/crates/v/clawhdf5-derive.svg)](https://crates.io/crates/clawhdf5-derive)
-[![docs.rs](https://docs.rs/rustyhdf5-derive/badge.svg)](https://docs.rs/rustyhdf5-derive)
+[![docs.rs](https://docs.rs/clawhdf5-derive/badge.svg)](https://docs.rs/clawhdf5-derive)
 
-Derive macros for rustyhdf5 HDF5 traits.
+Derive macros for clawhdf5 HDF5 traits.
 
 ## Features
 
@@ -13,7 +13,7 @@ Derive macros for rustyhdf5 HDF5 traits.
 ## Usage
 
 ```rust
-use rustyhdf5_derive::HDF5Type;
+use clawhdf5_derive::HDF5Type;
 
 #[derive(HDF5Type)]
 struct Point {

crates/clawhdf5-filters/README.md

@@ -1,9 +1,9 @@
-# rustyhdf5-filters
+# clawhdf5-filters
 
-[![crates.io](https://img.shields.io/crates/v/rustyhdf5-filters.svg)](https://crates.io/crates/rustyhdf5-filters)
+[![crates.io](https://img.shields.io/crates/v/clawhdf5-filters.svg)](https://crates.io/crates/clawhdf5-filters)
-[![docs.rs](https://docs.rs/rustyhdf5-filters/badge.svg)](https://docs.rs/rustyhdf5-filters)
+[![docs.rs](https://docs.rs/clawhdf5-filters/badge.svg)](https://docs.rs/clawhdf5-filters)
 
-Filter and compression pipeline for rustyhdf5.
+Filter and compression pipeline for clawhdf5.
 
 ## Features
 
@@ -14,7 +14,7 @@ Filter and compression pipeline for rustyhdf5.
 ## Usage
 
 ```rust
-use rustyhdf5_filters::{deflate_decode, deflate_encode};
+use clawhdf5_filters::{deflate_decode, deflate_encode};
 
 let compressed = deflate_encode(&data, 6).unwrap();
 let decompressed = deflate_decode(&compressed).unwrap();

crates/clawhdf5-format/README.md

@@ -1,7 +1,7 @@
-# rustyhdf5-format
+# clawhdf5-format
 
-[![crates.io](https://img.shields.io/crates/v/rustyhdf5-format.svg)](https://crates.io/crates/rustyhdf5-format)
+[![crates.io](https://img.shields.io/crates/v/clawhdf5-format.svg)](https://crates.io/crates/clawhdf5-format)
-[![docs.rs](https://docs.rs/rustyhdf5-format/badge.svg)](https://docs.rs/rustyhdf5-format)
+[![docs.rs](https://docs.rs/clawhdf5-format/badge.svg)](https://docs.rs/clawhdf5-format)
 
 Pure-Rust HDF5 binary format parsing and writing — no C dependencies.
 
@@ -16,7 +16,7 @@ Pure-Rust HDF5 binary format parsing and writing — no C dependencies.
 ## Usage
 
 ```rust
-use rustyhdf5_format::Superblock;
+use clawhdf5_format::Superblock;
 
 let data = std::fs::read("data.h5").unwrap();
 let sb = Superblock::from_bytes(&data).unwrap();

crates/clawhdf5-format/src/chunk_cache.rs

@@ -256,6 +256,14 @@ struct CacheInner {
     /// Populated once per dataset on first access.
     index: Option<HashMap<ChunkCoord, ChunkInfo>>,
 
+    /// Address of the dataset (its chunk-index base address) that the cached
+    /// index, chunk index, layout, and decompressed slots currently belong to.
+    /// The cache is shared per file across datasets, so every cached-read entry
+    /// checks this and resets the per-dataset state when the dataset changes —
+    /// otherwise one dataset's chunk index (with its own rank) would be reused
+    /// for another, corrupting reads.
+    index_addr: Option<u64>,
+
     /// LRU cache of decompressed chunk data.
     slots: Vec<CachedChunk>,
 
@@ -334,6 +342,7 @@ impl ChunkCache {
         Self {
             inner: std::sync::Mutex::new(CacheInner {
                 index: None,
+                index_addr: None,
                 slots: Vec::with_capacity(max_slots.min(64)),
                 current_bytes: 0,
                 max_bytes,
@@ -349,6 +358,29 @@ impl ChunkCache {
 
     // ----- Index operations -----
 
+    /// Bind the cache to the dataset at chunk-index address `addr`.
+    ///
+    /// The cache is shared per file across all of its datasets. If the cache
+    /// currently holds state for a different dataset, all per-dataset state
+    /// (chunk index, chunk-index map, layout, and decompressed slots) is
+    /// dropped so the next access rebuilds it for this dataset. Reading the
+    /// same dataset again is a no-op, preserving the cache's benefit for
+    /// repeated/sequential access. Returns `true` if a reset occurred.
+    pub fn ensure_dataset(&self, addr: u64) -> bool {
+        let mut inner = self.inner.lock().unwrap_or_else(|e| e.into_inner());
+        if inner.index_addr == Some(addr) {
+            return false;
+        }
+        inner.index = None;
+        inner.chunk_index = None;
+        inner.chunk_layout = None;
+        inner.slots.clear();
+        inner.current_bytes = 0;
+        inner.last_coord = None;
+        inner.index_addr = Some(addr);
+        true
+    }
+
     /// Returns `true` if the chunk index has been built.
     pub fn has_index(&self) -> bool {
         self.inner
@@ -565,6 +597,7 @@ impl ChunkCache {
     pub fn clear(&self) {
         let mut inner = self.inner.lock().unwrap_or_else(|e| e.into_inner());
         inner.index = None;
+        inner.index_addr = None;
         inner.slots.clear();
         inner.current_bytes = 0;
         inner.tick = 0;

crates/clawhdf5-format/src/chunked_read.rs

@@ -593,6 +593,10 @@ pub fn read_chunked_data_cached(
         )));
     }
 
+    // The per-file cache is shared across datasets; bind it to this one so a
+    // different dataset's chunk index is never reused for this read.
+    cache.ensure_dataset(addr);
+
     // Populate chunk index on first access
     if !cache.has_index() {
         let chunks = match (version, chunk_index_type) {
@@ -946,6 +950,10 @@ pub fn read_chunked_data_sweep(
         )));
     }
 
+    // The per-file cache is shared across datasets; bind it to this one so a
+    // different dataset's chunk index is never reused for this read.
+    cache.ensure_dataset(addr);
+
     // Populate chunk index on first access
     if !cache.has_index() {
         let chunks = match (version, chunk_index_type) {
@@ -1169,6 +1177,10 @@ pub fn read_chunked_data_indexed(
         )));
     }
 
+    // The per-file cache is shared across datasets; bind it to this one so a
+    // different dataset's chunk index is never reused for this read.
+    cache.ensure_dataset(addr);
+
     // Build chunk index on first access
     if !cache.has_chunk_index() {
         let chunks = match (version, chunk_index_type) {

crates/clawhdf5-format/src/chunked_write.rs

@@ -238,11 +238,15 @@ pub fn split_into_chunks(
 }
 
 /// Parallel compression threshold: use rayon when chunk count exceeds this.
-#[allow(dead_code)]
+#[cfg(feature = "parallel")]
 const PARALLEL_COMPRESS_THRESHOLD: usize = 4;
 
 /// Compress all chunks, using parallel compression when beneficial.
-#[allow(dead_code)]
+///
+/// With the `parallel` feature and more than [`PARALLEL_COMPRESS_THRESHOLD`]
+/// filtered chunks, compression runs across rayon threads; otherwise it is
+/// sequential. Output order matches input order, so per-chunk bytes are
+/// identical to the sequential path.
 fn compress_all_chunks(
     chunks: &[(Vec<u64>, Vec<u8>)],
     pipeline: &Option<FilterPipeline>,
@@ -578,17 +582,16 @@ pub fn build_chunked_data_at_ext(
     let num_chunks = chunks.len();
     let has_filters = pipeline.is_some();
 
-    // Compress each chunk, padding to cache-line boundaries for aligned access
+    // Compress all chunks up front (parallel under the `parallel` feature),
+    // then lay them out sequentially with cache-line padding for aligned access.
+    // Compression order matches chunk order, so the on-disk layout is identical
+    // to the previous per-chunk sequential path.
+    let compressed_chunks = compress_all_chunks(&chunks, &pipeline, element_size as u32)?;
+
     let mut data_buf = Vec::new();
     let mut written_chunks = Vec::with_capacity(num_chunks);
 
-    for (_offsets, chunk_bytes) in &chunks {
+    for ((_offsets, chunk_bytes), compressed) in chunks.iter().zip(compressed_chunks.iter()) {
-        let compressed = if let Some(pl) = pipeline.as_ref() {
-            compress_chunk(chunk_bytes, pl, element_size as u32)?
-        } else {
-            chunk_bytes.clone()
-        };
-
         // Pad current position to cache-line boundary
         let aligned_offset = align_to_cache_line(data_buf.len());
         if aligned_offset > data_buf.len() {
@@ -599,7 +602,7 @@ pub fn build_chunked_data_at_ext(
         let compressed_size = compressed.len() as u64;
         let raw_size = chunk_bytes.len() as u64;
 
-        data_buf.extend_from_slice(&compressed);
+        data_buf.extend_from_slice(compressed);
 
         written_chunks.push(WrittenChunk {
             address,

crates/clawhdf5-format/src/data_layout.rs

@@ -67,74 +67,81 @@ pub enum DataLayout {
     },
 }
 
-/// Parse VDS mappings from global heap object data.
+/// Parse VDS mappings from global-heap object data.
 ///
-/// The global heap object for a VDS layout contains a serialized list of
+/// The global-heap block holding a VDS mapping list is laid out as
-/// source mappings. Each mapping has:
+/// (reverse-engineered and validated against HDF5 2.0):
-/// - Virtual selection (serialized dataspace selection, variable length)
-/// - Source file name (null-terminated string)
-/// - Source dataset name (null-terminated string)
-/// - Source selection (serialized dataspace selection, variable length)
 ///
-/// The overall format starts with:
+/// ```text
-/// - version (4 bytes LE) — currently 0
+/// version(1) · nused(length_size, LE) · entry[nused] · checksum(4)
-/// - entry count (not explicitly stored; parse until data exhausted)
+/// ```
 ///
-/// This is a best-effort parser that handles common VDS files. The exact
+/// Each entry is:
-/// binary format is not fully specified publicly and may vary by HDF5 version.
+/// - source file name — a null-terminated string in **block version 0**; in
-pub fn parse_vds_mappings(heap_data: &[u8]) -> Result<Vec<VdsMapping>, FormatError> {
+///   **block version 1** a same-file reference is encoded as a single `0x04`
-    if heap_data.len() < 4 {
+///   marker byte (the source file is the virtual file itself) in place of the
+///   name;
+/// - source dataset name (null-terminated string);
+/// - source selection (serialized `H5S` dataspace selection — self-describing
+///   in length);
+/// - virtual selection (serialized `H5S` dataspace selection).
+///
+/// The selections are decoded with [`crate::selection::Selection`] purely to
+/// learn their byte length so the entry list can be walked; the raw selection
+/// bytes are retained on each [`VdsMapping`] for the reader to interpret.
+pub fn parse_vds_mappings(
+    heap_data: &[u8],
+    length_size: u8,
+) -> Result<Vec<VdsMapping>, FormatError> {
+    use crate::selection::Selection;
+
+    let ls = length_size as usize;
+    if heap_data.len() < 1 + ls {
         return Ok(Vec::new());
     }
-    // VDS global heap object starts with version(4)
+    let version = heap_data[0];
-    let _version = u32::from_le_bytes([heap_data[0], heap_data[1], heap_data[2], heap_data[3]]);
+    let mut pos = 1;
-    let mut pos = 4;
+    let nused = read_length(heap_data, pos, length_size)?;
+    pos += ls;
+
+    // `nused` is untrusted; don't pre-allocate from it. Each entry consumes at
+    // least a few bytes, so the loop is naturally bounded by the heap data and
+    // a bogus `nused` simply errors out on the first short read.
     let mut mappings = Vec::new();
+    // Reads one self-describing selection at `pos`, returning its raw bytes and
+    // advancing past it — bounds-checked so a corrupt selection can't overrun.
+    let read_selection = |heap_data: &[u8], pos: &mut usize| -> Result<Vec<u8>, FormatError> {
+        let rest = heap_data.get(*pos..).ok_or(FormatError::UnexpectedEof {
+            expected: *pos,
+            available: heap_data.len(),
+        })?;
+        let (_, len) = Selection::decode_serialized(rest)?;
+        let bytes = rest
+            .get(..len)
+            .ok_or(FormatError::UnexpectedEof {
+                expected: pos.saturating_add(len),
+                available: heap_data.len(),
+            })?
+            .to_vec();
+        *pos += len;
+        Ok(bytes)
+    };
 
-    while pos < heap_data.len() {
+    for _ in 0..nused {
-        // Each entry: virtual_selection_size(4) + virtual_selection(N) +
+        // Source file name (with the version-1 same-file marker handled).
-        //             source_file_name(null-term) + source_dataset_name(null-term) +
+        let source_file = if version >= 1 && heap_data.get(pos) == Some(&0x04) {
-        //             source_selection_size(4) + source_selection(N)
+            pos += 1;
-        if pos + 4 > heap_data.len() {
+            String::from(".")
-            break;
+        } else {
-        }
+            read_null_terminated_string(heap_data, &mut pos)?
+        };
 
-        // Virtual selection
+        // Source dataset name.
-        let vsel_size = u32::from_le_bytes([
-            heap_data[pos],
-            heap_data[pos + 1],
-            heap_data[pos + 2],
-            heap_data[pos + 3],
-        ]) as usize;
-        pos += 4;
-        if pos + vsel_size > heap_data.len() {
-            break;
-        }
-        let virtual_selection = heap_data[pos..pos + vsel_size].to_vec();
-        pos += vsel_size;
-
-        // Source file name (null-terminated)
-        let source_file = read_null_terminated_string(heap_data, &mut pos)?;
-
-        // Source dataset name (null-terminated)
         let source_dataset = read_null_terminated_string(heap_data, &mut pos)?;
 
-        // Source selection
+        // Source selection, then virtual selection (both self-describing length).
-        if pos + 4 > heap_data.len() {
+        let source_selection = read_selection(heap_data, &mut pos)?;
-            break;
+        let virtual_selection = read_selection(heap_data, &mut pos)?;
-        }
-        let ssel_size = u32::from_le_bytes([
-            heap_data[pos],
-            heap_data[pos + 1],
-            heap_data[pos + 2],
-            heap_data[pos + 3],
-        ]) as usize;
-        pos += 4;
-        if pos + ssel_size > heap_data.len() {
-            break;
-        }
-        let source_selection = heap_data[pos..pos + ssel_size].to_vec();
-        pos += ssel_size;
 
         mappings.push(VdsMapping {
             source_file,
@@ -235,7 +242,7 @@ impl DataLayout {
                     index: *global_heap_index as u16,
                 },
             )?;
-            *mappings = parse_vds_mappings(&obj.data)?;
+            *mappings = parse_vds_mappings(&obj.data, length_size)?;
         }
         Ok(())
     }
@@ -250,7 +257,9 @@ impl DataLayout {
 
         match version {
             3 => Self::parse_v3(data, layout_class, offset_size, length_size),
-            4 => Self::parse_v4(data, layout_class, offset_size, length_size),
+            // v5 (emitted by HDF5 1.14+/2.0 with `libver=latest`) uses the same
+            // message structure as v4 — only the version number was bumped.
+            4 | 5 => Self::parse_v4(data, layout_class, offset_size, length_size),
             _ => Err(FormatError::InvalidLayoutVersion(version)),
         }
     }
@@ -626,6 +635,30 @@ mod tests {
         );
     }
 
+    #[test]
+    fn v5_chunked_from_hdf5_2_0() {
+        // Real data layout message from h5py 3.16 / HDF5 2.0 (`libver=latest`)
+        // for a gzip-compressed 1-D chunked dataset. Version 5 uses the same
+        // structure as v4 (here: chunked, Fixed Array index). Regression guard
+        // for reading modern-format chunked datasets.
+        let bytes: [u8; 17] = [
+            0x05, 0x02, 0x00, 0x02, 0x01, 0x0a, 0x08, 0x03, 0x0a, 0xef, 0x05, 0x00, 0x00, 0x00,
+            0x00, 0x00, 0x00,
+        ];
+        let layout = DataLayout::parse(&bytes, 8, 8).unwrap();
+        match layout {
+            DataLayout::Chunked {
+                chunk_dimensions,
+                chunk_index_type,
+                ..
+            } => {
+                assert_eq!(chunk_dimensions, vec![10, 8]);
+                assert_eq!(chunk_index_type, Some(3)); // Fixed Array
+            }
+            other => panic!("expected Chunked, got {other:?}"),
+        }
+    }
+
     #[test]
     fn v4_chunked_single_chunk_no_filters() {
         let mut buf = vec![4u8, 2]; // version=4, class=2
@@ -678,9 +711,10 @@ mod tests {
 
     #[test]
     fn invalid_version() {
-        let buf = vec![5u8, 0, 0, 0];
+        // v3-v5 are supported; v6 is not a real layout message version.
+        let buf = vec![6u8, 0, 0, 0];
         let err = DataLayout::parse(&buf, 8, 8).unwrap_err();
-        assert_eq!(err, FormatError::InvalidLayoutVersion(5));
+        assert_eq!(err, FormatError::InvalidLayoutVersion(6));
     }
 
     #[test]
@@ -747,32 +781,84 @@ mod tests {
     }
 
     #[test]
-    fn parse_vds_mappings_basic() {
+    fn parse_vds_mappings_same_file_v1() {
-        // Build a simple VDS mapping blob
+        // The exact global-heap block written by HDF5 2.0 for a same-file VDS
-        let mut blob = Vec::new();
+        // with two sources: src_a -> virtual[0:4], src_b -> virtual[4:8].
-        blob.extend_from_slice(&0u32.to_le_bytes()); // version=0
+        let blob = [
+            0x01u8, // block version 1
+            0x02, 0, 0, 0, 0, 0, 0, 0, // nused = 2 (length_size = 8)
+            // entry 0
+            0x04, // same-file marker (replaces file name)
+            0x73, 0x72, 0x63, 0x5f, 0x61, 0x00, // "src_a\0"
+            0x03, 0, 0, 0, 0x01, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, // source sel = ALL
+            0x02, 0, 0, 0, 0x03, 0, 0, 0, 0x01, 0x02, 0x01, 0, 0, 0, // virtual sel: HYPER v3
+            0x00, 0x00, 0x01, 0x00, 0x01, 0x00, 0x04, 0x00, // start0 stride1 count1 block4
+            // entry 1
+            0x04, 0x73, 0x72, 0x63, 0x5f, 0x62, 0x00, // "src_b\0"
+            0x03, 0, 0, 0, 0x01, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, // source sel = ALL
+            0x02, 0, 0, 0, 0x03, 0, 0, 0, 0x01, 0x02, 0x01, 0, 0, 0, // virtual sel: HYPER v3
+            0x04, 0x00, 0x01, 0x00, 0x01, 0x00, 0x04, 0x00, // start4 stride1 count1 block4
+            0x68, 0xf0, 0x3e, 0xe4, // checksum (ignored)
+        ];
+        let mappings = parse_vds_mappings(&blob, 8).unwrap();
+        assert_eq!(mappings.len(), 2);
+        assert_eq!(mappings[0].source_file, ".");
+        assert_eq!(mappings[0].source_dataset, "src_a");
+        assert_eq!(mappings[1].source_file, ".");
+        assert_eq!(mappings[1].source_dataset, "src_b");
 
-        // Virtual selection (8 bytes of dummy data)
+        // Virtual selections decode to [0:4] and [4:8].
-        let vsel = vec![1, 2, 3, 4, 5, 6, 7, 8];
+        use crate::selection::Selection;
-        blob.extend_from_slice(&(vsel.len() as u32).to_le_bytes());
+        let (v0, _) = Selection::decode_serialized(&mappings[0].virtual_selection).unwrap();
-        blob.extend_from_slice(&vsel);
+        let (v1, _) = Selection::decode_serialized(&mappings[1].virtual_selection).unwrap();
+        assert_eq!(v0.iter_linear_1d(8).unwrap(), vec![0, 1, 2, 3]);
+        assert_eq!(v1.iter_linear_1d(8).unwrap(), vec![4, 5, 6, 7]);
+    }
 
-        // Source file name
+    #[test]
-        blob.extend_from_slice(b"source.h5\0");
+    fn parse_vds_mappings_external_v0() {
-
+        // Block version 0 with an explicit (external) source file name.
-        // Source dataset name
+        let blob = [
-        blob.extend_from_slice(b"/data\0");
+            0x00u8, // block version 0
-
+            0x01, 0, 0, 0, 0, 0, 0, 0, // nused = 1
-        // Source selection (4 bytes)
+            0x73, 0x72, 0x63, 0x5f, 0x65, 0x78, 0x74, 0x2e, 0x68, 0x35, 0x00, // "src_ext.h5\0"
-        let ssel = vec![10, 20, 30, 40];
+            0x64, 0x61, 0x74, 0x61, 0x00, // "data\0"
-        blob.extend_from_slice(&(ssel.len() as u32).to_le_bytes());
+            0x03, 0, 0, 0, 0x01, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, // source sel = ALL
-        blob.extend_from_slice(&ssel);
+            0x03, 0, 0, 0, 0x01, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, // virtual sel = ALL
-
+        ];
-        let mappings = parse_vds_mappings(&blob).unwrap();
+        let mappings = parse_vds_mappings(&blob, 8).unwrap();
         assert_eq!(mappings.len(), 1);
-        assert_eq!(mappings[0].source_file, "source.h5");
+        assert_eq!(mappings[0].source_file, "src_ext.h5");
-        assert_eq!(mappings[0].source_dataset, "/data");
+        assert_eq!(mappings[0].source_dataset, "data");
-        assert_eq!(mappings[0].virtual_selection, vsel);
+    }
-        assert_eq!(mappings[0].source_selection, ssel);
+
+    #[test]
+    fn parse_vds_mappings_huge_nused_does_not_oom_or_panic() {
+        // nused = u64::MAX with no entry data: must error, not pre-allocate or
+        // overrun.
+        let mut blob = vec![0x01u8];
+        blob.extend_from_slice(&u64::MAX.to_le_bytes());
+        assert!(parse_vds_mappings(&blob, 8).is_err());
+    }
+
+    #[test]
+    fn parse_vds_mappings_truncated_selection_does_not_overrun() {
+        // One entry whose source selection (ALL) is truncated to 8 of 16 bytes.
+        let blob = [
+            0x01u8, // version 1
+            0x01, 0, 0, 0, 0, 0, 0, 0, // nused = 1
+            0x04, // same-file marker
+            0x78, 0x00, // "x\0"
+            0x03, 0, 0, 0, 0x01, 0, 0, 0, // ALL header, truncated (8 of 16 bytes)
+        ];
+        assert!(parse_vds_mappings(&blob, 8).is_err());
+    }
+
+    #[test]
+    fn parse_vds_mappings_empty_is_ok_empty() {
+        assert!(parse_vds_mappings(&[], 8).unwrap().is_empty());
+        // Header present, nused = 0.
+        let blob = [0x01u8, 0, 0, 0, 0, 0, 0, 0, 0];
+        assert!(parse_vds_mappings(&blob, 8).unwrap().is_empty());
     }
 }

crates/clawhdf5-format/src/data_read.rs

@@ -73,6 +73,16 @@ pub fn read_raw_data(
     read_raw_data_full(file_data, layout, dataspace, datatype, None, 8, 8)
 }
 
+/// Resolves a Virtual Dataset source **file name** (as stored in the mapping,
+/// e.g. `"ext_src.h5"`) to that file's raw bytes.
+///
+/// The pure-byte read API has no filesystem of its own, so external-file VDS
+/// sources are read through a caller-supplied resolver. The std file API wires
+/// one that reads relative to the virtual file's directory; callers can supply
+/// their own (e.g. an in-memory map) in `no_std` builds. Returning `None` means
+/// the source file is unavailable and the mapping is skipped.
+pub type VdsSourceResolver<'a> = dyn Fn(&str) -> Option<Vec<u8>> + 'a;
+
 /// Read raw bytes with full parameters including filter pipeline and sizes.
 pub fn read_raw_data_full(
     file_data: &[u8],
@@ -82,6 +92,40 @@ pub fn read_raw_data_full(
     pipeline: Option<&FilterPipeline>,
     offset_size: u8,
     length_size: u8,
+) -> Result<Vec<u8>, FormatError> {
+    read_raw_data_full_impl(
+        file_data, layout, dataspace, datatype, pipeline, offset_size, length_size, None,
+    )
+}
+
+/// Like [`read_raw_data_full`], but with a resolver for external-file Virtual
+/// Dataset sources. For non-virtual layouts the resolver is ignored.
+#[allow(clippy::too_many_arguments)]
+pub fn read_raw_data_full_with_resolver(
+    file_data: &[u8],
+    layout: &DataLayout,
+    dataspace: &Dataspace,
+    datatype: &Datatype,
+    pipeline: Option<&FilterPipeline>,
+    offset_size: u8,
+    length_size: u8,
+    resolver: Option<&VdsSourceResolver>,
+) -> Result<Vec<u8>, FormatError> {
+    read_raw_data_full_impl(
+        file_data, layout, dataspace, datatype, pipeline, offset_size, length_size, resolver,
+    )
+}
+
+#[allow(clippy::too_many_arguments)]
+fn read_raw_data_full_impl(
+    file_data: &[u8],
+    layout: &DataLayout,
+    dataspace: &Dataspace,
+    datatype: &Datatype,
+    pipeline: Option<&FilterPipeline>,
+    offset_size: u8,
+    length_size: u8,
+    resolver: Option<&VdsSourceResolver>,
 ) -> Result<Vec<u8>, FormatError> {
     let num_elements = dataspace.num_elements() as usize;
     let elem_size = datatype.type_size() as usize;
@@ -128,7 +172,20 @@ pub fn read_raw_data_full(
             offset_size,
             length_size,
         ),
-        DataLayout::Virtual { .. } => Err(FormatError::UnsupportedVersion(0)),
+        DataLayout::Virtual {
+            global_heap_address,
+            global_heap_index,
+            ..
+        } => read_virtual_data(
+            file_data,
+            *global_heap_address,
+            *global_heap_index,
+            dataspace,
+            datatype,
+            offset_size,
+            length_size,
+            resolver,
+        ),
     }
 }
 
@@ -355,8 +412,170 @@ pub fn read_raw_data_selection(
             )?;
             extract_selection_from_buffer(&full_data, dims, elem_size, selection)
         }
-        DataLayout::Virtual { .. } => Err(FormatError::UnsupportedVersion(0)),
+        DataLayout::Virtual { .. } => {
+            // Assemble the full virtual dataset, then apply the read selection.
+            let full_data = read_raw_data_full(
+                file_data,
+                layout,
+                dataspace,
+                datatype,
+                pipeline,
+                offset_size,
+                length_size,
+            )?;
+            extract_selection_from_buffer(&full_data, dims, elem_size, selection)
         }
+    }
+}
+
+/// Assemble a **Virtual Dataset (VDS)** from its source mappings.
+///
+/// Supports virtual datasets of any rank. Same-file sources are read directly;
+/// **external-file** sources are read through the caller-supplied `resolver`,
+/// which maps a stored source file name to that file's bytes. Each mapping's
+/// selected source elements are scattered into the virtual buffer at the
+/// positions given by the virtual selection (both enumerated in row-major
+/// order, as HDF5 pairs them). Unmapped regions are left at the zero fill value.
+///
+/// A mapping whose external source file the resolver cannot supply (`None`) is
+/// skipped, leaving its region at fill — matching HDF5's tolerance of missing
+/// sources. An external source with no resolver at all is a hard error.
+#[allow(clippy::too_many_arguments)]
+fn read_virtual_data(
+    file_data: &[u8],
+    global_heap_address: Option<u64>,
+    global_heap_index: u32,
+    dataspace: &Dataspace,
+    datatype: &Datatype,
+    offset_size: u8,
+    length_size: u8,
+    resolver: Option<&VdsSourceResolver>,
+) -> Result<Vec<u8>, FormatError> {
+    use crate::data_layout::parse_vds_mappings;
+    use crate::global_heap::GlobalHeapCollection;
+    use crate::selection::Selection;
+
+    let elem_size = datatype.type_size() as usize;
+    let total_elems = dataspace.num_elements() as usize;
+    let mut out = vec![0u8; total_elems.saturating_mul(elem_size)];
+
+    let virtual_dims = &dataspace.dimensions;
+
+    let addr = global_heap_address.ok_or_else(|| {
+        FormatError::ChunkedReadError("virtual dataset has no mapping global heap".into())
+    })?;
+    let coll = GlobalHeapCollection::parse(file_data, addr as usize, length_size)?;
+    let obj = coll
+        .get_object(global_heap_index as u16)
+        .ok_or(FormatError::GlobalHeapObjectNotFound {
+            collection_address: addr,
+            index: global_heap_index as u16,
+        })?;
+    let mappings = parse_vds_mappings(&obj.data, length_size)?;
+
+    for m in &mappings {
+        let same_file = m.source_file.is_empty() || m.source_file == ".";
+
+        // Resolve the bytes of the file holding this source dataset.
+        let external;
+        let src_file_data: &[u8] = if same_file {
+            file_data
+        } else {
+            let r = resolver.ok_or_else(|| {
+                FormatError::ChunkedReadError(
+                    "external-file virtual dataset sources require a file resolver".into(),
+                )
+            })?;
+            match r(&m.source_file) {
+                Some(bytes) => {
+                    external = bytes;
+                    &external
+                }
+                // Source file unavailable: leave this region at fill value.
+                None => continue,
+            }
+        };
+
+        let (vsel, _) = Selection::decode_serialized(&m.virtual_selection)?;
+        let (ssel, _) = Selection::decode_serialized(&m.source_selection)?;
+
+        let (src_raw, src_dims) =
+            read_named_dataset_raw(src_file_data, &m.source_dataset, offset_size, length_size)?;
+
+        let vidx = vsel.iter_linear(virtual_dims)?;
+        let sidx = ssel.iter_linear(&src_dims)?;
+        if vidx.len() != sidx.len() {
+            return Err(FormatError::ChunkedReadError(
+                "virtual/source selection element counts differ".into(),
+            ));
+        }
+
+        for (&v, &s) in vidx.iter().zip(sidx.iter()) {
+            let (vo, so) = (v as usize * elem_size, s as usize * elem_size);
+            if vo + elem_size > out.len() || so + elem_size > src_raw.len() {
+                return Err(FormatError::ChunkedReadError(
+                    "virtual dataset selection out of bounds".into(),
+                ));
+            }
+            out[vo..vo + elem_size].copy_from_slice(&src_raw[so..so + elem_size]);
+        }
+    }
+
+    Ok(out)
+}
+
+/// Read a named dataset's raw (decoded) bytes and its dimensions, navigating
+/// from the superblock. Used to pull VDS source datasets out of the same file.
+fn read_named_dataset_raw(
+    file_data: &[u8],
+    path: &str,
+    _offset_size: u8,
+    _length_size: u8,
+) -> Result<(Vec<u8>, Vec<u64>), FormatError> {
+    use crate::filter_pipeline::FilterPipeline;
+    use crate::group_v2::resolve_path_any;
+    use crate::message_type::MessageType;
+    use crate::object_header::ObjectHeader;
+    use crate::signature::find_signature;
+    use crate::superblock::Superblock;
+
+    let sig = find_signature(file_data)?;
+    let sb = Superblock::parse(file_data, sig)?;
+    let addr = resolve_path_any(file_data, &sb, path)?;
+    let hdr = ObjectHeader::parse(file_data, addr as usize, sb.offset_size, sb.length_size)?;
+
+    let find = |t: MessageType| hdr.messages.iter().find(|m| m.msg_type == t);
+    let ds_msg = find(MessageType::Dataspace)
+        .ok_or_else(|| FormatError::ChunkedReadError("VDS source has no dataspace".into()))?;
+    let dataspace = Dataspace::parse(&ds_msg.data, sb.length_size)?;
+    let dt_msg = find(MessageType::Datatype)
+        .ok_or_else(|| FormatError::ChunkedReadError("VDS source has no datatype".into()))?;
+    let (datatype, _) = Datatype::parse(&dt_msg.data)?;
+    let dl_msg = find(MessageType::DataLayout)
+        .ok_or_else(|| FormatError::ChunkedReadError("VDS source has no data layout".into()))?;
+    let layout = DataLayout::parse(&dl_msg.data, sb.offset_size, sb.length_size)?;
+    // A virtual dataset whose source is itself another virtual dataset could
+    // form a cycle (A -> B -> A) and recurse into a stack overflow. Nested
+    // virtual sources are exotic and unsupported, so stop here cleanly.
+    if matches!(layout, DataLayout::Virtual { .. }) {
+        return Err(FormatError::ChunkedReadError(
+            "virtual dataset source is itself virtual (unsupported)".into(),
+        ));
+    }
+    let pipeline = find(MessageType::FilterPipeline)
+        .map(|m| FilterPipeline::parse(&m.data))
+        .transpose()?;
+
+    let raw = read_raw_data_full(
+        file_data,
+        &layout,
+        &dataspace,
+        &datatype,
+        pipeline.as_ref(),
+        sb.offset_size,
+        sb.length_size,
+    )?;
+    Ok((raw, dataspace.dimensions.clone()))
 }
 
 /// Extract selected elements from a full dataset buffer.
@@ -618,6 +837,11 @@ fn get_size(dt: &Datatype) -> usize {
 
 /// Convert raw bytes to `f64` values.
 pub fn read_as_f64(raw: &[u8], datatype: &Datatype) -> Result<Vec<f64>, FormatError> {
+    // Array datatypes (e.g. an array-typed compound member) are read as a flat
+    // sequence of their base elements.
+    if let Datatype::Array { base_type, .. } = datatype {
+        return read_as_f64(raw, base_type);
+    }
     ensure_numeric(datatype, "FloatingPoint or FixedPoint")?;
     let elem_size = get_size(datatype);
     if elem_size == 0 || !raw.len().is_multiple_of(elem_size) {
@@ -671,19 +895,27 @@ fn convert_to_f64(
                 Ok(v as f64)
             }
             8 => Ok(read_f64_bytes(bytes, order)),
+            2 => Ok(read_f16_bytes(bytes, order) as f64),
             _ => Err(FormatError::DataSizeMismatch {
                 expected: 8,
                 actual: *size as usize,
             }),
         },
-        Datatype::FixedPoint { size, signed, .. } => {
+        Datatype::FixedPoint {
-            if *signed {
+            size,
-                let v = read_signed_int(bytes, *size as usize, order);
+            signed,
-                Ok(v as f64)
+            bit_offset,
+            bit_precision,
+            ..
+        } => {
+            let full = read_unsigned_int(bytes, *size as usize, order);
+            let (off, prec) = effective_bits(*size as usize, *bit_offset, *bit_precision);
+            let v = if *signed {
+                extract_signed(full, off, prec) as f64
             } else {
-                let v = read_unsigned_int(bytes, *size as usize, order);
+                extract_unsigned(full, off, prec) as f64
-                Ok(v as f64)
+            };
-            }
+            Ok(v)
         }
         _ => Err(FormatError::TypeMismatch {
             expected: "numeric",
@@ -694,6 +926,9 @@ fn convert_to_f64(
 
 /// Convert raw bytes to `i64` values.
 pub fn read_as_i64(raw: &[u8], datatype: &Datatype) -> Result<Vec<i64>, FormatError> {
+    if let Datatype::Array { base_type, .. } = datatype {
+        return read_as_i64(raw, base_type);
+    }
     ensure_numeric(datatype, "FixedPoint (signed)")?;
     let elem_size = get_size(datatype);
     if elem_size == 0 || !raw.len().is_multiple_of(elem_size) {
@@ -707,6 +942,7 @@ pub fn read_as_i64(raw: &[u8], datatype: &Datatype) -> Result<Vec<i64>, FormatEr
     // Fast path: native LE i64 — single bulk memcpy
     #[cfg(target_endian = "little")]
     if elem_size == 8
+        && is_full_width(datatype)
         && matches!(
             datatype,
             Datatype::FixedPoint {
@@ -725,17 +961,21 @@ pub fn read_as_i64(raw: &[u8], datatype: &Datatype) -> Result<Vec<i64>, FormatEr
     }
 
     let order = get_byte_order(datatype);
+    let (off, prec) = fixed_bits(datatype);
     let mut result = Vec::with_capacity(count);
     for i in 0..count {
         let chunk = &raw[i * elem_size..(i + 1) * elem_size];
-        let v = read_signed_int(chunk, elem_size, &order);
+        let full = read_unsigned_int(chunk, elem_size, &order);
-        result.push(v);
+        result.push(extract_signed(full, off, prec));
     }
     Ok(result)
 }
 
 /// Convert raw bytes to `u64` values.
 pub fn read_as_u64(raw: &[u8], datatype: &Datatype) -> Result<Vec<u64>, FormatError> {
+    if let Datatype::Array { base_type, .. } = datatype {
+        return read_as_u64(raw, base_type);
+    }
     ensure_numeric(datatype, "FixedPoint (unsigned)")?;
     let elem_size = get_size(datatype);
     if elem_size == 0 || !raw.len().is_multiple_of(elem_size) {
@@ -746,17 +986,21 @@ pub fn read_as_u64(raw: &[u8], datatype: &Datatype) -> Result<Vec<u64>, FormatEr
     }
     let count = raw.len() / elem_size;
     let order = get_byte_order(datatype);
+    let (off, prec) = fixed_bits(datatype);
     let mut result = Vec::with_capacity(count);
     for i in 0..count {
         let chunk = &raw[i * elem_size..(i + 1) * elem_size];
-        let v = read_unsigned_int(chunk, elem_size, &order);
+        let full = read_unsigned_int(chunk, elem_size, &order);
-        result.push(v);
+        result.push(extract_unsigned(full, off, prec));
     }
     Ok(result)
 }
 
 /// Convert raw bytes to `f32` values.
 pub fn read_as_f32(raw: &[u8], datatype: &Datatype) -> Result<Vec<f32>, FormatError> {
+    if let Datatype::Array { base_type, .. } = datatype {
+        return read_as_f32(raw, base_type);
+    }
     ensure_numeric(datatype, "FloatingPoint")?;
     let elem_size = get_size(datatype);
     if elem_size == 0 || !raw.len().is_multiple_of(elem_size) {
@@ -796,17 +1040,30 @@ pub fn read_as_f32(raw: &[u8], datatype: &Datatype) -> Result<Vec<f32>, FormatEr
             Datatype::FloatingPoint { size: 8, .. } => {
                 result.push(read_f64_bytes(chunk, &order) as f32);
             }
+            Datatype::FloatingPoint { size: 2, .. } => {
+                result.push(read_f16_bytes(chunk, &order));
+            }
             Datatype::FixedPoint {
-                signed: true, size, ..
+                signed: true,
+                size,
+                bit_offset,
+                bit_precision,
+                ..
             } => {
-                result.push(read_signed_int(chunk, *size as usize, &order) as f32);
+                let full = read_unsigned_int(chunk, *size as usize, &order);
+                let (off, prec) = effective_bits(*size as usize, *bit_offset, *bit_precision);
+                result.push(extract_signed(full, off, prec) as f32);
             }
             Datatype::FixedPoint {
                 signed: false,
                 size,
+                bit_offset,
+                bit_precision,
                 ..
             } => {
-                result.push(read_unsigned_int(chunk, *size as usize, &order) as f32);
+                let full = read_unsigned_int(chunk, *size as usize, &order);
+                let (off, prec) = effective_bits(*size as usize, *bit_offset, *bit_precision);
+                result.push(extract_unsigned(full, off, prec) as f32);
             }
             _ => {
                 return Err(FormatError::TypeMismatch {
@@ -821,6 +1078,9 @@ pub fn read_as_f32(raw: &[u8], datatype: &Datatype) -> Result<Vec<f32>, FormatEr
 
 /// Convert raw bytes to `i32` values.
 pub fn read_as_i32(raw: &[u8], datatype: &Datatype) -> Result<Vec<i32>, FormatError> {
+    if let Datatype::Array { base_type, .. } = datatype {
+        return read_as_i32(raw, base_type);
+    }
     ensure_numeric(datatype, "FixedPoint")?;
     let elem_size = get_size(datatype);
     if elem_size == 0 || !raw.len().is_multiple_of(elem_size) {
@@ -834,6 +1094,7 @@ pub fn read_as_i32(raw: &[u8], datatype: &Datatype) -> Result<Vec<i32>, FormatEr
     // Fast path: native LE i32 — single bulk memcpy
     #[cfg(target_endian = "little")]
     if elem_size == 4
+        && is_full_width(datatype)
         && matches!(
             datatype,
             Datatype::FixedPoint {
@@ -851,11 +1112,12 @@ pub fn read_as_i32(raw: &[u8], datatype: &Datatype) -> Result<Vec<i32>, FormatEr
     }
 
     let order = get_byte_order(datatype);
+    let (off, prec) = fixed_bits(datatype);
     let mut result = Vec::with_capacity(count);
     for i in 0..count {
         let chunk = &raw[i * elem_size..(i + 1) * elem_size];
-        let v = read_signed_int(chunk, elem_size, &order);
+        let full = read_unsigned_int(chunk, elem_size, &order);
-        result.push(v as i32);
+        result.push(extract_signed(full, off, prec) as i32);
     }
     Ok(result)
 }
@@ -1232,6 +1494,53 @@ fn read_f64_bytes(bytes: &[u8], order: &DatatypeByteOrder) -> f64 {
     f64::from_le_bytes(buf)
 }
 
+/// Decode an IEEE-754 half-precision (binary16) value to `f32`. Pure integer
+/// bit manipulation (no_std-safe, no `powi`/`libm`).
+fn read_f16_bytes(bytes: &[u8], order: &DatatypeByteOrder) -> f32 {
+    let mut buf = [0u8; 2];
+    let len = bytes.len().min(2);
+    match order {
+        DatatypeByteOrder::BigEndian => {
+            for i in 0..len {
+                buf[i] = bytes[len - 1 - i];
+            }
+        }
+        _ => buf[..len].copy_from_slice(&bytes[..len]),
+    }
+    f16_bits_to_f32(u16::from_le_bytes(buf))
+}
+
+/// Convert the bit pattern of an IEEE-754 half (binary16) to an `f32`.
+fn f16_bits_to_f32(h: u16) -> f32 {
+    let h = h as u32;
+    let sign = (h & 0x8000) << 16;
+    let exp = (h >> 10) & 0x1f;
+    let mant = h & 0x3ff;
+    let bits = if exp == 0 {
+        if mant == 0 {
+            sign // signed zero
+        } else {
+            // Subnormal: normalize into an f32 normal.
+            let mut e: i32 = -1;
+            let mut m = mant;
+            loop {
+                e += 1;
+                m <<= 1;
+                if m & 0x400 != 0 {
+                    break;
+                }
+            }
+            let m = m & 0x3ff;
+            sign | (((127 - 15 - e) as u32) << 23) | (m << 13)
+        }
+    } else if exp == 0x1f {
+        sign | 0x7f80_0000 | (mant << 13) // inf / NaN
+    } else {
+        sign | ((exp + (127 - 15)) << 23) | (mant << 13)
+    };
+    f32::from_bits(bits)
+}
+
 fn read_f32_bytes(bytes: &[u8], order: &DatatypeByteOrder) -> f32 {
     let mut buf = [0u8; 4];
     let len = bytes.len().min(4);
@@ -1248,6 +1557,68 @@ fn read_f32_bytes(bytes: &[u8], order: &DatatypeByteOrder) -> f32 {
     f32::from_le_bytes(buf)
 }
 
+/// Effective (bit offset, bit precision) for a fixed-point field, defaulting a
+/// zero precision to the full storage width.
+fn effective_bits(size: usize, bit_offset: u16, bit_precision: u16) -> (u32, u32) {
+    let prec = if bit_precision == 0 {
+        (size * 8) as u32
+    } else {
+        bit_precision as u32
+    };
+    (bit_offset as u32, prec)
+}
+
+/// `(bit_offset, bit_precision)` for a fixed-point datatype, full width for
+/// other types.
+fn fixed_bits(datatype: &Datatype) -> (u32, u32) {
+    match datatype {
+        Datatype::FixedPoint {
+            size,
+            bit_offset,
+            bit_precision,
+            ..
+        } => effective_bits(*size as usize, *bit_offset, *bit_precision),
+        _ => (0, 0),
+    }
+}
+
+/// Whether a datatype occupies its full storage width (bit offset 0, precision
+/// == size·8), in which case the bulk-copy fast read paths apply. Non
+/// fixed-point types are treated as full width.
+fn is_full_width(datatype: &Datatype) -> bool {
+    match datatype {
+        Datatype::FixedPoint {
+            size,
+            bit_offset,
+            bit_precision,
+            ..
+        } => *bit_offset == 0 && *bit_precision as u32 == *size * 8,
+        _ => true,
+    }
+}
+
+/// Extract the `precision`-bit field at `offset` from a full-width integer read
+/// and sign-extend it. Full-width fields read as an ordinary signed integer;
+/// reduced-precision fields sign-extend from the field's top bit (HDF5 stores
+/// reduced-precision values zero-filled, so the sign lives in the precision
+/// field, not the storage word).
+fn extract_signed(full: u64, offset: u32, precision: u32) -> i64 {
+    if precision == 0 || precision >= 64 {
+        return full as i64;
+    }
+    let field = (full >> offset) & ((1u64 << precision) - 1);
+    let shift = 64 - precision;
+    ((field << shift) as i64) >> shift
+}
+
+/// Extract the `precision`-bit field at `offset` from a full-width integer read.
+fn extract_unsigned(full: u64, offset: u32, precision: u32) -> u64 {
+    if precision == 0 || precision >= 64 {
+        return full;
+    }
+    (full >> offset) & ((1u64 << precision) - 1)
+}
+
 fn read_unsigned_int(bytes: &[u8], size: usize, order: &DatatypeByteOrder) -> u64 {
     let buf = reorder_bytes(bytes, order);
     match size {
@@ -1266,22 +1637,6 @@ fn read_unsigned_int(bytes: &[u8], size: usize, order: &DatatypeByteOrder) -> u6
     }
 }
 
-fn read_signed_int(bytes: &[u8], size: usize, order: &DatatypeByteOrder) -> i64 {
-    let buf = reorder_bytes(bytes, order);
-    match size {
-        1 => buf[0] as i8 as i64,
-        2 => i16::from_le_bytes([buf[0], buf[1]]) as i64,
-        4 => i32::from_le_bytes([buf[0], buf[1], buf[2], buf[3]]) as i64,
-        8 => i64::from_le_bytes(buf),
-        _ => {
-            let u = read_unsigned_int(bytes, size, order);
-            // Sign extend
-            let shift = 64 - (size * 8);
-            ((u as i64) << shift) >> shift
-        }
-    }
-}
-
 // --- Type conversion cost analysis ---
 
 /// Cost classification for type conversions.
@@ -1362,6 +1717,113 @@ mod tests {
     use crate::dataspace::{Dataspace, DataspaceType};
     use crate::datatype::{CharacterSet, StringPadding};
 
+    fn f16_datatype() -> Datatype {
+        Datatype::FloatingPoint {
+            size: 2,
+            byte_order: DatatypeByteOrder::LittleEndian,
+            bit_offset: 0,
+            bit_precision: 16,
+            exponent_location: 10,
+            exponent_size: 5,
+            mantissa_location: 0,
+            mantissa_size: 10,
+            exponent_bias: 15,
+        }
+    }
+
+    // IEEE-754 half bit patterns for known values.
+    fn f16_bits(v: f32) -> u16 {
+        // Encode a few exact values used by the test.
+        match v {
+            x if x == 0.0 => 0x0000,
+            x if x == 1.0 => 0x3c00,
+            x if x == -2.0 => 0xc000,
+            x if x == 0.5 => 0x3800,
+            x if x == 65504.0 => 0x7bff, // f16 max
+            _ => panic!("unsupported test value {v}"),
+        }
+    }
+
+    #[test]
+    fn read_f16_as_f32_and_f64() {
+        let values = [0.0f32, 1.0, -2.0, 0.5, 65504.0];
+        let raw: Vec<u8> = values
+            .iter()
+            .flat_map(|&v| f16_bits(v).to_le_bytes())
+            .collect();
+        let dt = f16_datatype();
+        let got32 = read_as_f32(&raw, &dt).unwrap();
+        assert_eq!(got32, values);
+        let got64 = read_as_f64(&raw, &dt).unwrap();
+        let expect64: Vec<f64> = values.iter().map(|&v| v as f64).collect();
+        assert_eq!(got64, expect64);
+    }
+
+    fn reduced_int(signed: bool, precision: u16) -> Datatype {
+        Datatype::FixedPoint {
+            size: 4,
+            byte_order: DatatypeByteOrder::LittleEndian,
+            signed,
+            bit_offset: 0,
+            bit_precision: precision,
+        }
+    }
+
+    #[test]
+    fn reduced_precision_signed_sign_extends() {
+        // 16-bit-precision signed values stored zero-filled (HDF5's canonical
+        // layout, e.g. after N-Bit): the reader must sign-extend from bit 15.
+        let dt = reduced_int(true, 16);
+        // [-1, 100, -50, -32768] as 0x0000ffff / 0x00000064 / 0x0000ffce / 0x00008000
+        let raw: Vec<u8> = vec![
+            0xff, 0xff, 0x00, 0x00, 0x64, 0x00, 0x00, 0x00, 0xce, 0xff, 0x00, 0x00, 0x00, 0x80,
+            0x00, 0x00,
+        ];
+        assert_eq!(read_as_i32(&raw, &dt).unwrap(), vec![-1, 100, -50, -32768]);
+        assert_eq!(read_as_i64(&raw, &dt).unwrap(), vec![-1, 100, -50, -32768]);
+    }
+
+    #[test]
+    fn reduced_precision_unsigned_masks() {
+        // 12-bit-precision unsigned: high bits must read as zero, not sign.
+        let dt = reduced_int(false, 12);
+        // [4095, 1, 2048] as 0x00000fff / 0x00000001 / 0x00000800
+        let raw: Vec<u8> = vec![
+            0xff, 0x0f, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x00, 0x08, 0x00, 0x00,
+        ];
+        assert_eq!(read_as_u64(&raw, &dt).unwrap(), vec![4095, 1, 2048]);
+    }
+
+    #[test]
+    fn full_width_signed_unchanged() {
+        // Regression: full-width 32-bit signed must be unaffected.
+        let dt = reduced_int(true, 32);
+        let raw: Vec<u8> = vec![0xff, 0xff, 0xff, 0xff, 0x2a, 0x00, 0x00, 0x00];
+        assert_eq!(read_as_i32(&raw, &dt).unwrap(), vec![-1, 42]);
+    }
+
+    #[test]
+    fn array_datatype_reads_flat_base_elements() {
+        // An array-typed (e.g. compound member) datatype reads as a flat
+        // sequence of its base elements, applying base-type precision rules.
+        let arr = Datatype::Array {
+            base_type: Box::new(reduced_int(true, 16)),
+            dimensions: vec![2],
+        };
+        // [-1, 100, 1000, -32768] stored zero-filled at 16-bit precision.
+        let raw: Vec<u8> = vec![
+            0xff, 0xff, 0x00, 0x00, 0x64, 0x00, 0x00, 0x00, 0xe8, 0x03, 0x00, 0x00, 0x00, 0x80,
+            0x00, 0x00,
+        ];
+        assert_eq!(read_as_i32(&raw, &arr).unwrap(), vec![-1, 100, 1000, -32768]);
+        // Nested array-of-array unwraps recursively.
+        let nested = Datatype::Array {
+            base_type: Box::new(arr),
+            dimensions: vec![2],
+        };
+        assert_eq!(read_as_i32(&raw, &nested).unwrap(), vec![-1, 100, 1000, -32768]);
+    }
+
     fn make_f64_le_type() -> Datatype {
         Datatype::FloatingPoint {
             size: 8,

crates/clawhdf5-format/src/datatype.rs

@@ -348,7 +348,10 @@ impl Datatype {
                 let num_members = (bf0 as u16) | ((bf1 as u16) << 8);
                 let mut members = Vec::with_capacity(num_members as usize);
 
-                if version == 3 || version == 4 {
+                if (3..=5).contains(&version) {
+                    // v3, v4 and v5 share the compact member encoding (name,
+                    // variable-width offset, member datatype). HDF5 1.14+/2.0
+                    // with `libver=latest` emits v5 compound types.
                     let ob = offset_bytes_for_size(size);
                     for _ in 0..num_members {
                         let (name, name_len) = read_null_terminated_string(data, pos)?;
@@ -500,7 +503,9 @@ impl Datatype {
                         },
                         pos,
                     ))
-                } else if version == 3 {
+                } else if (3..=5).contains(&version) {
+                    // v3, v4 and v5 share the array encoding (ndims, dims, base
+                    // type); HDF5 1.14+/2.0 with `libver=latest` emits v5.
                     ensure_len(data, pos, 1)?;
                     let ndims = data[pos] as usize;
                     pos += 1;
@@ -1007,6 +1012,76 @@ mod tests {
         }
     }
 
+    #[test]
+    fn test_compound_v5_from_hdf5_2_0() {
+        // Real datatype message bytes emitted by h5py 3.16 / HDF5 2.0 with
+        // `libver=latest` for a compound dtype [('x','f8'),('y','f8'),('id','i4')].
+        // The wrapper is datatype version 5; members reuse the v3 compact
+        // encoding. Regression guard for reading modern-format compound types.
+        let bytes: [u8; 70] = [
+            0x56, 0x03, 0x00, 0x00, 0x14, 0x00, 0x00, 0x00, 0x78, 0x00, 0x00, 0x11, 0x20, 0x3f,
+            0x00, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x40, 0x00, 0x34, 0x0b, 0x00, 0x34, 0xff,
+            0x03, 0x00, 0x00, 0x79, 0x00, 0x08, 0x11, 0x20, 0x3f, 0x00, 0x08, 0x00, 0x00, 0x00,
+            0x00, 0x00, 0x40, 0x00, 0x34, 0x0b, 0x00, 0x34, 0xff, 0x03, 0x00, 0x00, 0x69, 0x64,
+            0x00, 0x10, 0x10, 0x08, 0x00, 0x00, 0x04, 0x00, 0x00, 0x00, 0x00, 0x00, 0x20, 0x00,
+        ];
+        let (dt, _) = Datatype::parse(&bytes).unwrap();
+        match dt {
+            Datatype::Compound { size, members } => {
+                assert_eq!(size, 20);
+                assert_eq!(members.len(), 3);
+                assert_eq!(
+                    (members[0].name.as_str(), members[0].byte_offset),
+                    ("x", 0)
+                );
+                assert_eq!(
+                    (members[1].name.as_str(), members[1].byte_offset),
+                    ("y", 8)
+                );
+                assert_eq!(
+                    (members[2].name.as_str(), members[2].byte_offset),
+                    ("id", 16)
+                );
+                assert!(matches!(
+                    members[0].datatype,
+                    Datatype::FloatingPoint { size: 8, .. }
+                ));
+                assert!(matches!(
+                    members[2].datatype,
+                    Datatype::FixedPoint {
+                        size: 4,
+                        signed: true,
+                        ..
+                    }
+                ));
+            }
+            _ => panic!("expected Compound"),
+        }
+    }
+
+    #[test]
+    fn test_array_v5_from_hdf5_2_0() {
+        // Real datatype message from h5py 3.16 / HDF5 2.0 (`libver=latest`) for
+        // an array dtype `('f8', (3,))`: datatype version 5, class 10, reusing
+        // the v3 array encoding (ndims, dims, base type).
+        let bytes: [u8; 33] = [
+            0x5a, 0x00, 0x00, 0x00, 0x18, 0x00, 0x00, 0x00, 0x01, 0x03, 0x00, 0x00, 0x00, 0x11,
+            0x20, 0x3f, 0x00, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x40, 0x00, 0x34, 0x0b, 0x00,
+            0x34, 0xff, 0x03, 0x00, 0x00,
+        ];
+        let (dt, _) = Datatype::parse(&bytes).unwrap();
+        match dt {
+            Datatype::Array {
+                base_type,
+                dimensions,
+            } => {
+                assert_eq!(dimensions, vec![3]);
+                assert!(matches!(*base_type, Datatype::FloatingPoint { size: 8, .. }));
+            }
+            other => panic!("expected Array, got {other:?}"),
+        }
+    }
+
     #[test]
     fn test_reference_object() {
         let buf = build_dt_header(7, 1, [0, 0, 0], 8);

crates/clawhdf5-format/src/file_writer.rs

@@ -33,6 +33,11 @@ const SUPERBLOCK_SIZE: usize = 48;
 /// Threshold for switching from compact (inline) to dense attribute storage.
 const DENSE_ATTR_THRESHOLD: usize = 8;
 
+/// Threshold for switching a group from compact (inline Link messages) to dense
+/// link storage (fractal heap + v2 B-tree), matching libhdf5's default
+/// `max_compact` of 8 links.
+const DENSE_LINK_THRESHOLD: usize = 8;
+
 // ---- OH builders ----
 
 pub(crate) fn build_chunked_dataset_oh(
@@ -123,10 +128,16 @@ pub(crate) fn build_compact_dataset_oh(
 
 pub(crate) fn build_group_oh(
     links: &[LinkMessage],
+    dense_link_info: Option<&[u8]>,
     attrs: &[AttributeMessage],
     dense_blob: Option<&DenseAttrBlob>,
 ) -> Vec<u8> {
     let mut w = ObjectHeaderWriter::new();
+    if let Some(li) = dense_link_info {
+        // Dense link storage: a LinkInfo pointing at the fractal heap + name
+        // B-tree, and no inline Link messages.
+        w.add_message(MessageType::LinkInfo, li.to_vec());
+    } else {
         let mut li = Vec::new();
         li.push(0); // version
         li.push(0); // flags
@@ -136,6 +147,7 @@ pub(crate) fn build_group_oh(
         for link in links {
             w.add_message(MessageType::Link, link.serialize(OFFSET_SIZE));
         }
+    }
     if let Some(blob) = dense_blob {
         w.add_message(MessageType::AttributeInfo, blob.attr_info_message.clone());
     } else {
@@ -167,21 +179,39 @@ pub(crate) struct DenseAttrBlob {
     pub(crate) blob: Vec<u8>,
 }
 
-/// Build dense attribute storage for a set of attributes.
+/// A fractal heap holding a set of serialized objects, plus the heap IDs that
-pub(crate) fn build_dense_attrs(attrs: &[AttributeMessage], base_address: u64) -> DenseAttrBlob {
+/// address them. Shared by dense attribute and dense link storage, which differ
-    // Dense attrs use v3 attribute messages (adds character set encoding byte).
+/// only in their v2 B-tree record layout.
-    let serialized: Vec<Vec<u8>> = attrs.iter().map(|a| a.serialize_v3(LENGTH_SIZE)).collect();
+pub(crate) struct FractalHeapBlock {
-
+    /// The complete heap bytes: FRHP header, then either a single root direct
-    let name_hashes: Vec<u32> = attrs
+    /// block, or a root indirect block (FHIB) followed by its direct blocks.
-        .iter()
+    blob: Vec<u8>,
-        .map(|a| crate::checksum::jenkins_lookup3(a.name.as_bytes()))
+    /// Address of the fractal heap header.
-        .collect();
+    frhp_addr: u64,
+    /// Address where the v2 B-tree should be placed (right after the heap).
+    btree_addr: u64,
+    /// Heap ID for each object, in input order.
+    heap_ids: Vec<Vec<u8>>,
+    /// Heap ID length (bytes).
+    heap_id_length: u16,
+}
 
+/// Build a fractal heap for `serialized` objects, laid out at `base_address`.
+///
+/// Uses a single root direct block when the data fits in one (≤ the maximum
+/// direct block size), otherwise a root indirect block over multiple direct
+/// blocks following the doubling table. The caller builds the matching v2
+/// B-tree (type 5 for links, type 8 for attributes) at the returned
+/// `btree_addr`.
+pub(crate) fn build_single_block_fractal_heap(
+    serialized: &[Vec<u8>],
+    base_address: u64,
+    max_heap_size: u16,
+    heap_id_length: u16,
+) -> FractalHeapBlock {
     let os = OFFSET_SIZE as usize;
     let ls = LENGTH_SIZE as usize;
-    let max_heap_size: u16 = 40;
+    let block_offset_bytes = (max_heap_size as usize).div_ceil(8);
-    let block_offset_bytes = (max_heap_size as usize).div_ceil(8); // 5
-    let heap_id_length: u16 = 8;
     let max_direct_block_size: u64 = 65536;
 
     // Direct block layout: sig(4) + ver(1) + heap_addr(os) + block_offset(bo_bytes)
@@ -191,6 +221,17 @@ pub(crate) fn build_dense_attrs(attrs: &[AttributeMessage], base_address: u64) -
     let dblock_content_size = dblock_header_size + total_data_size;
     let starting_block_size = dblock_content_size.next_power_of_two().max(512) as u64;
 
+    // When the objects don't fit in a single direct block, fall back to a
+    // multi-block heap with a root indirect block.
+    if starting_block_size > max_direct_block_size {
+        return build_multiblock_fractal_heap(
+            serialized,
+            base_address,
+            max_heap_size,
+            heap_id_length,
+        );
+    }
+
     // Fractal heap header size
     let frhp_size = 4
         + 1
@@ -242,7 +283,7 @@ pub(crate) fn build_dense_attrs(attrs: &[AttributeMessage], base_address: u64) -
     write_length(&mut frhp, starting_block_size, LENGTH_SIZE); // managed_space_in_heap
     write_length(&mut frhp, starting_block_size, LENGTH_SIZE); // allocated_managed_space
     write_length(&mut frhp, 0, LENGTH_SIZE); // dblock_alloc_iter
-    write_length(&mut frhp, attrs.len() as u64, LENGTH_SIZE); // managed_objects_count
+    write_length(&mut frhp, serialized.len() as u64, LENGTH_SIZE); // managed_objects_count
     write_length(&mut frhp, 0, LENGTH_SIZE); // huge_objects_size
     write_length(&mut frhp, 0, LENGTH_SIZE); // huge_objects_count
     write_length(&mut frhp, 0, LENGTH_SIZE); // tiny_objects_size
@@ -270,10 +311,10 @@ pub(crate) fn build_dense_attrs(attrs: &[AttributeMessage], base_address: u64) -
     debug_assert_eq!(dblock.len(), dblock_header_size);
 
     // Data area starts after header
-    let mut attr_offsets: Vec<(u64, u64)> = Vec::with_capacity(attrs.len());
+    let mut obj_offsets: Vec<(u64, u64)> = Vec::with_capacity(serialized.len());
-    for s in &serialized {
+    for s in serialized {
         let offset_in_heap = dblock.len() as u64;
-        attr_offsets.push((offset_in_heap, s.len() as u64));
+        obj_offsets.push((offset_in_heap, s.len() as u64));
         dblock.extend_from_slice(s);
     }
 
@@ -286,11 +327,283 @@ pub(crate) fn build_dense_attrs(attrs: &[AttributeMessage], base_address: u64) -
     debug_assert_eq!(dblock.len(), starting_block_size as usize);
 
     // Build heap IDs
-    let heap_ids: Vec<Vec<u8>> = attr_offsets
+    let heap_ids: Vec<Vec<u8>> = obj_offsets
         .iter()
         .map(|(off, len)| encode_managed_id(*off, *len, max_heap_size, heap_id_length))
         .collect();
 
+    let mut blob = Vec::with_capacity(frhp.len() + dblock.len());
+    blob.extend_from_slice(&frhp);
+    blob.extend_from_slice(&dblock);
+
+    FractalHeapBlock {
+        blob,
+        frhp_addr,
+        btree_addr,
+        heap_ids,
+        heap_id_length,
+    }
+}
+
+/// Build a multi-block fractal heap: a root indirect block (FHIB) over multiple
+/// direct blocks sized by the doubling table. Used when the objects don't fit
+/// in a single direct block. Objects do not span blocks (no huge-object path).
+fn build_multiblock_fractal_heap(
+    serialized: &[Vec<u8>],
+    base_address: u64,
+    max_heap_size: u16,
+    heap_id_length: u16,
+) -> FractalHeapBlock {
+    let os = OFFSET_SIZE as usize;
+    let block_offset_bytes = (max_heap_size as usize).div_ceil(8);
+    let max_direct_block_size: u64 = 65536;
+    let table_width: u16 = 4;
+    let starting_block_size: u64 = 512;
+    let dblock_header_size = 4 + 1 + os + block_offset_bytes + 4;
+    let block_capacity = |row: usize| block_size_for_row(starting_block_size, row) - dblock_header_size as u64;
+
+    // ---- Pack objects into direct blocks (row-major over the doubling table) ----
+    struct Blk {
+        row: usize,
+        size: u64,
+        heap_offset: u64,
+        data: Vec<u8>,
+    }
+    let mut blocks: Vec<Blk> = Vec::new();
+    // Each object's (heap_offset, length) for the heap ID.
+    let mut obj_loc: Vec<(u64, u64)> = vec![(0, 0); serialized.len()];
+
+    let mut row = 0usize;
+    let mut col = 0u16;
+    let mut heap_off = 0u64;
+    let mut cur: Option<Blk> = None;
+
+    for (idx, s) in serialized.iter().enumerate() {
+        loop {
+            if cur.is_none() {
+                let size = block_size_for_row(starting_block_size, row);
+                cur = Some(Blk {
+                    row,
+                    size,
+                    heap_offset: heap_off,
+                    data: Vec::new(),
+                });
+            }
+            let blk = cur.as_mut().unwrap();
+            let cap = block_capacity(blk.row) as usize;
+            if !blk.data.is_empty() && blk.data.len() + s.len() > cap {
+                // Doesn't fit; finalize this block and advance to the next slot.
+                let finished = cur.take().unwrap();
+                heap_off += finished.size;
+                blocks.push(finished);
+                col += 1;
+                if col >= table_width {
+                    col = 0;
+                    row += 1;
+                }
+                continue;
+            }
+            // Place the object (a fresh block always accepts at least one object
+            // up to its capacity; objects larger than a max block are unsupported).
+            let pos_in_block = dblock_header_size + blk.data.len();
+            obj_loc[idx] = (blk.heap_offset + pos_in_block as u64, s.len() as u64);
+            blk.data.extend_from_slice(s);
+            break;
+        }
+    }
+    if let Some(b) = cur.take() {
+        blocks.push(b);
+    }
+
+    let cur_rows = (blocks.last().map(|b| b.row).unwrap_or(0) + 1) as u16;
+
+    // ---- Addresses ----
+    let frhp_size = frhp_header_size(os, LENGTH_SIZE as usize);
+    let frhp_addr = base_address;
+    let fhib_addr = frhp_addr + frhp_size as u64;
+    let fhib_entries = cur_rows as usize * table_width as usize;
+    let fhib_size = 5 + os + block_offset_bytes + fhib_entries * os + 4;
+    let first_dblock_addr = fhib_addr + fhib_size as u64;
+
+    // Assign each used block an address (laid out consecutively after the FHIB).
+    let mut blk_addrs: Vec<u64> = Vec::with_capacity(blocks.len());
+    let mut a = first_dblock_addr;
+    for b in &blocks {
+        blk_addrs.push(a);
+        a += b.size;
+    }
+    let heap_end = a;
+    let btree_addr = heap_end;
+
+    // Bookkeeping totals.
+    let managed_space: u64 = (0..cur_rows as usize)
+        .map(|r| block_size_for_row(starting_block_size, r) * table_width as u64)
+        .sum();
+    let alloc_space: u64 = blocks.iter().map(|b| b.size).sum();
+    let used: u64 = blocks
+        .iter()
+        .map(|b| dblock_header_size as u64 + b.data.len() as u64)
+        .sum();
+    let free_space = alloc_space.saturating_sub(used);
+
+    // ---- FRHP header ----
+    let max_managed = max_direct_block_size as u32 - dblock_header_size as u32;
+    let frhp = write_frhp(WriteFrhp {
+        heap_id_length,
+        max_managed,
+        free_space,
+        managed_space,
+        alloc_space,
+        nobjects: serialized.len() as u64,
+        table_width,
+        starting_block_size,
+        max_direct_block_size,
+        max_heap_size,
+        root_addr: fhib_addr,
+        cur_rows,
+    });
+    debug_assert_eq!(frhp.len(), frhp_size);
+
+    // ---- Root indirect block (FHIB) ----
+    let mut fhib = Vec::with_capacity(fhib_size);
+    fhib.extend_from_slice(b"FHIB");
+    fhib.push(0); // version
+    write_offset(&mut fhib, frhp_addr, OFFSET_SIZE);
+    fhib.extend_from_slice(&vec![0u8; block_offset_bytes]); // block offset = 0 (root)
+    for &addr in &blk_addrs {
+        write_offset(&mut fhib, addr, OFFSET_SIZE);
+    }
+    // Remaining slots within the current rows are unallocated.
+    for _ in blk_addrs.len()..fhib_entries {
+        write_undef_offset(&mut fhib, OFFSET_SIZE);
+    }
+    let fhib_checksum = crate::checksum::jenkins_lookup3(&fhib);
+    fhib.extend_from_slice(&fhib_checksum.to_le_bytes());
+    debug_assert_eq!(fhib.len(), fhib_size);
+
+    // ---- Direct blocks ----
+    let mut blob = frhp;
+    blob.extend_from_slice(&fhib);
+    for b in &blocks {
+        let mut dblock = Vec::with_capacity(b.size as usize);
+        dblock.extend_from_slice(b"FHDB");
+        dblock.push(0); // version
+        write_offset(&mut dblock, frhp_addr, OFFSET_SIZE);
+        let mut bo = b.heap_offset.to_le_bytes().to_vec();
+        bo.truncate(block_offset_bytes);
+        dblock.extend_from_slice(&bo);
+        let cksum_pos = dblock.len();
+        dblock.extend_from_slice(&[0u8; 4]); // checksum placeholder
+        dblock.extend_from_slice(&b.data);
+        dblock.resize(b.size as usize, 0);
+        let cksum = crate::checksum::jenkins_lookup3(&dblock);
+        dblock[cksum_pos..cksum_pos + 4].copy_from_slice(&cksum.to_le_bytes());
+        blob.extend_from_slice(&dblock);
+    }
+
+    let heap_ids: Vec<Vec<u8>> = obj_loc
+        .iter()
+        .map(|(off, len)| encode_managed_id(*off, *len, max_heap_size, heap_id_length))
+        .collect();
+
+    FractalHeapBlock {
+        blob,
+        frhp_addr,
+        btree_addr,
+        heap_ids,
+        heap_id_length,
+    }
+}
+
+/// Doubling-table block size for `row`: rows 0 and 1 share the starting size;
+/// row r (r ≥ 1) is `start * 2^(r-1)`.
+fn block_size_for_row(starting_block_size: u64, row: usize) -> u64 {
+    if row <= 1 {
+        starting_block_size
+    } else {
+        starting_block_size << (row - 1)
+    }
+}
+
+/// Size in bytes of the FRHP header for the given offset/length sizes.
+fn frhp_header_size(os: usize, ls: usize) -> usize {
+    4 + 1 + 2 + 2 + 1 + 4 + ls + os + ls + os + ls + ls + ls + ls + ls + ls + ls + ls + 2 + ls + ls
+        + 2
+        + 2
+        + os
+        + 2
+        + 4
+}
+
+/// Parameters for [`write_frhp`].
+struct WriteFrhp {
+    heap_id_length: u16,
+    max_managed: u32,
+    free_space: u64,
+    managed_space: u64,
+    alloc_space: u64,
+    nobjects: u64,
+    table_width: u16,
+    starting_block_size: u64,
+    max_direct_block_size: u64,
+    max_heap_size: u16,
+    root_addr: u64,
+    cur_rows: u16,
+}
+
+/// Serialize a fractal heap header (FRHP).
+fn write_frhp(p: WriteFrhp) -> Vec<u8> {
+    let mut frhp = Vec::with_capacity(frhp_header_size(OFFSET_SIZE as usize, LENGTH_SIZE as usize));
+    frhp.extend_from_slice(b"FRHP");
+    frhp.push(0); // version
+    frhp.extend_from_slice(&p.heap_id_length.to_le_bytes());
+    frhp.extend_from_slice(&0u16.to_le_bytes()); // io_filter_encoded_length
+    frhp.push(0x02); // flags: bit 1 = checksum direct blocks
+    frhp.extend_from_slice(&p.max_managed.to_le_bytes());
+    write_length(&mut frhp, 0, LENGTH_SIZE); // next_huge_object_id
+    write_undef_offset(&mut frhp, OFFSET_SIZE); // btree_huge_objects_address
+    write_length(&mut frhp, p.free_space, LENGTH_SIZE); // free_space_managed_blocks
+    write_undef_offset(&mut frhp, OFFSET_SIZE); // free_space_mgr_addr
+    write_length(&mut frhp, p.managed_space, LENGTH_SIZE); // managed_space_in_heap
+    write_length(&mut frhp, p.alloc_space, LENGTH_SIZE); // allocated_managed_space
+    write_length(&mut frhp, 0, LENGTH_SIZE); // dblock_alloc_iter
+    write_length(&mut frhp, p.nobjects, LENGTH_SIZE); // managed_objects_count
+    write_length(&mut frhp, 0, LENGTH_SIZE); // huge_objects_size
+    write_length(&mut frhp, 0, LENGTH_SIZE); // huge_objects_count
+    write_length(&mut frhp, 0, LENGTH_SIZE); // tiny_objects_size
+    write_length(&mut frhp, 0, LENGTH_SIZE); // tiny_objects_count
+    frhp.extend_from_slice(&p.table_width.to_le_bytes());
+    write_length(&mut frhp, p.starting_block_size, LENGTH_SIZE);
+    write_length(&mut frhp, p.max_direct_block_size, LENGTH_SIZE);
+    frhp.extend_from_slice(&p.max_heap_size.to_le_bytes());
+    frhp.extend_from_slice(&1u16.to_le_bytes()); // starting # rows in root indirect block
+    write_offset(&mut frhp, p.root_addr, OFFSET_SIZE);
+    frhp.extend_from_slice(&p.cur_rows.to_le_bytes());
+    let checksum = crate::checksum::jenkins_lookup3(&frhp);
+    frhp.extend_from_slice(&checksum.to_le_bytes());
+    frhp
+}
+
+/// Build dense attribute storage for a set of attributes.
+pub(crate) fn build_dense_attrs(attrs: &[AttributeMessage], base_address: u64) -> DenseAttrBlob {
+    // Dense attrs use v3 attribute messages (adds character set encoding byte).
+    let serialized: Vec<Vec<u8>> = attrs.iter().map(|a| a.serialize_v3(LENGTH_SIZE)).collect();
+
+    let name_hashes: Vec<u32> = attrs
+        .iter()
+        .map(|a| crate::checksum::jenkins_lookup3(a.name.as_bytes()))
+        .collect();
+
+    let os = OFFSET_SIZE as usize;
+    let ls = LENGTH_SIZE as usize;
+
+    // Attribute heaps use max_heap_size 40 / heap ID length 8 (matching libhdf5).
+    let heap = build_single_block_fractal_heap(&serialized, base_address, 40, 8);
+    let frhp_addr = heap.frhp_addr;
+    let btree_addr = heap.btree_addr;
+    let heap_id_length = heap.heap_id_length;
+    let heap_ids = &heap.heap_ids;
+
     // Build B-tree v2 type 8 records (17 bytes each)
     let record_size: u16 = heap_id_length + 1 + 4 + 4;
     let mut records: Vec<(u32, u32, Vec<u8>)> = Vec::with_capacity(attrs.len());
@@ -342,9 +655,9 @@ pub(crate) fn build_dense_attrs(attrs: &[AttributeMessage], base_address: u64) -
     // Pad to node_size
     btlf.resize(node_size as usize, 0);
 
-    let mut blob = Vec::with_capacity(frhp.len() + dblock.len() + bthd.len() + btlf.len());
+    let mut blob =
-    blob.extend_from_slice(&frhp);
+        Vec::with_capacity(heap.blob.len() + bthd.len() + btlf.len());
-    blob.extend_from_slice(&dblock);
+    blob.extend_from_slice(&heap.blob);
     blob.extend_from_slice(&bthd);
     blob.extend_from_slice(&btlf);
 
@@ -356,6 +669,107 @@ pub(crate) fn build_dense_attrs(attrs: &[AttributeMessage], base_address: u64) -
     }
 }
 
+// ---- Dense link blob ----
+
+/// Pre-built dense link storage (fractal heap + B-tree v2 + link-info message).
+pub(crate) struct DenseLinkBlob {
+    /// Serialized LinkInfo message (to embed in the group's object header).
+    pub(crate) link_info_message: Vec<u8>,
+    /// The combined fractal heap header + direct block + B-tree v2 bytes.
+    pub(crate) blob: Vec<u8>,
+}
+
+/// Build dense link storage for a group's links, laid out at `base_address`.
+///
+/// Mirrors [`build_dense_attrs`]: each link is stored as a serialized Link
+/// message in a single-direct-block fractal heap, indexed by a v2 B-tree of
+/// **type 5** (link-name index, record = name hash + heap ID). The returned
+/// LinkInfo message points at the heap and the name B-tree.
+pub(crate) fn build_dense_links(links: &[LinkMessage], base_address: u64) -> DenseLinkBlob {
+    let serialized: Vec<Vec<u8>> = links.iter().map(|l| l.serialize(OFFSET_SIZE)).collect();
+    let name_hashes: Vec<u32> = links
+        .iter()
+        .map(|l| crate::checksum::jenkins_lookup3(l.name.as_bytes()))
+        .collect();
+
+    let os = OFFSET_SIZE as usize;
+    let ls = LENGTH_SIZE as usize;
+
+    // libhdf5's link heap uses max_heap_size 32 / heap ID length 7 (vs 40/8 for
+    // attributes), giving a 7-byte heap ID and an 11-byte type-5 record.
+    let heap = build_single_block_fractal_heap(&serialized, base_address, 32, 7);
+    let heap_id_length = heap.heap_id_length;
+
+    // B-tree v2 type 5 records: hash(4) + heap_id(heap_id_length). The B-tree
+    // search key is the name hash, so records are sorted by (hash, order).
+    let record_size: u16 = 4 + heap_id_length;
+    let mut records: Vec<(u32, u32, Vec<u8>)> = Vec::with_capacity(links.len());
+    for (i, heap_id) in heap.heap_ids.iter().enumerate() {
+        let mut rec = Vec::with_capacity(record_size as usize);
+        rec.extend_from_slice(&name_hashes[i].to_le_bytes()); // hash
+        rec.extend_from_slice(heap_id); // heap ID
+        records.push((name_hashes[i], i as u32, rec));
+    }
+    records.sort_by(|a, b| a.0.cmp(&b.0).then(a.1.cmp(&b.1)));
+
+    let bthd_size = 4 + 1 + 1 + 4 + 2 + 2 + 1 + 1 + os + 2 + ls + 4;
+    let num_records = links.len();
+    let btlf_size = 4 + 1 + 1 + (num_records * record_size as usize) + 4;
+    let node_size = btlf_size.next_power_of_two().max(512) as u32;
+
+    let bthd_addr = heap.btree_addr;
+    let btlf_addr = bthd_addr + bthd_size as u64;
+
+    let mut bthd = Vec::with_capacity(bthd_size);
+    bthd.extend_from_slice(b"BTHD");
+    bthd.push(0); // version
+    bthd.push(5); // type = link name index
+    bthd.extend_from_slice(&node_size.to_le_bytes());
+    bthd.extend_from_slice(&record_size.to_le_bytes());
+    bthd.extend_from_slice(&0u16.to_le_bytes()); // depth = 0 (single leaf)
+    bthd.push(100); // split_percent
+    bthd.push(40); // merge_percent
+    write_offset(&mut bthd, btlf_addr, OFFSET_SIZE);
+    bthd.extend_from_slice(&(num_records as u16).to_le_bytes());
+    write_length(&mut bthd, num_records as u64, LENGTH_SIZE);
+    let bthd_checksum = crate::checksum::jenkins_lookup3(&bthd);
+    bthd.extend_from_slice(&bthd_checksum.to_le_bytes());
+    debug_assert_eq!(bthd.len(), bthd_size);
+
+    let mut btlf = Vec::with_capacity(node_size as usize);
+    btlf.extend_from_slice(b"BTLF");
+    btlf.push(0); // version
+    btlf.push(5); // type
+    for (_, _, rec) in &records {
+        btlf.extend_from_slice(rec);
+    }
+    let btlf_checksum = crate::checksum::jenkins_lookup3(&btlf);
+    btlf.extend_from_slice(&btlf_checksum.to_le_bytes());
+    btlf.resize(node_size as usize, 0);
+
+    let mut blob =
+        Vec::with_capacity(heap.blob.len() + bthd.len() + btlf.len());
+    blob.extend_from_slice(&heap.blob);
+    blob.extend_from_slice(&bthd);
+    blob.extend_from_slice(&btlf);
+
+    DenseLinkBlob {
+        link_info_message: serialize_link_info(heap.frhp_addr, bthd_addr),
+        blob,
+    }
+}
+
+/// Serialize a LinkInfo message (version 0, no creation-order index) pointing
+/// at a fractal heap and a v2 B-tree name index.
+fn serialize_link_info(fh_addr: u64, btree_name_addr: u64) -> Vec<u8> {
+    let mut data = Vec::new();
+    data.push(0); // version
+    data.push(0x00); // flags: no creation-order tracking
+    write_offset(&mut data, fh_addr, OFFSET_SIZE);
+    write_offset(&mut data, btree_name_addr, OFFSET_SIZE);
+    data
+}
+
 fn encode_managed_id(offset: u64, length: u64, max_heap_size: u16, id_length: u16) -> Vec<u8> {
     let mut id = vec![0u8; id_length as usize];
     id[0] = 0x00; // type = 0 (managed)
@@ -599,6 +1013,18 @@ impl FileWriter {
             .map(|d| d.attrs.len() > DENSE_ATTR_THRESHOLD)
             .collect();
 
+        // Dense link decision: a group with more than the compact threshold of
+        // links stores them in a fractal heap + v2 B-tree instead of inline.
+        let root_link_count = root_ds_indices.len() + groups.len();
+        let root_links_dense = root_link_count > DENSE_LINK_THRESHOLD;
+        let group_links_dense: Vec<bool> = groups
+            .iter()
+            .map(|g| g.ds_indices.len() > DENSE_LINK_THRESHOLD)
+            .collect();
+        // The dense LinkInfo message is a fixed size regardless of address, so a
+        // dummy is sufficient for OH size computation.
+        let dummy_link_info = serialize_link_info(0, 0);
+
         // Pass 1: compute OH sizes with dummy addresses
         let group_oh_sizes: Vec<usize> = groups
             .iter()
@@ -609,12 +1035,9 @@ impl FileWriter {
                     .iter()
                     .map(|&i| make_link(&all_ds[i].name, 0))
                     .collect();
-                if group_dense[gi] {
+                let attr_blob = group_dense[gi].then(|| build_dense_attrs(&g.attrs, 0));
-                    let dummy_blob = build_dense_attrs(&g.attrs, 0);
+                let dl = group_links_dense[gi].then_some(dummy_link_info.as_slice());
-                    build_group_oh(&dummy_links, &g.attrs, Some(&dummy_blob)).len()
+                build_group_oh(&dummy_links, dl, &g.attrs, attr_blob.as_ref()).len()
-                } else {
-                    build_group_oh(&dummy_links, &g.attrs, None).len()
-                }
             })
             .collect();
 
@@ -628,11 +1051,10 @@ impl FileWriter {
             }
             links
         };
-        let root_oh_size = if root_dense {
+        let root_oh_size = {
-            let dummy_blob = build_dense_attrs(&root_attrs, 0);
+            let attr_blob = root_dense.then(|| build_dense_attrs(&root_attrs, 0));
-            build_group_oh(&root_dummy_links, &root_attrs, Some(&dummy_blob)).len()
+            let dl = root_links_dense.then_some(dummy_link_info.as_slice());
-        } else {
+            build_group_oh(&root_dummy_links, dl, &root_attrs, attr_blob.as_ref()).len()
-            build_group_oh(&root_dummy_links, &root_attrs, None).len()
         };
 
         struct DataBlob {
@@ -720,6 +1142,17 @@ impl FileWriter {
         let root_group_addr = SUPERBLOCK_SIZE as u64;
         let mut cursor2 = SUPERBLOCK_SIZE + root_oh_size;
 
+        // Each group is laid out as: object header, then (if dense) its link
+        // blob, then (if dense) its attribute blob. Link blobs are sized with
+        // dummy target addresses here — link message size is address-independent
+        // — and rebuilt with real addresses in the final pass.
+        let root_link_blob_addr = if root_links_dense {
+            let addr = cursor2 as u64;
+            cursor2 += build_dense_links(&root_dummy_links, addr).blob.len();
+            Some(addr)
+        } else {
+            None
+        };
         let root_dense_blob = if root_dense {
             let blob = build_dense_attrs(&root_attrs, cursor2 as u64);
             cursor2 += blob.blob.len();
@@ -728,6 +1161,7 @@ impl FileWriter {
             None
         };
 
+        let mut group_link_blob_addrs: Vec<Option<u64>> = Vec::new();
         let mut group_dense_blobs: Vec<Option<DenseAttrBlob>> = Vec::new();
         let group_addrs2: Vec<u64> = group_oh_sizes
             .iter()
@@ -735,6 +1169,18 @@ impl FileWriter {
             .map(|(gi, &sz)| {
                 let addr = cursor2 as u64;
                 cursor2 += sz;
+                if group_links_dense[gi] {
+                    let dummy_links: Vec<LinkMessage> = groups[gi]
+                        .ds_indices
+                        .iter()
+                        .map(|&i| make_link(&all_ds[i].name, 0))
+                        .collect();
+                    let blob_addr = cursor2 as u64;
+                    cursor2 += build_dense_links(&dummy_links, blob_addr).blob.len();
+                    group_link_blob_addrs.push(Some(blob_addr));
+                } else {
+                    group_link_blob_addrs.push(None);
+                }
                 if group_dense[gi] {
                     let blob = build_dense_attrs(&groups[gi].attrs, cursor2 as u64);
                     cursor2 += blob.blob.len();
@@ -868,27 +1314,41 @@ impl FileWriter {
         for (gi, g) in groups.iter().enumerate() {
             root_links.push(make_link(&g.name, group_addrs2[gi]));
         }
+        // Rebuild the root link blob with real target addresses (same size as
+        // the dummy used for layout); its LinkInfo goes in the OH.
+        let root_link_blob = root_link_blob_addr.map(|addr| build_dense_links(&root_links, addr));
+        let root_dl = root_link_blob.as_ref().map(|b| b.link_info_message.as_slice());
         buf.extend_from_slice(&build_group_oh(
             &root_links,
+            root_dl,
             &root_attrs,
             root_dense_blob.as_ref(),
         ));
+        if let Some(ref b) = root_link_blob {
+            buf.extend_from_slice(&b.blob);
+        }
         if let Some(ref blob) = root_dense_blob {
             buf.extend_from_slice(&blob.blob);
         }
 
-        // Group OHs + dense blobs
+        // Group OHs + dense blobs (link blob, then attr blob, matching pass 2)
         for (gi, g) in groups.iter().enumerate() {
             let links: Vec<LinkMessage> = g
                 .ds_indices
                 .iter()
                 .map(|&i| make_link(&all_ds[i].name, ds_oh_addrs2[i]))
                 .collect();
+            let link_blob = group_link_blob_addrs[gi].map(|addr| build_dense_links(&links, addr));
+            let dl = link_blob.as_ref().map(|b| b.link_info_message.as_slice());
             buf.extend_from_slice(&build_group_oh(
                 &links,
+                dl,
                 &g.attrs,
                 group_dense_blobs[gi].as_ref(),
             ));
+            if let Some(ref b) = link_blob {
+                buf.extend_from_slice(&b.blob);
+            }
             if let Some(ref blob) = group_dense_blobs[gi] {
                 buf.extend_from_slice(&blob.blob);
             }

crates/clawhdf5-format/src/filters.rs

@@ -8,7 +8,8 @@ use alloc::{vec, vec::Vec};
 
 use crate::error::FormatError;
 use crate::filter_pipeline::{
-    FILTER_DEFLATE, FILTER_FLETCHER32, FILTER_LZ4, FILTER_SHUFFLE, FILTER_ZSTD, FilterPipeline,
+    FILTER_DEFLATE, FILTER_FLETCHER32, FILTER_LZ4, FILTER_NBIT, FILTER_SCALEOFFSET, FILTER_SHUFFLE,
+    FILTER_ZSTD, FilterPipeline,
 };
 
 /// Apply a filter pipeline to decompress a chunk.
@@ -16,7 +17,7 @@ use crate::filter_pipeline::{
 pub fn decompress_chunk(
     compressed: &[u8],
     pipeline: &FilterPipeline,
-    _chunk_size: usize,
+    chunk_size: usize,
     element_size: u32,
 ) -> Result<Vec<u8>, FormatError> {
     let mut data = compressed.to_vec();
@@ -28,6 +29,10 @@ pub fn decompress_chunk(
             FILTER_LZ4 => lz4_decompress(&data)?,
             FILTER_ZSTD => zstd_decompress(&data)?,
             FILTER_FLETCHER32 => fletcher32_verify(&data)?,
+            // `chunk_size` is the expected decompressed size; pass it so these
+            // decoders can reject an element count that would over-allocate.
+            FILTER_SCALEOFFSET => scaleoffset_decompress(&data, &filter.client_data, chunk_size)?,
+            FILTER_NBIT => nbit_decompress(&data, &filter.client_data, chunk_size)?,
             other => return Err(FormatError::UnsupportedFilter(other)),
         };
     }
@@ -64,6 +69,501 @@ pub fn compress_chunk(
     Ok(result)
 }
 
+/// Decode the HDF5 scale-offset filter (id 6).
+///
+/// Supports the integer variant (`H5Z_SO_INT`) and the floating-point
+/// **D-scale** variant (`H5Z_SO_FLOAT_DSCALE`); the float E-scale variant is
+/// reported as unsupported.
+///
+/// Compressed buffer layout (reverse-engineered against HDF5 2.0 and verified
+/// across signed/unsigned int sizes, f32/f64, negatives, fill values and chunk
+/// sizes): `minbits` (u32 LE) · `minval_width` (1 byte) · `minval`
+/// (`minval_width` bytes — a little-endian integer for the int variant, or the
+/// minimum float for D-scale) · 8 reserved bytes · MSB-first packed codes
+/// (`nelmts * minbits` bits). The all-ones code is reserved for the (defined)
+/// fill value. Integer reconstruction is `value = minval + code`; D-scale float
+/// is `value = minval + code / 10^scale_factor`.
+///
+/// `cd` is the `H5Zscaleoffset.c` parameter block: `[0]`=scale type
+/// (0 = float D-scale, 2 = integer), `[1]`=scale factor (decimal digits for
+/// D-scale), `[2]`=element count, `[4]`=element size, `[5]`=signed flag,
+/// `[6]`=byte order (1 = big-endian), `[7]`=fill defined, `[8..]`=fill value.
+fn scaleoffset_decompress(
+    data: &[u8],
+    cd: &[u32],
+    expected_bytes: usize,
+) -> Result<Vec<u8>, FormatError> {
+    const H5Z_SO_FLOAT_DSCALE: u32 = 0;
+    const H5Z_SO_INT: u32 = 2;
+    if cd.len() < 8 {
+        return Err(FormatError::ChunkedReadError(
+            "scale-offset: missing filter client data".into(),
+        ));
+    }
+    let scale_type = cd[0];
+    let is_float = scale_type == H5Z_SO_FLOAT_DSCALE;
+    if scale_type != H5Z_SO_INT && !is_float {
+        // Float E-scale (scale type 1) uses a different algorithm.
+        return Err(FormatError::UnsupportedFilter(FILTER_SCALEOFFSET));
+    }
+    let nelmts = cd[2] as usize;
+    let elem_size = cd[4] as usize;
+    if elem_size == 0 || elem_size > 8 || (is_float && elem_size != 4 && elem_size != 8) {
+        return Err(FormatError::ChunkedReadError(
+            "scale-offset: unsupported element size".into(),
+        ));
+    }
+    // The decoded output must match the chunk's uncompressed size; reject an
+    // element count that would over-allocate (e.g. minbits == 0 with a huge
+    // nelmts and no packed payload to bound it).
+    let out_bytes = nelmts
+        .checked_mul(elem_size)
+        .ok_or_else(|| FormatError::ChunkedReadError("scale-offset: size overflow".into()))?;
+    if expected_bytes != 0 && out_bytes > expected_bytes {
+        return Err(FormatError::ChunkedReadError(
+            "scale-offset: element count exceeds chunk size".into(),
+        ));
+    }
+    let signed = cd[5] == 1;
+    let big_endian = cd[6] == 1;
+    let fill_defined = cd[7] == 1;
+
+    // --- header: minbits, then minval, then 8 reserved bytes ---
+    if data.len() < 5 {
+        return Err(FormatError::ChunkedReadError(
+            "scale-offset: truncated header".into(),
+        ));
+    }
+    let minbits = u32::from_le_bytes([data[0], data[1], data[2], data[3]]) as usize;
+    let minval_width = data[4] as usize;
+    let minval_end = 5 + minval_width;
+    if data.len() < minval_end {
+        return Err(FormatError::ChunkedReadError(
+            "scale-offset: truncated minval".into(),
+        ));
+    }
+    let minval_bytes = &data[5..minval_end];
+
+    // --- unpack the per-element codes (MSB-first), shared by both variants ---
+    if minbits > 64 {
+        return Err(FormatError::ChunkedReadError(
+            "scale-offset: implausible minbits".into(),
+        ));
+    }
+    let codes: Vec<u64> = if minbits == 0 {
+        // No packed payload: every element equals minval.
+        vec![0u64; nelmts]
+    } else {
+        let packed = data.get(minval_end + 8..).ok_or_else(|| {
+            FormatError::ChunkedReadError("scale-offset: truncated packed data".into())
+        })?;
+        let need_bits = nelmts
+            .checked_mul(minbits)
+            .ok_or_else(|| FormatError::ChunkedReadError("scale-offset: size overflow".into()))?;
+        if packed.len() * 8 < need_bits {
+            return Err(FormatError::ChunkedReadError(
+                "scale-offset: packed data too short".into(),
+            ));
+        }
+        let mut out = Vec::with_capacity(nelmts);
+        let mut bitpos = 0usize;
+        for _ in 0..nelmts {
+            let mut code: u64 = 0;
+            for _ in 0..minbits {
+                let bit = (packed[bitpos / 8] >> (7 - (bitpos % 8))) & 1;
+                code = (code << 1) | bit as u64;
+                bitpos += 1;
+            }
+            out.push(code);
+        }
+        out
+    };
+    // The fill code (all ones) only exists when there are bits to pack.
+    let has_fill_code = fill_defined && minbits > 0 && minbits < 64;
+    // Computed for all 1..=64 widths; `1 << 64` would overflow, so saturate.
+    let fill_code: u64 = if minbits == 0 {
+        0
+    } else if minbits >= 64 {
+        u64::MAX
+    } else {
+        (1u64 << minbits) - 1
+    };
+
+    if is_float {
+        let scale = 10f64.powi(cd[1] as i32);
+        let minval = read_le_float(minval_bytes, elem_size);
+        let fill_value = if fill_defined {
+            let lo = *cd.get(8).unwrap_or(&0) as u64;
+            let hi = *cd.get(9).unwrap_or(&0) as u64;
+            bits_to_float(lo | (hi << 32), elem_size)
+        } else {
+            0.0
+        };
+        let values: Vec<f64> = codes
+            .iter()
+            .map(|&code| {
+                if has_fill_code && code == fill_code {
+                    fill_value
+                } else {
+                    minval + code as f64 / scale
+                }
+            })
+            .collect();
+        Ok(write_floats(&values, elem_size, big_endian))
+    } else {
+        let minval = read_le_int(minval_bytes, signed);
+        let fill_value: i64 = if fill_defined {
+            let lo = *cd.get(8).unwrap_or(&0) as u64;
+            let hi = *cd.get(9).unwrap_or(&0) as u64;
+            sign_extend(lo | (hi << 32), elem_size, signed)
+        } else {
+            0
+        };
+        let values: Vec<i64> = codes
+            .iter()
+            .map(|&code| {
+                if has_fill_code && code == fill_code {
+                    fill_value
+                } else {
+                    minval.wrapping_add(code as i64)
+                }
+            })
+            .collect();
+        Ok(write_elements(&values, elem_size, big_endian))
+    }
+}
+
+/// Read a little-endian float of `size` bytes (4 = f32, otherwise f64) as f64.
+fn read_le_float(bytes: &[u8], size: usize) -> f64 {
+    if size == 4 {
+        let mut b = [0u8; 4];
+        let n = bytes.len().min(4);
+        b[..n].copy_from_slice(&bytes[..n]);
+        f32::from_le_bytes(b) as f64
+    } else {
+        let mut b = [0u8; 8];
+        let n = bytes.len().min(8);
+        b[..n].copy_from_slice(&bytes[..n]);
+        f64::from_le_bytes(b)
+    }
+}
+
+/// Interpret the low bits of `raw` as an IEEE float of `size` bytes.
+fn bits_to_float(raw: u64, size: usize) -> f64 {
+    if size == 4 {
+        f32::from_bits(raw as u32) as f64
+    } else {
+        f64::from_bits(raw)
+    }
+}
+
+/// Serialize reconstructed float values as `elem_size`-byte (f32/f64) elements
+/// in the requested byte order.
+fn write_floats(values: &[f64], elem_size: usize, big_endian: bool) -> Vec<u8> {
+    let mut out = Vec::with_capacity(values.len() * elem_size);
+    for &v in values {
+        let bytes: [u8; 8] = if elem_size == 4 {
+            let mut b = [0u8; 8];
+            b[..4].copy_from_slice(&(v as f32).to_le_bytes());
+            b
+        } else {
+            v.to_le_bytes()
+        };
+        if big_endian {
+            for i in (0..elem_size).rev() {
+                out.push(bytes[i]);
+            }
+        } else {
+            out.extend_from_slice(&bytes[..elem_size]);
+        }
+    }
+    out
+}
+
+/// Read a little-endian integer of `bytes.len()` bytes, sign-extending when
+/// `signed`. Used for the scale-offset `minval` field.
+fn read_le_int(bytes: &[u8], signed: bool) -> i64 {
+    let mut raw: u64 = 0;
+    for (i, &b) in bytes.iter().enumerate().take(8) {
+        raw |= (b as u64) << (i * 8);
+    }
+    sign_extend(raw, bytes.len().min(8), signed)
+}
+
+/// Interpret the low `size` bytes of `raw` as a (possibly signed) integer.
+fn sign_extend(raw: u64, size: usize, signed: bool) -> i64 {
+    if size == 0 || size >= 8 {
+        return raw as i64;
+    }
+    let bits = size * 8;
+    let mask = (1u64 << bits) - 1;
+    let val = raw & mask;
+    if signed && (val & (1u64 << (bits - 1))) != 0 {
+        (val | !mask) as i64
+    } else {
+        val as i64
+    }
+}
+
+/// Serialize reconstructed integer values as `elem_size`-byte elements in the
+/// requested byte order.
+fn write_elements(values: &[i64], elem_size: usize, big_endian: bool) -> Vec<u8> {
+    let mut out = Vec::with_capacity(values.len() * elem_size);
+    for &v in values {
+        let le = (v as u64).to_le_bytes();
+        if big_endian {
+            for i in (0..elem_size).rev() {
+                out.push(le[i]);
+            }
+        } else {
+            out.extend_from_slice(&le[..elem_size]);
+        }
+    }
+    out
+}
+
+/// A node of the N-Bit datatype tree (reconstructed from the filter's client
+/// data) describing how one element is bit-packed.
+enum NbitNode {
+    /// Leaf: `precision` significant bits at `bit_offset` of a `size`-byte field.
+    Atomic {
+        size: usize,
+        big_endian: bool,
+        precision: u32,
+        bit_offset: u32,
+    },
+    /// Fixed-size struct of members, each at a byte offset within the element.
+    Compound {
+        size: usize,
+        members: Vec<(usize, NbitNode)>,
+    },
+    /// `count` consecutive copies of `base`, each `base_size` bytes apart.
+    Array {
+        base: Box<NbitNode>,
+        count: usize,
+        base_size: usize,
+    },
+}
+
+impl NbitNode {
+    fn byte_size(&self) -> usize {
+        match self {
+            NbitNode::Atomic { size, .. } => *size,
+            NbitNode::Compound { size, .. } => *size,
+            NbitNode::Array {
+                count, base_size, ..
+            } => count * base_size,
+        }
+    }
+}
+
+fn nbit_cd(cd: &[u32], i: usize) -> Result<u32, FormatError> {
+    cd.get(i)
+        .copied()
+        .ok_or_else(|| FormatError::ChunkedReadError("nbit: truncated client data".into()))
+}
+
+/// Maximum N-Bit type-tree nesting depth. Real types nest only a few levels;
+/// the bound stops a crafted tree from recursing into a stack overflow.
+const NBIT_MAX_DEPTH: u32 = 64;
+
+/// Parse one N-Bit type node from the client-data tree, advancing `idx`.
+fn parse_nbit_node(cd: &[u32], idx: &mut usize, depth: u32) -> Result<NbitNode, FormatError> {
+    const ATOMIC: u32 = 1;
+    const ARRAY: u32 = 2;
+    const COMPOUND: u32 = 3;
+    if depth > NBIT_MAX_DEPTH {
+        return Err(FormatError::ChunkedReadError(
+            "nbit: type tree nested too deeply".into(),
+        ));
+    }
+    let class = nbit_cd(cd, *idx)?;
+    match class {
+        ATOMIC => {
+            // class, size, byte order, precision, bit offset
+            let size = nbit_cd(cd, *idx + 1)? as usize;
+            let big_endian = nbit_cd(cd, *idx + 2)? == 1;
+            let precision = nbit_cd(cd, *idx + 3)?;
+            let bit_offset = nbit_cd(cd, *idx + 4)?;
+            *idx += 5;
+            let end_bit = bit_offset.checked_add(precision);
+            if size == 0
+                || size > 8
+                || precision == 0
+                || end_bit.is_none_or(|e| e > (size * 8) as u32)
+            {
+                return Err(FormatError::ChunkedReadError(
+                    "nbit: invalid atomic parameters".into(),
+                ));
+            }
+            Ok(NbitNode::Atomic {
+                size,
+                big_endian,
+                precision,
+                bit_offset,
+            })
+        }
+        ARRAY => {
+            // class, total size, base type node
+            let total = nbit_cd(cd, *idx + 1)? as usize;
+            *idx += 2;
+            let base = parse_nbit_node(cd, idx, depth + 1)?;
+            let base_size = base.byte_size();
+            if base_size == 0 || !total.is_multiple_of(base_size) {
+                return Err(FormatError::ChunkedReadError(
+                    "nbit: invalid array layout".into(),
+                ));
+            }
+            Ok(NbitNode::Array {
+                count: total / base_size,
+                base_size,
+                base: Box::new(base),
+            })
+        }
+        COMPOUND => {
+            // class, total size, member count, (member byte offset, node)*
+            let total = nbit_cd(cd, *idx + 1)? as usize;
+            let nmembers = nbit_cd(cd, *idx + 2)? as usize;
+            *idx += 3;
+            // Don't pre-allocate from the untrusted member count; the loop is
+            // bounded by `nbit_cd` running out of client data.
+            let mut members = Vec::new();
+            for _ in 0..nmembers {
+                let moff = nbit_cd(cd, *idx)? as usize;
+                *idx += 1;
+                let node = parse_nbit_node(cd, idx, depth + 1)?;
+                if moff
+                    .checked_add(node.byte_size())
+                    .is_none_or(|end| end > total)
+                {
+                    return Err(FormatError::ChunkedReadError(
+                        "nbit: member exceeds compound size".into(),
+                    ));
+                }
+                members.push((moff, node));
+            }
+            Ok(NbitNode::Compound {
+                size: total,
+                members,
+            })
+        }
+        // Class 4 is H5Z_NBIT_NOOPTYPE (members copied verbatim) — not seen in
+        // practice for the supported leaf types and left unsupported.
+        _ => Err(FormatError::UnsupportedFilter(FILTER_NBIT)),
+    }
+}
+
+/// MSB-first bit reader over the packed N-Bit stream.
+struct BitReader<'a> {
+    data: &'a [u8],
+    pos: usize,
+}
+
+impl BitReader<'_> {
+    fn read(&mut self, nbits: u32) -> Result<u64, FormatError> {
+        let mut value = 0u64;
+        for _ in 0..nbits {
+            let byte = *self.data.get(self.pos / 8).ok_or_else(|| {
+                FormatError::ChunkedReadError("nbit: packed data too short".into())
+            })?;
+            let bit = (byte >> (7 - (self.pos % 8))) & 1;
+            value = (value << 1) | bit as u64;
+            self.pos += 1;
+        }
+        Ok(value)
+    }
+}
+
+/// Decode one element node into `elem[base..]` (the rest stays zero-filled).
+fn decode_nbit_node(
+    node: &NbitNode,
+    br: &mut BitReader,
+    elem: &mut [u8],
+    base: usize,
+) -> Result<(), FormatError> {
+    match node {
+        NbitNode::Atomic {
+            size,
+            big_endian,
+            precision,
+            bit_offset,
+        } => {
+            let value = br.read(*precision)? << bit_offset;
+            let le = value.to_le_bytes();
+            let dst = &mut elem[base..base + size];
+            if *big_endian {
+                for (j, slot) in dst.iter_mut().enumerate() {
+                    *slot = le[size - 1 - j];
+                }
+            } else {
+                dst.copy_from_slice(&le[..*size]);
+            }
+        }
+        NbitNode::Compound { members, .. } => {
+            for (moff, child) in members {
+                decode_nbit_node(child, br, elem, base + moff)?;
+            }
+        }
+        NbitNode::Array {
+            base: bnode,
+            count,
+            base_size,
+        } => {
+            for i in 0..*count {
+                decode_nbit_node(bnode, br, elem, base + i * base_size)?;
+            }
+        }
+    }
+    Ok(())
+}
+
+/// Decode the HDF5 N-Bit filter (id 5).
+///
+/// N-Bit strips the unused leading/trailing bits of each (possibly nested)
+/// datatype field and packs the significant `precision` bits MSB-first,
+/// contiguously, with no header. The filter's client data carries a recursive
+/// type tree — atomic (`[1, size, order, precision, offset]`), array
+/// (`[2, total_size, <base>]`) and compound
+/// (`[3, total_size, nmembers, (offset, <node>)*]`) — preceded by
+/// `[nparms, flag, nelmts]`. Decompression walks the tree once per element,
+/// placing each field's bits at its byte/bit offset in a zero-filled element
+/// (HDF5's canonical reduced-precision layout). Sign-extension of reduced
+/// precision signed integers is the datatype reader's job. Atomic floats are
+/// encoded as full-precision atomics and handled transparently.
+fn nbit_decompress(data: &[u8], cd: &[u32], expected_bytes: usize) -> Result<Vec<u8>, FormatError> {
+    if cd.len() < 4 {
+        return Err(FormatError::ChunkedReadError(
+            "nbit: missing filter client data".into(),
+        ));
+    }
+    let nelmts = cd[2] as usize;
+    let mut idx = 3;
+    let root = parse_nbit_node(cd, &mut idx, 0)?;
+    let elem_size = root.byte_size();
+    if elem_size == 0 {
+        return Err(FormatError::ChunkedReadError(
+            "nbit: zero element size".into(),
+        ));
+    }
+
+    let total = nelmts
+        .checked_mul(elem_size)
+        .ok_or_else(|| FormatError::ChunkedReadError("nbit: size overflow".into()))?;
+    // The decoded output must match the chunk's uncompressed size; reject a
+    // count that would over-allocate.
+    if expected_bytes != 0 && total > expected_bytes {
+        return Err(FormatError::ChunkedReadError(
+            "nbit: element count exceeds chunk size".into(),
+        ));
+    }
+    let mut out = vec![0u8; total];
+    let mut br = BitReader { data, pos: 0 };
+    for elem in out.chunks_exact_mut(elem_size) {
+        decode_nbit_node(&root, &mut br, elem, 0)?;
+    }
+    Ok(out)
+}
+
 /// Decompress zlib-compressed data.
 #[cfg(feature = "deflate")]
 fn deflate_decompress(data: &[u8]) -> Result<Vec<u8>, FormatError> {
@@ -707,4 +1207,242 @@ mod tests {
         let decompressed = decompress_chunk(&compressed, &pipeline, data.len(), 8).unwrap();
         assert_eq!(decompressed, data);
     }
+
+    // --- scale-offset (filter id 6) -------------------------------------------
+    // Inputs below are real compressed chunks + client data captured from
+    // h5py 3.16 / HDF5 2.0 (`scaleoffset=0`), so they guard the decoder against
+    // the reference implementation without needing h5py at test time.
+
+    fn i32_le(vals: &[i32]) -> Vec<u8> {
+        vals.iter().flat_map(|v| v.to_le_bytes()).collect()
+    }
+
+    #[test]
+    fn scaleoffset_int_basic() {
+        // i32 [0,1,2,3], chunk of 4, default fill 0 (element 0 is the fill).
+        let cd = [2u32, 0, 4, 0, 4, 1, 0, 1, 0];
+        let raw = [
+            0x02, 0x00, 0x00, 0x00, 0x08, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+            0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xc6, 0x00,
+        ];
+        assert_eq!(scaleoffset_decompress(&raw, &cd, 0).unwrap(), i32_le(&[0, 1, 2, 3]));
+    }
+
+    #[test]
+    fn scaleoffset_int_negative() {
+        // i32 [-5,-3,-1,0,2,4,7,9], chunk of 8, fill 0 (minval = -5).
+        let cd = [2u32, 0, 8, 0, 4, 1, 0, 1, 0];
+        let raw = [
+            0x04, 0x00, 0x00, 0x00, 0x08, 0xfb, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0x00,
+            0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0x4f, 0x79, 0xce, 0x00,
+        ];
+        assert_eq!(
+            scaleoffset_decompress(&raw, &cd, 0).unwrap(),
+            i32_le(&[-5, -3, -1, 0, 2, 4, 7, 9])
+        );
+    }
+
+    #[test]
+    fn scaleoffset_uint() {
+        // u32 [100..109], chunk of 10, unsigned (cd[5]==0), minval 100.
+        let cd = [2u32, 0, 10, 0, 4, 0, 0, 1, 0];
+        let raw = [
+            0x04, 0x00, 0x00, 0x00, 0x08, 0x64, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+            0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x23, 0x45, 0x67, 0x89, 0x00,
+        ];
+        let expected: Vec<u8> = (100u32..110).flat_map(|v| v.to_le_bytes()).collect();
+        assert_eq!(scaleoffset_decompress(&raw, &cd, 0).unwrap(), expected);
+    }
+
+    fn as_f32(bytes: &[u8]) -> Vec<f32> {
+        bytes
+            .chunks_exact(4)
+            .map(|c| f32::from_le_bytes(c.try_into().unwrap()))
+            .collect()
+    }
+
+    #[test]
+    fn scaleoffset_float_dscale_d1() {
+        // f32 [0,1,2,3], D=1, default fill 0 (element 0 is the fill).
+        let cd = [0u32, 1, 4, 1, 4, 0, 0, 1, 0];
+        let raw = [
+            0x05, 0x00, 0x00, 0x00, 0x08, 0x00, 0x00, 0x80, 0x3f, 0x00, 0x00, 0x00, 0x00, 0x00,
+            0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xf8, 0x15, 0x40,
+        ];
+        let got = as_f32(&scaleoffset_decompress(&raw, &cd, 0).unwrap());
+        assert_eq!(got, vec![0.0, 1.0, 2.0, 3.0]);
+    }
+
+    #[test]
+    fn scaleoffset_float_dscale_d3() {
+        // f32 [0,0.1,0.2,0.3], D=3 (lossy reconstruction within 10^-3).
+        let cd = [0u32, 3, 4, 1, 4, 0, 0, 1, 0];
+        let raw = [
+            0x08, 0x00, 0x00, 0x00, 0x08, 0xcd, 0xcc, 0xcc, 0x3d, 0x00, 0x00, 0x00, 0x00, 0x00,
+            0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xff, 0x00, 0x64, 0xc8, 0x00,
+        ];
+        let got = as_f32(&scaleoffset_decompress(&raw, &cd, 0).unwrap());
+        let exp = [0.0f32, 0.1, 0.2, 0.3];
+        assert_eq!(got.len(), 4);
+        for (g, e) in got.iter().zip(exp.iter()) {
+            assert!((g - e).abs() < 1e-3, "got {g} expected {e}");
+        }
+    }
+
+    #[test]
+    fn scaleoffset_float_escale_unsupported() {
+        // scale_type 1 = float E-scale — a different algorithm, must be rejected.
+        let cd = [1u32, 3, 50, 1, 4, 0, 0, 1, 0];
+        let raw = [0u8; 24];
+        assert!(matches!(
+            scaleoffset_decompress(&raw, &cd, 0),
+            Err(FormatError::UnsupportedFilter(FILTER_SCALEOFFSET))
+        ));
+    }
+
+    // --- N-Bit (filter id 5) --------------------------------------------------
+    // Real compressed chunks + client data from h5py 3.16 / HDF5 2.0. The
+    // expected outputs are the canonical zero-filled element bytes (verified
+    // equal to the contiguous, un-filtered on-disk layout of the same type).
+
+    #[test]
+    fn nbit_unsigned_12bit() {
+        // u32 storage, 12-bit precision: [0, 1, 4095, 2048].
+        let cd = [8u32, 0, 4, 1, 4, 0, 12, 0];
+        let raw = [0x00, 0x00, 0x01, 0xff, 0xf8, 0x00, 0x00];
+        let expected: Vec<u8> = [0u32, 1, 4095, 2048]
+            .iter()
+            .flat_map(|v| v.to_le_bytes())
+            .collect();
+        assert_eq!(nbit_decompress(&raw, &cd, 0).unwrap(), expected);
+    }
+
+    #[test]
+    fn nbit_signed_16bit_zero_filled() {
+        // i32 storage, 16-bit precision: [-1, -50, 100, -1000]. N-Bit restores
+        // the canonical zero-filled layout (high 16 bits zero); the datatype
+        // reader is responsible for sign-extending reduced-precision integers.
+        let cd = [8u32, 0, 4, 1, 4, 0, 16, 0];
+        let raw = [0xff, 0xff, 0xff, 0xce, 0x00, 0x64, 0xfc, 0x18, 0x00];
+        // Canonical bytes captured from an equivalent un-filtered dataset.
+        let expected: Vec<u8> = vec![
+            0xff, 0xff, 0x00, 0x00, // 0x0000ffff
+            0xce, 0xff, 0x00, 0x00, // 0x0000ffce
+            0x64, 0x00, 0x00, 0x00, // 0x00000064
+            0x18, 0xfc, 0x00, 0x00, // 0x0000fc18
+        ];
+        assert_eq!(nbit_decompress(&raw, &cd, 0).unwrap(), expected);
+    }
+
+    #[test]
+    fn nbit_compound_int_members() {
+        // Compound { a: i32@0 prec 16, b: u32@4 prec 8 }, 3 elements.
+        // data = [(-1,200),(1000,7),(-32768,255)]. Captured from HDF5 2.0.
+        let cd = [18u32, 0, 3, 3, 8, 2, 0, 1, 4, 0, 16, 0, 4, 1, 4, 0, 8, 0];
+        let raw = [0xff, 0xff, 0xc8, 0x03, 0xe8, 0x07, 0x80, 0x00, 0xff, 0x00];
+        #[rustfmt::skip]
+        let expected: Vec<u8> = vec![
+            0xff,0xff,0x00,0x00, 0xc8,0x00,0x00,0x00, // (-1, 200)
+            0xe8,0x03,0x00,0x00, 0x07,0x00,0x00,0x00, // (1000, 7)
+            0x00,0x80,0x00,0x00, 0xff,0x00,0x00,0x00, // (-32768, 255)
+        ];
+        assert_eq!(nbit_decompress(&raw, &cd, 0).unwrap(), expected);
+    }
+
+    #[test]
+    fn nbit_compound_with_array_member() {
+        // Compound { a: array(2,) of i32 prec 16 @0; b: u32@8 prec 8 }, 2 elements.
+        // data = [([-1,100],200), ([1000,-32768],7)].
+        let cd = [20u32, 0, 2, 3, 12, 2, 0, 2, 8, 1, 4, 0, 16, 0, 8, 1, 4, 0, 8, 0];
+        let raw = [0xff, 0xff, 0x00, 0x64, 0xc8, 0x03, 0xe8, 0x80, 0x00, 0x07, 0x00];
+        #[rustfmt::skip]
+        let expected: Vec<u8> = vec![
+            0xff,0xff,0x00,0x00, 0x64,0x00,0x00,0x00, 0xc8,0x00,0x00,0x00, // ([-1,100], 200)
+            0xe8,0x03,0x00,0x00, 0x00,0x80,0x00,0x00, 0x07,0x00,0x00,0x00, // ([1000,-32768], 7)
+        ];
+        assert_eq!(nbit_decompress(&raw, &cd, 0).unwrap(), expected);
+    }
+
+    #[test]
+    fn nbit_compound_with_float_member() {
+        // Compound { i: i32@0 prec 16; f: f32@4 prec 32 }, 2 elements.
+        // data = [(-1, 1.5), (100, -2.5)]. The float member is a full-precision
+        // atomic; its bits are packed MSB-first and restored to LE storage.
+        let cd = [18u32, 0, 2, 3, 8, 2, 0, 1, 4, 0, 16, 0, 4, 1, 4, 0, 32, 0];
+        let raw = [
+            0xff, 0xff, 0x3f, 0xc0, 0x00, 0x00, 0x00, 0x64, 0xc0, 0x20, 0x00, 0x00, 0x00,
+        ];
+        #[rustfmt::skip]
+        let expected: Vec<u8> = vec![
+            0xff,0xff,0x00,0x00, 0x00,0x00,0xc0,0x3f, // (-1, 1.5)
+            0x64,0x00,0x00,0x00, 0x00,0x00,0x20,0xc0, // (100, -2.5)
+        ];
+        assert_eq!(nbit_decompress(&raw, &cd, 0).unwrap(), expected);
+    }
+
+    // ----- Adversarial / hardening: malformed filter data must not panic -----
+
+    #[test]
+    fn scaleoffset_minbits_64_does_not_panic() {
+        // minbits == 64 previously overflowed `1u64 << minbits` computing the
+        // fill code. cd: int, nelmts=1, elem_size=8, fill_defined=1.
+        let cd = [2u32, 0, 1, 0, 8, 1, 0, 1];
+        let mut data = Vec::new();
+        data.extend_from_slice(&64u32.to_le_bytes()); // minbits = 64
+        data.push(8); // minval_width
+        data.extend_from_slice(&0i64.to_le_bytes()); // minval
+        data.extend_from_slice(&[0u8; 8]); // reserved
+        data.extend_from_slice(&[0u8; 8]); // one 64-bit code
+        // Must return a Result (Ok or Err) without panicking.
+        let _ = scaleoffset_decompress(&data, &cd, 8);
+    }
+
+    #[test]
+    fn scaleoffset_huge_nelmts_bounded_by_chunk_size() {
+        // minbits == 0 (no packed payload) with a giant nelmts must not try to
+        // allocate when the expected chunk size is small.
+        let cd = [2u32, 0, u32::MAX, 0, 4, 0, 0, 0];
+        let mut data = Vec::new();
+        data.extend_from_slice(&0u32.to_le_bytes()); // minbits = 0
+        data.push(4);
+        data.extend_from_slice(&[0u8; 4]);
+        assert!(scaleoffset_decompress(&data, &cd, 64).is_err());
+    }
+
+    #[test]
+    fn nbit_deeply_nested_array_is_rejected() {
+        // A chain of ARRAY nodes (class 2) far deeper than NBIT_MAX_DEPTH must
+        // error rather than recurse into a stack overflow. Layout per level:
+        // [class=2, total]. Terminate with a (never-reached) atomic.
+        let mut cd = vec![0u32, 0, 1]; // nparms, flag, nelmts
+        for _ in 0..500 {
+            cd.push(2); // ARRAY
+            cd.push(8); // total size
+        }
+        cd.extend_from_slice(&[1, 8, 0, 8, 0]); // atomic leaf
+        assert!(nbit_decompress(&[], &cd, 0).is_err());
+    }
+
+    #[test]
+    fn nbit_atomic_bit_offset_overflow_is_rejected() {
+        // bit_offset + precision near u32::MAX must not overflow the check.
+        let cd = [0u32, 0, 1, 1, 8, 0, u32::MAX, u32::MAX];
+        assert!(nbit_decompress(&[0u8; 8], &cd, 0).is_err());
+    }
+
+    #[test]
+    fn nbit_huge_nelmts_bounded_by_chunk_size() {
+        // Valid 1-byte atomic but an enormous element count; the expected chunk
+        // size bounds the allocation.
+        let cd = [0u32, 0, u32::MAX, 1, 1, 0, 8, 0];
+        assert!(nbit_decompress(&[0u8; 4], &cd, 16).is_err());
+    }
+
+    #[test]
+    fn scaleoffset_truncated_inputs_do_not_panic() {
+        assert!(scaleoffset_decompress(&[], &[2, 0, 1, 0, 4, 0, 0, 0], 4).is_err());
+        assert!(scaleoffset_decompress(&[0u8; 3], &[2, 0, 1, 0, 4, 0, 0, 0], 4).is_err());
+        // Missing client data entirely.
+        assert!(scaleoffset_decompress(&[0u8; 32], &[2, 0], 4).is_err());
+    }
 }

crates/clawhdf5-format/src/fixed_array.rs

@@ -140,27 +140,36 @@ pub fn read_fixed_array_chunks(
         ));
     }
 
-    // Skip version(1) + client_id(1) + header_address(offset_size)
+    // Elements start immediately after the data block prefix.
-    let mut pos = db_header_size;
+    let elements_start = db_offset + db_header_size;
 
-    // Check if paged
+    let num_elements = header.num_elements as usize;
-    let page_size = 1u64 << header.max_nelmts_bits;
+    // A chunk index cannot describe more elements than the file has bytes (each
-    let is_paged = header.num_elements > page_size;
+    // element occupies at least `offset_size` bytes). Reject a corrupt count
-
+    // before it can drive a huge loop or overflow an offset computation.
-    if is_paged {
+    if num_elements > file_data.len() {
-        // For paged data blocks, we need to handle page bitmap + pages
-        // For now, implement non-paged path (covers most real-world cases)
         return Err(FormatError::ChunkedReadError(
-            "paged Fixed Array data blocks not yet supported".into(),
+            "Fixed Array element count exceeds file size".into(),
         ));
     }
-
-    // Non-paged: elements stored directly
-    let num_elements = header.num_elements as usize;
     let os = offset_size as usize;
+    // On-disk stride of one element. For non-filtered arrays the element is just
+    // the chunk address (== offset_size); for filtered arrays it is
+    // address + chunk_size + filter_mask (== header.element_size).
+    let elem_stride = (header.element_size as usize).max(os);
 
-    // Compute chunk offsets based on index
+    // Absolute file offset of element `idx` within a run starting at `base`,
-    // Chunks are stored in row-major order within the dataset space
+    // with overflow surfaced as a clean error rather than a panic/wrap.
+    let elem_at = |base: usize, idx: usize| -> Result<usize, FormatError> {
+        idx.checked_mul(elem_stride)
+            .and_then(|o| base.checked_add(o))
+            .ok_or(FormatError::ChunkedReadError(
+                "Fixed Array element offset overflow".into(),
+            ))
+    };
+
+    // Compute chunk offsets based on index.
+    // Chunks are stored in row-major order within the dataset space.
     let mut num_chunks_per_dim = Vec::with_capacity(rank);
     for d_idx in 0..rank {
         let ch_dim = chunk_dimensions[d_idx] as u64;
@@ -177,97 +186,150 @@ pub fn read_fixed_array_chunks(
         chunk_dimensions.iter().map(|&d| d as u64).product::<u64>() * element_size as u64;
 
     let mut chunks = Vec::new();
-
+    let push_element = |i: usize, abs: usize, chunks: &mut Vec<ChunkInfo>| -> Result<(), FormatError> {
-    for i in 0..num_elements {
+        if let Some((address, chunk_size, filter_mask)) = parse_fa_element(
-        let abs_pos = db_offset
+            file_data,
-            .checked_add(pos)
+            abs,
-            .ok_or(FormatError::UnexpectedEof {
+            header.client_id,
-                expected: usize::MAX,
+            offset_size,
-                available: file_data.len(),
+            header.element_size,
-            })?;
+            chunk_byte_size,
-        if abs_pos > file_data.len() {
+        )? {
-            return Err(FormatError::UnexpectedEof {
-                expected: abs_pos,
-                available: file_data.len(),
-            });
-        }
-        let elem_data = &file_data[abs_pos..];
-        if header.client_id == 0 {
-            // Non-filtered: just address
-            if db_offset
-                .checked_add(pos)
-                .and_then(|p| p.checked_add(os))
-                .is_none_or(|end| end > file_data.len())
-            {
-                return Err(FormatError::UnexpectedEof {
-                    expected: db_offset.saturating_add(pos).saturating_add(os),
-                    available: file_data.len(),
-                });
-            }
-            let address = read_offset(elem_data, 0, offset_size)?;
-            pos += os;
-
-            if is_undefined(file_data, db_offset + pos - os, offset_size) {
-                continue; // unallocated chunk
-            }
-
             let offsets = index_to_chunk_offsets(i, &num_chunks_per_dim, chunk_dimensions);
             chunks.push(ChunkInfo {
-                chunk_size: chunk_byte_size as u32,
+                chunk_size,
-                filter_mask: 0,
+                filter_mask,
                 offsets,
                 address,
             });
+        }
+        Ok(())
+    };
+
+    // A data block is paged when it holds more elements than fit in one page.
+    // `max_nelmts_bits` is an untrusted u8; a shift >= the pointer width would
+    // panic, so reject it (real page-size bits are tiny — 10 by default).
+    if header.max_nelmts_bits as u32 >= usize::BITS {
+        return Err(FormatError::ChunkedReadError(
+            "Fixed Array max_nelmts_bits too large".into(),
+        ));
+    }
+    let page_nelmts = 1usize << header.max_nelmts_bits;
+    let is_paged = num_elements > page_nelmts;
+
+    if !is_paged {
+        // Non-paged: prefix, then `num_elements` elements packed directly,
+        // then a trailing checksum (which we don't validate).
+        for i in 0..num_elements {
+            push_element(i, elem_at(elements_start, i)?, &mut chunks)?;
+        }
+        return Ok(chunks);
+    }
+
+    // Paged layout: prefix, then a page-init bitmap (one bit per page, MSB-first
+    // within each byte), then a 4-byte checksum, then the pages. Every page
+    // occupies a full slot of `page_nelmts` elements plus a 4-byte checksum;
+    // only the final page holds fewer elements. Uninitialized pages (bit clear)
+    // still occupy their slot on disk but are zero-filled, so the bitmap — not a
+    // 0xFF sentinel — is what marks a whole page as unallocated.
+    let stride_overflow = || {
+        FormatError::ChunkedReadError("Fixed Array page offset overflow".into())
+    };
+    let npages = num_elements.div_ceil(page_nelmts);
+    let bitmap_size = npages.div_ceil(8);
+    let bitmap_start = elements_start;
+    // prefix(db_header_size) + bitmap + checksum(4)
+    let pages_start = db_offset + db_header_size + bitmap_size + 4;
+    let page_stride = page_nelmts
+        .checked_mul(elem_stride)
+        .and_then(|x| x.checked_add(4))
+        .ok_or_else(stride_overflow)?;
+
+    if bitmap_start + bitmap_size > file_data.len() {
+        return Err(FormatError::UnexpectedEof {
+            expected: bitmap_start + bitmap_size,
+            available: file_data.len(),
+        });
+    }
+
+    for p in 0..npages {
+        let page_first = p * page_nelmts; // < num_elements, cannot overflow
+        let page_count = core::cmp::min(page_nelmts, num_elements - page_first);
+
+        // Check the page-init bit (MSB-first within each byte).
+        let bit_byte = file_data[bitmap_start + p / 8];
+        let bit_mask = 1u8 << (7 - (p % 8));
+        if bit_byte & bit_mask == 0 {
+            continue; // entire page unallocated
+        }
+
+        let page_off = p
+            .checked_mul(page_stride)
+            .and_then(|o| pages_start.checked_add(o))
+            .ok_or_else(stride_overflow)?;
+        for e in 0..page_count {
+            push_element(page_first + e, elem_at(page_off, e)?, &mut chunks)?;
+        }
+    }
+
+    Ok(chunks)
+}
+
+/// Parse a single Fixed Array element at absolute file offset `abs`.
+///
+/// Returns `Some((address, chunk_size, filter_mask))` for an allocated chunk, or
+/// `None` if the element is undefined (an unallocated chunk, address all-`0xFF`).
+fn parse_fa_element(
+    file_data: &[u8],
+    abs: usize,
+    client_id: u8,
+    offset_size: u8,
+    element_size: u8,
+    chunk_byte_size: u64,
+) -> Result<Option<(u64, u32, u32)>, FormatError> {
+    let os = offset_size as usize;
+    if client_id == 0 {
+        // Non-filtered: element is just the chunk address.
+        if abs + os > file_data.len() {
+            return Err(FormatError::UnexpectedEof {
+                expected: abs + os,
+                available: file_data.len(),
+            });
+        }
+        if is_undefined(file_data, abs, offset_size) {
+            return Ok(None);
+        }
+        let address = read_offset(file_data, abs, offset_size)?;
+        Ok(Some((address, chunk_byte_size as u32, 0)))
     } else {
         // Filtered: address(offset_size) + chunk_size(variable) + filter_mask(4)
-            let es = header.element_size as usize;
+        let es = element_size as usize;
         if es < os + 4 {
             return Err(FormatError::ChunkedReadError(
                 "element_size too small for filtered element".into(),
             ));
         }
         let chunk_size_bytes = es - os - 4;
-            let elem_total = os + chunk_size_bytes + 4;
+        if abs + es > file_data.len() {
-            if db_offset
-                .checked_add(pos)
-                .and_then(|p| p.checked_add(elem_total))
-                .is_none_or(|end| end > file_data.len())
-            {
             return Err(FormatError::UnexpectedEof {
-                    expected: db_offset.saturating_add(pos).saturating_add(elem_total),
+                expected: abs + es,
                 available: file_data.len(),
             });
         }
-
+        if is_undefined(file_data, abs, offset_size) {
-            let address = read_offset(elem_data, 0, offset_size)?;
+            return Ok(None);
-
+        }
-            // Read chunk_size (variable length, little-endian)
+        let address = read_offset(file_data, abs, offset_size)?;
-            let chunk_size = read_variable_length(&elem_data[os..], chunk_size_bytes)?;
+        let chunk_size = read_variable_length(&file_data[abs + os..], chunk_size_bytes)?;
-
+        let fm_off = abs + os + chunk_size_bytes;
-            let fm_off = os + chunk_size_bytes;
         let filter_mask = u32::from_le_bytes([
-                elem_data[fm_off],
+            file_data[fm_off],
-                elem_data[fm_off + 1],
+            file_data[fm_off + 1],
-                elem_data[fm_off + 2],
+            file_data[fm_off + 2],
-                elem_data[fm_off + 3],
+            file_data[fm_off + 3],
         ]);
-            pos += elem_total;
+        Ok(Some((address, chunk_size as u32, filter_mask)))
-
-            if is_undefined(file_data, db_offset + pos - elem_total, offset_size) {
-                continue; // unallocated chunk
     }
-
-            let offsets = index_to_chunk_offsets(i, &num_chunks_per_dim, chunk_dimensions);
-            chunks.push(ChunkInfo {
-                chunk_size: chunk_size as u32,
-                filter_mask,
-                offsets,
-                address,
-            });
-        }
-    }
-
-    Ok(chunks)
 }
 
 /// Convert a linear chunk index to N-dimensional chunk offsets in dataset space.
@@ -392,6 +454,41 @@ mod tests {
         assert!(result.is_err());
     }
 
+    /// Malformed headers must error, never panic (shift overflow, huge counts).
+    #[test]
+    fn read_rejects_oversized_max_nelmts_bits() {
+        let mut buf = vec![0u8; 512];
+        let fahd = 0x40usize;
+        buf[fahd..fahd + 4].copy_from_slice(b"FAHD");
+        buf[fahd + 4] = 0; // version
+        buf[fahd + 5] = 0; // client_id
+        buf[fahd + 6] = 8; // element_size
+        buf[fahd + 7] = 200; // max_nelmts_bits — absurd, would overflow a shift
+        buf[fahd + 8..fahd + 16].copy_from_slice(&3u64.to_le_bytes()); // num_elements
+        buf[fahd + 16..fahd + 24].copy_from_slice(&0x100u64.to_le_bytes());
+        // FADB so parsing reaches the paged check
+        let db = 0x100usize;
+        buf[db..db + 4].copy_from_slice(b"FADB");
+        let header = FixedArrayHeader::parse(&buf, fahd, 8, 8).unwrap();
+        let r = read_fixed_array_chunks(&buf, &header, &[100], &[20], 8, 8, 8);
+        assert!(r.is_err());
+    }
+
+    #[test]
+    fn read_rejects_num_elements_larger_than_file() {
+        let mut buf = vec![0u8; 256];
+        let fahd = 0x40usize;
+        buf[fahd..fahd + 4].copy_from_slice(b"FAHD");
+        buf[fahd + 6] = 8;
+        buf[fahd + 7] = 10;
+        buf[fahd + 8..fahd + 16].copy_from_slice(&u64::MAX.to_le_bytes()); // absurd count
+        buf[fahd + 16..fahd + 24].copy_from_slice(&0x80u64.to_le_bytes());
+        buf[0x80..0x84].copy_from_slice(b"FADB");
+        let header = FixedArrayHeader::parse(&buf, fahd, 8, 8).unwrap();
+        let r = read_fixed_array_chunks(&buf, &header, &[100], &[20], 8, 8, 8);
+        assert!(r.is_err());
+    }
+
     #[test]
     fn parse_fixed_array_header_invalid_version() {
         let mut buf = vec![0u8; 256];
@@ -535,4 +632,103 @@ mod tests {
         assert_eq!(chunks[2].address, 0x3000);
         assert_eq!(chunks[2].chunk_size, 100);
     }
+
+    /// Build a synthetic *paged* Fixed Array (non-filtered) and verify reading.
+    ///
+    /// Layout reverse-engineered and confirmed against an HDF5 2.0 file:
+    /// after the FADB prefix comes a page-init bitmap (MSB-first within each
+    /// byte), a 4-byte checksum, then full-size page slots (`page_nelmts`
+    /// elements + a 4-byte checksum each), with only the last page shorter.
+    /// Uninitialized pages occupy their slot but are skipped via the bitmap.
+    #[test]
+    fn read_paged_non_filtered_chunks() {
+        let offset_size: u8 = 8;
+        let length_size: u8 = 8;
+        let os = offset_size as usize;
+
+        // page_nelmts = 1 << 2 = 4. Use 11 elements => 3 pages
+        // (page0: 4, page1: 4, page2: 3 short). Initialize pages 0 and 2; leave
+        // page 1 uninitialized. 3 pages still fits one bitmap byte, but we place
+        // the set bits at positions 7 and 5 to lock the MSB-first ordering.
+        let max_nelmts_bits = 2u8;
+        let page_nelmts = 1usize << max_nelmts_bits; // 4
+        let num_elements = 11u64;
+        let db_header_size = 4 + 1 + 1 + os; // FADB sig+ver+client+header_addr
+        let bitmap_size = 1usize; // ceil(3/8)
+        let page_total = page_nelmts * os + 4; // elements + checksum
+
+        let fahd_offset = 0x100usize;
+        let db_offset = 0x400usize;
+        let mut file_data = vec![0u8; 0x4000];
+
+        // FAHD
+        file_data[fahd_offset..fahd_offset + 4].copy_from_slice(b"FAHD");
+        file_data[fahd_offset + 4] = 0; // version
+        file_data[fahd_offset + 5] = 0; // client_id = non-filtered
+        file_data[fahd_offset + 6] = os as u8; // element_size = address only
+        file_data[fahd_offset + 7] = max_nelmts_bits;
+        file_data[fahd_offset + 8..fahd_offset + 16].copy_from_slice(&num_elements.to_le_bytes());
+        file_data[fahd_offset + 16..fahd_offset + 24]
+            .copy_from_slice(&(db_offset as u64).to_le_bytes());
+
+        // FADB prefix
+        file_data[db_offset..db_offset + 4].copy_from_slice(b"FADB");
+        file_data[db_offset + 4] = 0; // version
+        file_data[db_offset + 5] = 0; // client_id
+        file_data[db_offset + 6..db_offset + 6 + os]
+            .copy_from_slice(&(fahd_offset as u64).to_le_bytes());
+
+        // Page-init bitmap: pages 0 and 2 initialized, page 1 not.
+        // MSB-first => page0 -> bit7 (0x80), page2 -> bit5 (0x20) => 0xA0.
+        let bitmap_off = db_offset + db_header_size;
+        file_data[bitmap_off] = 0b1010_0000;
+
+        // Pages start after bitmap + 4-byte checksum.
+        let pages_start = db_offset + db_header_size + bitmap_size + 4;
+
+        let base_addr = 0x1000u64;
+        // Page 0 (elements 0..4) and page 2 (elements 8..11) carry addresses;
+        // page 1's slot is left zero-filled and must be skipped.
+        for &p in &[0usize, 2usize] {
+            let page_off = pages_start + p * page_total;
+            let count = core::cmp::min(page_nelmts, num_elements as usize - p * page_nelmts);
+            for e in 0..count {
+                let i = p * page_nelmts + e;
+                let addr = base_addr + i as u64 * 0x100;
+                let pos = page_off + e * os;
+                file_data[pos..pos + os].copy_from_slice(&addr.to_le_bytes());
+            }
+        }
+
+        let header =
+            FixedArrayHeader::parse(&file_data, fahd_offset, offset_size, length_size).unwrap();
+        assert_eq!(header.num_elements, 11);
+
+        let ds_dims = vec![11u64 * 20];
+        let chunk_dims = vec![20u32];
+        let chunks = read_fixed_array_chunks(
+            &file_data,
+            &header,
+            &ds_dims,
+            &chunk_dims,
+            8,
+            offset_size,
+            length_size,
+        )
+        .unwrap();
+
+        // Page 1 (elements 4,5,6,7) is uninitialized => skipped. The remaining
+        // 7 chunks (0..4 and 8..11) come back with their original linear index.
+        assert_eq!(chunks.len(), 7);
+        let mut got: Vec<(u64, u64)> = chunks
+            .iter()
+            .map(|c| (c.offsets[0], c.address))
+            .collect();
+        got.sort();
+        let expect: Vec<(u64, u64)> = [0usize, 1, 2, 3, 8, 9, 10]
+            .iter()
+            .map(|&i| (i as u64 * 20, base_addr + i as u64 * 0x100))
+            .collect();
+        assert_eq!(got, expect);
+    }
 }

crates/clawhdf5-format/src/fractal_heap.rs

@@ -379,8 +379,9 @@ impl FractalHeapHeader {
         // Build table of (block_size, heap_offset) for each child entry
         let mut current_heap_offset = iblock_heap_offset;
 
-        // Count direct block entries vs indirect block entries
+        // Rows below max_direct_rows hold direct blocks; rows at/above hold
-        let start_indirect = self.starting_row_of_indirect_blocks as usize;
+        // child indirect blocks. (NOT the FRHP "starting rows" field.)
+        let start_indirect = self.max_direct_rows();
 
         // Read child addresses for direct block rows
         let max_direct_rows = nrows_usize.min(start_indirect);
@@ -455,6 +456,25 @@ impl FractalHeapHeader {
         })
     }
 
+    /// Number of rows in the doubling table whose block size is at most the
+    /// maximum *direct* block size. Rows below this hold direct blocks; rows at
+    /// or above it hold child indirect blocks.
+    ///
+    /// This is derived from the heap geometry, NOT the FRHP
+    /// "Starting # of Rows in Root Indirect Block" field (a constant, often 1)
+    /// — confusing the two makes a multi-direct-block heap unreadable.
+    fn max_direct_rows(&self) -> usize {
+        if self.starting_block_size == 0 {
+            return usize::MAX;
+        }
+        // Rows 0 and 1 share the starting block size; row r (r >= 1) is
+        // starting_block_size * 2^(r-1). The largest direct row reaches
+        // max_direct_block_size, giving log2(max/start) + 2 direct rows.
+        let ratio = (self.max_direct_block_size / self.starting_block_size).max(1);
+        let log2 = 63 - ratio.leading_zeros() as usize;
+        log2 + 2
+    }
+
     /// Get block size for a given row in the doubling table.
     fn block_size_for_row(&self, row: usize) -> u64 {
         let sbs = self.starting_block_size;

crates/clawhdf5-format/src/selection.rs

@@ -19,6 +19,8 @@ use alloc::{vec, vec::Vec};
 
 use core::ops::Range;
 
+use crate::error::FormatError;
+
 /// A selection describing which elements of a dataset to access.
 #[derive(Debug, Clone, PartialEq)]
 pub enum Selection {
@@ -220,6 +222,262 @@ impl Selection {
             }
         }
     }
+
+    /// Decode a selection from its on-disk **`H5S_select_serialize`** form.
+    ///
+    /// Returns the selection and the number of bytes consumed (selections are
+    /// self-describing in length, so the count lets a caller walk a packed list
+    /// of selections — as the Virtual Dataset global-heap block does).
+    ///
+    /// Only the forms needed for VDS assembly are decoded: `ALL`, `NONE`, and
+    /// **regular** hyperslabs serialized at **version 3** (the encoding HDF5
+    /// 1.10+/2.0 emit). Point selections, irregular hyperslabs, and older
+    /// hyperslab versions return an error rather than mis-decoding.
+    pub fn decode_serialized(data: &[u8]) -> Result<(Selection, usize), FormatError> {
+        if data.len() < 8 {
+            return Err(FormatError::UnexpectedEof {
+                expected: 8,
+                available: data.len(),
+            });
+        }
+        let sel_type = u32::from_le_bytes([data[0], data[1], data[2], data[3]]);
+        let version = u32::from_le_bytes([data[4], data[5], data[6], data[7]]);
+
+        match sel_type {
+            // ALL / NONE: type(4) + version(4) + reserved(4) + length(4) = 16 bytes.
+            3 | 0 => {
+                if data.len() < 16 {
+                    return Err(FormatError::UnexpectedEof {
+                        expected: 16,
+                        available: data.len(),
+                    });
+                }
+                let sel = if sel_type == 3 {
+                    Selection::All
+                } else {
+                    Selection::None
+                };
+                Ok((sel, 16))
+            }
+            2 => decode_hyperslab_serialized(data, version),
+            1 => Err(FormatError::ChunkedReadError(
+                "VDS point selections are not supported".into(),
+            )),
+            _ => Err(FormatError::ChunkedReadError(
+                "unknown dataspace selection type".into(),
+            )),
+        }
+    }
+
+    /// Enumerate the selected element indices of a **1-D** dataspace of the
+    /// given `extent`, in row-major selection order.
+    ///
+    /// Convenience wrapper over [`Selection::iter_linear`] for rank-1 spaces.
+    pub fn iter_linear_1d(&self, extent: u64) -> Result<Vec<u64>, FormatError> {
+        self.iter_linear(&[extent])
+    }
+
+    /// Enumerate the **row-major linear indices** of the selected elements of a
+    /// dataspace with shape `dims`, in row-major (C) iteration order.
+    ///
+    /// This is the order HDF5 uses to pair a virtual selection with a source
+    /// selection in a Virtual Dataset, so the i-th index returned here for the
+    /// virtual selection corresponds to the i-th index for the source
+    /// selection. Hyperslab/point selections whose rank differs from
+    /// `dims.len()` are rejected.
+    pub fn iter_linear(&self, dims: &[u64]) -> Result<Vec<u64>, FormatError> {
+        let overflow = || FormatError::Overflow("VDS selection index overflow".into());
+        let total: u64 = dims
+            .iter()
+            .try_fold(1u64, |acc, &d| acc.checked_mul(d))
+            .ok_or_else(overflow)?;
+        // Row-major strides: row_stride[d] = product(dims[d+1..]).
+        let rank = dims.len();
+        let mut row_stride = vec![1u64; rank];
+        for d in (0..rank.saturating_sub(1)).rev() {
+            row_stride[d] = row_stride[d + 1]
+                .checked_mul(dims[d + 1])
+                .ok_or_else(overflow)?;
+        }
+
+        match self {
+            Selection::All => Ok((0..total).collect()),
+            Selection::None => Ok(Vec::new()),
+            Selection::Hyperslab {
+                start,
+                stride,
+                count,
+                block,
+            } => {
+                if start.len() != rank {
+                    return Err(FormatError::ChunkedReadError(
+                        "VDS selection rank does not match dataspace rank".into(),
+                    ));
+                }
+                // Selected coordinates along each dimension, in order.
+                let mut per_dim: Vec<Vec<u64>> = Vec::with_capacity(rank);
+                for d in 0..rank {
+                    let mut coords = Vec::new();
+                    for ci in 0..count[d] {
+                        let base = ci
+                            .checked_mul(stride[d])
+                            .and_then(|o| start[d].checked_add(o))
+                            .ok_or_else(overflow)?;
+                        for bi in 0..block[d] {
+                            let coord = base.checked_add(bi).ok_or_else(overflow)?;
+                            // Anything past the extent is malformed; bail before the
+                            // coordinate list can grow without bound.
+                            if coord >= dims[d] {
+                                return Err(FormatError::ChunkedReadError(
+                                    "VDS hyperslab selection exceeds dataspace extent".into(),
+                                ));
+                            }
+                            coords.push(coord);
+                        }
+                    }
+                    per_dim.push(coords);
+                }
+                if per_dim.iter().any(|c| c.is_empty()) {
+                    return Ok(Vec::new());
+                }
+                // Cartesian product in row-major order (dim 0 slowest-varying).
+                let out_len: usize = per_dim
+                    .iter()
+                    .try_fold(1usize, |acc, c| acc.checked_mul(c.len()))
+                    .ok_or_else(overflow)?;
+                let mut out = Vec::with_capacity(out_len);
+                let mut idx = vec![0usize; rank];
+                loop {
+                    let mut lin = 0u64;
+                    for d in 0..rank {
+                        lin = per_dim[d][idx[d]]
+                            .checked_mul(row_stride[d])
+                            .and_then(|o| lin.checked_add(o))
+                            .ok_or_else(overflow)?;
+                    }
+                    out.push(lin);
+                    // Increment the mixed-radix counter, last dimension fastest.
+                    let mut carry = true;
+                    for d in (0..rank).rev() {
+                        idx[d] += 1;
+                        if idx[d] < per_dim[d].len() {
+                            carry = false;
+                            break;
+                        }
+                        idx[d] = 0;
+                    }
+                    if carry {
+                        break;
+                    }
+                }
+                Ok(out)
+            }
+            Selection::Points(pts) => {
+                let mut out = Vec::with_capacity(pts.len());
+                for p in pts {
+                    if p.len() != rank {
+                        return Err(FormatError::ChunkedReadError(
+                            "VDS point selection rank does not match dataspace rank".into(),
+                        ));
+                    }
+                    let mut lin = 0u64;
+                    for d in 0..rank {
+                        if p[d] >= dims[d] {
+                            return Err(FormatError::ChunkedReadError(
+                                "VDS point selection exceeds dataspace extent".into(),
+                            ));
+                        }
+                        lin = p[d]
+                            .checked_mul(row_stride[d])
+                            .and_then(|o| lin.checked_add(o))
+                            .ok_or_else(overflow)?;
+                    }
+                    out.push(lin);
+                }
+                Ok(out)
+            }
+        }
+    }
+}
+
+/// Decode an `H5S_SEL_HYPER` selection in its serialized form. Only version-3
+/// **regular** hyperslabs are supported.
+fn decode_hyperslab_serialized(
+    data: &[u8],
+    version: u32,
+) -> Result<(Selection, usize), FormatError> {
+    if version != 3 {
+        return Err(FormatError::ChunkedReadError(
+            "only version-3 hyperslab selections are supported".into(),
+        ));
+    }
+    // type(4) ver(4) flags(1) enc_size(1) rank(4) [start,stride,count,block]*rank
+    if data.len() < 14 {
+        return Err(FormatError::UnexpectedEof {
+            expected: 14,
+            available: data.len(),
+        });
+    }
+    let flags = data[8];
+    let enc_size = data[9] as usize;
+    // Bit 0 set => regular hyperslab. Irregular hyperslabs list explicit blocks.
+    if flags & 0x01 == 0 {
+        return Err(FormatError::ChunkedReadError(
+            "irregular VDS hyperslab selections are not supported".into(),
+        ));
+    }
+    if enc_size != 2 && enc_size != 4 && enc_size != 8 {
+        return Err(FormatError::ChunkedReadError(
+            "unsupported hyperslab coordinate encoding size".into(),
+        ));
+    }
+    let rank = u32::from_le_bytes([data[10], data[11], data[12], data[13]]) as usize;
+    // HDF5 caps dataspace rank at 32 (H5S_MAX_RANK). Reject anything larger so a
+    // corrupt rank can't drive a huge allocation or read loop.
+    if rank > 32 {
+        return Err(FormatError::ChunkedReadError(
+            "hyperslab selection rank exceeds maximum (32)".into(),
+        ));
+    }
+    let mut pos = 14;
+    let read_coord = |data: &[u8], pos: usize| -> Result<u64, FormatError> {
+        if pos + enc_size > data.len() {
+            return Err(FormatError::UnexpectedEof {
+                expected: pos + enc_size,
+                available: data.len(),
+            });
+        }
+        let mut v = 0u64;
+        for (i, &b) in data[pos..pos + enc_size].iter().enumerate() {
+            v |= (b as u64) << (i * 8);
+        }
+        Ok(v)
+    };
+    let (mut start, mut stride, mut count, mut block) = (
+        Vec::with_capacity(rank),
+        Vec::with_capacity(rank),
+        Vec::with_capacity(rank),
+        Vec::with_capacity(rank),
+    );
+    for _ in 0..rank {
+        start.push(read_coord(data, pos)?);
+        pos += enc_size;
+        stride.push(read_coord(data, pos)?);
+        pos += enc_size;
+        count.push(read_coord(data, pos)?);
+        pos += enc_size;
+        block.push(read_coord(data, pos)?);
+        pos += enc_size;
+    }
+    Ok((
+        Selection::Hyperslab {
+            start,
+            stride,
+            count,
+            block,
+        },
+        pos,
+    ))
 }
 
 // ---------------------------------------------------------------------------
@@ -313,4 +571,173 @@ mod tests {
         // Chunk [9..10] should not intersect (only row 9, but selection ends at row 8)
         assert!(!sel.intersects_chunk(&[9], &[1]));
     }
+
+    #[test]
+    fn decode_all_selection_16_bytes() {
+        let bytes = [3u8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0];
+        let (sel, consumed) = Selection::decode_serialized(&bytes).unwrap();
+        assert_eq!(sel, Selection::All);
+        assert_eq!(consumed, 16);
+        assert_eq!(sel.iter_linear_1d(4).unwrap(), vec![0, 1, 2, 3]);
+    }
+
+    #[test]
+    fn decode_regular_hyperslab_matches_vds_fixture() {
+        // Exact virtual selection for src_a in the VDS fixture:
+        // start=0 stride=1 count=1 block=4, version 3, enc_size 2, rank 1.
+        let bytes = [
+            0x02, 0, 0, 0, // type = HYPER
+            0x03, 0, 0, 0, // version 3
+            0x01, // flags = regular
+            0x02, // enc_size = 2
+            0x01, 0, 0, 0, // rank = 1
+            0x00, 0x00, // start
+            0x01, 0x00, // stride
+            0x01, 0x00, // count
+            0x04, 0x00, // block
+        ];
+        let (sel, consumed) = Selection::decode_serialized(&bytes).unwrap();
+        assert_eq!(consumed, 22);
+        assert_eq!(
+            sel,
+            Selection::Hyperslab {
+                start: vec![0],
+                stride: vec![1],
+                count: vec![1],
+                block: vec![4],
+            }
+        );
+        assert_eq!(sel.iter_linear_1d(8).unwrap(), vec![0, 1, 2, 3]);
+    }
+
+    #[test]
+    fn decode_hyperslab_start4() {
+        let bytes = [
+            0x02, 0, 0, 0, 0x03, 0, 0, 0, 0x01, 0x02, 0x01, 0, 0, 0, //
+            0x04, 0x00, 0x01, 0x00, 0x01, 0x00, 0x04, 0x00,
+        ];
+        let (sel, _) = Selection::decode_serialized(&bytes).unwrap();
+        assert_eq!(sel.iter_linear_1d(8).unwrap(), vec![4, 5, 6, 7]);
+    }
+
+    #[test]
+    fn decode_strided_hyperslab_iter() {
+        // start=1 stride=3 count=2 block=2 => 1,2, 4,5
+        let bytes = [
+            0x02, 0, 0, 0, 0x03, 0, 0, 0, 0x01, 0x02, 0x01, 0, 0, 0, //
+            0x01, 0x00, 0x03, 0x00, 0x02, 0x00, 0x02, 0x00,
+        ];
+        let (sel, _) = Selection::decode_serialized(&bytes).unwrap();
+        assert_eq!(sel.iter_linear_1d(8).unwrap(), vec![1, 2, 4, 5]);
+    }
+
+    #[test]
+    fn decode_nd_hyperslab_iter_rejected() {
+        let bytes = [
+            0x02, 0, 0, 0, 0x03, 0, 0, 0, 0x01, 0x02, 0x02, 0, 0, 0, // rank 2
+            0, 0, 1, 0, 1, 0, 2, 0, 0, 0, 1, 0, 1, 0, 2, 0,
+        ];
+        let (sel, _) = Selection::decode_serialized(&bytes).unwrap();
+        assert!(sel.iter_linear_1d(16).is_err());
+    }
+
+    #[test]
+    fn decode_irregular_hyperslab_rejected() {
+        let bytes = [0x02u8, 0, 0, 0, 0x03, 0, 0, 0, 0x00, 0x02, 0x01, 0, 0, 0];
+        assert!(Selection::decode_serialized(&bytes).is_err());
+    }
+
+    #[test]
+    fn iter_linear_2d_block_row_major() {
+        // A 2x2 block at the top-left of a 4x4 space => linear 0,1,4,5.
+        let sel = Selection::Hyperslab {
+            start: vec![0, 0],
+            stride: vec![1, 1],
+            count: vec![1, 1],
+            block: vec![2, 2],
+        };
+        assert_eq!(sel.iter_linear(&[4, 4]).unwrap(), vec![0, 1, 4, 5]);
+
+        // The same block shifted to the bottom-right => 10,11,14,15.
+        let sel2 = Selection::Hyperslab {
+            start: vec![2, 2],
+            stride: vec![1, 1],
+            count: vec![1, 1],
+            block: vec![2, 2],
+        };
+        assert_eq!(sel2.iter_linear(&[4, 4]).unwrap(), vec![10, 11, 14, 15]);
+    }
+
+    #[test]
+    fn iter_linear_2d_strided() {
+        // start=(0,0) stride=(2,2) count=(2,2) block=(1,1) over 4x4 =>
+        // coords (0,0)(0,2)(2,0)(2,2) => linear 0,2,8,10.
+        let sel = Selection::Hyperslab {
+            start: vec![0, 0],
+            stride: vec![2, 2],
+            count: vec![2, 2],
+            block: vec![1, 1],
+        };
+        assert_eq!(sel.iter_linear(&[4, 4]).unwrap(), vec![0, 2, 8, 10]);
+    }
+
+    #[test]
+    fn iter_linear_all_2d() {
+        assert_eq!(
+            Selection::All.iter_linear(&[2, 3]).unwrap(),
+            (0..6).collect::<Vec<_>>()
+        );
+    }
+
+    #[test]
+    fn iter_linear_rank_mismatch_rejected() {
+        let sel = Selection::Hyperslab {
+            start: vec![0],
+            stride: vec![1],
+            count: vec![1],
+            block: vec![2],
+        };
+        assert!(sel.iter_linear(&[4, 4]).is_err());
+    }
+
+    // ----- Adversarial / hardening: malformed input must error, never panic -----
+
+    #[test]
+    fn decode_all_truncated_does_not_overrun() {
+        // ALL claims to consume 16 bytes but only 8 are present.
+        let bytes = [3u8, 0, 0, 0, 1, 0, 0, 0];
+        assert!(Selection::decode_serialized(&bytes).is_err());
+    }
+
+    #[test]
+    fn decode_hyperslab_huge_rank_rejected() {
+        // rank = 0xFFFFFFFF must not drive a giant allocation.
+        let bytes = [
+            0x02u8, 0, 0, 0, 0x03, 0, 0, 0, 0x01, 0x02, 0xFF, 0xFF, 0xFF, 0xFF,
+        ];
+        assert!(Selection::decode_serialized(&bytes).is_err());
+    }
+
+    #[test]
+    fn iter_linear_hyperslab_overflow_is_error() {
+        // start/stride/count near u64::MAX must not panic on multiply/add.
+        let sel = Selection::Hyperslab {
+            start: vec![u64::MAX - 1],
+            stride: vec![u64::MAX],
+            count: vec![u64::MAX],
+            block: vec![u64::MAX],
+        };
+        assert!(sel.iter_linear(&[100]).is_err());
+    }
+
+    #[test]
+    fn iter_linear_dims_product_overflow_is_error() {
+        assert!(Selection::All.iter_linear(&[u64::MAX, u64::MAX]).is_err());
+    }
+
+    #[test]
+    fn decode_empty_or_short_is_error_not_panic() {
+        assert!(Selection::decode_serialized(&[]).is_err());
+        assert!(Selection::decode_serialized(&[2, 0, 0, 0, 3, 0]).is_err());
+    }
 }

crates/clawhdf5-format/tests/integration_test.rs

@@ -662,6 +662,156 @@ fn v4_fixed_array_read() {
     }
 }
 
+#[test]
+fn v4_virtual_dataset_same_file_read() {
+    // A 1-D virtual dataset assembled from two same-file sources:
+    //   virt[0:4]  <- src_a[1:5]  (partial source hyperslab) => 11,12,13,14
+    //   virt[4:8]  <- (unmapped)                              => fill 0
+    //   virt[8:12] <- src_b[0:4]  (ALL)                       => 20,21,22,23
+    let file_data = include_bytes!("fixtures/vds_same_file.h5");
+    let (raw, datatype, _) = read_chunked_dataset(file_data, "virt");
+    let values = read_as_i32(&raw, &datatype).unwrap();
+    assert_eq!(
+        values,
+        vec![11, 12, 13, 14, 0, 0, 0, 0, 20, 21, 22, 23],
+        "VDS assembly (partial source slice + fill gap) mismatch"
+    );
+}
+
+#[test]
+fn v4_virtual_dataset_2d_same_file_read() {
+    // A 4x4 virtual dataset assembled from two 2x2 same-file sources placed as
+    // non-contiguous blocks (exercises N-dimensional row-major scatter):
+    //   virt[0:2,0:2] <- src_a = [[1,2],[3,4]]
+    //   virt[2:4,2:4] <- src_b = [[5,6],[7,8]]
+    //   everything else -> fill 0
+    let file_data = include_bytes!("fixtures/vds_2d_same_file.h5");
+    let (raw, datatype, _) = read_chunked_dataset(file_data, "virt");
+    let values = read_as_i32(&raw, &datatype).unwrap();
+    assert_eq!(
+        values,
+        vec![1, 2, 0, 0, 3, 4, 0, 0, 0, 0, 5, 6, 0, 0, 7, 8],
+        "2-D VDS block scatter mismatch"
+    );
+}
+
+#[test]
+fn scaleoffset_float_escale_reads_as_raw() {
+    // The scale-offset filter's floating-point *E-scale* mode (cd_values[0] = 1)
+    // is not actually implemented by the HDF5 library: when asked for it, HDF5
+    // stores the chunk raw (no minbits/minval header) and sets the chunk filter
+    // mask to skip the filter. So such a dataset must read back verbatim purely
+    // by honoring the per-chunk filter mask — no E-scale decoder is needed.
+    let file_data = include_bytes!("fixtures/scaleoffset_float_escale.h5");
+    let (raw, datatype, _) = read_chunked_dataset(file_data, "x");
+    let values = read_as_f64(&raw, &datatype).unwrap();
+    let expect: Vec<f64> = (0..20).map(|i| i as f64 * 0.25).collect();
+    assert_eq!(values, expect, "E-scale (raw + masked filter) must read verbatim");
+}
+
+#[test]
+fn v4_virtual_dataset_cycle_errors_not_overflow() {
+    // virt -> virt2 -> virt (both virtual, same file). The reader must reject
+    // the nested virtual source rather than recurse into a stack overflow.
+    let file_data = include_bytes!("fixtures/vds_cyclic.h5");
+    let offset = find_signature(file_data).unwrap();
+    let sb = Superblock::parse(file_data, offset).unwrap();
+    let addr = resolve_path_any(file_data, &sb, "virt").unwrap();
+    let hdr = ObjectHeader::parse(file_data, addr as usize, sb.offset_size, sb.length_size).unwrap();
+    let ds = Dataspace::parse(
+        &hdr.messages.iter().find(|m| m.msg_type == MessageType::Dataspace).unwrap().data,
+        sb.length_size,
+    )
+    .unwrap();
+    let (dt, _) = Datatype::parse(
+        &hdr.messages.iter().find(|m| m.msg_type == MessageType::Datatype).unwrap().data,
+    )
+    .unwrap();
+    let layout = DataLayout::parse(
+        &hdr.messages.iter().find(|m| m.msg_type == MessageType::DataLayout).unwrap().data,
+        sb.offset_size,
+        sb.length_size,
+    )
+    .unwrap();
+    let r = read_raw_data_full(
+        file_data, &layout, &ds, &dt, None, sb.offset_size, sb.length_size,
+    );
+    assert!(r.is_err(), "cyclic virtual dataset must error, not overflow");
+}
+
+#[test]
+fn v4_virtual_dataset_external_file_read() {
+    use clawhdf5_format::data_read::read_raw_data_full_with_resolver;
+    // The virtual file maps virt[0:8] <- (external) ext_src.h5:/data = [10..17].
+    let virt = include_bytes!("fixtures/vds_external_virt.h5");
+    let src = include_bytes!("fixtures/vds_external_src.h5").to_vec();
+
+    let sig = find_signature(virt).unwrap();
+    let sb = Superblock::parse(virt, sig).unwrap();
+    let addr = resolve_path_any(virt, &sb, "virt").unwrap();
+    let hdr = ObjectHeader::parse(virt, addr as usize, sb.offset_size, sb.length_size).unwrap();
+    let ds = Dataspace::parse(
+        &hdr.messages.iter().find(|m| m.msg_type == MessageType::Dataspace).unwrap().data,
+        sb.length_size,
+    )
+    .unwrap();
+    let (dt, _) = Datatype::parse(
+        &hdr.messages.iter().find(|m| m.msg_type == MessageType::Datatype).unwrap().data,
+    )
+    .unwrap();
+    let layout = DataLayout::parse(
+        &hdr.messages.iter().find(|m| m.msg_type == MessageType::DataLayout).unwrap().data,
+        sb.offset_size,
+        sb.length_size,
+    )
+    .unwrap();
+
+    // Resolver supplies the external source file's bytes by its stored name.
+    let resolver = |name: &str| -> Option<Vec<u8>> {
+        if name == "ext_src.h5" {
+            Some(src.clone())
+        } else {
+            None
+        }
+    };
+    let raw = read_raw_data_full_with_resolver(
+        virt,
+        &layout,
+        &ds,
+        &dt,
+        None,
+        sb.offset_size,
+        sb.length_size,
+        Some(&resolver),
+    )
+    .unwrap();
+    let values = read_as_i32(&raw, &dt).unwrap();
+    assert_eq!(values, vec![10, 11, 12, 13, 14, 15, 16, 17]);
+
+    // With no resolver, an external source is a clean error (not wrong data).
+    let no_resolver = read_raw_data_full_with_resolver(
+        virt, &layout, &ds, &dt, None, sb.offset_size, sb.length_size, None,
+    );
+    assert!(no_resolver.is_err());
+}
+
+#[test]
+fn v4_paged_fixed_array_read() {
+    // 1025 chunks of 16 int32s, gzip-filtered => Fixed Array index whose data
+    // block is *paged* (page holds 1024 elements). Page 0 is full, page 1 holds
+    // the single trailing chunk. Chunk k stores value k at its first element.
+    let file_data = include_bytes!("fixtures/v4_fixed_array_paged.h5");
+    let (raw, datatype, _) = read_chunked_dataset(file_data, "big");
+    let values = read_as_i32(&raw, &datatype).unwrap();
+    assert_eq!(values.len(), 1025 * 16);
+    for k in 0..1025usize {
+        assert_eq!(values[k * 16], k as i32, "chunk-start mismatch at chunk {k}");
+        for j in 1..16 {
+            assert_eq!(values[k * 16 + j], 0, "non-start element nonzero at {}", k * 16 + j);
+        }
+    }
+}
+
 #[test]
 fn v4_2d_fixed_array_read() {
     let file_data = include_bytes!("fixtures/v4_2d.h5");

crates/clawhdf5-gpu/README.md

@@ -1,9 +1,9 @@
-# rustyhdf5-gpu
+# clawhdf5-gpu
 
-[![crates.io](https://img.shields.io/crates/v/rustyhdf5-gpu.svg)](https://crates.io/crates/rustyhdf5-gpu)
+[![crates.io](https://img.shields.io/crates/v/clawhdf5-gpu.svg)](https://crates.io/crates/clawhdf5-gpu)
-[![docs.rs](https://docs.rs/rustyhdf5-gpu/badge.svg)](https://docs.rs/rustyhdf5-gpu)
+[![docs.rs](https://docs.rs/clawhdf5-gpu/badge.svg)](https://docs.rs/clawhdf5-gpu)
 
-GPU-accelerated vector operations for rustyhdf5 using wgpu compute shaders.
+GPU-accelerated vector operations for clawhdf5 using wgpu compute shaders.
 
 ## Features
 
@@ -14,7 +14,7 @@ GPU-accelerated vector operations for rustyhdf5 using wgpu compute shaders.
 ## Usage
 
 ```rust
-use rustyhdf5_gpu::GpuAccelerator;
+use clawhdf5_gpu::GpuAccelerator;
 
 let accel = GpuAccelerator::new().unwrap();
 let distances = accel.l2_distances(&query, &vectors).unwrap();

crates/clawhdf5-io/README.md

@@ -1,9 +1,9 @@
-# rustyhdf5-io
+# clawhdf5-io
 
-[![crates.io](https://img.shields.io/crates/v/rustyhdf5-io.svg)](https://crates.io/crates/rustyhdf5-io)
+[![crates.io](https://img.shields.io/crates/v/clawhdf5-io.svg)](https://crates.io/crates/clawhdf5-io)
-[![docs.rs](https://docs.rs/rustyhdf5-io/badge.svg)](https://docs.rs/rustyhdf5-io)
+[![docs.rs](https://docs.rs/clawhdf5-io/badge.svg)](https://docs.rs/clawhdf5-io)
 
-I/O abstraction layer for rustyhdf5.
+I/O abstraction layer for clawhdf5.
 
 ## Features
 
@@ -15,7 +15,7 @@ I/O abstraction layer for rustyhdf5.
 ## Usage
 
 ```rust
-use rustyhdf5_io::MmapReader;
+use clawhdf5_io::MmapReader;
 
 let reader = MmapReader::open("data.h5").unwrap();
 ```

crates/clawhdf5-migrate/README.md

@@ -1,22 +1,22 @@
-# edgehdf5-migrate
+# clawhdf5-migrate
 
-[![crates.io](https://img.shields.io/crates/v/edgehdf5-migrate.svg)](https://crates.io/crates/edgehdf5-migrate)
+[![crates.io](https://img.shields.io/crates/v/clawhdf5-migrate.svg)](https://crates.io/crates/clawhdf5-migrate)
-[![docs.rs](https://img.shields.io/docsrs/edgehdf5-migrate)](https://docs.rs/edgehdf5-migrate)
+[![docs.rs](https://img.shields.io/docsrs/clawhdf5-migrate)](https://docs.rs/clawhdf5-migrate)
 
 CLI tool to migrate SQLite agent memory databases to HDF5 format.
 
-Converts existing SQLite-based agent memory stores (embeddings, text chunks, metadata) into the HDF5 format used by [edgehdf5-memory](https://crates.io/crates/edgehdf5-memory).
+Converts existing SQLite-based agent memory stores (embeddings, text chunks, metadata) into the HDF5 format used by [clawhdf5-agent](https://crates.io/crates/clawhdf5-agent).
 
 ## Installation
 
 ```bash
-cargo install edgehdf5-migrate
+cargo install clawhdf5-migrate
 ```
 
 ## Usage
 
 ```bash
-edgehdf5-migrate --input agent.db --output agent.h5
+clawhdf5-migrate --input agent.db --output agent.h5
 ```
 
 ## License

crates/clawhdf5-migrate/src/hdf5_reader.rs

@@ -0,0 +1,159 @@
+//! Read a migration HDF5 file back into the in-memory data model.
+//!
+//! Used to verify migrated content (real validation) and to merge new rows into
+//! an existing output (incremental migration). Mirrors the layout produced by
+//! [`crate::hdf5_writer`].
+
+use clawhdf5::reader::{File, Group};
+use clawhdf5_format::type_builders::AttrValue;
+
+use crate::sqlite_reader::{Entity, MemoryChunk, Relation, Session, SqliteData};
+
+type BoxErr = Box<dyn std::error::Error>;
+
+fn read_strings(group: &Group<'_>, name: &str) -> Result<Vec<String>, BoxErr> {
+    Ok(group.dataset(name)?.read_string()?)
+}
+
+fn read_i64s(group: &Group<'_>, name: &str) -> Result<Vec<i64>, BoxErr> {
+    Ok(group.dataset(name)?.read_i64()?)
+}
+
+fn read_f64s(group: &Group<'_>, name: &str) -> Result<Vec<f64>, BoxErr> {
+    Ok(group.dataset(name)?.read_f64()?)
+}
+
+/// Read the embeddings dataset as a flat `Vec<f32>` of `n * dim` values,
+/// handling both f32 and (lossy) f16 storage.
+fn read_embeddings_flat(group: &Group<'_>) -> Result<Vec<f32>, BoxErr> {
+    Ok(group.dataset("embeddings")?.read_f32()?)
+}
+
+/// Read a migration HDF5 file into a [`SqliteData`].
+pub fn read_hdf5(path: &str) -> Result<SqliteData, BoxErr> {
+    let file = File::open(path)?;
+
+    let embedding_dim = match file.root().attrs()?.get("embedding_dim") {
+        Some(AttrValue::I64(d)) => *d as usize,
+        _ => 0,
+    };
+
+    let chunks = read_chunks(&file, embedding_dim)?;
+    let sessions = read_sessions(&file)?;
+    let entities = read_entities(&file)?;
+    let relations = read_relations(&file)?;
+
+    Ok(SqliteData {
+        chunks,
+        sessions,
+        entities,
+        relations,
+        embedding_dim,
+    })
+}
+
+fn read_chunks(file: &File, dim: usize) -> Result<Vec<MemoryChunk>, BoxErr> {
+    let g = file.group("chunks")?;
+    let count = group_count(&g)?;
+    if count == 0 {
+        return Ok(Vec::new());
+    }
+    let ids = read_i64s(&g, "id")?;
+    let texts = read_strings(&g, "text")?;
+    let channels = read_strings(&g, "source_channel")?;
+    let timestamps = read_f64s(&g, "timestamp")?;
+    let session_ids = read_strings(&g, "session_id")?;
+    let tags = read_strings(&g, "tags")?;
+    let deleted = g.dataset("deleted")?.read_i32()?;
+    let emb_flat = read_embeddings_flat(&g)?;
+    let dim = dim.max(1);
+
+    let mut chunks = Vec::with_capacity(ids.len());
+    for (i, &id) in ids.iter().enumerate() {
+        let embedding = emb_flat
+            .get(i * dim..(i + 1) * dim)
+            .map(|s| s.to_vec())
+            .unwrap_or_default();
+        chunks.push(MemoryChunk {
+            id,
+            chunk: texts.get(i).cloned().unwrap_or_default(),
+            embedding,
+            source_channel: channels.get(i).cloned().unwrap_or_default(),
+            timestamp: timestamps.get(i).copied().unwrap_or(0.0),
+            session_id: session_ids.get(i).cloned().unwrap_or_default(),
+            tags: tags.get(i).cloned().unwrap_or_default(),
+            deleted: deleted.get(i).copied().unwrap_or(0),
+        });
+    }
+    Ok(chunks)
+}
+
+fn read_sessions(file: &File) -> Result<Vec<Session>, BoxErr> {
+    let g = file.group("sessions")?;
+    if group_count(&g)? == 0 {
+        return Ok(Vec::new());
+    }
+    let ids = read_strings(&g, "id")?;
+    let starts = read_i64s(&g, "start_idx")?;
+    let ends = read_i64s(&g, "end_idx")?;
+    let channels = read_strings(&g, "channel")?;
+    let timestamps = read_f64s(&g, "timestamp")?;
+    let summaries = read_strings(&g, "summary")?;
+    Ok((0..ids.len())
+        .map(|i| Session {
+            id: ids[i].clone(),
+            start_idx: starts.get(i).copied().unwrap_or(0),
+            end_idx: ends.get(i).copied().unwrap_or(0),
+            channel: channels.get(i).cloned().unwrap_or_default(),
+            timestamp: timestamps.get(i).copied().unwrap_or(0.0),
+            summary: summaries.get(i).cloned().unwrap_or_default(),
+        })
+        .collect())
+}
+
+fn read_entities(file: &File) -> Result<Vec<Entity>, BoxErr> {
+    let g = file.group("entities")?;
+    if group_count(&g)? == 0 {
+        return Ok(Vec::new());
+    }
+    let ids = read_i64s(&g, "id")?;
+    let names = read_strings(&g, "name")?;
+    let types = read_strings(&g, "type")?;
+    let emb_idxs = read_i64s(&g, "embedding_idx")?;
+    Ok((0..ids.len())
+        .map(|i| Entity {
+            id: ids[i],
+            name: names.get(i).cloned().unwrap_or_default(),
+            entity_type: types.get(i).cloned().unwrap_or_default(),
+            embedding_idx: emb_idxs.get(i).copied().unwrap_or(-1),
+        })
+        .collect())
+}
+
+fn read_relations(file: &File) -> Result<Vec<Relation>, BoxErr> {
+    let g = file.group("relations")?;
+    if group_count(&g)? == 0 {
+        return Ok(Vec::new());
+    }
+    let srcs = read_i64s(&g, "src")?;
+    let tgts = read_i64s(&g, "tgt")?;
+    let rels = read_strings(&g, "relation")?;
+    let weights = read_f64s(&g, "weight")?;
+    let timestamps = read_f64s(&g, "timestamp")?;
+    Ok((0..srcs.len())
+        .map(|i| Relation {
+            src: srcs[i],
+            tgt: tgts.get(i).copied().unwrap_or(0),
+            relation: rels.get(i).cloned().unwrap_or_default(),
+            weight: weights.get(i).copied().unwrap_or(1.0),
+            timestamp: timestamps.get(i).copied().unwrap_or(0.0),
+        })
+        .collect())
+}
+
+fn group_count(group: &Group<'_>) -> Result<u64, BoxErr> {
+    match group.attrs()?.get("count") {
+        Some(AttrValue::I64(n)) => Ok(*n as u64),
+        _ => Ok(0),
+    }
+}

crates/clawhdf5-migrate/src/main.rs

@@ -1,9 +1,12 @@
+mod hdf5_reader;
 mod hdf5_writer;
 mod sqlite_reader;
 mod validate;
 
 use clap::Parser;
 
+use sqlite_reader::SchemaConfig;
+
 /// Migrate ZeroClaw agent memory from SQLite to HDF5 format.
 #[derive(Parser, Debug)]
 #[command(name = "clawhdf5-migrate", version, about)]
@@ -48,45 +51,126 @@ struct Cli {
     #[arg(long)]
     dry_run: bool,
 
+    /// Content-check every migrated row (default: a representative sample)
+    #[arg(long)]
+    validate_full: bool,
+
+    /// Append only rows newer than the existing output (by chunk id), merging
+    /// into the file at --hdf5 if it exists
+    #[arg(long)]
+    incremental: bool,
+
+    /// Override the SQLite table name for memory chunks
+    #[arg(long)]
+    chunks_table: Option<String>,
+
+    /// Override the SQLite table name for sessions
+    #[arg(long)]
+    sessions_table: Option<String>,
+
+    /// Override the SQLite table name for entities
+    #[arg(long)]
+    entities_table: Option<String>,
+
+    /// Override the SQLite table name for relations
+    #[arg(long)]
+    relations_table: Option<String>,
+
     /// Print progress
     #[arg(long)]
     verbose: bool,
 }
 
+/// Build the schema config from CLI table-name overrides (defaults otherwise).
+fn schema_from_cli(cli: &Cli) -> SchemaConfig {
+    let mut c = SchemaConfig::default();
+    if let Some(t) = &cli.chunks_table {
+        c.chunks.table = t.clone();
+    }
+    if let Some(t) = &cli.sessions_table {
+        c.sessions.table = t.clone();
+    }
+    if let Some(t) = &cli.entities_table {
+        c.entities.table = t.clone();
+    }
+    if let Some(t) = &cli.relations_table {
+        c.relations.table = t.clone();
+    }
+    c
+}
+
 fn main() -> Result<(), Box<dyn std::error::Error>> {
     let cli = Cli::parse();
+    let schema = schema_from_cli(&cli);
+
+    // Dry run: a fast count-only pass that does not buffer the database.
+    if cli.dry_run {
+        let counts = sqlite_reader::read_counts(&cli.sqlite, cli.skip_deleted, &schema)?;
+        eprintln!("Dry run — no output file written.");
+        eprintln!(
+            "Would migrate: {} chunks, {} sessions, {} entities, {} relations",
+            counts.chunks, counts.sessions, counts.entities, counts.relations
+        );
+        return Ok(());
+    }
 
     if cli.verbose {
         eprintln!("Reading SQLite database: {}", cli.sqlite);
     }
 
-    let data = sqlite_reader::read_sqlite(&cli.sqlite, cli.skip_deleted, cli.embedding_dim)?;
+    // Incremental: merge new rows into the existing output (if present).
+    let incremental_base = if cli.incremental && std::path::Path::new(&cli.hdf5).exists() {
+        Some(hdf5_reader::read_hdf5(&cli.hdf5)?)
+    } else {
+        None
+    };
+    let min_chunk_id = incremental_base
+        .as_ref()
+        .map(|d| d.chunks.iter().map(|c| c.id).max().unwrap_or(0))
+        .unwrap_or(0);
+    let dim_hint = cli
+        .embedding_dim
+        .or_else(|| incremental_base.as_ref().map(|d| d.embedding_dim));
+
+    let source = if min_chunk_id > 0 {
+        sqlite_reader::read_sqlite_filtered(
+            &cli.sqlite,
+            cli.skip_deleted,
+            dim_hint,
+            &schema,
+            min_chunk_id,
+        )?
+    } else {
+        sqlite_reader::read_sqlite(&cli.sqlite, cli.skip_deleted, dim_hint, &schema)?
+    };
+
+    // Build the dataset to write: either the source alone, or the existing
+    // output plus the newly-read rows (metadata groups refreshed from source).
+    let data = match incremental_base {
+        Some(mut base) => {
+            let added = source.chunks.len();
+            base.chunks.extend(source.chunks);
+            base.sessions = source.sessions;
+            base.entities = source.entities;
+            base.relations = source.relations;
+            base.embedding_dim = source.embedding_dim.max(base.embedding_dim);
+            if cli.verbose {
+                eprintln!("Incremental: appended {added} new chunks (id > {min_chunk_id})");
+            }
+            base
+        }
+        None => source,
+    };
 
     if cli.verbose {
         eprintln!(
-            "Read {} chunks, {} sessions, {} entities, {} relations",
+            "Migrating {} chunks, {} sessions, {} entities, {} relations (dim={})",
-            data.chunks.len(),
-            data.sessions.len(),
-            data.entities.len(),
-            data.relations.len()
-        );
-        eprintln!("Embedding dimension: {}", data.embedding_dim);
-    }
-
-    if cli.dry_run {
-        eprintln!("Dry run — no output file written.");
-        eprintln!(
-            "Would migrate: {} chunks, {} sessions, {} entities, {} relations (dim={})",
             data.chunks.len(),
             data.sessions.len(),
             data.entities.len(),
             data.relations.len(),
             data.embedding_dim
         );
-        return Ok(());
-    }
-
-    if cli.verbose {
         eprintln!("Writing HDF5 file: {}", cli.hdf5);
     }
 
@@ -101,25 +185,19 @@ fn main() -> Result<(), Box<dyn std::error::Error>> {
     hdf5_writer::write_hdf5(&cli.hdf5, &data, &opts)?;
 
     if cli.verbose {
-        eprintln!("Validating output...");
+        eprintln!("Validating output (content check)...");
     }
 
-    let summary = validate::validate_hdf5(
+    let summary = validate::validate_hdf5(&cli.hdf5, &data, cli.validate_full, cli.float16)?;
-        &cli.hdf5,
-        data.chunks.len(),
-        data.sessions.len(),
-        data.entities.len(),
-        data.relations.len(),
-        data.embedding_dim,
-    )?;
 
     eprintln!(
-        "Migration complete: {} chunks, {} sessions, {} entities, {} relations (dim={})",
+        "Migration complete: {} chunks, {} sessions, {} entities, {} relations (dim={}); {} rows content-verified",
         summary.chunks,
         summary.sessions,
         summary.entities,
         summary.relations,
-        summary.embedding_dim
+        summary.embedding_dim,
+        summary.rows_checked,
     );
 
     Ok(())
@@ -231,7 +309,7 @@ mod tests {
         insert_relation(&conn, 1, 1, "self");
         drop(conn);
 
-        let data = sqlite_reader::read_sqlite(&db_path, false, None).unwrap();
+        let data = sqlite_reader::read_sqlite(&db_path, false, None, &SchemaConfig::default()).unwrap();
         let opts = hdf5_writer::WriteOptions {
             agent_id: "test-agent".into(),
             embedder: "test-embed".into(),
@@ -241,7 +319,7 @@ mod tests {
         };
         hdf5_writer::write_hdf5(h5_path.to_str().unwrap(), &data, &opts).unwrap();
 
-        let summary = validate::validate_hdf5(h5_path.to_str().unwrap(), 2, 1, 1, 1, 8).unwrap();
+        let summary = validate::validate_hdf5(h5_path.to_str().unwrap(), &data, false, false).unwrap();
         assert_eq!(summary.chunks, 2);
         assert_eq!(summary.sessions, 1);
         assert_eq!(summary.entities, 1);
@@ -262,7 +340,7 @@ mod tests {
         insert_chunk(&conn, 3, "also active", &make_embedding(4, 3.0), 0);
         drop(conn);
 
-        let data = sqlite_reader::read_sqlite(&db_path, true, None).unwrap();
+        let data = sqlite_reader::read_sqlite(&db_path, true, None, &SchemaConfig::default()).unwrap();
         assert_eq!(data.chunks.len(), 2);
 
         let opts = hdf5_writer::WriteOptions {
@@ -274,7 +352,7 @@ mod tests {
         };
         hdf5_writer::write_hdf5(h5_path.to_str().unwrap(), &data, &opts).unwrap();
 
-        let summary = validate::validate_hdf5(h5_path.to_str().unwrap(), 2, 0, 0, 0, 4).unwrap();
+        let summary = validate::validate_hdf5(h5_path.to_str().unwrap(), &data, false, false).unwrap();
         assert_eq!(summary.chunks, 2);
     }
 
@@ -289,7 +367,7 @@ mod tests {
         insert_chunk(&conn, 2, "deleted", &make_embedding(4, 2.0), 1);
         drop(conn);
 
-        let data = sqlite_reader::read_sqlite(&db_path, false, None).unwrap();
+        let data = sqlite_reader::read_sqlite(&db_path, false, None, &SchemaConfig::default()).unwrap();
         assert_eq!(data.chunks.len(), 2);
     }
 
@@ -303,7 +381,7 @@ mod tests {
         insert_chunk(&conn, 1, "test", &make_embedding(16, 0.5), 0);
         drop(conn);
 
-        let data = sqlite_reader::read_sqlite(&db_path, false, None).unwrap();
+        let data = sqlite_reader::read_sqlite(&db_path, false, None, &SchemaConfig::default()).unwrap();
         assert_eq!(data.embedding_dim, 16);
     }
 
@@ -317,7 +395,7 @@ mod tests {
         insert_chunk(&conn, 1, "test", &make_embedding(16, 0.5), 0);
         drop(conn);
 
-        let data = sqlite_reader::read_sqlite(&db_path, false, Some(8)).unwrap();
+        let data = sqlite_reader::read_sqlite(&db_path, false, Some(8), &SchemaConfig::default()).unwrap();
         assert_eq!(data.embedding_dim, 8);
         // Embedding truncated to dim 8
         assert_eq!(data.chunks[0].embedding.len(), 8);
@@ -335,7 +413,7 @@ mod tests {
         insert_chunk(&conn, 1, "test", &emb, 0);
         drop(conn);
 
-        let data = sqlite_reader::read_sqlite(&db_path, false, None).unwrap();
+        let data = sqlite_reader::read_sqlite(&db_path, false, None, &SchemaConfig::default()).unwrap();
         let opts = hdf5_writer::WriteOptions {
             agent_id: "t".into(),
             embedder: "t".into(),
@@ -345,8 +423,8 @@ mod tests {
         };
         hdf5_writer::write_hdf5(h5_path.to_str().unwrap(), &data, &opts).unwrap();
 
-        // Verify file was created and is valid
+        // Content-validate with the float16 tolerance enabled.
-        let summary = validate::validate_hdf5(h5_path.to_str().unwrap(), 1, 0, 0, 0, 4).unwrap();
+        let summary = validate::validate_hdf5(h5_path.to_str().unwrap(), &data, true, true).unwrap();
         assert_eq!(summary.chunks, 1);
 
         // Verify float16 values are within tolerance
@@ -375,7 +453,7 @@ mod tests {
         }
         drop(conn);
 
-        let data = sqlite_reader::read_sqlite(&db_path, false, None).unwrap();
+        let data = sqlite_reader::read_sqlite(&db_path, false, None, &SchemaConfig::default()).unwrap();
 
         let opts_compressed = hdf5_writer::WriteOptions {
             agent_id: "t".into(),
@@ -415,7 +493,7 @@ mod tests {
         drop(conn);
 
         // Simulate dry-run: read data but don't write
-        let data = sqlite_reader::read_sqlite(&db_path, false, None).unwrap();
+        let data = sqlite_reader::read_sqlite(&db_path, false, None, &SchemaConfig::default()).unwrap();
         assert_eq!(data.chunks.len(), 1);
         assert!(!h5_path.exists());
     }
@@ -427,7 +505,7 @@ mod tests {
         let db_path = create_test_db(&dir);
         let h5_path = dir.path().join("out.h5");
 
-        let data = sqlite_reader::read_sqlite(&db_path, false, None).unwrap();
+        let data = sqlite_reader::read_sqlite(&db_path, false, None, &SchemaConfig::default()).unwrap();
         assert_eq!(data.chunks.len(), 0);
         assert_eq!(data.sessions.len(), 0);
         assert_eq!(data.entities.len(), 0);
@@ -442,7 +520,7 @@ mod tests {
         };
         hdf5_writer::write_hdf5(h5_path.to_str().unwrap(), &data, &opts).unwrap();
 
-        let summary = validate::validate_hdf5(h5_path.to_str().unwrap(), 0, 0, 0, 0, 0).unwrap();
+        let summary = validate::validate_hdf5(h5_path.to_str().unwrap(), &data, false, false).unwrap();
         assert_eq!(summary.chunks, 0);
     }
 
@@ -465,7 +543,7 @@ mod tests {
         }
         drop(conn);
 
-        let data = sqlite_reader::read_sqlite(&db_path, false, None).unwrap();
+        let data = sqlite_reader::read_sqlite(&db_path, false, None, &SchemaConfig::default()).unwrap();
         assert_eq!(data.chunks.len(), 1000);
 
         let opts = hdf5_writer::WriteOptions {
@@ -478,7 +556,7 @@ mod tests {
         hdf5_writer::write_hdf5(h5_path.to_str().unwrap(), &data, &opts).unwrap();
 
         let summary =
-            validate::validate_hdf5(h5_path.to_str().unwrap(), 1000, 0, 0, 0, 64).unwrap();
+            validate::validate_hdf5(h5_path.to_str().unwrap(), &data, false, false).unwrap();
         assert_eq!(summary.chunks, 1000);
     }
 
@@ -495,7 +573,7 @@ mod tests {
         insert_session(&conn, "session-gamma", 21, 30);
         drop(conn);
 
-        let data = sqlite_reader::read_sqlite(&db_path, false, None).unwrap();
+        let data = sqlite_reader::read_sqlite(&db_path, false, None, &SchemaConfig::default()).unwrap();
         assert_eq!(data.sessions.len(), 3);
 
         let opts = hdf5_writer::WriteOptions {
@@ -507,7 +585,7 @@ mod tests {
         };
         hdf5_writer::write_hdf5(h5_path.to_str().unwrap(), &data, &opts).unwrap();
 
-        let summary = validate::validate_hdf5(h5_path.to_str().unwrap(), 0, 3, 0, 0, 0).unwrap();
+        let summary = validate::validate_hdf5(h5_path.to_str().unwrap(), &data, false, false).unwrap();
         assert_eq!(summary.sessions, 3);
     }
 
@@ -527,7 +605,7 @@ mod tests {
         insert_relation(&conn, 2, 3, "uses");
         drop(conn);
 
-        let data = sqlite_reader::read_sqlite(&db_path, false, None).unwrap();
+        let data = sqlite_reader::read_sqlite(&db_path, false, None, &SchemaConfig::default()).unwrap();
         assert_eq!(data.entities.len(), 3);
         assert_eq!(data.relations.len(), 3);
 
@@ -540,7 +618,7 @@ mod tests {
         };
         hdf5_writer::write_hdf5(h5_path.to_str().unwrap(), &data, &opts).unwrap();
 
-        let summary = validate::validate_hdf5(h5_path.to_str().unwrap(), 0, 0, 3, 3, 0).unwrap();
+        let summary = validate::validate_hdf5(h5_path.to_str().unwrap(), &data, false, false).unwrap();
         assert_eq!(summary.entities, 3);
         assert_eq!(summary.relations, 3);
     }
@@ -556,7 +634,7 @@ mod tests {
         insert_chunk(&conn, 1, "test", &make_embedding(4, 1.0), 0);
         drop(conn);
 
-        let data = sqlite_reader::read_sqlite(&db_path, false, None).unwrap();
+        let data = sqlite_reader::read_sqlite(&db_path, false, None, &SchemaConfig::default()).unwrap();
         let opts = hdf5_writer::WriteOptions {
             agent_id: "t".into(),
             embedder: "t".into(),
@@ -566,15 +644,15 @@ mod tests {
         };
         hdf5_writer::write_hdf5(h5_path.to_str().unwrap(), &data, &opts).unwrap();
 
-        // Expect 5 chunks but only 1 was written
+        // Validating against a source with an extra (unwritten) chunk must fail.
-        let result = validate::validate_hdf5(h5_path.to_str().unwrap(), 5, 0, 0, 0, 4);
+        let mut bigger = sqlite_reader::read_sqlite(&db_path, false, None, &SchemaConfig::default())
+            .unwrap();
+        let mut extra = bigger.chunks[0].clone();
+        extra.id = 999;
+        bigger.chunks.push(extra);
+        let result = validate::validate_hdf5(h5_path.to_str().unwrap(), &bigger, false, false);
         assert!(result.is_err());
-        assert!(
+        assert!(result.unwrap_err().to_string().contains("count mismatch"));
-            result
-                .unwrap_err()
-                .to_string()
-                .contains("Chunk count mismatch")
-        );
     }
 
     // ---------- Test 14: Metadata attributes are stored ----------
@@ -588,7 +666,7 @@ mod tests {
         insert_chunk(&conn, 1, "test", &make_embedding(8, 1.0), 0);
         drop(conn);
 
-        let data = sqlite_reader::read_sqlite(&db_path, false, None).unwrap();
+        let data = sqlite_reader::read_sqlite(&db_path, false, None, &SchemaConfig::default()).unwrap();
         let opts = hdf5_writer::WriteOptions {
             agent_id: "my-agent-42".into(),
             embedder: "openai-ada".into(),
@@ -634,7 +712,7 @@ mod tests {
         insert_chunk(&conn, 1, "test", &emb, 0);
         drop(conn);
 
-        let data = sqlite_reader::read_sqlite(&db_path, false, None).unwrap();
+        let data = sqlite_reader::read_sqlite(&db_path, false, None, &SchemaConfig::default()).unwrap();
         let opts = hdf5_writer::WriteOptions {
             agent_id: "t".into(),
             embedder: "t".into(),
@@ -680,7 +758,7 @@ mod tests {
         drop(conn);
 
         // Skip deleted
-        let data = sqlite_reader::read_sqlite(&db_path, true, None).unwrap();
+        let data = sqlite_reader::read_sqlite(&db_path, true, None, &SchemaConfig::default()).unwrap();
         assert_eq!(data.chunks.len(), 4); // chunk 3 is deleted
 
         let opts = hdf5_writer::WriteOptions {
@@ -692,7 +770,7 @@ mod tests {
         };
         hdf5_writer::write_hdf5(h5_path.to_str().unwrap(), &data, &opts).unwrap();
 
-        let summary = validate::validate_hdf5(h5_path.to_str().unwrap(), 4, 2, 2, 1, 16).unwrap();
+        let summary = validate::validate_hdf5(h5_path.to_str().unwrap(), &data, false, false).unwrap();
         assert_eq!(summary.chunks, 4);
         assert_eq!(summary.sessions, 2);
         assert_eq!(summary.entities, 2);
@@ -711,7 +789,7 @@ mod tests {
         insert_session(&conn, "s1", 0, 10);
         drop(conn);
 
-        let data = sqlite_reader::read_sqlite(&db_path, false, None).unwrap();
+        let data = sqlite_reader::read_sqlite(&db_path, false, None, &SchemaConfig::default()).unwrap();
         let opts = hdf5_writer::WriteOptions {
             agent_id: "t".into(),
             embedder: "t".into(),
@@ -721,13 +799,125 @@ mod tests {
         };
         hdf5_writer::write_hdf5(h5_path.to_str().unwrap(), &data, &opts).unwrap();
 
-        let result = validate::validate_hdf5(h5_path.to_str().unwrap(), 0, 99, 0, 0, 0);
+        // Validating against a source whose session content differs must fail.
+        let mut tampered = sqlite_reader::read_sqlite(&db_path, false, None, &SchemaConfig::default())
+            .unwrap();
+        tampered.sessions[0].summary = "DIFFERENT".into();
+        let result = validate::validate_hdf5(h5_path.to_str().unwrap(), &tampered, false, false);
         assert!(result.is_err());
-        assert!(
+        assert!(result.unwrap_err().to_string().contains("session"));
-            result
+    }
-                .unwrap_err()
+
-                .to_string()
+    // ---------- Real content validation catches corrupt embeddings ----------
-                .contains("Session count mismatch")
+    #[test]
+    fn test_content_validation_catches_embedding_corruption() {
+        let dir = TempDir::new().unwrap();
+        let db_path = create_test_db(&dir);
+        let h5_path = dir.path().join("out.h5");
+
+        let conn = Connection::open(&db_path).unwrap();
+        insert_chunk(&conn, 1, "hello", &make_embedding(8, 1.0), 0);
+        drop(conn);
+
+        let data = sqlite_reader::read_sqlite(&db_path, false, None, &SchemaConfig::default()).unwrap();
+        let opts = hdf5_writer::WriteOptions {
+            agent_id: "t".into(),
+            embedder: "t".into(),
+            compression: false,
+            compression_level: 4,
+            float16: false,
+        };
+        hdf5_writer::write_hdf5(h5_path.to_str().unwrap(), &data, &opts).unwrap();
+
+        // A source whose embedding differs (but counts match) must fail validation.
+        let mut tampered = sqlite_reader::read_sqlite(&db_path, false, None, &SchemaConfig::default())
+            .unwrap();
+        tampered.chunks[0].embedding[3] += 9.0;
+        let result = validate::validate_hdf5(h5_path.to_str().unwrap(), &tampered, true, false);
+        assert!(result.is_err());
+        assert!(result.unwrap_err().to_string().contains("embedding"));
+    }
+
+    // ---------- Configurable schema: custom table names ----------
+    #[test]
+    fn test_configurable_table_names() {
+        let dir = TempDir::new().unwrap();
+        let db_path = dir.path().join("custom.db");
+        let path_str = db_path.to_str().unwrap().to_string();
+        let conn = Connection::open(&path_str).unwrap();
+        // Chunks live in a differently-named table; the others use defaults.
+        conn.execute_batch(
+            "CREATE TABLE my_chunks (
+                id INTEGER PRIMARY KEY, chunk TEXT, embedding BLOB,
+                source_channel TEXT, timestamp REAL, session_id TEXT, tags TEXT, deleted INTEGER
             );
+            CREATE TABLE sessions (id TEXT, start_idx INTEGER, end_idx INTEGER, channel TEXT, timestamp REAL, summary TEXT);
+            CREATE TABLE entities (id INTEGER, name TEXT, type TEXT, embedding_idx INTEGER);
+            CREATE TABLE relations (src INTEGER, tgt INTEGER, relation TEXT, weight REAL, timestamp REAL);",
+        )
+        .unwrap();
+        let blob: Vec<u8> = make_embedding(4, 1.0).iter().flat_map(|v| v.to_le_bytes()).collect();
+        conn.execute(
+            "INSERT INTO my_chunks VALUES (1, 'hi', ?1, 'api', 1.0, 's', '', 0)",
+            rusqlite::params![blob],
+        )
+        .unwrap();
+        drop(conn);
+
+        let mut schema = SchemaConfig::default();
+        schema.chunks.table = "my_chunks".into();
+        let data = sqlite_reader::read_sqlite(&path_str, false, None, &schema).unwrap();
+        assert_eq!(data.chunks.len(), 1);
+        assert_eq!(data.chunks[0].chunk, "hi");
+        assert_eq!(data.embedding_dim, 4);
+
+        // Counts pass should also honor the custom table name.
+        let counts = sqlite_reader::read_counts(&path_str, false, &schema).unwrap();
+        assert_eq!(counts.chunks, 1);
+    }
+
+    // ---------- Incremental migration appends only new rows ----------
+    #[test]
+    fn test_incremental_migration() {
+        let dir = TempDir::new().unwrap();
+        let db_path = create_test_db(&dir);
+        let h5_path = dir.path().join("out.h5");
+        let cfg = SchemaConfig::default();
+        let opts = hdf5_writer::WriteOptions {
+            agent_id: "t".into(),
+            embedder: "t".into(),
+            compression: false,
+            compression_level: 4,
+            float16: false,
+        };
+
+        // First migration: 2 chunks.
+        let conn = Connection::open(&db_path).unwrap();
+        insert_chunk(&conn, 1, "one", &make_embedding(4, 1.0), 0);
+        insert_chunk(&conn, 2, "two", &make_embedding(4, 2.0), 0);
+        drop(conn);
+        let data = sqlite_reader::read_sqlite(&db_path, false, None, &cfg).unwrap();
+        hdf5_writer::write_hdf5(h5_path.to_str().unwrap(), &data, &opts).unwrap();
+
+        // Add two more rows, then migrate incrementally.
+        let conn = Connection::open(&db_path).unwrap();
+        insert_chunk(&conn, 3, "three", &make_embedding(4, 3.0), 0);
+        insert_chunk(&conn, 4, "four", &make_embedding(4, 4.0), 0);
+        drop(conn);
+
+        let base = hdf5_reader::read_hdf5(h5_path.to_str().unwrap()).unwrap();
+        let max_id = base.chunks.iter().map(|c| c.id).max().unwrap_or(0);
+        assert_eq!(max_id, 2);
+        let new = sqlite_reader::read_sqlite_filtered(&db_path, false, Some(4), &cfg, max_id).unwrap();
+        assert_eq!(new.chunks.len(), 2); // only id 3 and 4
+
+        let mut merged = base;
+        merged.chunks.extend(new.chunks);
+        hdf5_writer::write_hdf5(h5_path.to_str().unwrap(), &merged, &opts).unwrap();
+
+        let final_data = hdf5_reader::read_hdf5(h5_path.to_str().unwrap()).unwrap();
+        assert_eq!(final_data.chunks.len(), 4);
+        let texts: Vec<&str> = final_data.chunks.iter().map(|c| c.chunk.as_str()).collect();
+        assert_eq!(texts, vec!["one", "two", "three", "four"]);
     }
 }

crates/clawhdf5-migrate/src/sqlite_reader.rs

@@ -53,9 +53,115 @@ pub struct SqliteData {
     pub embedding_dim: usize,
 }
 
+/// A table name plus the ordered column names the reader maps by position.
+#[derive(Debug, Clone)]
+pub struct TableSchema {
+    pub table: String,
+    pub columns: Vec<&'static str>,
+}
+
+/// Configurable mapping from a SQLite layout to the migration's data model.
+///
+/// Defaults to the ZeroClaw schema; the CLI can override the table names so the
+/// tool can migrate databases whose tables are named differently. Column names
+/// (and order) are part of the config too, so a library caller can remap them.
+#[derive(Debug, Clone)]
+pub struct SchemaConfig {
+    pub chunks: TableSchema,
+    pub sessions: TableSchema,
+    pub entities: TableSchema,
+    pub relations: TableSchema,
+}
+
+impl Default for SchemaConfig {
+    fn default() -> Self {
+        SchemaConfig {
+            chunks: TableSchema {
+                table: "memory_chunks".into(),
+                columns: vec![
+                    "id",
+                    "chunk",
+                    "embedding",
+                    "source_channel",
+                    "timestamp",
+                    "session_id",
+                    "tags",
+                    "deleted",
+                ],
+            },
+            sessions: TableSchema {
+                table: "sessions".into(),
+                columns: vec!["id", "start_idx", "end_idx", "channel", "timestamp", "summary"],
+            },
+            entities: TableSchema {
+                table: "entities".into(),
+                columns: vec!["id", "name", "type", "embedding_idx"],
+            },
+            relations: TableSchema {
+                table: "relations".into(),
+                columns: vec!["src", "tgt", "relation", "weight", "timestamp"],
+            },
+        }
+    }
+}
+
+impl TableSchema {
+    fn select(&self, where_clause: &str) -> String {
+        format!(
+            "SELECT {} FROM {}{}",
+            self.columns.join(", "),
+            self.table,
+            where_clause
+        )
+    }
+}
+
+/// Row counts for each table — a fast pass that does not load row contents.
+/// Used for `--dry-run` and progress without buffering the whole database.
+#[derive(Debug, Default, Clone, Copy)]
+pub struct RowCounts {
+    pub chunks: u64,
+    pub sessions: u64,
+    pub entities: u64,
+    pub relations: u64,
+}
+
+fn count_rows(conn: &Connection, table: &str, where_clause: &str) -> SqlResult<u64> {
+    conn.query_row(
+        &format!("SELECT COUNT(*) FROM {table}{where_clause}"),
+        [],
+        |r| r.get(0),
+    )
+}
+
+/// Count rows in each table without reading their contents.
+pub fn read_counts(
+    path: &str,
+    skip_deleted: bool,
+    config: &SchemaConfig,
+) -> Result<RowCounts, Box<dyn std::error::Error>> {
+    let conn = Connection::open(path)?;
+    let deleted_col = config.chunks.columns.get(7).copied().unwrap_or("deleted");
+    let chunk_where = if skip_deleted {
+        format!(" WHERE {deleted_col} = 0")
+    } else {
+        String::new()
+    };
+    Ok(RowCounts {
+        chunks: count_rows(&conn, &config.chunks.table, &chunk_where)?,
+        sessions: count_rows(&conn, &config.sessions.table, "")?,
+        entities: count_rows(&conn, &config.entities.table, "")?,
+        relations: count_rows(&conn, &config.relations.table, "")?,
+    })
+}
+
 /// Auto-detect embedding dimension from the first chunk's BLOB size.
-fn detect_embedding_dim(conn: &Connection) -> SqlResult<Option<usize>> {
+fn detect_embedding_dim(conn: &Connection, config: &SchemaConfig) -> SqlResult<Option<usize>> {
-    let mut stmt = conn.prepare("SELECT embedding FROM memory_chunks LIMIT 1")?;
+    let emb_col = config.chunks.columns.get(2).copied().unwrap_or("embedding");
+    let mut stmt = conn.prepare(&format!(
+        "SELECT {emb_col} FROM {} LIMIT 1",
+        config.chunks.table
+    ))?;
     let mut rows = stmt.query([])?;
     if let Some(row) = rows.next()? {
         let blob: Vec<u8> = row.get(0)?;
@@ -80,18 +186,31 @@ pub fn read_sqlite(
     path: &str,
     skip_deleted: bool,
     embedding_dim: Option<usize>,
+    config: &SchemaConfig,
+) -> Result<SqliteData, Box<dyn std::error::Error>> {
+    read_sqlite_filtered(path, skip_deleted, embedding_dim, config, 0)
+}
+
+/// Like [`read_sqlite`] but only reads chunks whose id is greater than
+/// `min_chunk_id` (0 = all). Used for incremental migration.
+pub fn read_sqlite_filtered(
+    path: &str,
+    skip_deleted: bool,
+    embedding_dim: Option<usize>,
+    config: &SchemaConfig,
+    min_chunk_id: i64,
 ) -> Result<SqliteData, Box<dyn std::error::Error>> {
     let conn = Connection::open(path)?;
 
     let dim = match embedding_dim {
         Some(d) => d,
-        None => detect_embedding_dim(&conn)?.unwrap_or(0),
+        None => detect_embedding_dim(&conn, config)?.unwrap_or(0),
     };
 
-    let chunks = read_chunks(&conn, skip_deleted, dim)?;
+    let chunks = read_chunks(&conn, skip_deleted, dim, config, min_chunk_id)?;
-    let sessions = read_sessions(&conn)?;
+    let sessions = read_sessions(&conn, config)?;
-    let entities = read_entities(&conn)?;
+    let entities = read_entities(&conn, config)?;
-    let relations = read_relations(&conn)?;
+    let relations = read_relations(&conn, config)?;
 
     Ok(SqliteData {
         chunks,
@@ -106,16 +225,26 @@ fn read_chunks(
     conn: &Connection,
     skip_deleted: bool,
     expected_dim: usize,
+    config: &SchemaConfig,
+    min_chunk_id: i64,
 ) -> SqlResult<Vec<MemoryChunk>> {
-    let sql = if skip_deleted {
+    let id_col = config.chunks.columns.first().copied().unwrap_or("id");
-        "SELECT id, chunk, embedding, source_channel, timestamp, session_id, tags, deleted \
+    let deleted_col = config.chunks.columns.get(7).copied().unwrap_or("deleted");
-         FROM memory_chunks WHERE deleted = 0"
+    let mut conds = Vec::new();
+    if skip_deleted {
+        conds.push(format!("{deleted_col} = 0"));
+    }
+    if min_chunk_id > 0 {
+        conds.push(format!("{id_col} > {min_chunk_id}"));
+    }
+    let where_clause = if conds.is_empty() {
+        String::new()
     } else {
-        "SELECT id, chunk, embedding, source_channel, timestamp, session_id, tags, deleted \
+        format!(" WHERE {}", conds.join(" AND "))
-         FROM memory_chunks"
     };
+    let sql = config.chunks.select(&where_clause);
 
-    let mut stmt = conn.prepare(sql)?;
+    let mut stmt = conn.prepare(&sql)?;
     let rows = stmt.query_map([], |row| {
         let blob: Vec<u8> = row.get(2)?;
         let mut embedding = blob_to_f32(&blob);
@@ -140,9 +269,8 @@ fn read_chunks(
     rows.collect()
 }
 
-fn read_sessions(conn: &Connection) -> SqlResult<Vec<Session>> {
+fn read_sessions(conn: &Connection, config: &SchemaConfig) -> SqlResult<Vec<Session>> {
-    let mut stmt =
+    let mut stmt = conn.prepare(&config.sessions.select(""))?;
-        conn.prepare("SELECT id, start_idx, end_idx, channel, timestamp, summary FROM sessions")?;
     let rows = stmt.query_map([], |row| {
         Ok(Session {
             id: row.get(0)?,
@@ -156,8 +284,8 @@ fn read_sessions(conn: &Connection) -> SqlResult<Vec<Session>> {
     rows.collect()
 }
 
-fn read_entities(conn: &Connection) -> SqlResult<Vec<Entity>> {
+fn read_entities(conn: &Connection, config: &SchemaConfig) -> SqlResult<Vec<Entity>> {
-    let mut stmt = conn.prepare("SELECT id, name, type, embedding_idx FROM entities")?;
+    let mut stmt = conn.prepare(&config.entities.select(""))?;
     let rows = stmt.query_map([], |row| {
         Ok(Entity {
             id: row.get(0)?,
@@ -169,8 +297,8 @@ fn read_entities(conn: &Connection) -> SqlResult<Vec<Entity>> {
     rows.collect()
 }
 
-fn read_relations(conn: &Connection) -> SqlResult<Vec<Relation>> {
+fn read_relations(conn: &Connection, config: &SchemaConfig) -> SqlResult<Vec<Relation>> {
-    let mut stmt = conn.prepare("SELECT src, tgt, relation, weight, timestamp FROM relations")?;
+    let mut stmt = conn.prepare(&config.relations.select(""))?;
     let rows = stmt.query_map([], |row| {
         Ok(Relation {
             src: row.get(0)?,

crates/clawhdf5-migrate/src/validate.rs

@@ -1,5 +1,7 @@
-use clawhdf5::reader::File;
+use crate::hdf5_reader::read_hdf5;
-use clawhdf5_format::type_builders::AttrValue;
+use crate::sqlite_reader::SqliteData;
+
+type BoxErr = Box<dyn std::error::Error>;
 
 /// Summary of a migration validation.
 #[derive(Debug)]
@@ -9,119 +11,149 @@ pub struct ValidationSummary {
     pub entities: u64,
     pub relations: u64,
     pub embedding_dim: u64,
+    /// Number of rows whose full content was compared against the source.
+    pub rows_checked: u64,
 }
 
-/// Validate an HDF5 file written by the migration tool.
+/// Validate a migrated HDF5 file against the source data.
 ///
-/// Checks that row counts and embedding dimensions match expectations.
+/// Reads the written file back and compares actual content — chunk text,
+/// embeddings, and every session/entity/relation field — to the source, not
+/// just the row counts. When `full` is false a representative sample of chunk
+/// rows is content-checked (counts and all other groups are always checked in
+/// full); when `full` is true every chunk row is compared too. `float16` widens
+/// the embedding tolerance to allow for half-precision quantization.
 pub fn validate_hdf5(
     path: &str,
-    expected_chunks: usize,
+    source: &SqliteData,
-    expected_sessions: usize,
+    full: bool,
-    expected_entities: usize,
+    float16: bool,
-    expected_relations: usize,
+) -> Result<ValidationSummary, BoxErr> {
-    expected_dim: usize,
+    let got = read_hdf5(path)?;
-) -> Result<ValidationSummary, Box<dyn std::error::Error>> {
-    let file = File::open(path)?;
-    let root = file.root();
 
-    // Read root attributes
+    // ---- Counts ----
-    let attrs = root.attrs()?;
+    check_count("chunk", got.chunks.len(), source.chunks.len())?;
-    let stored_dim = match attrs.get("embedding_dim") {
+    check_count("session", got.sessions.len(), source.sessions.len())?;
-        Some(AttrValue::I64(d)) => *d as u64,
+    check_count("entity", got.entities.len(), source.entities.len())?;
-        _ => 0,
+    check_count("relation", got.relations.len(), source.relations.len())?;
-    };
+    if got.embedding_dim != source.embedding_dim {
-
-    // Validate chunks group
-    let chunks_group = file.group("chunks")?;
-    let chunk_attrs = chunks_group.attrs()?;
-    let chunk_count = match chunk_attrs.get("count") {
-        Some(AttrValue::I64(n)) => *n as u64,
-        _ => 0,
-    };
-
-    if chunk_count != expected_chunks as u64 {
         return Err(format!(
-            "Chunk count mismatch: HDF5 has {}, expected {}",
+            "embedding_dim mismatch: HDF5 has {}, source has {}",
-            chunk_count, expected_chunks
+            got.embedding_dim, source.embedding_dim
         )
         .into());
     }
 
-    // Validate embedding dimensions if chunks exist
+    // ---- Chunk content (sampled or full) ----
-    if chunk_count > 0 && expected_dim > 0 {
+    let (emb_abs, emb_rel) = if float16 { (1e-2, 1e-2) } else { (1e-4, 0.0) };
-        let emb_ds = chunks_group.dataset("embeddings")?;
+    let mut rows_checked = 0u64;
-        let shape = emb_ds.shape()?;
+    for i in sample_indices(source.chunks.len(), full) {
-        if shape.len() == 2 && shape[1] != expected_dim as u64 {
+        let (s, g) = (&source.chunks[i], &got.chunks[i]);
+        if s.id != g.id {
+            return Err(field_err("chunk", i, "id", s.id, g.id));
+        }
+        if s.chunk != g.chunk {
             return Err(format!(
-                "Embedding dim mismatch: HDF5 has {}, expected {}",
+                "chunk[{i}].text mismatch: source {:?}, HDF5 {:?}",
-                shape[1], expected_dim
+                truncate(&s.chunk),
+                truncate(&g.chunk)
             )
             .into());
         }
-
+        if s.session_id != g.session_id || s.source_channel != g.source_channel || s.tags != g.tags
-        if stored_dim != expected_dim as u64 {
+        {
+            return Err(format!("chunk[{i}] string field mismatch").into());
+        }
+        if s.deleted != g.deleted {
+            return Err(field_err("chunk", i, "deleted", s.deleted, g.deleted));
+        }
+        if s.embedding.len() != g.embedding.len() {
             return Err(format!(
-                "Embedding dim attr mismatch: HDF5 attr={}, expected {}",
+                "chunk[{i}] embedding length mismatch: {} vs {}",
-                stored_dim, expected_dim
+                s.embedding.len(),
+                g.embedding.len()
+            )
+            .into());
+        }
+        for (k, (&a, &b)) in s.embedding.iter().zip(g.embedding.iter()).enumerate() {
+            if (a - b).abs() > emb_abs + emb_rel * a.abs() {
+                return Err(format!(
+                    "chunk[{i}].embedding[{k}] mismatch: source {a}, HDF5 {b}"
                 )
                 .into());
             }
         }
-
+        rows_checked += 1;
-    // Validate sessions group
-    let sessions_group = file.group("sessions")?;
-    let sess_attrs = sessions_group.attrs()?;
-    let session_count = match sess_attrs.get("count") {
-        Some(AttrValue::I64(n)) => *n as u64,
-        _ => 0,
-    };
-
-    if session_count != expected_sessions as u64 {
-        return Err(format!(
-            "Session count mismatch: HDF5 has {}, expected {}",
-            session_count, expected_sessions
-        )
-        .into());
     }
 
-    // Validate entities group
+    // ---- Other groups (always full — they are small) ----
-    let entities_group = file.group("entities")?;
+    for (i, (s, g)) in source.sessions.iter().zip(got.sessions.iter()).enumerate() {
-    let ent_attrs = entities_group.attrs()?;
+        if s.id != g.id
-    let entity_count = match ent_attrs.get("count") {
+            || s.start_idx != g.start_idx
-        Some(AttrValue::I64(n)) => *n as u64,
+            || s.end_idx != g.end_idx
-        _ => 0,
+            || s.channel != g.channel
-    };
+            || s.summary != g.summary
-
+        {
-    if entity_count != expected_entities as u64 {
+            return Err(format!("session[{i}] mismatch").into());
-        return Err(format!(
-            "Entity count mismatch: HDF5 has {}, expected {}",
-            entity_count, expected_entities
-        )
-        .into());
         }
-
+        rows_checked += 1;
-    // Validate relations group
+    }
-    let relations_group = file.group("relations")?;
+    for (i, (s, g)) in source.entities.iter().zip(got.entities.iter()).enumerate() {
-    let rel_attrs = relations_group.attrs()?;
+        if s.id != g.id
-    let relation_count = match rel_attrs.get("count") {
+            || s.name != g.name
-        Some(AttrValue::I64(n)) => *n as u64,
+            || s.entity_type != g.entity_type
-        _ => 0,
+            || s.embedding_idx != g.embedding_idx
-    };
+        {
-
+            return Err(format!("entity[{i}] mismatch").into());
-    if relation_count != expected_relations as u64 {
+        }
-        return Err(format!(
+        rows_checked += 1;
-            "Relation count mismatch: HDF5 has {}, expected {}",
+    }
-            relation_count, expected_relations
+    for (i, (s, g)) in source.relations.iter().zip(got.relations.iter()).enumerate() {
-        )
+        if s.src != g.src || s.tgt != g.tgt || s.relation != g.relation {
-        .into());
+            return Err(format!("relation[{i}] mismatch").into());
+        }
+        rows_checked += 1;
     }
 
     Ok(ValidationSummary {
-        chunks: chunk_count,
+        chunks: got.chunks.len() as u64,
-        sessions: session_count,
+        sessions: got.sessions.len() as u64,
-        entities: entity_count,
+        entities: got.entities.len() as u64,
-        relations: relation_count,
+        relations: got.relations.len() as u64,
-        embedding_dim: stored_dim,
+        embedding_dim: got.embedding_dim as u64,
+        rows_checked,
     })
 }
+
+fn check_count(kind: &str, got: usize, expected: usize) -> Result<(), BoxErr> {
+    if got != expected {
+        return Err(format!("{kind} count mismatch: HDF5 has {got}, source has {expected}").into());
+    }
+    Ok(())
+}
+
+fn field_err<T: std::fmt::Display>(kind: &str, i: usize, field: &str, s: T, g: T) -> BoxErr {
+    format!("{kind}[{i}].{field} mismatch: source {s}, HDF5 {g}").into()
+}
+
+fn truncate(s: &str) -> String {
+    if s.len() <= 40 {
+        s.to_string()
+    } else {
+        format!("{}…", &s[..40])
+    }
+}
+
+/// Indices of chunk rows to content-check. Full = all; otherwise a spread of
+/// representative rows (first/last and evenly-spaced interior samples).
+fn sample_indices(n: usize, full: bool) -> Vec<usize> {
+    if n == 0 {
+        return Vec::new();
+    }
+    if full || n <= 16 {
+        return (0..n).collect();
+    }
+    let mut idx: Vec<usize> = (0..16).map(|k| k * (n - 1) / 15).collect();
+    idx.dedup();
+    idx
+}

crates/clawhdf5-netcdf4/README.md

@@ -1,9 +1,9 @@
-# rustyhdf5-netcdf4
+# clawhdf5-netcdf4
 
-[![crates.io](https://img.shields.io/crates/v/rustyhdf5-netcdf4.svg)](https://crates.io/crates/rustyhdf5-netcdf4)
+[![crates.io](https://img.shields.io/crates/v/clawhdf5-netcdf4.svg)](https://crates.io/crates/clawhdf5-netcdf4)
-[![docs.rs](https://docs.rs/rustyhdf5-netcdf4/badge.svg)](https://docs.rs/rustyhdf5-netcdf4)
+[![docs.rs](https://docs.rs/clawhdf5-netcdf4/badge.svg)](https://docs.rs/clawhdf5-netcdf4)
 
-NetCDF-4 read support built on rustyhdf5 — pure Rust, no C dependencies.
+NetCDF-4 read support built on clawhdf5 — pure Rust, no C dependencies.
 
 ## Features
 
@@ -14,7 +14,7 @@ NetCDF-4 read support built on rustyhdf5 — pure Rust, no C dependencies.
 ## Usage
 
 ```rust
-use rustyhdf5_netcdf4::NetCDF4File;
+use clawhdf5_netcdf4::NetCDF4File;
 
 let nc = NetCDF4File::open("climate.nc").unwrap();
 let temp = nc.variable("temperature").unwrap();

crates/clawhdf5-py/README.md

@@ -1,9 +1,9 @@
-# rustyhdf5-py
+# clawhdf5-py
 
-[![crates.io](https://img.shields.io/crates/v/rustyhdf5-py.svg)](https://crates.io/crates/rustyhdf5-py)
+[![crates.io](https://img.shields.io/crates/v/clawhdf5-py.svg)](https://crates.io/crates/clawhdf5-py)
-[![docs.rs](https://docs.rs/rustyhdf5-py/badge.svg)](https://docs.rs/rustyhdf5-py)
+[![docs.rs](https://docs.rs/clawhdf5-py/badge.svg)](https://docs.rs/clawhdf5-py)
 
-Python bindings for rustyhdf5 — a pure-Rust HDF5 library.
+Python bindings for clawhdf5 — a pure-Rust HDF5 library.
 
 ## Features
 
@@ -14,9 +14,9 @@ Python bindings for rustyhdf5 — a pure-Rust HDF5 library.
 ## Usage
 
 ```python
-import rustyhdf5
+import clawhdf5
 
-with rustyhdf5.File('data.h5', 'r') as f:
+with clawhdf5.File('data.h5', 'r') as f:
     data = f['/dataset'][:]
 ```
 

crates/clawhdf5-types/README.md

@@ -1,9 +1,9 @@
-# rustyhdf5-types
+# clawhdf5-types
 
-[![crates.io](https://img.shields.io/crates/v/rustyhdf5-types.svg)](https://crates.io/crates/rustyhdf5-types)
+[![crates.io](https://img.shields.io/crates/v/clawhdf5-types.svg)](https://crates.io/crates/clawhdf5-types)
-[![docs.rs](https://docs.rs/rustyhdf5-types/badge.svg)](https://docs.rs/rustyhdf5-types)
+[![docs.rs](https://docs.rs/clawhdf5-types/badge.svg)](https://docs.rs/clawhdf5-types)
 
-HDF5 type system definitions for the rustyhdf5 ecosystem.
+HDF5 type system definitions for the clawhdf5 ecosystem.
 
 ## Features
 
@@ -13,7 +13,7 @@ HDF5 type system definitions for the rustyhdf5 ecosystem.
 ## Usage
 
 ```rust
-use rustyhdf5_types::HDF5Type;
+use clawhdf5_types::HDF5Type;
 ```
 
 ## License

crates/clawhdf5/README.md

@@ -1,7 +1,7 @@
-# rustyhdf5
+# clawhdf5
 
-[![crates.io](https://img.shields.io/crates/v/rustyhdf5.svg)](https://crates.io/crates/rustyhdf5)
+[![crates.io](https://img.shields.io/crates/v/clawhdf5.svg)](https://crates.io/crates/clawhdf5)
-[![docs.rs](https://docs.rs/rustyhdf5/badge.svg)](https://docs.rs/rustyhdf5)
+[![docs.rs](https://docs.rs/clawhdf5/badge.svg)](https://docs.rs/clawhdf5)
 
 Pure-Rust HDF5 reader/writer — no C dependencies.
 
@@ -16,7 +16,7 @@ Pure-Rust HDF5 reader/writer — no C dependencies.
 ## Usage
 
 ```rust
-use rustyhdf5::File;
+use clawhdf5::File;
 
 let file = File::open("data.h5").unwrap();
 let dataset = file.dataset("/group/data").unwrap();

crates/clawhdf5/src/reader.rs

@@ -66,6 +66,9 @@ pub struct File {
     superblock: Superblock,
     /// Per-file chunk cache shared across all dataset reads.
     chunk_cache: ChunkCache,
+    /// Directory the file was opened from, used to resolve external Virtual
+    /// Dataset source files relative to this file. `None` for in-memory files.
+    base_dir: Option<std::path::PathBuf>,
 }
 
 impl File {
@@ -74,6 +77,7 @@ impl File {
     /// When the `mmap` feature is enabled (default), this uses memory-mapped
     /// I/O.  Otherwise it reads the entire file into a `Vec<u8>`.
     pub fn open<P: AsRef<std::path::Path>>(path: P) -> Result<Self, Error> {
+        let base_dir = path.as_ref().parent().map(|p| p.to_path_buf());
         #[cfg(feature = "mmap")]
         {
             let reader = clawhdf5_io::MmapReader::open(path).map_err(Error::Io)?;
@@ -84,12 +88,15 @@ impl File {
                 data: FileData::Mmap(reader),
                 superblock,
                 chunk_cache: ChunkCache::new(),
+                base_dir,
             })
         }
         #[cfg(not(feature = "mmap"))]
         {
             let bytes = std::fs::read(path.as_ref()).map_err(Error::Io)?;
-            Self::from_bytes(bytes)
+            let mut f = Self::from_bytes(bytes)?;
+            f.base_dir = base_dir;
+            Ok(f)
         }
     }
 
@@ -99,10 +106,15 @@ impl File {
     /// undesirable (e.g. network filesystems, very small files, etc.).
     pub fn open_buffered<P: AsRef<std::path::Path>>(path: P) -> Result<Self, Error> {
         let bytes = std::fs::read(path.as_ref()).map_err(Error::Io)?;
-        Self::from_bytes(bytes)
+        let mut f = Self::from_bytes(bytes)?;
+        f.base_dir = path.as_ref().parent().map(|p| p.to_path_buf());
+        Ok(f)
     }
 
     /// Open an HDF5 file from an in-memory byte vector.
+    ///
+    /// In-memory files have no directory, so external Virtual Dataset sources
+    /// cannot be resolved automatically (same-file VDS still works).
     pub fn from_bytes(data: Vec<u8>) -> Result<Self, Error> {
         let sig_offset = signature::find_signature(&data)?;
         let superblock = Superblock::parse(&data, sig_offset)?;
@@ -110,6 +122,7 @@ impl File {
             data: FileData::Owned(data),
             superblock,
             chunk_cache: ChunkCache::new(),
+            base_dir: None,
         })
     }
 
@@ -710,6 +723,28 @@ impl<'f> Dataset<'f> {
         let ds = self.dataspace()?;
         let dl = self.data_layout()?;
         let pipeline = self.filter_pipeline();
+
+        // Virtual datasets are assembled from source datasets; the per-file
+        // chunk cache does not apply. Route them through the resolver path so
+        // external sibling files resolve relative to this file's directory.
+        if matches!(dl, DataLayout::Virtual { .. }) {
+            let base_dir = self.file.base_dir.clone();
+            let resolver = move |name: &str| -> Option<Vec<u8>> {
+                let dir = base_dir.as_ref()?;
+                std::fs::read(dir.join(name)).ok()
+            };
+            return Ok(data_read::read_raw_data_full_with_resolver(
+                self.file.data.as_bytes(),
+                &dl,
+                &ds,
+                &dt,
+                pipeline.as_ref(),
+                self.file.offset_size(),
+                self.file.length_size(),
+                Some(&resolver),
+            )?);
+        }
+
         Ok(data_read::read_raw_data_cached(
             self.file.data.as_bytes(),
             &dl,

crates/clawhdf5/tests/h5py_interop_tests.rs

@@ -481,3 +481,77 @@ fn serde_json_minimal_parse(s: &str) -> Vec<f64> {
         .map(|v| v.trim().parse::<f64>().unwrap())
         .collect()
 }
+
+// ---------------------------------------------------------------------------
+// A_dense. Write a group with many links (dense storage) -> h5py reads all
+// ---------------------------------------------------------------------------
+
+#[test]
+fn clawhdf5_writes_dense_group_h5py_reads() {
+    skip_if_no_python!();
+    let dir = tempfile::tempdir().unwrap();
+    let path = dir.path().join("dense_group.h5");
+    let path_str = path.display().to_string();
+
+    // 20 links exceeds the compact threshold (8) -> dense fractal-heap storage.
+    let mut b = FileBuilder::new();
+    let mut g = b.create_group("big");
+    for i in 0..20 {
+        g.create_dataset(&format!("dataset_{i:03}"))
+            .with_i32_data(&[i, i * 10]);
+    }
+    b.add_group(g.finish());
+    b.write(&path).unwrap();
+
+    let script = format!(
+        r#"
+import h5py
+with h5py.File("{path_str}", "r") as f:
+    big = f["big"]
+    names = sorted(big.keys())
+    assert len(names) == 20, f"expected 20 links, got {{len(names)}}"
+    for i in range(20):
+        v = big[f"dataset_{{i:03}}"][()].tolist()
+        assert v == [i, i*10], f"link {{i}} = {{v}}"
+print("OK")
+"#
+    );
+    let out = run_python_output(&script);
+    assert_eq!(out, "OK");
+}
+
+// ---------------------------------------------------------------------------
+// A_multiblock. Write a multi-direct-block fractal heap -> h5py reads
+// ---------------------------------------------------------------------------
+
+#[test]
+fn clawhdf5_writes_multiblock_heap_h5py_reads() {
+    skip_if_no_python!();
+    let dir = tempfile::tempdir().unwrap();
+    let path = dir.path().join("multiblock.h5");
+    let path_str = path.display().to_string();
+
+    // ~1600 dense attributes overflow a single 64KiB fractal-heap direct block.
+    let mut b = FileBuilder::new();
+    let mut g = b.create_group("g");
+    for i in 0..1600i64 {
+        g.set_attr(&format!("attribute_number_{i:05}"), AttrValue::I64(i * 2));
+    }
+    g.create_dataset("d").with_i32_data(&[1]);
+    b.add_group(g.finish());
+    b.write(&path).unwrap();
+
+    let script = format!(
+        r#"
+import h5py
+with h5py.File("{path_str}", "r") as f:
+    a = f["g"].attrs
+    assert len(a) == 1600, f"expected 1600 attrs, got {{len(a)}}"
+    for i in (0, 1, 999, 1599):
+        v = int(a[f"attribute_number_{{i:05}}"])
+        assert v == i*2, f"attr {{i}} = {{v}}"
+print("OK")
+"#
+    );
+    assert_eq!(run_python_output(&script), "OK");
+}

crates/clawhdf5/tests/integration_tests.rs

@@ -706,6 +706,90 @@ fn fletcher32_roundtrip() {
 // 15. Multiple groups with same-named datasets
 // ---------------------------------------------------------------------------
 
+#[test]
+fn multiple_chunked_datasets_share_file_cache() {
+    // The per-file ChunkCache is shared across datasets. Two chunked datasets
+    // of *different rank* must each read correctly: a 1-D dataset's chunk index
+    // (rank 1) must not be reused for a 2-D dataset (rank 2). Read the 1-D one
+    // first so it seeds the shared cache, then the 2-D one.
+    use clawhdf5_format::datatype::{CharacterSet, Datatype, StringPadding};
+
+    // 1-D chunked + compressed fixed-length strings (payload > compress threshold).
+    let strings: Vec<String> = (0..64).map(|i| format!("entry-{i:06}-{}", "x".repeat(80))).collect();
+    let max_len = strings.iter().map(|s| s.len()).max().unwrap();
+    let mut sraw = Vec::new();
+    for s in &strings {
+        let mut b = s.as_bytes().to_vec();
+        b.resize(max_len, 0);
+        sraw.extend_from_slice(&b);
+    }
+    let sdt = Datatype::String {
+        size: max_len as u32,
+        padding: StringPadding::NullPad,
+        charset: CharacterSet::Utf8,
+    };
+
+    // 2-D chunked + compressed f32 matrix.
+    let (n, d) = (40usize, 8usize);
+    let mat: Vec<f32> = (0..n * d).map(|i| i as f32).collect();
+
+    let mut b = FileBuilder::new();
+    {
+        let ds = b.create_dataset("strs");
+        ds.with_compound_data(sdt, sraw, strings.len() as u64);
+        ds.with_chunks(&[16]);
+        ds.with_deflate(6);
+    }
+    {
+        let ds = b.create_dataset("mat");
+        ds.with_f32_data(&mat).with_shape(&[n as u64, d as u64]);
+        ds.with_chunks(&[10, d as u64]).with_shuffle().with_deflate(6);
+    }
+    let bytes = b.finish().unwrap();
+    let file = File::from_bytes(bytes).unwrap();
+
+    // Read the 1-D dataset first (seeds the shared cache with a rank-1 index),
+    // then the 2-D dataset through the same File/cache.
+    let got_strs = file.dataset("strs").unwrap().read_string().unwrap();
+    assert_eq!(got_strs, strings);
+    let got_mat = file.dataset("mat").unwrap().read_f32().unwrap();
+    assert_eq!(got_mat, mat);
+    // Read the 1-D one again to confirm the cache rebinds back correctly.
+    assert_eq!(file.dataset("strs").unwrap().read_string().unwrap(), strings);
+}
+
+#[test]
+fn virtual_dataset_external_file_auto_resolved() {
+    // The facade resolves external Virtual Dataset sources relative to the
+    // opened file's directory automatically. Drop both files side by side in a
+    // temp dir and open the virtual one through the public File API.
+    let virt = include_bytes!("../../clawhdf5-format/tests/fixtures/vds_external_virt.h5");
+    let src = include_bytes!("../../clawhdf5-format/tests/fixtures/vds_external_src.h5");
+
+    let dir = tempfile::tempdir().unwrap();
+    std::fs::write(dir.path().join("ext_src.h5"), src).unwrap();
+    let virt_path = dir.path().join("ext_virt.h5");
+    std::fs::write(&virt_path, virt).unwrap();
+
+    let file = File::open(&virt_path).unwrap();
+    let values = file.dataset("virt").unwrap().read_i32().unwrap();
+    assert_eq!(values, vec![10, 11, 12, 13, 14, 15, 16, 17]);
+}
+
+#[test]
+fn virtual_dataset_external_missing_source_is_fill() {
+    // If the external source file is absent, its region reads as the zero fill
+    // value rather than erroring.
+    let virt = include_bytes!("../../clawhdf5-format/tests/fixtures/vds_external_virt.h5");
+    let dir = tempfile::tempdir().unwrap();
+    let virt_path = dir.path().join("ext_virt.h5");
+    std::fs::write(&virt_path, virt).unwrap();
+
+    let file = File::open(&virt_path).unwrap();
+    let values = file.dataset("virt").unwrap().read_i32().unwrap();
+    assert_eq!(values, vec![0; 8]);
+}
+
 #[test]
 fn same_dataset_name_in_different_groups() {
     let mut b = FileBuilder::new();
@@ -730,3 +814,108 @@ fn same_dataset_name_in_different_groups() {
         vec![3.0, 4.0]
     );
 }
+
+#[test]
+fn dense_group_links_roundtrip() {
+    // A group with more than the compact threshold (8) of links is stored
+    // densely (fractal heap + v2 B-tree). It must round-trip; a small sibling
+    // group stays compact. Names are chosen so hashes are non-trivial.
+    let mut b = FileBuilder::new();
+    let mut big = b.create_group("big");
+    for i in 0..20 {
+        big.create_dataset(&format!("dataset_{i:03}"))
+            .with_i32_data(&[i, i * 2, i * 3]);
+    }
+    b.add_group(big.finish());
+    let mut small = b.create_group("small");
+    small.create_dataset("a").with_f64_data(&[1.0]);
+    small.create_dataset("b").with_f64_data(&[2.0]);
+    b.add_group(small.finish());
+    let bytes = b.finish().unwrap();
+
+    let file = File::from_bytes(bytes).unwrap();
+
+    // All 20 dense-group links resolve, with correct data.
+    let mut names = file.group("big").unwrap().datasets().unwrap();
+    names.sort();
+    assert_eq!(names.len(), 20);
+    for i in 0..20 {
+        assert_eq!(
+            file.dataset(&format!("big/dataset_{i:03}"))
+                .unwrap()
+                .read_i32()
+                .unwrap(),
+            vec![i, i * 2, i * 3],
+            "dense link {i} mismatch"
+        );
+    }
+    // The small (compact) group still works.
+    assert_eq!(file.dataset("small/a").unwrap().read_f64().unwrap(), vec![1.0]);
+    assert_eq!(file.dataset("small/b").unwrap().read_f64().unwrap(), vec![2.0]);
+}
+
+#[test]
+fn reads_libhdf5_multiblock_fractal_heap() {
+    // A group whose dense attributes overflow a single fractal-heap direct
+    // block, so libhdf5 stored them under a root indirect block (FHIB) with
+    // multiple direct blocks. Reading requires deriving the direct/indirect row
+    // split from the heap geometry, not the FRHP "starting rows" field.
+    let bytes = include_bytes!("../../clawhdf5-format/tests/fixtures/fractal_heap_multiblock.h5");
+    let file = File::from_bytes(bytes.to_vec()).unwrap();
+    let attrs = file.group("g").unwrap().attrs().unwrap();
+    for i in 0..80i64 {
+        let name = format!("a{i:03}");
+        match attrs.get(&name) {
+            Some(AttrValue::I64(v)) => assert_eq!(*v, i * 3, "{name}"),
+            other => panic!("{name} = {other:?}"),
+        }
+    }
+}
+
+#[test]
+fn dense_attrs_multiblock_fractal_heap_roundtrip() {
+    // Enough dense attributes to overflow a single 64 KiB fractal-heap direct
+    // block, forcing a root indirect block over multiple direct blocks.
+    let mut b = FileBuilder::new();
+    let mut g = b.create_group("g");
+    let n = 1600i64;
+    for i in 0..n {
+        g.set_attr(&format!("attribute_number_{i:05}"), AttrValue::I64(i * 2));
+    }
+    g.create_dataset("d").with_i32_data(&[1]);
+    b.add_group(g.finish());
+    let file = File::from_bytes(b.finish().unwrap()).unwrap();
+
+    let attrs = file.group("g").unwrap().attrs().unwrap();
+    for i in 0..n {
+        match attrs.get(&format!("attribute_number_{i:05}")) {
+            Some(AttrValue::I64(v)) => assert_eq!(*v, i * 2, "attr {i}"),
+            other => panic!("attr {i} = {other:?}"),
+        }
+    }
+}
+
+#[test]
+fn dense_links_multiblock_fractal_heap_roundtrip() {
+    // Enough links to overflow a single fractal-heap direct block.
+    let mut b = FileBuilder::new();
+    let mut g = b.create_group("big");
+    let n = 2200;
+    for i in 0..n {
+        g.create_dataset(&format!("dataset_number_{i:05}"))
+            .with_i32_data(&[i]);
+    }
+    b.add_group(g.finish());
+    let file = File::from_bytes(b.finish().unwrap()).unwrap();
+
+    assert_eq!(file.group("big").unwrap().datasets().unwrap().len(), n as usize);
+    for i in [0, 1, 1234, n - 1] {
+        assert_eq!(
+            file.dataset(&format!("big/dataset_number_{i:05}"))
+                .unwrap()
+                .read_i32()
+                .unwrap(),
+            vec![i]
+        );
+    }
+}