4 changed files with 156 additions and 21 deletions
@@ -119,6 +119,65 @@
  opens in HDF5 1.8.23's h5dump and in h5py). `test.mjs` no longer
  hard-codes the fixture's length.

+### Chunk dimensions of 2^32 or more; 4 GiB chunks written without copies (2026-09-29)
+- **Chunk dimensions of 2^32 or more** (libhdf5 2.x writes them with
+  layout message version 5, up to 8 bytes per dimension; a 4 GiB chunk of
+  1-byte elements needs one) are read and written. They were refused
+  (`InvalidChunkDimensions`). **Public API changes:**
+  `DataLayout::Chunked::chunk_dimensions` is `Vec<u64>` (was `Vec<u32>`),
+  and the `chunk_dimensions` argument of `chunked_read::{collect_chunk_info_checked,
+  collect_chunk_info_checked_in, generate_implicit_chunks_in_grid}`,
+  `fixed_array::{read_fixed_array_chunks, read_fixed_array_chunks_in}`,
+  `extensible_array::{read_extensible_array_chunks,
+  read_extensible_array_chunks_in}`, and the `chunk_dims` argument of
+  `chunked_write::serialize_v4_single_chunk_pub`, are `&[u64]` (were
+  `&[u32]`). A chunk whose size overflows 64 bits is refused when a dataset is read
+  (`InvalidChunkDimensions`), and the writer refuses a chunk dimension of
+  0 up front. A version-3 layout (4-byte dimensions) is never written for
+  such a chunk: over 4 GiB, it always takes version 5, as in libhdf5.
+- **Writing without copies:** `FileWriter` no longer copies chunks into a
+  buffer per pass and then into the file's buffer. A chunk that is a
+  contiguous run of the dataset's data (a dataset stored as one chunk of
+  its shape, blocks of whole rows) is borrowed when unfiltered and
+  compressed straight from it when filtered, and contiguous datasets are
+  not copied. New `FileWriter::finish_with(put)` hands the file out piece
+  by piece; `FileBuilder::write` uses it (through a 1 MiB `BufWriter`,
+  still to a temporary file renamed into place), so a file is never
+  assembled in memory. New `DatasetBuilder::with_u8_data_owned(Vec<u8>)`
+  takes the data without the copy `with_u8_data` makes. Peak resident
+  memory of one unfiltered 1 GiB chunk of `u8`
+  (`cargo run --release -p clawhdf5 --example write_one_chunk --
+  1073741824 OUT owned|slice`, tank, 2026-09-29): 5.0 GiB before (the
+  writer of `4260af4`), 1.0 GiB after with `owned`; 6.0 and 2.0 GiB with
+  `slice`; 4 GiB + 7 bytes `owned`, 4.00 GiB. Same bytes written. Write
+  speed was not re-measured (the machine was not idle): the A/B of
+  `FileBuilder::write` and `finish` on small and large files is still to
+  be run.
+- `chunked_write::extract_chunk` (behind `split_into_chunks`) no longer
+  panics when the data is shorter than the shape; the missing part is
+  zeros, as it was meant to be.
+- Tests: `crates/clawhdf5/tests/fixtures/huge_chunk_dims.h5` (31 KB,
+  libhdf5 2.0.0 via h5py 3.16, `gen_huge_chunks.py dims`: `u8` chunks of
+  2^32 + 7 deflated twice, Single Chunk and Extensible Array), and in
+  `huge_chunks_interop`: `huge_chunk_dims_list` (always), and with
+  `CLAWHDF5_HUGE_CHUNKS=1` `huge_chunk_dims_read`,
+  `huge_chunk_dims_unfiltered_read` (a sparse file h5py writes, Single
+  Chunk and Fixed Array, mmap and positioned reads),
+  `writer_huge_chunk_dims_round_trip` (deflated, read back by clawhdf5,
+  h5py 3.16 and — layouts — h5dump 2.2.0),
+  `writer_unfiltered_huge_chunk_round_trip` (an unfiltered 4 GiB + 7 byte
+  chunk: clawhdf5, h5py, and h5dump 2.2.0 printing values past 2^32) and
+  `writer_huge_chunks_lz4_zstd_round_trip` (with the `lz4`/`zstd`
+  features; h5py through hdf5plugin 7.1.0). The whole opt-in suite
+  (`cargo test --release -p clawhdf5 --features lz4,zstd --test
+  huge_chunks_interop -- --test-threads=1`, h5py and h5dump 2.2.0
+  included) passed with a peak of 10.04 GiB resident (tank, 2026-09-29,
+  commit `0065c6b`). Unit tests: 5-byte dimension encoding, contiguous
+  chunk detection, `finish_with` against `finish`. The wasm package test
+  lists the new fixture and gets `FormatError::Overflow` reading it.
+  `docs/known-issues.md`: two fixed entries; the open "Chunks of 4 GiB or
+  more: limits" keeps the refused filters, the editor and memory.
+
 ### NetCDF-4: variables' dimensions come from the file (2026-09-28)
 - `clawhdf5-netcdf4` gave each variable the first unused dimension of
  equal size (else an anonymous `dim_<n>`), so a variable on an unlimited
@@ -102,7 +102,8 @@ writes 85.2 µs vs 877 µs (10.3x); 64 group creates 130 µs vs 1.37 ms
 (10.6x); a 512×512 `f32` chunked deflate-6 write 1.44 ms vs 65.0 ms
 (re-measured 2026-09-23 with the pure-Rust deflate: 1.46 ms vs 51.4 ms,
 35x); a 100K `f32` sequential write is a tie. The writer (`FileBuilder`)
-assembles a file in memory and writes it once, which is part of that
+lays out the whole file before writing it (when these were measured, in
+one buffer in memory), which is part of that
 difference; read the caveats in [BENCHMARKS.md](BENCHMARKS.md#caveats)
 before quoting these.

@@ -116,7 +117,7 @@ Limits and open issues, with dates, are in
 | **File format** | Superblock v0–v3, user blocks, v1/v2 object headers; writing the HDF5 1.10 format (default) or, with `libver_bounds(LibVer::V18, LibVer::V18)`, files HDF5 1.8 reads (checked with HDF5 1.8.23) | Metadata cache images | Writing the pre-1.8 format (version-0 superblock, symbol-table groups) |
 | **Groups and links** | Symbol-table, compact and dense groups (tested to 100 000 links), creation order, soft and hard links; writing external links | | Following external links (explicit error); user-defined links are skipped |
 | **Datatypes** | Integers and IEEE floats of every width and byte order (incl. `f16`), enums, compounds (every version, incl. HDF5 2.0's v5), arrays, fixed-length strings, opaque, complex: h5py's `{r, i}` compound (`with_complex_f64_data`) and HDF5 2.0's native class 11 (`with_native_complex_f64_data`, opt-in: only libhdf5 2.0+ reads it; read as its own type, `DType::Complex`, and printed by `h5rs` as h5dump 2.x prints it) | Variable-length strings and sequences, object references; HDF5 2.x's small floats (bfloat16, FP8 E4M3/E5M2, FP6 E2M3/E3M2, FP4 E2M1: every bit pattern decoded as libhdf5 2.2.0 decodes it) and other non-IEEE floats up to 64 bits | Writing variable-length data; writing non-IEEE floats; decoding region and attribute references; x87 long double and binary128 |
-| **Layouts and chunk indexes** | Compact, contiguous and chunked; chunk indexes single chunk, Fixed Array, Extensible Array and v2 B-tree (the writer picks one as libhdf5 does), and v1 B-tree (every chunked dataset under the 1.8 bound, split as libhdf5 splits it); chunks of 4 GiB or more (HDF5 2.0's layout message version 5); fill values; resizable datasets; virtual datasets (read limits in known-issues) | The implicit chunk index (the editor also changes it) | External raw data files (explicit error); chunk dimensions of 2^32 or more |
+| **Layouts and chunk indexes** | Compact, contiguous and chunked; chunk indexes single chunk, Fixed Array, Extensible Array and v2 B-tree (the writer picks one as libhdf5 does), and v1 B-tree (every chunked dataset under the 1.8 bound, split as libhdf5 splits it); chunks of 4 GiB or more and chunk dimensions of 2^32 or more (HDF5 2.0's layout message version 5); fill values; resizable datasets; virtual datasets (read limits in known-issues) | The implicit chunk index (the editor also changes it) | External raw data files (explicit error) |
 | **Filters** | deflate (pure-Rust zlib-rs), shuffle, Fletcher-32, LZ4 (opt-in), Zstd (C, opt-in); plugins LZF, bitshuffle, bzip2, Blosc 1 | N-Bit, scale-offset, SZIP (C, opt-in); plugins Blosc2 and ZFP | Other filter IDs, unless you register a codec (`filter_registry::register_filter`) |
 | **Editing in place** | `FileEditor`: overwrite values, grow and shrink chunked datasets (every index), set attributes (compact and dense), in files from h5py or clawhdf5 | | Creating or deleting objects in an existing file; deleting attributes; new chunks in implicit indexes; VL data; rewriting chunks of 4 GiB or more; filters this build cannot encode (refused before any write) |
 | **Access** | Local files (mmap or buffered), bytes in memory, any `Storage` backend, HTTP(S) and S3/GCS/Azure via `clawhdf5-remote`, SWMR reading (`File::open_swmr`, `Dataset::refresh`) | Remote files and the browser are read-only | SWMR writing; remote SWMR; MPI collective I/O (`clawhdf5-io`'s `mpi-io` reads on one rank and broadcasts) |
@@ -17,6 +17,11 @@ Checked against `main` at `9b5803f` on 2026-09-28.
 |---|---|---|
 | [NetCDF-4: differences from netCDF-C](#netcdf-4-differences-from-netcdf-c) | deliberate (floats of other than 4 or 8 bytes, axes netCDF-C leaves without a dimension), and 9 corpus files (external links, values the HDF5 reader refuses) | 2026-09-29 |
 | [Chunks of 4 GiB or more: limits](#chunks-of-4-gib-or-more-limits) | refused (chunk dimensions of 2^32 or more, five filters, editor rewrites), memory (a decoded chunk is held whole) | 2026-09-28 |
+
+
+| [NetCDF-4: variables' dimensions are guessed from sizes](#netcdf-4-variables-dimensions-are-guessed-from-sizes) | **wrong metadata** (`Variable::dimensions`, pure dimension scales listed as variables; values and dimension sizes are right) | 2026-09-28 |
+| [Chunks of 4 GiB or more: limits](#chunks-of-4-gib-or-more-limits) | refused (five filters, editor rewrites), memory (a decoded chunk is held whole) | 2026-09-28 |
+| [NetCDF-4: an unlimited dimension reports size 0](#netcdf-4-an-unlimited-dimension-reports-size-0) | **wrong metadata** (dimension size; variable shapes and values are right) | 2026-09-28 |
 | [Small floats decode as libhdf5 does, not as the OCP MX specification](#small-floats-decode-as-libhdf5-does-not-as-the-ocp-mx-specification) | deliberate: libhdf5's values (FP4/FP6/FP8 E4M3 all-ones exponent is inf/NaN) | 2026-09-28 |
 | [In-place modification (`FileEditor`) limits](#in-place-modification-fileeditor-limits) | refused edits (`Error::Unsupported`), space reuse per editor, no journal | 2026-09-26 |
 | [Python in-place editing limits](#python-in-place-editing-clawhdf5filepath-r-limits) | refused writes (`NotImplementedError`), deliberate conversion differences | 2026-09-27 |
@@ -241,7 +246,10 @@ Values are correct in every case; this is cost only. Selections other than

 ## Chunks of 4 GiB or more: limits

-**Status:** open (documented 2026-09-28). HDF5 2.0 writes a chunk of more
+**Status:** open (documented 2026-09-28; updated 2026-09-29, when
+[chunk dimensions of 2^32 or more](#chunk-dimensions-of-232-or-more-were-refused)
+and [unfiltered writes](#writing-a-4-gib-unfiltered-chunk-held-several-copies-of-it)
+were fixed). HDF5 2.0 writes a chunk of more
 than 0xFFFFFFFF bytes with layout message version 5 (`H5D__chunk_construct`
 in libhdf5 2.2.0: "chunk size > 4GB requires H5F_LIBVER_V200"), which
 also makes a filtered chunk index element store the chunk's size in "size
@@ -253,34 +261,40 @@ The [fix](#chunks-of-4-gib-or-more-could-not-be-read) is tested by
 `crates/clawhdf5/tests/huge_chunks_interop.rs`; its decoding tests are
 opt-in (`CLAWHDF5_HUGE_CHUNKS=1`, one at a time: `--test-threads=1`).
 What remains:
- **Chunk dimensions of 2^32 or more** (which libhdf5 2.x also allows under
-  `H5F_LIBVER_V200`) are refused, when read (`InvalidChunkDimensions`)
-  and when written: `DataLayout::Chunked::chunk_dimensions` is `Vec<u32>`.
-  A 4 GiB chunk of 1-byte elements needs one.
 - **Filters the writer refuses for such a chunk** (`FilterError`, before
  anything is written): LZF, bitshuffle, bzip2 and Blosc (their HDF5
  filters record sizes or block lengths in 32 bits, or cannot take such a
  buffer) and pcodec. Deflate, shuffle, Fletcher-32, LZ4 and Zstd are
-  written; only shuffle + deflate is tested end to end (LZ4's framing and
-  the 256 MiB limit it had are unit-tested; Zstd is not tested at this
-  size).
- **Unfiltered chunks this size are written but not tested end to end:**
-  `FileBuilder` assembles the whole file in memory, several copies of the
-  chunk (well over the 12 GiB the tests may use). Their layout messages
-  are unit-tested (`chunked_write::tests::huge_chunk_layout_messages_and_index_elements`).
+  written and tested end to end (`writer_huge_chunks_round_trip`,
+  `writer_huge_chunk_dims_round_trip`,
+  `writer_huge_chunks_lz4_zstd_round_trip`: read back by clawhdf5 and
+  h5py 3.16, LZ4 and Zstd through hdf5plugin 7.1.0).
 - **`FileEditor`** refuses to rewrite such chunks (see
  [its limits](#in-place-modification-fileeditor-limits)).
 - **Memory:** decoding a filtered chunk holds all of it (4 GiB and more),
  twice when it is shuffled (the inflated and the unshuffled copy, as
-  libhdf5's filters do too); writing one holds the chunk and its shuffled
-  copy. A selection decodes only the chunks it touches; one of an
-  unfiltered chunk reads only the rows it selects, through a memory map or
+  libhdf5's filters do too). Writing a filtered chunk holds its compressed
+  copy and, where it is not a contiguous run of the dataset's data (a
+  chunk past the dataset's edge is zero-padded), the raw chunk; shuffled,
+  the shuffled copy too. An unfiltered chunk is not copied when it is such
+  a run (a dataset stored as one chunk of its own shape): writing one
+  unfiltered chunk of 2^32 + 7 bytes with `with_u8_data_owned` and
+  `FileBuilder::write` peaked at 4.00 GiB resident, the data itself
+  (`write_one_chunk 4294967303 OUT owned`, tank, 2026-09-29, commit
+  `0065c6b`; see [the fix](#writing-a-4-gib-unfiltered-chunk-held-several-copies-of-it)).
+  `FileWriter::finish` (a `Vec`) holds the data and the file.
+  A selection decodes only the chunks it touches; one of an unfiltered
+  chunk reads only the rows it selects, through a memory map or
  positioned reads (`File::open_storage`): every selection of
  `unfiltered_huge_chunks_read` together read under 1 MiB. Peak resident
-  memory of `cargo test --release -p clawhdf5 --test huge_chunks_interop
-  -- --test-threads=1` with `CLAWHDF5_HUGE_CHUNKS=1` (all six tests, h5py
-  included) was 8.06 GiB (tank, 2026-09-28, commit `9143737`); selection
-  reads of the double-deflated fixture alone, 4.0 GiB.
+  memory of `cargo test --release -p clawhdf5 --features lz4,zstd --test
+  huge_chunks_interop -- --test-threads=1` with `CLAWHDF5_HUGE_CHUNKS=1`
+  (all twelve tests, h5py and h5dump included) was 10.04 GiB (tank,
+  2026-09-29, commit `0065c6b`), in `writer_huge_chunks_lz4_zstd_round_trip`
+  (the other tests, 8.2 GiB or less; the first run of the suite, six
+  tests, 8.06 GiB on 2026-09-28). Which step of that test holds 10 GiB
+  (clawhdf5's LZ4 or Zstd write or read, or hdf5plugin's) was not
+  measured separately.
 - **32-bit targets (wasm32):** such a chunk cannot be held in memory:
  reading or writing one is `FormatError::Overflow` ("exceeds the
  addressable size" / "address space"); the file still opens and lists
@@ -694,6 +708,58 @@ browser reads `[re, im]` pairs.
 `Datatype::complex_as_compound`; exhaustive `match`es over `DType` need a
 `Complex` arm. Files are unchanged; nothing needs rewriting.

+## Chunk dimensions of 2^32 or more were refused
+
+**Status:** fixed 2026-09-29 (branch `feat/huge-chunk-dims`, PR not yet
+opened); affected every release (v2.1.0 to v2.7.0). An error, never wrong
+data. Nothing for users to do but upgrade; code that matches on
+`DataLayout::Chunked::chunk_dimensions` gets `u64`s now.
+
+libhdf5 2.x writes a chunk larger than 4 GiB with layout message version
+5, whose dimensions take up to 8 bytes each, so a chunk dimension can be
+2^32 or more (a 4 GiB chunk of 1-byte elements needs one).
+`DataLayout::Chunked::chunk_dimensions` was `Vec<u32>`: such a file was
+refused when opened for reading (`InvalidChunkDimensions`, "larger than
+2^32 - 1"), and the writer refused such a chunk. Chunk dimensions are
+`u64` throughout (format crate, facade, editor, `h5rs`, Python bindings);
+a version-3 layout, whose dimensions are 4 bytes, is never written for
+them (a chunk that large always takes version 5, as in libhdf5). Tests:
+`huge_chunks_interop` (`huge_chunk_dims_list` always, over
+`fixtures/huge_chunk_dims.h5`, which libhdf5 2.0.0 wrote: `u8` chunks of
+2^32 + 7 in a Single Chunk and an Extensible Array index; with
+`CLAWHDF5_HUGE_CHUNKS=1`, reads of it and of an unfiltered sparse file
+h5py writes, and clawhdf5's own such chunks read back by h5py 3.16 and
+h5dump 2.2.0), unit tests of the 5-byte encoding, and the wasm package
+test (the file lists; reading such a chunk on wasm32 is
+`FormatError::Overflow`).
+
+## Writing a 4 GiB unfiltered chunk held several copies of it
+
+**Status:** fixed 2026-09-29 (branch `feat/huge-chunk-dims`, PR not yet
+opened); affected every release (v2.1.0 to v2.7.0). Memory only. To write
+large data with the least memory, hand the builder the data
+(`DatasetBuilder::with_u8_data_owned`) and write with
+`FileBuilder::write` (or `FileWriter::finish_with`).
+
+`FileWriter::finish` copied every chunk into a per-chunk buffer, laid the
+chunks out into one buffer per pass (two passes, the first kept), then
+copied everything into the file's buffer, and `FileBuilder::write` wrote
+that buffer out: about five times a dataset's size for one unfiltered
+chunk, six with `with_u8_data` (which copies its argument). Measured
+before and after on tank, 2026-09-29, peak resident memory of
+`write_one_chunk 1073741824 OUT <mode>` (`crates/clawhdf5/examples`, one
+unfiltered 1 GiB chunk of `u8`): `owned` 5.0 GiB before (the writer at
+`4260af4`), 1.0 GiB after (commit `0065c6b`); `slice` 6.0 GiB before,
+2.0 GiB after. With 4 GiB + 7 bytes and `owned`: 4.00 GiB after. The
+files written are byte for byte the same. Now a chunk that is a contiguous
+run of the dataset's data is borrowed (and a filtered one compressed
+straight from it), the layout refers to chunks instead of copying them,
+and `FileBuilder::write` streams the file to disk
+(`FileWriter::finish_with`); contiguous datasets are not copied either.
+Test: `writer_unfiltered_huge_chunk_round_trip` (with
+`CLAWHDF5_HUGE_CHUNKS=1`: 4 GiB + 7 bytes written and read back by
+clawhdf5, h5py 3.16 and h5dump 2.2.0; the test process peaked at 4.00 GiB).
+
 ## NetCDF-4: variables' dimensions are guessed from sizes


@@ -514,6 +514,15 @@ async function limitTests() {
      `huge chunk: ${name}`);
  }
  hc.free();
+  // Likewise chunks whose dimension is 2^32 or more (u8 chunks of 2^32 + 7).
+  const dims = pkg.open(new Uint8Array(readFileSync(join(import.meta.dirname, "..", "..", "..",
+    "crates/clawhdf5/tests/fixtures/huge_chunk_dims.h5"))));
+  eq(dims.list("/").map((e) => e.name), ["earray", "single"], "huge chunk dims: list");
+  for (const name of ["single", "earray"]) {
+    await fails(() => dims.readHyperslab(`/${name}`, [0], [4]), /exceeds this platform.s address space/,
+      `huge chunk dims: ${name}`);
+  }
+  dims.free();
 }

 // A body of `total` bytes in 64 KiB pieces, made as they are read; `pulled()`