CHANGELOG.md

@@ -111,6 +111,80 @@
   the pre-1.8 format (version-0 superblock, symbol-table groups) cannot be
   written. `docs/known-issues.md`.
 
+### Chunks of 4 GiB or more (HDF5 2.0's layout message version 5) (2026-09-28)
+- **What libhdf5 does** (read in libhdf5 2.2.0's `H5Dchunk.c`, `H5Dfarray.c`,
+  `H5Dearray.c`, `H5Dbtree2.c`, `H5Dbtree.c`, `H5Olayout.c`): a chunk of
+  more than 0xFFFFFFFF bytes requires layout message version 5, so a high
+  bound of `H5F_LIBVER_V200` ("chunk size > 4GB requires H5F_LIBVER_V200"),
+  whatever the low bound, and version 5 always uses the newer indexes.
+  Under version 5 a filtered Fixed Array, Extensible Array or v2 B-tree
+  element stores the chunk's size in "size of lengths" bytes (8); under
+  version 4 in one byte more than the chunk needs. A filtered Single Chunk
+  stores it in "size of lengths" bytes under both. Unfiltered elements
+  store no size, and the Implicit index none at all. A v1 B-tree key keeps
+  a 32-bit size: libhdf5 never writes a larger chunk there and refuses to
+  open one ("chunk size must be < 4GB with v1 b-tree index").
+- **Reading** such chunks works in every index: `ChunkInfo::chunk_size` and
+  `ChunkMapping::file_size` are now `u64` (**public API change**). The
+  Single Chunk, Implicit, Fixed and Extensible Array readers truncated an
+  unfiltered chunk's size (the read failed), and the v2 B-tree reader
+  refused any chunk over 4 GiB. Checked against a fixture libhdf5 2.0.0
+  wrote (`tests/fixtures/huge_chunks_filtered.h5`, 188 KB: a 4 GiB + 8
+  byte chunk in each filtered index, deflated twice) and a 44 GiB sparse
+  file h5py writes at test time (every unfiltered index).
+- **Selections read what they touch:** a selection of a chunked dataset with
+  a non-default fill value is read over a box of fill values from the
+  chunks it touches (new `partial_read::read_selection_filled_in`); it
+  used to decode the whole dataset. An unfiltered chunk of a file that is
+  not in memory (`File::open_storage`) is read row by row instead of whole.
+  A filtered chunk is still decoded whole.
+- **Writing:** a chunk of more than `u32::MAX` bytes gets layout message
+  version 5 and libhdf5's index element widths; the Fixed and Extensible
+  Array structures match libhdf5 2.0.0's byte for byte. New public helpers
+  `chunked_write::{layout_version_for, chunk_size_len, MAX_V4_CHUNK_BYTES}`.
+  Refused before anything is written: chunk dimensions of 2^32 or more
+  (they were cut to 32 bits) and, for such chunks, LZF, bitshuffle, bzip2,
+  Blosc and pcodec. Chunks are extracted row by row and one at a time
+  (compressed in parallel only up to 64 MiB), and a filter pipeline no
+  longer copies its input first.
+- **Deflate** compressed only the first 4 GiB - 1 bytes of a larger chunk
+  (zlib takes at most that much per call, and `Finish` ended the stream
+  there; since the one-pass deflate of 2026-09-23, in `clawhdf5-format` and
+  `clawhdf5-filters`), and it reserved the compressor's worst-case bound,
+  which flate2's Rust backends zero on every call: 4 GiB kept per 4 GiB
+  chunk. Inputs over 64 MiB now start at 1/16 of the bound and grow, and a
+  result more than 64 MiB too large is shrunk. On the read side, an
+  intermediate deflate stage (one followed by a filter other than shuffle
+  or Fletcher-32) no longer reserves the chunk's whole bound: reading the
+  double-deflated fixture peaked at 8.5 GiB, now 4.0 GiB.
+- **LZ4:** a chunk decoding to more than 256 MiB was refused ("lz4:
+  declared size exceeds limit") though the chunk size bounded it; the
+  ceiling now applies only to a decode of unknown size. A chunk of 4 GiB
+  or more is read as the registered HDF5 framing, whose 64-bit size no
+  longer starts with four zero bytes.
+- **`FileEditor`** refuses, before writing anything, to write values into
+  or to prune, fill or allocate chunks of 4 GiB or more; growing the extent
+  (late allocation) and setting attributes work. It sizes a version-5
+  filtered index element from the file's size of lengths (it assumed 8).
+- **32-bit targets:** reading or writing such a chunk is
+  `FormatError::Overflow`; `scripts/check-32bit-casts.sh` passes, and the
+  wasm package test reads the fixture and gets that error in every index.
+- Tests: `crates/clawhdf5/tests/huge_chunks_interop.rs` (index listing,
+  byte-for-byte array indexes and the editor always; with
+  `CLAWHDF5_HUGE_CHUNKS=1`, reads of every index filtered and unfiltered,
+  and a write of every index read back by clawhdf5, h5py 3.16 and — layout
+  and storage size — h5dump 2.2.0 via `CLAWHDF5_H5DUMP2`),
+  `partial_read_equivalence::fill_value_selections_match_full_reads`, unit
+  tests of the encodings (`chunked_write::tests`) and of LZ4. The opt-in
+  run (`cargo test --release -p clawhdf5 --test huge_chunks_interop --
+  --test-threads=1`) peaked at 8.06 GiB resident, h5py included (tank,
+  2026-09-28, commit `9143737`). libhdf5 2.2.0's h5dump cannot print these
+  chunks' values (its deflate filter fails on any chunk over 4 GiB,
+  libhdf5's own included); h5py 3.16 reads them. `docs/known-issues.md`: three fixed
+  entries and the open "Chunks of 4 GiB or more: limits" (chunk dimensions
+  of 2^32 or more, the refused filters, memory, the untested unfiltered
+  write).
+
 ### A dropped `FileEditor` releases its lock at once (2026-09-28)
 - `FileEditor`'s `flock` could outlive the editor for a moment when
   another thread forked to spawn a process: the child shared the locked

README.md

@@ -116,9 +116,9 @@ Limits and open issues, with dates, are in
 | **File format** | Superblock v0–v3, user blocks, v1/v2 object headers; writing the HDF5 1.10 format (default) or, with `libver_bounds(LibVer::V18, LibVer::V18)`, files HDF5 1.8 reads (checked with HDF5 1.8.23) | Metadata cache images | Writing the pre-1.8 format (version-0 superblock, symbol-table groups) |
 | **Groups and links** | Symbol-table, compact and dense groups (tested to 100 000 links), creation order, soft and hard links; writing external links | | Following external links (explicit error); user-defined links are skipped |
 | **Datatypes** | Integers and IEEE floats of every width and byte order (incl. `f16`), enums, compounds (every version, incl. HDF5 2.0's v5), arrays, fixed-length strings, opaque, complex: h5py's `{r, i}` compound (`with_complex_f64_data`) and HDF5 2.0's native class 11 (`with_native_complex_f64_data`, opt-in: only libhdf5 2.0+ reads it; reads surface it as `{r, i}`) | Variable-length strings and sequences, object references; HDF5 2.x's small floats (bfloat16, FP8 E4M3/E5M2, FP6 E2M3/E3M2, FP4 E2M1: every bit pattern decoded as libhdf5 2.2.0 decodes it) and other non-IEEE floats up to 64 bits | Writing variable-length data; writing non-IEEE floats; decoding region and attribute references; x87 long double and binary128 |
-| **Layouts and chunk indexes** | Compact, contiguous and chunked; chunk indexes single chunk, Fixed Array, Extensible Array and v2 B-tree (the writer picks one as libhdf5 does), and v1 B-tree (every chunked dataset under the 1.8 bound, split as libhdf5 splits it); fill values; resizable datasets; virtual datasets (read limits in known-issues) | The implicit chunk index (the editor also changes it) | External raw data files (explicit error) |
+| **Layouts and chunk indexes** | Compact, contiguous and chunked; chunk indexes single chunk, Fixed Array, Extensible Array and v2 B-tree (the writer picks one as libhdf5 does), and v1 B-tree (every chunked dataset under the 1.8 bound, split as libhdf5 splits it); chunks of 4 GiB or more (HDF5 2.0's layout message version 5); fill values; resizable datasets; virtual datasets (read limits in known-issues) | The implicit chunk index (the editor also changes it) | External raw data files (explicit error); chunk dimensions of 2^32 or more |
 | **Filters** | deflate (pure-Rust zlib-rs), shuffle, Fletcher-32, LZ4 (opt-in), Zstd (C, opt-in); plugins LZF, bitshuffle, bzip2, Blosc 1 | N-Bit, scale-offset, SZIP (C, opt-in); plugins Blosc2 and ZFP | Other filter IDs, unless you register a codec (`filter_registry::register_filter`) |
-| **Editing in place** | `FileEditor`: overwrite values, grow and shrink chunked datasets (every index), set attributes (compact and dense), in files from h5py or clawhdf5 | | Creating or deleting objects in an existing file; deleting attributes; new chunks in implicit indexes; VL data; filters this build cannot encode (refused before any write) |
+| **Editing in place** | `FileEditor`: overwrite values, grow and shrink chunked datasets (every index), set attributes (compact and dense), in files from h5py or clawhdf5 | | Creating or deleting objects in an existing file; deleting attributes; new chunks in implicit indexes; VL data; rewriting chunks of 4 GiB or more; filters this build cannot encode (refused before any write) |
 | **Access** | Local files (mmap or buffered), bytes in memory, any `Storage` backend, HTTP(S) and S3/GCS/Azure via `clawhdf5-remote`, SWMR reading (`File::open_swmr`, `Dataset::refresh`) | Remote files and the browser are read-only | SWMR writing; remote SWMR; MPI collective I/O (`clawhdf5-io`'s `mpi-io` reads on one rank and broadcasts) |
 | **Bindings** | Python (read, `'w'` for numeric and complex arrays, `'r+'` editing, URLs), NetCDF-4 (CF scale/offset/fill) | WebAssembly (`open(bytes)`, `openUrl`); no Zstd/SZIP/pcodec, no compound, reference, opaque, bitfield, time or VL-sequence datasets | Node.js (the package does not work; see known-issues) |
 

crates/clawhdf5/tests/fixtures/gen_huge_chunks.py

@@ -32,10 +32,11 @@ chunk still inflates to 4 GiB + 8 bytes.
 writes only the elements written (an unfiltered chunk larger than the chunk
 cache is written in place) and the file is sparse: tens of GiB long but a few
 blocks on disk. Unwritten elements read as whatever the file holds there:
-zeros. Do not put it on tmpfs, which is memory.
+zeros. Do not put it on tmpfs, which is memory. Its Single Chunk is
+allocated early (see `make`).
 
 libhdf5 holds a whole filtered chunk in memory while it writes it, so the
-`filtered` run needs about 4.5 GiB of memory (and about 3 minutes).
+`filtered` run needs about 4 GiB of memory and 100 s (tank).
 
 Generated 2026-09-28 with h5py 3.16.0 (libhdf5 2.0.0).
 """

docs/known-issues.md

@@ -15,6 +15,10 @@ Checked against `main` at `9b5803f` on 2026-09-28.
 
 | Issue | Kind | Since |
 |---|---|---|
+
+| [NetCDF-4: variables' dimensions are guessed from sizes](#netcdf-4-variables-dimensions-are-guessed-from-sizes) | **wrong metadata** (`Variable::dimensions`, pure dimension scales listed as variables; values and dimension sizes are right) | 2026-09-28 |
+| [Chunks of 4 GiB or more: limits](#chunks-of-4-gib-or-more-limits) | refused (chunk dimensions of 2^32 or more, five filters, editor rewrites), memory (a decoded chunk is held whole) | 2026-09-28 |
+| [NetCDF-4: an unlimited dimension reports size 0](#netcdf-4-an-unlimited-dimension-reports-size-0) | **wrong metadata** (dimension size; variable shapes and values are right) | 2026-09-28 |
 | [Small floats decode as libhdf5 does, not as the OCP MX specification](#small-floats-decode-as-libhdf5-does-not-as-the-ocp-mx-specification) | deliberate: libhdf5's values (FP4/FP6/FP8 E4M3 all-ones exponent is inf/NaN) | 2026-09-28 |
 | [In-place modification (`FileEditor`) limits](#in-place-modification-fileeditor-limits) | refused edits (`Error::Unsupported`), space reuse per editor, no journal | 2026-09-26 |
 | [Python in-place editing limits](#python-in-place-editing-clawhdf5filepath-r-limits) | refused writes (`NotImplementedError`), deliberate conversion differences | 2026-09-27 |
@@ -57,6 +61,10 @@ writing anything:
   message (they have no dense storage);
 - partial edge chunks stored unfiltered (`H5Pset_chunk_opts`), external
   raw data files, virtual datasets;
+- values of a dataset whose chunks are 4 GiB or more (HDF5 2.0), and a
+  resize of one that prunes, fills or allocates chunks (added 2026-09-28;
+  growing its extent under late allocation, and its attributes, work):
+  see [Chunks of 4 GiB or more](#chunks-of-4-gib-or-more-limits);
 - files with a metadata cache image, paged or persistent free-space
   management, a driver info block, or version-3 consistency flags set.
 
@@ -141,18 +149,79 @@ before anything is written. On top of them:
 read (and so the Python `ds[...]`) of contiguous data copies just the
 selected runs; of chunked data it materialises only the selection's
 bounding box when that box covers at most half the dataset
-(`partial_read`). It decodes the whole dataset and extracts the selection
-instead when:
+(`partial_read`), with the dataset's fill value where no chunk was
+written. It decodes the whole dataset and extracts the selection instead
+when:
 - the bounding box covers more than half the dataset — which a strided
   selection across a chunked dataset (`ds[::100]`) always does, although
   it may touch few chunks;
-- the dataset is compact or virtual, or has no storage;
-- it is chunked with a non-default fill value (the box path does not fill
-  unallocated chunks, so the fill-aware full read is used).
+- the dataset is compact or virtual, or has no storage.
+
+A chunked dataset with a non-default fill value was read whole for any
+selection until 2026-09-28 (`partial_read::read_selection_filled_in` fills
+the box now; `partial_read_equivalence::fill_value_selections_match_full_reads`).
 
 Values are correct in every case; this is cost only. Selections other than
 `Selection::All` also bypass the file's chunk cache.
 
+## Chunks of 4 GiB or more: limits
+
+**Status:** open (documented 2026-09-28). HDF5 2.0 writes a chunk of more
+than 0xFFFFFFFF bytes with layout message version 5 (`H5D__chunk_construct`
+in libhdf5 2.2.0: "chunk size > 4GB requires H5F_LIBVER_V200"), which
+also makes a filtered chunk index element store the chunk's size in "size
+of lengths" bytes (8) instead of one byte more than the chunk needs.
+clawhdf5 reads and writes such chunks in every index libhdf5 gives them
+(Single Chunk, Implicit, Fixed Array, Extensible Array, v2 B-tree; libhdf5
+never puts one in a v1 B-tree and refuses to open one, as clawhdf5 does).
+The [fix](#chunks-of-4-gib-or-more-could-not-be-read) is tested by
+`crates/clawhdf5/tests/huge_chunks_interop.rs`; its decoding tests are
+opt-in (`CLAWHDF5_HUGE_CHUNKS=1`, one at a time: `--test-threads=1`).
+What remains:
+- **Chunk dimensions of 2^32 or more** (which libhdf5 2.x also allows under
+  `H5F_LIBVER_V200`) are refused, when read (`InvalidChunkDimensions`)
+  and when written: `DataLayout::Chunked::chunk_dimensions` is `Vec<u32>`.
+  A 4 GiB chunk of 1-byte elements needs one.
+- **Filters the writer refuses for such a chunk** (`FilterError`, before
+  anything is written): LZF, bitshuffle, bzip2 and Blosc (their HDF5
+  filters record sizes or block lengths in 32 bits, or cannot take such a
+  buffer) and pcodec. Deflate, shuffle, Fletcher-32, LZ4 and Zstd are
+  written; only shuffle + deflate is tested end to end (LZ4's framing and
+  the 256 MiB limit it had are unit-tested; Zstd is not tested at this
+  size).
+- **Unfiltered chunks this size are written but not tested end to end:**
+  `FileBuilder` assembles the whole file in memory, several copies of the
+  chunk (well over the 12 GiB the tests may use). Their layout messages
+  are unit-tested (`chunked_write::tests::huge_chunk_layout_messages_and_index_elements`).
+- **`FileEditor`** refuses to rewrite such chunks (see
+  [its limits](#in-place-modification-fileeditor-limits)).
+- **Memory:** decoding a filtered chunk holds all of it (4 GiB and more),
+  twice when it is shuffled (the inflated and the unshuffled copy, as
+  libhdf5's filters do too); writing one holds the chunk and its shuffled
+  copy. A selection decodes only the chunks it touches; one of an
+  unfiltered chunk reads only the rows it selects, through a memory map or
+  positioned reads (`File::open_storage`): every selection of
+  `unfiltered_huge_chunks_read` together read under 1 MiB. Peak resident
+  memory of `cargo test --release -p clawhdf5 --test huge_chunks_interop
+  -- --test-threads=1` with `CLAWHDF5_HUGE_CHUNKS=1` (all six tests, h5py
+  included) was 8.06 GiB (tank, 2026-09-28, commit `9143737`); selection
+  reads of the double-deflated fixture alone, 4.0 GiB.
+- **32-bit targets (wasm32):** such a chunk cannot be held in memory:
+  reading or writing one is `FormatError::Overflow` ("exceeds the
+  addressable size" / "address space"); the file still opens and lists
+  (`examples/wasm-viewer/test/test.mjs`, "huge chunk").
+- **libhdf5 2.2.0's h5dump cannot print these values:** built here from tag
+  `2.2.0`, it fails to inflate any deflated chunk over 4 GiB, those
+  libhdf5 2.0.0 writes included ("memory allocation failed for deflate
+  uncompression", `H5Zdeflate.c`), while h5py 3.16 (libhdf5 2.0.0) reads
+  them. The tests check h5dump 2.2.0 on layouts and storage sizes only
+  (`CLAWHDF5_H5DUMP2`).
+- libhdf5 2.0.0 itself (h5py 3.16) drops a write to an unallocated,
+  unfiltered Single Chunk this large when the fill time is "never" (the
+  dataset stays unallocated); `fixtures/gen_huge_chunks.py` allocates it
+  early instead. Not a clawhdf5 issue; recorded because the generator
+  depends on it.
+
 ## HDF5 features still unsupported
 
 **Status:** open. What remains of the gaps the
@@ -564,6 +633,70 @@ under load from other builds) a freshly opened file with 8192 chunks
 per dataset reads small selections 1.2x to 2.3x slower through a version-1
 B-tree (see `BENCHMARKS.md`, "HDF5 1.8 format").
 
+## Chunks of 4 GiB or more could not be read
+
+**Status:** fixed 2026-09-28 (branch `feat/huge-chunks`); affected every
+release (v2.1.0 to v2.7.0). Errors, and cost; no wrong values were
+returned. Nothing for users to do but upgrade.
+
+HDF5 2.0 writes chunks of more than 4 GiB - 1 bytes (layout message
+version 5). `ChunkInfo::chunk_size` was a `u32`, so the Single Chunk,
+Implicit, Fixed Array and Extensible Array readers truncated an
+unfiltered chunk's size and the read failed ("incorrect chunk size
+returned from index for unfiltered chunk"); a v2 B-tree index refused any
+such chunk ("chunk larger than 4 GiB"), filtered or not. Filtered chunks
+in the other indexes read, because their stored sizes were small. A
+selection of a chunked dataset with a non-default fill value decoded the
+whole dataset (8 GiB of output for the fixture's 2-D dataset), and a
+selection of an unfiltered chunk fetched the whole chunk from a file that
+is not in memory. `ChunkInfo::chunk_size` and `ChunkMapping::file_size` are now
+`u64`; the selection path fills its box with the fill value and reads an
+unfiltered chunk's rows only. Tests: `huge_chunks_interop`
+(`filtered_huge_chunk_indexes_list` always; with `CLAWHDF5_HUGE_CHUNKS=1`
+`filtered_huge_chunks_read`, over a fixture libhdf5 2.0.0 wrote, and
+`unfiltered_huge_chunks_read`, over a 44 GiB sparse file h5py writes at
+test time). Limits that remain:
+[Chunks of 4 GiB or more](#chunks-of-4-gib-or-more-limits).
+
+## Chunks of 4 GiB or more were written unreadable
+
+**Status:** fixed 2026-09-28 (branch `feat/huge-chunks`). The deflate
+truncation was before any release (the one-pass deflate dates from
+2026-09-23); the rest affected every release (v2.1.0 to v2.7.0). Files
+clawhdf5 wrote with a chunk of 4 GiB or more should be written again.
+
+The writer gave such a chunk layout message version 4 (which libhdf5
+before 2.0 cannot read, and which libhdf5 2.x never writes for it; whether
+2.x reads what clawhdf5 wrote was not checked), cut a chunk dimension of
+2^32 or more to 32 bits, and, since 2026-09-23, deflated only the first
+4 GiB - 1 bytes of the chunk (zlib takes at most that much per call and
+`Finish` ended the stream there): the chunk failed to decode ("decoded to
+4294967295 bytes, expected 4294967304"). It now writes layout version 5
+with libhdf5's index element widths (the Fixed and Extensible Array
+structures match libhdf5 2.0.0's byte for byte:
+`huge_chunk_array_indexes_match_libhdf5`), refuses chunk dimensions of
+2^32 or more and the filters that cannot take such a chunk, deflates the
+whole chunk, and no longer keeps the compressor's worst-case bound (4 GiB
+of zeroed memory for a chunk that deflates to 4 MiB): writing the four
+datasets of `writer_huge_chunks_round_trip` went past the tests' 12 GiB
+cap, and compressing one shuffled 4 GiB chunk now peaks at 4.3 GiB
+(tank, 2026-09-28). h5py 3.16 reads what it writes.
+
+## LZ4 chunks larger than 256 MiB were refused
+
+**Status:** fixed 2026-09-28 (branch `feat/huge-chunks`); affected every
+release (v2.1.0 to v2.7.0). An error, never wrong data. Nothing for users
+to do but upgrade.
+
+The LZ4 decoder refused a chunk that decodes to more than 256 MiB
+("lz4: declared size exceeds limit") even when the dataset's chunk size
+bounded it; that ceiling is meant for a decode whose size is unknown, and
+now applies only then (deflate never had it). A chunk of 4 GiB or more is
+always read as the registered HDF5 framing, whose 64-bit size then no
+longer starts with four zero bytes. Tests:
+`filters::tests::lz4_chunks_over_256_mib_decode`,
+`lz4_chunks_of_4_gib_use_the_registered_framing`.
+
 ## A dropped `FileEditor` could keep its file locked for a moment
 
 **Status:** fixed 2026-09-28 (#23), before any release