diff --git a/CHANGELOG.md b/CHANGELOG.md index 95b763c..88e255b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -111,6 +111,80 @@ the pre-1.8 format (version-0 superblock, symbol-table groups) cannot be written. `docs/known-issues.md`. +### Chunks of 4 GiB or more (HDF5 2.0's layout message version 5) (2026-09-28) +- **What libhdf5 does** (read in libhdf5 2.2.0's `H5Dchunk.c`, `H5Dfarray.c`, + `H5Dearray.c`, `H5Dbtree2.c`, `H5Dbtree.c`, `H5Olayout.c`): a chunk of + more than 0xFFFFFFFF bytes requires layout message version 5, so a high + bound of `H5F_LIBVER_V200` ("chunk size > 4GB requires H5F_LIBVER_V200"), + whatever the low bound, and version 5 always uses the newer indexes. + Under version 5 a filtered Fixed Array, Extensible Array or v2 B-tree + element stores the chunk's size in "size of lengths" bytes (8); under + version 4 in one byte more than the chunk needs. A filtered Single Chunk + stores it in "size of lengths" bytes under both. Unfiltered elements + store no size, and the Implicit index none at all. A v1 B-tree key keeps + a 32-bit size: libhdf5 never writes a larger chunk there and refuses to + open one ("chunk size must be < 4GB with v1 b-tree index"). +- **Reading** such chunks works in every index: `ChunkInfo::chunk_size` and + `ChunkMapping::file_size` are now `u64` (**public API change**). The + Single Chunk, Implicit, Fixed and Extensible Array readers truncated an + unfiltered chunk's size (the read failed), and the v2 B-tree reader + refused any chunk over 4 GiB. Checked against a fixture libhdf5 2.0.0 + wrote (`tests/fixtures/huge_chunks_filtered.h5`, 188 KB: a 4 GiB + 8 + byte chunk in each filtered index, deflated twice) and a 44 GiB sparse + file h5py writes at test time (every unfiltered index). +- **Selections read what they touch:** a selection of a chunked dataset with + a non-default fill value is read over a box of fill values from the + chunks it touches (new `partial_read::read_selection_filled_in`); it + used to decode the whole dataset. An unfiltered chunk of a file that is + not in memory (`File::open_storage`) is read row by row instead of whole. + A filtered chunk is still decoded whole. +- **Writing:** a chunk of more than `u32::MAX` bytes gets layout message + version 5 and libhdf5's index element widths; the Fixed and Extensible + Array structures match libhdf5 2.0.0's byte for byte. New public helpers + `chunked_write::{layout_version_for, chunk_size_len, MAX_V4_CHUNK_BYTES}`. + Refused before anything is written: chunk dimensions of 2^32 or more + (they were cut to 32 bits) and, for such chunks, LZF, bitshuffle, bzip2, + Blosc and pcodec. Chunks are extracted row by row and one at a time + (compressed in parallel only up to 64 MiB), and a filter pipeline no + longer copies its input first. +- **Deflate** compressed only the first 4 GiB - 1 bytes of a larger chunk + (zlib takes at most that much per call, and `Finish` ended the stream + there; since the one-pass deflate of 2026-09-23, in `clawhdf5-format` and + `clawhdf5-filters`), and it reserved the compressor's worst-case bound, + which flate2's Rust backends zero on every call: 4 GiB kept per 4 GiB + chunk. Inputs over 64 MiB now start at 1/16 of the bound and grow, and a + result more than 64 MiB too large is shrunk. On the read side, an + intermediate deflate stage (one followed by a filter other than shuffle + or Fletcher-32) no longer reserves the chunk's whole bound: reading the + double-deflated fixture peaked at 8.5 GiB, now 4.0 GiB. +- **LZ4:** a chunk decoding to more than 256 MiB was refused ("lz4: + declared size exceeds limit") though the chunk size bounded it; the + ceiling now applies only to a decode of unknown size. A chunk of 4 GiB + or more is read as the registered HDF5 framing, whose 64-bit size no + longer starts with four zero bytes. +- **`FileEditor`** refuses, before writing anything, to write values into + or to prune, fill or allocate chunks of 4 GiB or more; growing the extent + (late allocation) and setting attributes work. It sizes a version-5 + filtered index element from the file's size of lengths (it assumed 8). +- **32-bit targets:** reading or writing such a chunk is + `FormatError::Overflow`; `scripts/check-32bit-casts.sh` passes, and the + wasm package test reads the fixture and gets that error in every index. +- Tests: `crates/clawhdf5/tests/huge_chunks_interop.rs` (index listing, + byte-for-byte array indexes and the editor always; with + `CLAWHDF5_HUGE_CHUNKS=1`, reads of every index filtered and unfiltered, + and a write of every index read back by clawhdf5, h5py 3.16 and — layout + and storage size — h5dump 2.2.0 via `CLAWHDF5_H5DUMP2`), + `partial_read_equivalence::fill_value_selections_match_full_reads`, unit + tests of the encodings (`chunked_write::tests`) and of LZ4. The opt-in + run (`cargo test --release -p clawhdf5 --test huge_chunks_interop -- + --test-threads=1`) peaked at 8.06 GiB resident, h5py included (tank, + 2026-09-28, commit `9143737`). libhdf5 2.2.0's h5dump cannot print these + chunks' values (its deflate filter fails on any chunk over 4 GiB, + libhdf5's own included); h5py 3.16 reads them. `docs/known-issues.md`: three fixed + entries and the open "Chunks of 4 GiB or more: limits" (chunk dimensions + of 2^32 or more, the refused filters, memory, the untested unfiltered + write). + ### A dropped `FileEditor` releases its lock at once (2026-09-28) - `FileEditor`'s `flock` could outlive the editor for a moment when another thread forked to spawn a process: the child shared the locked diff --git a/README.md b/README.md index a87b5a2..d7f61da 100644 --- a/README.md +++ b/README.md @@ -116,9 +116,9 @@ Limits and open issues, with dates, are in | **File format** | Superblock v0–v3, user blocks, v1/v2 object headers; writing the HDF5 1.10 format (default) or, with `libver_bounds(LibVer::V18, LibVer::V18)`, files HDF5 1.8 reads (checked with HDF5 1.8.23) | Metadata cache images | Writing the pre-1.8 format (version-0 superblock, symbol-table groups) | | **Groups and links** | Symbol-table, compact and dense groups (tested to 100 000 links), creation order, soft and hard links; writing external links | | Following external links (explicit error); user-defined links are skipped | | **Datatypes** | Integers and IEEE floats of every width and byte order (incl. `f16`), enums, compounds (every version, incl. HDF5 2.0's v5), arrays, fixed-length strings, opaque, complex: h5py's `{r, i}` compound (`with_complex_f64_data`) and HDF5 2.0's native class 11 (`with_native_complex_f64_data`, opt-in: only libhdf5 2.0+ reads it; reads surface it as `{r, i}`) | Variable-length strings and sequences, object references; HDF5 2.x's small floats (bfloat16, FP8 E4M3/E5M2, FP6 E2M3/E3M2, FP4 E2M1: every bit pattern decoded as libhdf5 2.2.0 decodes it) and other non-IEEE floats up to 64 bits | Writing variable-length data; writing non-IEEE floats; decoding region and attribute references; x87 long double and binary128 | -| **Layouts and chunk indexes** | Compact, contiguous and chunked; chunk indexes single chunk, Fixed Array, Extensible Array and v2 B-tree (the writer picks one as libhdf5 does), and v1 B-tree (every chunked dataset under the 1.8 bound, split as libhdf5 splits it); fill values; resizable datasets; virtual datasets (read limits in known-issues) | The implicit chunk index (the editor also changes it) | External raw data files (explicit error) | +| **Layouts and chunk indexes** | Compact, contiguous and chunked; chunk indexes single chunk, Fixed Array, Extensible Array and v2 B-tree (the writer picks one as libhdf5 does), and v1 B-tree (every chunked dataset under the 1.8 bound, split as libhdf5 splits it); chunks of 4 GiB or more (HDF5 2.0's layout message version 5); fill values; resizable datasets; virtual datasets (read limits in known-issues) | The implicit chunk index (the editor also changes it) | External raw data files (explicit error); chunk dimensions of 2^32 or more | | **Filters** | deflate (pure-Rust zlib-rs), shuffle, Fletcher-32, LZ4 (opt-in), Zstd (C, opt-in); plugins LZF, bitshuffle, bzip2, Blosc 1 | N-Bit, scale-offset, SZIP (C, opt-in); plugins Blosc2 and ZFP | Other filter IDs, unless you register a codec (`filter_registry::register_filter`) | -| **Editing in place** | `FileEditor`: overwrite values, grow and shrink chunked datasets (every index), set attributes (compact and dense), in files from h5py or clawhdf5 | | Creating or deleting objects in an existing file; deleting attributes; new chunks in implicit indexes; VL data; filters this build cannot encode (refused before any write) | +| **Editing in place** | `FileEditor`: overwrite values, grow and shrink chunked datasets (every index), set attributes (compact and dense), in files from h5py or clawhdf5 | | Creating or deleting objects in an existing file; deleting attributes; new chunks in implicit indexes; VL data; rewriting chunks of 4 GiB or more; filters this build cannot encode (refused before any write) | | **Access** | Local files (mmap or buffered), bytes in memory, any `Storage` backend, HTTP(S) and S3/GCS/Azure via `clawhdf5-remote`, SWMR reading (`File::open_swmr`, `Dataset::refresh`) | Remote files and the browser are read-only | SWMR writing; remote SWMR; MPI collective I/O (`clawhdf5-io`'s `mpi-io` reads on one rank and broadcasts) | | **Bindings** | Python (read, `'w'` for numeric and complex arrays, `'r+'` editing, URLs), NetCDF-4 (CF scale/offset/fill) | WebAssembly (`open(bytes)`, `openUrl`); no Zstd/SZIP/pcodec, no compound, reference, opaque, bitfield, time or VL-sequence datasets | Node.js (the package does not work; see known-issues) | diff --git a/crates/clawhdf5/tests/fixtures/gen_huge_chunks.py b/crates/clawhdf5/tests/fixtures/gen_huge_chunks.py index 16c18c4..f901fd5 100644 --- a/crates/clawhdf5/tests/fixtures/gen_huge_chunks.py +++ b/crates/clawhdf5/tests/fixtures/gen_huge_chunks.py @@ -32,10 +32,11 @@ chunk still inflates to 4 GiB + 8 bytes. writes only the elements written (an unfiltered chunk larger than the chunk cache is written in place) and the file is sparse: tens of GiB long but a few blocks on disk. Unwritten elements read as whatever the file holds there: -zeros. Do not put it on tmpfs, which is memory. +zeros. Do not put it on tmpfs, which is memory. Its Single Chunk is +allocated early (see `make`). libhdf5 holds a whole filtered chunk in memory while it writes it, so the -`filtered` run needs about 4.5 GiB of memory (and about 3 minutes). +`filtered` run needs about 4 GiB of memory and 100 s (tank). Generated 2026-09-28 with h5py 3.16.0 (libhdf5 2.0.0). """ diff --git a/docs/known-issues.md b/docs/known-issues.md index 9d932d0..89e0f1c 100644 --- a/docs/known-issues.md +++ b/docs/known-issues.md @@ -15,6 +15,10 @@ Checked against `main` at `9b5803f` on 2026-09-28. | Issue | Kind | Since | |---|---|---| + +| [NetCDF-4: variables' dimensions are guessed from sizes](#netcdf-4-variables-dimensions-are-guessed-from-sizes) | **wrong metadata** (`Variable::dimensions`, pure dimension scales listed as variables; values and dimension sizes are right) | 2026-09-28 | +| [Chunks of 4 GiB or more: limits](#chunks-of-4-gib-or-more-limits) | refused (chunk dimensions of 2^32 or more, five filters, editor rewrites), memory (a decoded chunk is held whole) | 2026-09-28 | +| [NetCDF-4: an unlimited dimension reports size 0](#netcdf-4-an-unlimited-dimension-reports-size-0) | **wrong metadata** (dimension size; variable shapes and values are right) | 2026-09-28 | | [Small floats decode as libhdf5 does, not as the OCP MX specification](#small-floats-decode-as-libhdf5-does-not-as-the-ocp-mx-specification) | deliberate: libhdf5's values (FP4/FP6/FP8 E4M3 all-ones exponent is inf/NaN) | 2026-09-28 | | [In-place modification (`FileEditor`) limits](#in-place-modification-fileeditor-limits) | refused edits (`Error::Unsupported`), space reuse per editor, no journal | 2026-09-26 | | [Python in-place editing limits](#python-in-place-editing-clawhdf5filepath-r-limits) | refused writes (`NotImplementedError`), deliberate conversion differences | 2026-09-27 | @@ -57,6 +61,10 @@ writing anything: message (they have no dense storage); - partial edge chunks stored unfiltered (`H5Pset_chunk_opts`), external raw data files, virtual datasets; +- values of a dataset whose chunks are 4 GiB or more (HDF5 2.0), and a + resize of one that prunes, fills or allocates chunks (added 2026-09-28; + growing its extent under late allocation, and its attributes, work): + see [Chunks of 4 GiB or more](#chunks-of-4-gib-or-more-limits); - files with a metadata cache image, paged or persistent free-space management, a driver info block, or version-3 consistency flags set. @@ -141,18 +149,79 @@ before anything is written. On top of them: read (and so the Python `ds[...]`) of contiguous data copies just the selected runs; of chunked data it materialises only the selection's bounding box when that box covers at most half the dataset -(`partial_read`). It decodes the whole dataset and extracts the selection -instead when: +(`partial_read`), with the dataset's fill value where no chunk was +written. It decodes the whole dataset and extracts the selection instead +when: - the bounding box covers more than half the dataset — which a strided selection across a chunked dataset (`ds[::100]`) always does, although it may touch few chunks; -- the dataset is compact or virtual, or has no storage; -- it is chunked with a non-default fill value (the box path does not fill - unallocated chunks, so the fill-aware full read is used). +- the dataset is compact or virtual, or has no storage. + +A chunked dataset with a non-default fill value was read whole for any +selection until 2026-09-28 (`partial_read::read_selection_filled_in` fills +the box now; `partial_read_equivalence::fill_value_selections_match_full_reads`). Values are correct in every case; this is cost only. Selections other than `Selection::All` also bypass the file's chunk cache. +## Chunks of 4 GiB or more: limits + +**Status:** open (documented 2026-09-28). HDF5 2.0 writes a chunk of more +than 0xFFFFFFFF bytes with layout message version 5 (`H5D__chunk_construct` +in libhdf5 2.2.0: "chunk size > 4GB requires H5F_LIBVER_V200"), which +also makes a filtered chunk index element store the chunk's size in "size +of lengths" bytes (8) instead of one byte more than the chunk needs. +clawhdf5 reads and writes such chunks in every index libhdf5 gives them +(Single Chunk, Implicit, Fixed Array, Extensible Array, v2 B-tree; libhdf5 +never puts one in a v1 B-tree and refuses to open one, as clawhdf5 does). +The [fix](#chunks-of-4-gib-or-more-could-not-be-read) is tested by +`crates/clawhdf5/tests/huge_chunks_interop.rs`; its decoding tests are +opt-in (`CLAWHDF5_HUGE_CHUNKS=1`, one at a time: `--test-threads=1`). +What remains: +- **Chunk dimensions of 2^32 or more** (which libhdf5 2.x also allows under + `H5F_LIBVER_V200`) are refused, when read (`InvalidChunkDimensions`) + and when written: `DataLayout::Chunked::chunk_dimensions` is `Vec`. + A 4 GiB chunk of 1-byte elements needs one. +- **Filters the writer refuses for such a chunk** (`FilterError`, before + anything is written): LZF, bitshuffle, bzip2 and Blosc (their HDF5 + filters record sizes or block lengths in 32 bits, or cannot take such a + buffer) and pcodec. Deflate, shuffle, Fletcher-32, LZ4 and Zstd are + written; only shuffle + deflate is tested end to end (LZ4's framing and + the 256 MiB limit it had are unit-tested; Zstd is not tested at this + size). +- **Unfiltered chunks this size are written but not tested end to end:** + `FileBuilder` assembles the whole file in memory, several copies of the + chunk (well over the 12 GiB the tests may use). Their layout messages + are unit-tested (`chunked_write::tests::huge_chunk_layout_messages_and_index_elements`). +- **`FileEditor`** refuses to rewrite such chunks (see + [its limits](#in-place-modification-fileeditor-limits)). +- **Memory:** decoding a filtered chunk holds all of it (4 GiB and more), + twice when it is shuffled (the inflated and the unshuffled copy, as + libhdf5's filters do too); writing one holds the chunk and its shuffled + copy. A selection decodes only the chunks it touches; one of an + unfiltered chunk reads only the rows it selects, through a memory map or + positioned reads (`File::open_storage`): every selection of + `unfiltered_huge_chunks_read` together read under 1 MiB. Peak resident + memory of `cargo test --release -p clawhdf5 --test huge_chunks_interop + -- --test-threads=1` with `CLAWHDF5_HUGE_CHUNKS=1` (all six tests, h5py + included) was 8.06 GiB (tank, 2026-09-28, commit `9143737`); selection + reads of the double-deflated fixture alone, 4.0 GiB. +- **32-bit targets (wasm32):** such a chunk cannot be held in memory: + reading or writing one is `FormatError::Overflow` ("exceeds the + addressable size" / "address space"); the file still opens and lists + (`examples/wasm-viewer/test/test.mjs`, "huge chunk"). +- **libhdf5 2.2.0's h5dump cannot print these values:** built here from tag + `2.2.0`, it fails to inflate any deflated chunk over 4 GiB, those + libhdf5 2.0.0 writes included ("memory allocation failed for deflate + uncompression", `H5Zdeflate.c`), while h5py 3.16 (libhdf5 2.0.0) reads + them. The tests check h5dump 2.2.0 on layouts and storage sizes only + (`CLAWHDF5_H5DUMP2`). +- libhdf5 2.0.0 itself (h5py 3.16) drops a write to an unallocated, + unfiltered Single Chunk this large when the fill time is "never" (the + dataset stays unallocated); `fixtures/gen_huge_chunks.py` allocates it + early instead. Not a clawhdf5 issue; recorded because the generator + depends on it. + ## HDF5 features still unsupported **Status:** open. What remains of the gaps the @@ -564,6 +633,70 @@ under load from other builds) a freshly opened file with 8192 chunks per dataset reads small selections 1.2x to 2.3x slower through a version-1 B-tree (see `BENCHMARKS.md`, "HDF5 1.8 format"). +## Chunks of 4 GiB or more could not be read + +**Status:** fixed 2026-09-28 (branch `feat/huge-chunks`); affected every +release (v2.1.0 to v2.7.0). Errors, and cost; no wrong values were +returned. Nothing for users to do but upgrade. + +HDF5 2.0 writes chunks of more than 4 GiB - 1 bytes (layout message +version 5). `ChunkInfo::chunk_size` was a `u32`, so the Single Chunk, +Implicit, Fixed Array and Extensible Array readers truncated an +unfiltered chunk's size and the read failed ("incorrect chunk size +returned from index for unfiltered chunk"); a v2 B-tree index refused any +such chunk ("chunk larger than 4 GiB"), filtered or not. Filtered chunks +in the other indexes read, because their stored sizes were small. A +selection of a chunked dataset with a non-default fill value decoded the +whole dataset (8 GiB of output for the fixture's 2-D dataset), and a +selection of an unfiltered chunk fetched the whole chunk from a file that +is not in memory. `ChunkInfo::chunk_size` and `ChunkMapping::file_size` are now +`u64`; the selection path fills its box with the fill value and reads an +unfiltered chunk's rows only. Tests: `huge_chunks_interop` +(`filtered_huge_chunk_indexes_list` always; with `CLAWHDF5_HUGE_CHUNKS=1` +`filtered_huge_chunks_read`, over a fixture libhdf5 2.0.0 wrote, and +`unfiltered_huge_chunks_read`, over a 44 GiB sparse file h5py writes at +test time). Limits that remain: +[Chunks of 4 GiB or more](#chunks-of-4-gib-or-more-limits). + +## Chunks of 4 GiB or more were written unreadable + +**Status:** fixed 2026-09-28 (branch `feat/huge-chunks`). The deflate +truncation was before any release (the one-pass deflate dates from +2026-09-23); the rest affected every release (v2.1.0 to v2.7.0). Files +clawhdf5 wrote with a chunk of 4 GiB or more should be written again. + +The writer gave such a chunk layout message version 4 (which libhdf5 +before 2.0 cannot read, and which libhdf5 2.x never writes for it; whether +2.x reads what clawhdf5 wrote was not checked), cut a chunk dimension of +2^32 or more to 32 bits, and, since 2026-09-23, deflated only the first +4 GiB - 1 bytes of the chunk (zlib takes at most that much per call and +`Finish` ended the stream there): the chunk failed to decode ("decoded to +4294967295 bytes, expected 4294967304"). It now writes layout version 5 +with libhdf5's index element widths (the Fixed and Extensible Array +structures match libhdf5 2.0.0's byte for byte: +`huge_chunk_array_indexes_match_libhdf5`), refuses chunk dimensions of +2^32 or more and the filters that cannot take such a chunk, deflates the +whole chunk, and no longer keeps the compressor's worst-case bound (4 GiB +of zeroed memory for a chunk that deflates to 4 MiB): writing the four +datasets of `writer_huge_chunks_round_trip` went past the tests' 12 GiB +cap, and compressing one shuffled 4 GiB chunk now peaks at 4.3 GiB +(tank, 2026-09-28). h5py 3.16 reads what it writes. + +## LZ4 chunks larger than 256 MiB were refused + +**Status:** fixed 2026-09-28 (branch `feat/huge-chunks`); affected every +release (v2.1.0 to v2.7.0). An error, never wrong data. Nothing for users +to do but upgrade. + +The LZ4 decoder refused a chunk that decodes to more than 256 MiB +("lz4: declared size exceeds limit") even when the dataset's chunk size +bounded it; that ceiling is meant for a decode whose size is unknown, and +now applies only then (deflate never had it). A chunk of 4 GiB or more is +always read as the registered HDF5 framing, whose 64-bit size then no +longer starts with four zero bytes. Tests: +`filters::tests::lz4_chunks_over_256_mib_decode`, +`lz4_chunks_of_4_gib_use_the_registered_framing`. + ## A dropped `FileEditor` could keep its file locked for a moment **Status:** fixed 2026-09-28 (#23), before any release