diff --git a/CHANGELOG.md b/CHANGELOG.md index 2d9e329..7831789 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -119,6 +119,65 @@ opens in HDF5 1.8.23's h5dump and in h5py). `test.mjs` no longer hard-codes the fixture's length. +### Chunk dimensions of 2^32 or more; 4 GiB chunks written without copies (2026-09-29) +- **Chunk dimensions of 2^32 or more** (libhdf5 2.x writes them with + layout message version 5, up to 8 bytes per dimension; a 4 GiB chunk of + 1-byte elements needs one) are read and written. They were refused + (`InvalidChunkDimensions`). **Public API changes:** + `DataLayout::Chunked::chunk_dimensions` is `Vec` (was `Vec`), + and the `chunk_dimensions` argument of `chunked_read::{collect_chunk_info_checked, + collect_chunk_info_checked_in, generate_implicit_chunks_in_grid}`, + `fixed_array::{read_fixed_array_chunks, read_fixed_array_chunks_in}`, + `extensible_array::{read_extensible_array_chunks, + read_extensible_array_chunks_in}`, and the `chunk_dims` argument of + `chunked_write::serialize_v4_single_chunk_pub`, are `&[u64]` (were + `&[u32]`). A chunk whose size overflows 64 bits is refused when a dataset is read + (`InvalidChunkDimensions`), and the writer refuses a chunk dimension of + 0 up front. A version-3 layout (4-byte dimensions) is never written for + such a chunk: over 4 GiB, it always takes version 5, as in libhdf5. +- **Writing without copies:** `FileWriter` no longer copies chunks into a + buffer per pass and then into the file's buffer. A chunk that is a + contiguous run of the dataset's data (a dataset stored as one chunk of + its shape, blocks of whole rows) is borrowed when unfiltered and + compressed straight from it when filtered, and contiguous datasets are + not copied. New `FileWriter::finish_with(put)` hands the file out piece + by piece; `FileBuilder::write` uses it (through a 1 MiB `BufWriter`, + still to a temporary file renamed into place), so a file is never + assembled in memory. New `DatasetBuilder::with_u8_data_owned(Vec)` + takes the data without the copy `with_u8_data` makes. Peak resident + memory of one unfiltered 1 GiB chunk of `u8` + (`cargo run --release -p clawhdf5 --example write_one_chunk -- + 1073741824 OUT owned|slice`, tank, 2026-09-29): 5.0 GiB before (the + writer of `4260af4`), 1.0 GiB after with `owned`; 6.0 and 2.0 GiB with + `slice`; 4 GiB + 7 bytes `owned`, 4.00 GiB. Same bytes written. Write + speed was not re-measured (the machine was not idle): the A/B of + `FileBuilder::write` and `finish` on small and large files is still to + be run. +- `chunked_write::extract_chunk` (behind `split_into_chunks`) no longer + panics when the data is shorter than the shape; the missing part is + zeros, as it was meant to be. +- Tests: `crates/clawhdf5/tests/fixtures/huge_chunk_dims.h5` (31 KB, + libhdf5 2.0.0 via h5py 3.16, `gen_huge_chunks.py dims`: `u8` chunks of + 2^32 + 7 deflated twice, Single Chunk and Extensible Array), and in + `huge_chunks_interop`: `huge_chunk_dims_list` (always), and with + `CLAWHDF5_HUGE_CHUNKS=1` `huge_chunk_dims_read`, + `huge_chunk_dims_unfiltered_read` (a sparse file h5py writes, Single + Chunk and Fixed Array, mmap and positioned reads), + `writer_huge_chunk_dims_round_trip` (deflated, read back by clawhdf5, + h5py 3.16 and — layouts — h5dump 2.2.0), + `writer_unfiltered_huge_chunk_round_trip` (an unfiltered 4 GiB + 7 byte + chunk: clawhdf5, h5py, and h5dump 2.2.0 printing values past 2^32) and + `writer_huge_chunks_lz4_zstd_round_trip` (with the `lz4`/`zstd` + features; h5py through hdf5plugin 7.1.0). The whole opt-in suite + (`cargo test --release -p clawhdf5 --features lz4,zstd --test + huge_chunks_interop -- --test-threads=1`, h5py and h5dump 2.2.0 + included) passed with a peak of 10.04 GiB resident (tank, 2026-09-29, + commit `0065c6b`). Unit tests: 5-byte dimension encoding, contiguous + chunk detection, `finish_with` against `finish`. The wasm package test + lists the new fixture and gets `FormatError::Overflow` reading it. + `docs/known-issues.md`: two fixed entries; the open "Chunks of 4 GiB or + more: limits" keeps the refused filters, the editor and memory. + ### NetCDF-4: variables' dimensions come from the file (2026-09-28) - `clawhdf5-netcdf4` gave each variable the first unused dimension of equal size (else an anonymous `dim_`), so a variable on an unlimited diff --git a/README.md b/README.md index 9acde11..567e230 100644 --- a/README.md +++ b/README.md @@ -102,7 +102,8 @@ writes 85.2 µs vs 877 µs (10.3x); 64 group creates 130 µs vs 1.37 ms (10.6x); a 512×512 `f32` chunked deflate-6 write 1.44 ms vs 65.0 ms (re-measured 2026-09-23 with the pure-Rust deflate: 1.46 ms vs 51.4 ms, 35x); a 100K `f32` sequential write is a tie. The writer (`FileBuilder`) -assembles a file in memory and writes it once, which is part of that +lays out the whole file before writing it (when these were measured, in +one buffer in memory), which is part of that difference; read the caveats in [BENCHMARKS.md](BENCHMARKS.md#caveats) before quoting these. @@ -116,7 +117,7 @@ Limits and open issues, with dates, are in | **File format** | Superblock v0–v3, user blocks, v1/v2 object headers; writing the HDF5 1.10 format (default) or, with `libver_bounds(LibVer::V18, LibVer::V18)`, files HDF5 1.8 reads (checked with HDF5 1.8.23) | Metadata cache images | Writing the pre-1.8 format (version-0 superblock, symbol-table groups) | | **Groups and links** | Symbol-table, compact and dense groups (tested to 100 000 links), creation order, soft and hard links; writing external links | | Following external links (explicit error); user-defined links are skipped | | **Datatypes** | Integers and IEEE floats of every width and byte order (incl. `f16`), enums, compounds (every version, incl. HDF5 2.0's v5), arrays, fixed-length strings, opaque, complex: h5py's `{r, i}` compound (`with_complex_f64_data`) and HDF5 2.0's native class 11 (`with_native_complex_f64_data`, opt-in: only libhdf5 2.0+ reads it; read as its own type, `DType::Complex`, and printed by `h5rs` as h5dump 2.x prints it) | Variable-length strings and sequences, object references; HDF5 2.x's small floats (bfloat16, FP8 E4M3/E5M2, FP6 E2M3/E3M2, FP4 E2M1: every bit pattern decoded as libhdf5 2.2.0 decodes it) and other non-IEEE floats up to 64 bits | Writing variable-length data; writing non-IEEE floats; decoding region and attribute references; x87 long double and binary128 | -| **Layouts and chunk indexes** | Compact, contiguous and chunked; chunk indexes single chunk, Fixed Array, Extensible Array and v2 B-tree (the writer picks one as libhdf5 does), and v1 B-tree (every chunked dataset under the 1.8 bound, split as libhdf5 splits it); chunks of 4 GiB or more (HDF5 2.0's layout message version 5); fill values; resizable datasets; virtual datasets (read limits in known-issues) | The implicit chunk index (the editor also changes it) | External raw data files (explicit error); chunk dimensions of 2^32 or more | +| **Layouts and chunk indexes** | Compact, contiguous and chunked; chunk indexes single chunk, Fixed Array, Extensible Array and v2 B-tree (the writer picks one as libhdf5 does), and v1 B-tree (every chunked dataset under the 1.8 bound, split as libhdf5 splits it); chunks of 4 GiB or more and chunk dimensions of 2^32 or more (HDF5 2.0's layout message version 5); fill values; resizable datasets; virtual datasets (read limits in known-issues) | The implicit chunk index (the editor also changes it) | External raw data files (explicit error) | | **Filters** | deflate (pure-Rust zlib-rs), shuffle, Fletcher-32, LZ4 (opt-in), Zstd (C, opt-in); plugins LZF, bitshuffle, bzip2, Blosc 1 | N-Bit, scale-offset, SZIP (C, opt-in); plugins Blosc2 and ZFP | Other filter IDs, unless you register a codec (`filter_registry::register_filter`) | | **Editing in place** | `FileEditor`: overwrite values, grow and shrink chunked datasets (every index), set attributes (compact and dense), in files from h5py or clawhdf5 | | Creating or deleting objects in an existing file; deleting attributes; new chunks in implicit indexes; VL data; rewriting chunks of 4 GiB or more; filters this build cannot encode (refused before any write) | | **Access** | Local files (mmap or buffered), bytes in memory, any `Storage` backend, HTTP(S) and S3/GCS/Azure via `clawhdf5-remote`, SWMR reading (`File::open_swmr`, `Dataset::refresh`) | Remote files and the browser are read-only | SWMR writing; remote SWMR; MPI collective I/O (`clawhdf5-io`'s `mpi-io` reads on one rank and broadcasts) | diff --git a/docs/known-issues.md b/docs/known-issues.md index 9c17945..2ef9b02 100644 --- a/docs/known-issues.md +++ b/docs/known-issues.md @@ -17,6 +17,11 @@ Checked against `main` at `9b5803f` on 2026-09-28. |---|---|---| | [NetCDF-4: differences from netCDF-C](#netcdf-4-differences-from-netcdf-c) | deliberate (floats of other than 4 or 8 bytes, axes netCDF-C leaves without a dimension), and 9 corpus files (external links, values the HDF5 reader refuses) | 2026-09-29 | | [Chunks of 4 GiB or more: limits](#chunks-of-4-gib-or-more-limits) | refused (chunk dimensions of 2^32 or more, five filters, editor rewrites), memory (a decoded chunk is held whole) | 2026-09-28 | + + +| [NetCDF-4: variables' dimensions are guessed from sizes](#netcdf-4-variables-dimensions-are-guessed-from-sizes) | **wrong metadata** (`Variable::dimensions`, pure dimension scales listed as variables; values and dimension sizes are right) | 2026-09-28 | +| [Chunks of 4 GiB or more: limits](#chunks-of-4-gib-or-more-limits) | refused (five filters, editor rewrites), memory (a decoded chunk is held whole) | 2026-09-28 | +| [NetCDF-4: an unlimited dimension reports size 0](#netcdf-4-an-unlimited-dimension-reports-size-0) | **wrong metadata** (dimension size; variable shapes and values are right) | 2026-09-28 | | [Small floats decode as libhdf5 does, not as the OCP MX specification](#small-floats-decode-as-libhdf5-does-not-as-the-ocp-mx-specification) | deliberate: libhdf5's values (FP4/FP6/FP8 E4M3 all-ones exponent is inf/NaN) | 2026-09-28 | | [In-place modification (`FileEditor`) limits](#in-place-modification-fileeditor-limits) | refused edits (`Error::Unsupported`), space reuse per editor, no journal | 2026-09-26 | | [Python in-place editing limits](#python-in-place-editing-clawhdf5filepath-r-limits) | refused writes (`NotImplementedError`), deliberate conversion differences | 2026-09-27 | @@ -241,7 +246,10 @@ Values are correct in every case; this is cost only. Selections other than ## Chunks of 4 GiB or more: limits -**Status:** open (documented 2026-09-28). HDF5 2.0 writes a chunk of more +**Status:** open (documented 2026-09-28; updated 2026-09-29, when +[chunk dimensions of 2^32 or more](#chunk-dimensions-of-232-or-more-were-refused) +and [unfiltered writes](#writing-a-4-gib-unfiltered-chunk-held-several-copies-of-it) +were fixed). HDF5 2.0 writes a chunk of more than 0xFFFFFFFF bytes with layout message version 5 (`H5D__chunk_construct` in libhdf5 2.2.0: "chunk size > 4GB requires H5F_LIBVER_V200"), which also makes a filtered chunk index element store the chunk's size in "size @@ -253,34 +261,40 @@ The [fix](#chunks-of-4-gib-or-more-could-not-be-read) is tested by `crates/clawhdf5/tests/huge_chunks_interop.rs`; its decoding tests are opt-in (`CLAWHDF5_HUGE_CHUNKS=1`, one at a time: `--test-threads=1`). What remains: -- **Chunk dimensions of 2^32 or more** (which libhdf5 2.x also allows under - `H5F_LIBVER_V200`) are refused, when read (`InvalidChunkDimensions`) - and when written: `DataLayout::Chunked::chunk_dimensions` is `Vec`. - A 4 GiB chunk of 1-byte elements needs one. - **Filters the writer refuses for such a chunk** (`FilterError`, before anything is written): LZF, bitshuffle, bzip2 and Blosc (their HDF5 filters record sizes or block lengths in 32 bits, or cannot take such a buffer) and pcodec. Deflate, shuffle, Fletcher-32, LZ4 and Zstd are - written; only shuffle + deflate is tested end to end (LZ4's framing and - the 256 MiB limit it had are unit-tested; Zstd is not tested at this - size). -- **Unfiltered chunks this size are written but not tested end to end:** - `FileBuilder` assembles the whole file in memory, several copies of the - chunk (well over the 12 GiB the tests may use). Their layout messages - are unit-tested (`chunked_write::tests::huge_chunk_layout_messages_and_index_elements`). + written and tested end to end (`writer_huge_chunks_round_trip`, + `writer_huge_chunk_dims_round_trip`, + `writer_huge_chunks_lz4_zstd_round_trip`: read back by clawhdf5 and + h5py 3.16, LZ4 and Zstd through hdf5plugin 7.1.0). - **`FileEditor`** refuses to rewrite such chunks (see [its limits](#in-place-modification-fileeditor-limits)). - **Memory:** decoding a filtered chunk holds all of it (4 GiB and more), twice when it is shuffled (the inflated and the unshuffled copy, as - libhdf5's filters do too); writing one holds the chunk and its shuffled - copy. A selection decodes only the chunks it touches; one of an - unfiltered chunk reads only the rows it selects, through a memory map or + libhdf5's filters do too). Writing a filtered chunk holds its compressed + copy and, where it is not a contiguous run of the dataset's data (a + chunk past the dataset's edge is zero-padded), the raw chunk; shuffled, + the shuffled copy too. An unfiltered chunk is not copied when it is such + a run (a dataset stored as one chunk of its own shape): writing one + unfiltered chunk of 2^32 + 7 bytes with `with_u8_data_owned` and + `FileBuilder::write` peaked at 4.00 GiB resident, the data itself + (`write_one_chunk 4294967303 OUT owned`, tank, 2026-09-29, commit + `0065c6b`; see [the fix](#writing-a-4-gib-unfiltered-chunk-held-several-copies-of-it)). + `FileWriter::finish` (a `Vec`) holds the data and the file. + A selection decodes only the chunks it touches; one of an unfiltered + chunk reads only the rows it selects, through a memory map or positioned reads (`File::open_storage`): every selection of `unfiltered_huge_chunks_read` together read under 1 MiB. Peak resident - memory of `cargo test --release -p clawhdf5 --test huge_chunks_interop - -- --test-threads=1` with `CLAWHDF5_HUGE_CHUNKS=1` (all six tests, h5py - included) was 8.06 GiB (tank, 2026-09-28, commit `9143737`); selection - reads of the double-deflated fixture alone, 4.0 GiB. + memory of `cargo test --release -p clawhdf5 --features lz4,zstd --test + huge_chunks_interop -- --test-threads=1` with `CLAWHDF5_HUGE_CHUNKS=1` + (all twelve tests, h5py and h5dump included) was 10.04 GiB (tank, + 2026-09-29, commit `0065c6b`), in `writer_huge_chunks_lz4_zstd_round_trip` + (the other tests, 8.2 GiB or less; the first run of the suite, six + tests, 8.06 GiB on 2026-09-28). Which step of that test holds 10 GiB + (clawhdf5's LZ4 or Zstd write or read, or hdf5plugin's) was not + measured separately. - **32-bit targets (wasm32):** such a chunk cannot be held in memory: reading or writing one is `FormatError::Overflow` ("exceeds the addressable size" / "address space"); the file still opens and lists @@ -694,6 +708,58 @@ browser reads `[re, im]` pairs. `Datatype::complex_as_compound`; exhaustive `match`es over `DType` need a `Complex` arm. Files are unchanged; nothing needs rewriting. +## Chunk dimensions of 2^32 or more were refused + +**Status:** fixed 2026-09-29 (branch `feat/huge-chunk-dims`, PR not yet +opened); affected every release (v2.1.0 to v2.7.0). An error, never wrong +data. Nothing for users to do but upgrade; code that matches on +`DataLayout::Chunked::chunk_dimensions` gets `u64`s now. + +libhdf5 2.x writes a chunk larger than 4 GiB with layout message version +5, whose dimensions take up to 8 bytes each, so a chunk dimension can be +2^32 or more (a 4 GiB chunk of 1-byte elements needs one). +`DataLayout::Chunked::chunk_dimensions` was `Vec`: such a file was +refused when opened for reading (`InvalidChunkDimensions`, "larger than +2^32 - 1"), and the writer refused such a chunk. Chunk dimensions are +`u64` throughout (format crate, facade, editor, `h5rs`, Python bindings); +a version-3 layout, whose dimensions are 4 bytes, is never written for +them (a chunk that large always takes version 5, as in libhdf5). Tests: +`huge_chunks_interop` (`huge_chunk_dims_list` always, over +`fixtures/huge_chunk_dims.h5`, which libhdf5 2.0.0 wrote: `u8` chunks of +2^32 + 7 in a Single Chunk and an Extensible Array index; with +`CLAWHDF5_HUGE_CHUNKS=1`, reads of it and of an unfiltered sparse file +h5py writes, and clawhdf5's own such chunks read back by h5py 3.16 and +h5dump 2.2.0), unit tests of the 5-byte encoding, and the wasm package +test (the file lists; reading such a chunk on wasm32 is +`FormatError::Overflow`). + +## Writing a 4 GiB unfiltered chunk held several copies of it + +**Status:** fixed 2026-09-29 (branch `feat/huge-chunk-dims`, PR not yet +opened); affected every release (v2.1.0 to v2.7.0). Memory only. To write +large data with the least memory, hand the builder the data +(`DatasetBuilder::with_u8_data_owned`) and write with +`FileBuilder::write` (or `FileWriter::finish_with`). + +`FileWriter::finish` copied every chunk into a per-chunk buffer, laid the +chunks out into one buffer per pass (two passes, the first kept), then +copied everything into the file's buffer, and `FileBuilder::write` wrote +that buffer out: about five times a dataset's size for one unfiltered +chunk, six with `with_u8_data` (which copies its argument). Measured +before and after on tank, 2026-09-29, peak resident memory of +`write_one_chunk 1073741824 OUT ` (`crates/clawhdf5/examples`, one +unfiltered 1 GiB chunk of `u8`): `owned` 5.0 GiB before (the writer at +`4260af4`), 1.0 GiB after (commit `0065c6b`); `slice` 6.0 GiB before, +2.0 GiB after. With 4 GiB + 7 bytes and `owned`: 4.00 GiB after. The +files written are byte for byte the same. Now a chunk that is a contiguous +run of the dataset's data is borrowed (and a filtered one compressed +straight from it), the layout refers to chunks instead of copying them, +and `FileBuilder::write` streams the file to disk +(`FileWriter::finish_with`); contiguous datasets are not copied either. +Test: `writer_unfiltered_huge_chunk_round_trip` (with +`CLAWHDF5_HUGE_CHUNKS=1`: 4 GiB + 7 bytes written and read back by +clawhdf5, h5py 3.16 and h5dump 2.2.0; the test process peaked at 4.00 GiB). + ## NetCDF-4: variables' dimensions are guessed from sizes diff --git a/examples/wasm-viewer/test/test.mjs b/examples/wasm-viewer/test/test.mjs index 4a2f26c..eccbb5d 100644 --- a/examples/wasm-viewer/test/test.mjs +++ b/examples/wasm-viewer/test/test.mjs @@ -514,6 +514,15 @@ async function limitTests() { `huge chunk: ${name}`); } hc.free(); + // Likewise chunks whose dimension is 2^32 or more (u8 chunks of 2^32 + 7). + const dims = pkg.open(new Uint8Array(readFileSync(join(import.meta.dirname, "..", "..", "..", + "crates/clawhdf5/tests/fixtures/huge_chunk_dims.h5")))); + eq(dims.list("/").map((e) => e.name), ["earray", "single"], "huge chunk dims: list"); + for (const name of ["single", "earray"]) { + await fails(() => dims.readHyperslab(`/${name}`, [0], [4]), /exceeds this platform.s address space/, + `huge chunk dims: ${name}`); + } + dims.free(); } // A body of `total` bytes in 64 KiB pieces, made as they are read; `pulled()`