docs: chunk dimensions of 2^32 or more and copy-free 4 GiB writes
known-issues: two fixed entries (chunk dimensions of 2^32 or more were refused; a 4 GiB unfiltered chunk was held several times when written), the open "Chunks of 4 GiB or more: limits" updated (LZ4/Zstd tested end to end, memory figures of 2026-09-29) and the open-issues table in sync. CHANGELOG (Unreleased) entry with the public API changes and the peak RSS measurements; README capability table. The wasm package test lists the new fixture and expects an Overflow error reading its chunks. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
@@ -119,6 +119,65 @@
|
||||
opens in HDF5 1.8.23's h5dump and in h5py). `test.mjs` no longer
|
||||
hard-codes the fixture's length.
|
||||
|
||||
### Chunk dimensions of 2^32 or more; 4 GiB chunks written without copies (2026-09-29)
|
||||
- **Chunk dimensions of 2^32 or more** (libhdf5 2.x writes them with
|
||||
layout message version 5, up to 8 bytes per dimension; a 4 GiB chunk of
|
||||
1-byte elements needs one) are read and written. They were refused
|
||||
(`InvalidChunkDimensions`). **Public API changes:**
|
||||
`DataLayout::Chunked::chunk_dimensions` is `Vec<u64>` (was `Vec<u32>`),
|
||||
and the `chunk_dimensions` argument of `chunked_read::{collect_chunk_info_checked,
|
||||
collect_chunk_info_checked_in, generate_implicit_chunks_in_grid}`,
|
||||
`fixed_array::{read_fixed_array_chunks, read_fixed_array_chunks_in}`,
|
||||
`extensible_array::{read_extensible_array_chunks,
|
||||
read_extensible_array_chunks_in}`, and the `chunk_dims` argument of
|
||||
`chunked_write::serialize_v4_single_chunk_pub`, are `&[u64]` (were
|
||||
`&[u32]`). A chunk whose size overflows 64 bits is refused when a dataset is read
|
||||
(`InvalidChunkDimensions`), and the writer refuses a chunk dimension of
|
||||
0 up front. A version-3 layout (4-byte dimensions) is never written for
|
||||
such a chunk: over 4 GiB, it always takes version 5, as in libhdf5.
|
||||
- **Writing without copies:** `FileWriter` no longer copies chunks into a
|
||||
buffer per pass and then into the file's buffer. A chunk that is a
|
||||
contiguous run of the dataset's data (a dataset stored as one chunk of
|
||||
its shape, blocks of whole rows) is borrowed when unfiltered and
|
||||
compressed straight from it when filtered, and contiguous datasets are
|
||||
not copied. New `FileWriter::finish_with(put)` hands the file out piece
|
||||
by piece; `FileBuilder::write` uses it (through a 1 MiB `BufWriter`,
|
||||
still to a temporary file renamed into place), so a file is never
|
||||
assembled in memory. New `DatasetBuilder::with_u8_data_owned(Vec<u8>)`
|
||||
takes the data without the copy `with_u8_data` makes. Peak resident
|
||||
memory of one unfiltered 1 GiB chunk of `u8`
|
||||
(`cargo run --release -p clawhdf5 --example write_one_chunk --
|
||||
1073741824 OUT owned|slice`, tank, 2026-09-29): 5.0 GiB before (the
|
||||
writer of `4260af4`), 1.0 GiB after with `owned`; 6.0 and 2.0 GiB with
|
||||
`slice`; 4 GiB + 7 bytes `owned`, 4.00 GiB. Same bytes written. Write
|
||||
speed was not re-measured (the machine was not idle): the A/B of
|
||||
`FileBuilder::write` and `finish` on small and large files is still to
|
||||
be run.
|
||||
- `chunked_write::extract_chunk` (behind `split_into_chunks`) no longer
|
||||
panics when the data is shorter than the shape; the missing part is
|
||||
zeros, as it was meant to be.
|
||||
- Tests: `crates/clawhdf5/tests/fixtures/huge_chunk_dims.h5` (31 KB,
|
||||
libhdf5 2.0.0 via h5py 3.16, `gen_huge_chunks.py dims`: `u8` chunks of
|
||||
2^32 + 7 deflated twice, Single Chunk and Extensible Array), and in
|
||||
`huge_chunks_interop`: `huge_chunk_dims_list` (always), and with
|
||||
`CLAWHDF5_HUGE_CHUNKS=1` `huge_chunk_dims_read`,
|
||||
`huge_chunk_dims_unfiltered_read` (a sparse file h5py writes, Single
|
||||
Chunk and Fixed Array, mmap and positioned reads),
|
||||
`writer_huge_chunk_dims_round_trip` (deflated, read back by clawhdf5,
|
||||
h5py 3.16 and — layouts — h5dump 2.2.0),
|
||||
`writer_unfiltered_huge_chunk_round_trip` (an unfiltered 4 GiB + 7 byte
|
||||
chunk: clawhdf5, h5py, and h5dump 2.2.0 printing values past 2^32) and
|
||||
`writer_huge_chunks_lz4_zstd_round_trip` (with the `lz4`/`zstd`
|
||||
features; h5py through hdf5plugin 7.1.0). The whole opt-in suite
|
||||
(`cargo test --release -p clawhdf5 --features lz4,zstd --test
|
||||
huge_chunks_interop -- --test-threads=1`, h5py and h5dump 2.2.0
|
||||
included) passed with a peak of 10.04 GiB resident (tank, 2026-09-29,
|
||||
commit `0065c6b`). Unit tests: 5-byte dimension encoding, contiguous
|
||||
chunk detection, `finish_with` against `finish`. The wasm package test
|
||||
lists the new fixture and gets `FormatError::Overflow` reading it.
|
||||
`docs/known-issues.md`: two fixed entries; the open "Chunks of 4 GiB or
|
||||
more: limits" keeps the refused filters, the editor and memory.
|
||||
|
||||
### NetCDF-4: variables' dimensions come from the file (2026-09-28)
|
||||
- `clawhdf5-netcdf4` gave each variable the first unused dimension of
|
||||
equal size (else an anonymous `dim_<n>`), so a variable on an unlimited
|
||||
|
||||
@@ -102,7 +102,8 @@ writes 85.2 µs vs 877 µs (10.3x); 64 group creates 130 µs vs 1.37 ms
|
||||
(10.6x); a 512×512 `f32` chunked deflate-6 write 1.44 ms vs 65.0 ms
|
||||
(re-measured 2026-09-23 with the pure-Rust deflate: 1.46 ms vs 51.4 ms,
|
||||
35x); a 100K `f32` sequential write is a tie. The writer (`FileBuilder`)
|
||||
assembles a file in memory and writes it once, which is part of that
|
||||
lays out the whole file before writing it (when these were measured, in
|
||||
one buffer in memory), which is part of that
|
||||
difference; read the caveats in [BENCHMARKS.md](BENCHMARKS.md#caveats)
|
||||
before quoting these.
|
||||
|
||||
@@ -116,7 +117,7 @@ Limits and open issues, with dates, are in
|
||||
| **File format** | Superblock v0–v3, user blocks, v1/v2 object headers; writing the HDF5 1.10 format (default) or, with `libver_bounds(LibVer::V18, LibVer::V18)`, files HDF5 1.8 reads (checked with HDF5 1.8.23) | Metadata cache images | Writing the pre-1.8 format (version-0 superblock, symbol-table groups) |
|
||||
| **Groups and links** | Symbol-table, compact and dense groups (tested to 100 000 links), creation order, soft and hard links; writing external links | | Following external links (explicit error); user-defined links are skipped |
|
||||
| **Datatypes** | Integers and IEEE floats of every width and byte order (incl. `f16`), enums, compounds (every version, incl. HDF5 2.0's v5), arrays, fixed-length strings, opaque, complex: h5py's `{r, i}` compound (`with_complex_f64_data`) and HDF5 2.0's native class 11 (`with_native_complex_f64_data`, opt-in: only libhdf5 2.0+ reads it; read as its own type, `DType::Complex`, and printed by `h5rs` as h5dump 2.x prints it) | Variable-length strings and sequences, object references; HDF5 2.x's small floats (bfloat16, FP8 E4M3/E5M2, FP6 E2M3/E3M2, FP4 E2M1: every bit pattern decoded as libhdf5 2.2.0 decodes it) and other non-IEEE floats up to 64 bits | Writing variable-length data; writing non-IEEE floats; decoding region and attribute references; x87 long double and binary128 |
|
||||
| **Layouts and chunk indexes** | Compact, contiguous and chunked; chunk indexes single chunk, Fixed Array, Extensible Array and v2 B-tree (the writer picks one as libhdf5 does), and v1 B-tree (every chunked dataset under the 1.8 bound, split as libhdf5 splits it); chunks of 4 GiB or more (HDF5 2.0's layout message version 5); fill values; resizable datasets; virtual datasets (read limits in known-issues) | The implicit chunk index (the editor also changes it) | External raw data files (explicit error); chunk dimensions of 2^32 or more |
|
||||
| **Layouts and chunk indexes** | Compact, contiguous and chunked; chunk indexes single chunk, Fixed Array, Extensible Array and v2 B-tree (the writer picks one as libhdf5 does), and v1 B-tree (every chunked dataset under the 1.8 bound, split as libhdf5 splits it); chunks of 4 GiB or more and chunk dimensions of 2^32 or more (HDF5 2.0's layout message version 5); fill values; resizable datasets; virtual datasets (read limits in known-issues) | The implicit chunk index (the editor also changes it) | External raw data files (explicit error) |
|
||||
| **Filters** | deflate (pure-Rust zlib-rs), shuffle, Fletcher-32, LZ4 (opt-in), Zstd (C, opt-in); plugins LZF, bitshuffle, bzip2, Blosc 1 | N-Bit, scale-offset, SZIP (C, opt-in); plugins Blosc2 and ZFP | Other filter IDs, unless you register a codec (`filter_registry::register_filter`) |
|
||||
| **Editing in place** | `FileEditor`: overwrite values, grow and shrink chunked datasets (every index), set attributes (compact and dense), in files from h5py or clawhdf5 | | Creating or deleting objects in an existing file; deleting attributes; new chunks in implicit indexes; VL data; rewriting chunks of 4 GiB or more; filters this build cannot encode (refused before any write) |
|
||||
| **Access** | Local files (mmap or buffered), bytes in memory, any `Storage` backend, HTTP(S) and S3/GCS/Azure via `clawhdf5-remote`, SWMR reading (`File::open_swmr`, `Dataset::refresh`) | Remote files and the browser are read-only | SWMR writing; remote SWMR; MPI collective I/O (`clawhdf5-io`'s `mpi-io` reads on one rank and broadcasts) |
|
||||
|
||||
+85
-19
@@ -17,6 +17,11 @@ Checked against `main` at `9b5803f` on 2026-09-28.
|
||||
|---|---|---|
|
||||
| [NetCDF-4: differences from netCDF-C](#netcdf-4-differences-from-netcdf-c) | deliberate (floats of other than 4 or 8 bytes, axes netCDF-C leaves without a dimension), and 9 corpus files (external links, values the HDF5 reader refuses) | 2026-09-29 |
|
||||
| [Chunks of 4 GiB or more: limits](#chunks-of-4-gib-or-more-limits) | refused (chunk dimensions of 2^32 or more, five filters, editor rewrites), memory (a decoded chunk is held whole) | 2026-09-28 |
|
||||
|
||||
|
||||
| [NetCDF-4: variables' dimensions are guessed from sizes](#netcdf-4-variables-dimensions-are-guessed-from-sizes) | **wrong metadata** (`Variable::dimensions`, pure dimension scales listed as variables; values and dimension sizes are right) | 2026-09-28 |
|
||||
| [Chunks of 4 GiB or more: limits](#chunks-of-4-gib-or-more-limits) | refused (five filters, editor rewrites), memory (a decoded chunk is held whole) | 2026-09-28 |
|
||||
| [NetCDF-4: an unlimited dimension reports size 0](#netcdf-4-an-unlimited-dimension-reports-size-0) | **wrong metadata** (dimension size; variable shapes and values are right) | 2026-09-28 |
|
||||
| [Small floats decode as libhdf5 does, not as the OCP MX specification](#small-floats-decode-as-libhdf5-does-not-as-the-ocp-mx-specification) | deliberate: libhdf5's values (FP4/FP6/FP8 E4M3 all-ones exponent is inf/NaN) | 2026-09-28 |
|
||||
| [In-place modification (`FileEditor`) limits](#in-place-modification-fileeditor-limits) | refused edits (`Error::Unsupported`), space reuse per editor, no journal | 2026-09-26 |
|
||||
| [Python in-place editing limits](#python-in-place-editing-clawhdf5filepath-r-limits) | refused writes (`NotImplementedError`), deliberate conversion differences | 2026-09-27 |
|
||||
@@ -241,7 +246,10 @@ Values are correct in every case; this is cost only. Selections other than
|
||||
|
||||
## Chunks of 4 GiB or more: limits
|
||||
|
||||
**Status:** open (documented 2026-09-28). HDF5 2.0 writes a chunk of more
|
||||
**Status:** open (documented 2026-09-28; updated 2026-09-29, when
|
||||
[chunk dimensions of 2^32 or more](#chunk-dimensions-of-232-or-more-were-refused)
|
||||
and [unfiltered writes](#writing-a-4-gib-unfiltered-chunk-held-several-copies-of-it)
|
||||
were fixed). HDF5 2.0 writes a chunk of more
|
||||
than 0xFFFFFFFF bytes with layout message version 5 (`H5D__chunk_construct`
|
||||
in libhdf5 2.2.0: "chunk size > 4GB requires H5F_LIBVER_V200"), which
|
||||
also makes a filtered chunk index element store the chunk's size in "size
|
||||
@@ -253,34 +261,40 @@ The [fix](#chunks-of-4-gib-or-more-could-not-be-read) is tested by
|
||||
`crates/clawhdf5/tests/huge_chunks_interop.rs`; its decoding tests are
|
||||
opt-in (`CLAWHDF5_HUGE_CHUNKS=1`, one at a time: `--test-threads=1`).
|
||||
What remains:
|
||||
- **Chunk dimensions of 2^32 or more** (which libhdf5 2.x also allows under
|
||||
`H5F_LIBVER_V200`) are refused, when read (`InvalidChunkDimensions`)
|
||||
and when written: `DataLayout::Chunked::chunk_dimensions` is `Vec<u32>`.
|
||||
A 4 GiB chunk of 1-byte elements needs one.
|
||||
- **Filters the writer refuses for such a chunk** (`FilterError`, before
|
||||
anything is written): LZF, bitshuffle, bzip2 and Blosc (their HDF5
|
||||
filters record sizes or block lengths in 32 bits, or cannot take such a
|
||||
buffer) and pcodec. Deflate, shuffle, Fletcher-32, LZ4 and Zstd are
|
||||
written; only shuffle + deflate is tested end to end (LZ4's framing and
|
||||
the 256 MiB limit it had are unit-tested; Zstd is not tested at this
|
||||
size).
|
||||
- **Unfiltered chunks this size are written but not tested end to end:**
|
||||
`FileBuilder` assembles the whole file in memory, several copies of the
|
||||
chunk (well over the 12 GiB the tests may use). Their layout messages
|
||||
are unit-tested (`chunked_write::tests::huge_chunk_layout_messages_and_index_elements`).
|
||||
written and tested end to end (`writer_huge_chunks_round_trip`,
|
||||
`writer_huge_chunk_dims_round_trip`,
|
||||
`writer_huge_chunks_lz4_zstd_round_trip`: read back by clawhdf5 and
|
||||
h5py 3.16, LZ4 and Zstd through hdf5plugin 7.1.0).
|
||||
- **`FileEditor`** refuses to rewrite such chunks (see
|
||||
[its limits](#in-place-modification-fileeditor-limits)).
|
||||
- **Memory:** decoding a filtered chunk holds all of it (4 GiB and more),
|
||||
twice when it is shuffled (the inflated and the unshuffled copy, as
|
||||
libhdf5's filters do too); writing one holds the chunk and its shuffled
|
||||
copy. A selection decodes only the chunks it touches; one of an
|
||||
unfiltered chunk reads only the rows it selects, through a memory map or
|
||||
libhdf5's filters do too). Writing a filtered chunk holds its compressed
|
||||
copy and, where it is not a contiguous run of the dataset's data (a
|
||||
chunk past the dataset's edge is zero-padded), the raw chunk; shuffled,
|
||||
the shuffled copy too. An unfiltered chunk is not copied when it is such
|
||||
a run (a dataset stored as one chunk of its own shape): writing one
|
||||
unfiltered chunk of 2^32 + 7 bytes with `with_u8_data_owned` and
|
||||
`FileBuilder::write` peaked at 4.00 GiB resident, the data itself
|
||||
(`write_one_chunk 4294967303 OUT owned`, tank, 2026-09-29, commit
|
||||
`0065c6b`; see [the fix](#writing-a-4-gib-unfiltered-chunk-held-several-copies-of-it)).
|
||||
`FileWriter::finish` (a `Vec`) holds the data and the file.
|
||||
A selection decodes only the chunks it touches; one of an unfiltered
|
||||
chunk reads only the rows it selects, through a memory map or
|
||||
positioned reads (`File::open_storage`): every selection of
|
||||
`unfiltered_huge_chunks_read` together read under 1 MiB. Peak resident
|
||||
memory of `cargo test --release -p clawhdf5 --test huge_chunks_interop
|
||||
-- --test-threads=1` with `CLAWHDF5_HUGE_CHUNKS=1` (all six tests, h5py
|
||||
included) was 8.06 GiB (tank, 2026-09-28, commit `9143737`); selection
|
||||
reads of the double-deflated fixture alone, 4.0 GiB.
|
||||
memory of `cargo test --release -p clawhdf5 --features lz4,zstd --test
|
||||
huge_chunks_interop -- --test-threads=1` with `CLAWHDF5_HUGE_CHUNKS=1`
|
||||
(all twelve tests, h5py and h5dump included) was 10.04 GiB (tank,
|
||||
2026-09-29, commit `0065c6b`), in `writer_huge_chunks_lz4_zstd_round_trip`
|
||||
(the other tests, 8.2 GiB or less; the first run of the suite, six
|
||||
tests, 8.06 GiB on 2026-09-28). Which step of that test holds 10 GiB
|
||||
(clawhdf5's LZ4 or Zstd write or read, or hdf5plugin's) was not
|
||||
measured separately.
|
||||
- **32-bit targets (wasm32):** such a chunk cannot be held in memory:
|
||||
reading or writing one is `FormatError::Overflow` ("exceeds the
|
||||
addressable size" / "address space"); the file still opens and lists
|
||||
@@ -694,6 +708,58 @@ browser reads `[re, im]` pairs.
|
||||
`Datatype::complex_as_compound`; exhaustive `match`es over `DType` need a
|
||||
`Complex` arm. Files are unchanged; nothing needs rewriting.
|
||||
|
||||
## Chunk dimensions of 2^32 or more were refused
|
||||
|
||||
**Status:** fixed 2026-09-29 (branch `feat/huge-chunk-dims`, PR not yet
|
||||
opened); affected every release (v2.1.0 to v2.7.0). An error, never wrong
|
||||
data. Nothing for users to do but upgrade; code that matches on
|
||||
`DataLayout::Chunked::chunk_dimensions` gets `u64`s now.
|
||||
|
||||
libhdf5 2.x writes a chunk larger than 4 GiB with layout message version
|
||||
5, whose dimensions take up to 8 bytes each, so a chunk dimension can be
|
||||
2^32 or more (a 4 GiB chunk of 1-byte elements needs one).
|
||||
`DataLayout::Chunked::chunk_dimensions` was `Vec<u32>`: such a file was
|
||||
refused when opened for reading (`InvalidChunkDimensions`, "larger than
|
||||
2^32 - 1"), and the writer refused such a chunk. Chunk dimensions are
|
||||
`u64` throughout (format crate, facade, editor, `h5rs`, Python bindings);
|
||||
a version-3 layout, whose dimensions are 4 bytes, is never written for
|
||||
them (a chunk that large always takes version 5, as in libhdf5). Tests:
|
||||
`huge_chunks_interop` (`huge_chunk_dims_list` always, over
|
||||
`fixtures/huge_chunk_dims.h5`, which libhdf5 2.0.0 wrote: `u8` chunks of
|
||||
2^32 + 7 in a Single Chunk and an Extensible Array index; with
|
||||
`CLAWHDF5_HUGE_CHUNKS=1`, reads of it and of an unfiltered sparse file
|
||||
h5py writes, and clawhdf5's own such chunks read back by h5py 3.16 and
|
||||
h5dump 2.2.0), unit tests of the 5-byte encoding, and the wasm package
|
||||
test (the file lists; reading such a chunk on wasm32 is
|
||||
`FormatError::Overflow`).
|
||||
|
||||
## Writing a 4 GiB unfiltered chunk held several copies of it
|
||||
|
||||
**Status:** fixed 2026-09-29 (branch `feat/huge-chunk-dims`, PR not yet
|
||||
opened); affected every release (v2.1.0 to v2.7.0). Memory only. To write
|
||||
large data with the least memory, hand the builder the data
|
||||
(`DatasetBuilder::with_u8_data_owned`) and write with
|
||||
`FileBuilder::write` (or `FileWriter::finish_with`).
|
||||
|
||||
`FileWriter::finish` copied every chunk into a per-chunk buffer, laid the
|
||||
chunks out into one buffer per pass (two passes, the first kept), then
|
||||
copied everything into the file's buffer, and `FileBuilder::write` wrote
|
||||
that buffer out: about five times a dataset's size for one unfiltered
|
||||
chunk, six with `with_u8_data` (which copies its argument). Measured
|
||||
before and after on tank, 2026-09-29, peak resident memory of
|
||||
`write_one_chunk 1073741824 OUT <mode>` (`crates/clawhdf5/examples`, one
|
||||
unfiltered 1 GiB chunk of `u8`): `owned` 5.0 GiB before (the writer at
|
||||
`4260af4`), 1.0 GiB after (commit `0065c6b`); `slice` 6.0 GiB before,
|
||||
2.0 GiB after. With 4 GiB + 7 bytes and `owned`: 4.00 GiB after. The
|
||||
files written are byte for byte the same. Now a chunk that is a contiguous
|
||||
run of the dataset's data is borrowed (and a filtered one compressed
|
||||
straight from it), the layout refers to chunks instead of copying them,
|
||||
and `FileBuilder::write` streams the file to disk
|
||||
(`FileWriter::finish_with`); contiguous datasets are not copied either.
|
||||
Test: `writer_unfiltered_huge_chunk_round_trip` (with
|
||||
`CLAWHDF5_HUGE_CHUNKS=1`: 4 GiB + 7 bytes written and read back by
|
||||
clawhdf5, h5py 3.16 and h5dump 2.2.0; the test process peaked at 4.00 GiB).
|
||||
|
||||
## NetCDF-4: variables' dimensions are guessed from sizes
|
||||
|
||||
|
||||
|
||||
@@ -514,6 +514,15 @@ async function limitTests() {
|
||||
`huge chunk: ${name}`);
|
||||
}
|
||||
hc.free();
|
||||
// Likewise chunks whose dimension is 2^32 or more (u8 chunks of 2^32 + 7).
|
||||
const dims = pkg.open(new Uint8Array(readFileSync(join(import.meta.dirname, "..", "..", "..",
|
||||
"crates/clawhdf5/tests/fixtures/huge_chunk_dims.h5"))));
|
||||
eq(dims.list("/").map((e) => e.name), ["earray", "single"], "huge chunk dims: list");
|
||||
for (const name of ["single", "earray"]) {
|
||||
await fails(() => dims.readHyperslab(`/${name}`, [0], [4]), /exceeds this platform.s address space/,
|
||||
`huge chunk dims: ${name}`);
|
||||
}
|
||||
dims.free();
|
||||
}
|
||||
|
||||
// A body of `total` bytes in 64 KiB pieces, made as they are read; `pulled()`
|
||||
|
||||
Reference in New Issue
Block a user