Write chunks of 4 GiB or more; FileEditor refuses to rewrite them

The writer gives a chunk of more than u32::MAX bytes layout message
version 5 and, filtered, index elements whose stored size takes the file's
size of lengths, as libhdf5 2.x does (the Fixed and Extensible Array
structures match libhdf5's byte for byte). Chunk dimensions of 2^32 or more
and filters that cannot take such a chunk (LZF, bitshuffle, bzip2, Blosc,
pcodec) are refused instead of truncated. Chunks are extracted row by row
and one at a time; deflate no longer cuts input at 4 GiB - 1 bytes, nor
holds the worst-case bound of a large chunk; an LZ4 chunk of 4 GiB or more
is read as the registered framing.

FileEditor refuses writing values into, or pruning/allocating, chunks of
4 GiB or more before anything is written; growing the extent and
attributes still work.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-28 23:44:48 -05:00
co-authored by Claude Opus 5.5
parent ac7871fff5
commit 5a20cf04e8
8 changed files with 788 additions and 175 deletions
+3 -2
View File
@@ -18,9 +18,10 @@ pub(crate) fn serialize_v4_extensible_array(
ea_address: u64,
offset_size: u8,
element_size: u32,
version: u8,
) -> Vec<u8> {
let mut buf = Vec::new();
buf.push(4); // version
buf.push(version);
buf.push(2); // class = chunked
buf.push(0x00); // flags
@@ -87,7 +88,7 @@ pub fn build_extensible_array_at(
ea_base_address: u64,
) -> Vec<u8> {
let os = offset_size as usize;
let chunk_size_bytes = has_filters.then(|| filtered_chunk_size_len(slots));
let chunk_size_bytes = has_filters.then(|| filtered_chunk_size_len(slots, length_size));
let elem_size = os + chunk_size_bytes.map_or(0, |n| n + 4);
let client_id: u8 = if has_filters { 1 } else { 0 };
let arr_off_size = (MAX_NELMTS_BITS as usize).div_ceil(8);