Write chunks of 4 GiB or more; FileEditor refuses to rewrite them
The writer gives a chunk of more than u32::MAX bytes layout message version 5 and, filtered, index elements whose stored size takes the file's size of lengths, as libhdf5 2.x does (the Fixed and Extensible Array structures match libhdf5's byte for byte). Chunk dimensions of 2^32 or more and filters that cannot take such a chunk (LZF, bitshuffle, bzip2, Blosc, pcodec) are refused instead of truncated. Chunks are extracted row by row and one at a time; deflate no longer cuts input at 4 GiB - 1 bytes, nor holds the worst-case bound of a large chunk; an LZ4 chunk of 4 GiB or more is read as the registered framing. FileEditor refuses writing values into, or pruning/allocating, chunks of 4 GiB or more before anything is written; growing the extent and attributes still work. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
@@ -18,9 +18,10 @@ pub(crate) fn serialize_v4_extensible_array(
|
||||
ea_address: u64,
|
||||
offset_size: u8,
|
||||
element_size: u32,
|
||||
version: u8,
|
||||
) -> Vec<u8> {
|
||||
let mut buf = Vec::new();
|
||||
buf.push(4); // version
|
||||
buf.push(version);
|
||||
buf.push(2); // class = chunked
|
||||
buf.push(0x00); // flags
|
||||
|
||||
@@ -87,7 +88,7 @@ pub fn build_extensible_array_at(
|
||||
ea_base_address: u64,
|
||||
) -> Vec<u8> {
|
||||
let os = offset_size as usize;
|
||||
let chunk_size_bytes = has_filters.then(|| filtered_chunk_size_len(slots));
|
||||
let chunk_size_bytes = has_filters.then(|| filtered_chunk_size_len(slots, length_size));
|
||||
let elem_size = os + chunk_size_bytes.map_or(0, |n| n + 4);
|
||||
let client_id: u8 = if has_filters { 1 } else { 0 };
|
||||
let arr_off_size = (MAX_NELMTS_BITS as usize).div_ceil(8);
|
||||
|
||||
Reference in New Issue
Block a user