format: bound ZFP chunk fetches at 4x, not n + n/4

The M2 fetch bound gave every codec n + n/4 + 4096 stored bytes, but
ZFP's fixed-rate mode stores up to 64 bits per value, doubling 4-byte
types: 60 of the 2205 zfp_interop datasets (rate 64, f4/i4) failed with
'compressed stream ends early' once M2 and ZFP were merged.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-26 19:33:03 -05:00
co-authored by Claude Opus 5.5
parent 7447dce121
commit f37e7ae326
+7 -1
View File
@@ -72,7 +72,8 @@ fn filter_output_bound(filter_id: u16, input: usize) -> usize {
/// `filter_mask`) is the chunk itself: `chunk_bytes`, and bytes past them /// `filter_mask`) is the chunk itself: `chunk_bytes`, and bytes past them
/// were never used. A filtered chunk is bounded by each applied filter's /// were never used. A filtered chunk is bounded by each applied filter's
/// worst-case growth in the write direction: shuffle keeps the size, /// worst-case growth in the write direction: shuffle keeps the size,
/// Fletcher32 adds 4 bytes, and any other codec gets `n + n/4 + 4096` — a /// Fletcher32 adds 4 bytes, ZFP gets `4n + 4096` (its fixed-rate mode stores
/// up to 64 bits per value), and any other codec gets `n + n/4 + 4096` — a
/// deliberately generous bound (bzip2 grows 1000 random bytes by 252, more /// deliberately generous bound (bzip2 grows 1000 random bytes by 252, more
/// than the decoders' own `n + n/8 + 64` output bound), since a legitimate /// than the decoders' own `n + n/8 + 64` output bound), since a legitimate
/// chunk cut short here would fail to read. A filter handled by a codec the /// chunk cut short here would fail to read. A filter handled by a codec the
@@ -95,6 +96,11 @@ pub(crate) fn stored_chunk_limit(
FILTER_SHUFFLE => size, FILTER_SHUFFLE => size,
FILTER_FLETCHER32 => size.saturating_add(4), FILTER_FLETCHER32 => size.saturating_add(4),
id if filter_registry::may_be_registered(id) => return usize::MAX, id if filter_registry::may_be_registered(id) => return usize::MAX,
// ZFP's fixed-rate mode stores up to 64 bits per value, so a
// 4-byte type doubles, and precision/accuracy modes add group
// test bits per bit plane on top: 4x plus a header is still a
// bound, where `n + n/4` cut rate-64 chunks short.
crate::filter_pipeline::FILTER_ZFP => size.saturating_mul(4).saturating_add(4096),
_ => size.saturating_add(size / 4).saturating_add(4096), _ => size.saturating_add(size / 4).saturating_add(4096),
}; };
} }