fix(format): cap the Zstandard window at what the output can need
ruzstd reserves a frame's declared window (up to its 100 MiB default) when a decoder is reset for a new frame, before decoding anything. The Blosc, Blosc2 and bitshuffle decoders reuse one decoder per chunk, so a Blosc2 chunk of two 16-byte streams, each declaring a 96 MiB window, allocated 128 MiB. zstd_decode_into now sets the decoder's maximum window to twice the stream's output (at least 128 KiB): c-blosc, c-blosc2 and bitshuffle compress each block in one call with its size known, so libzstd's window never exceeds the block. Found by tracking peak allocation in the Blosc2 fuzz test. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
@@ -218,12 +218,20 @@ pub(crate) fn bitshuffle_decode(
|
||||
}
|
||||
|
||||
/// Decode Zstandard frames into exactly `dst`, failing if they hold more.
|
||||
///
|
||||
/// ruzstd reserves a frame's declared window (by default up to 100 MiB)
|
||||
/// before decoding it, so the window is capped at what the output could
|
||||
/// need: twice `dst` (window sizes are rounded up), and at least 128 KiB.
|
||||
/// The encoders behind these filters (c-blosc, c-blosc2, bitshuffle)
|
||||
/// compress each block in one call with its size known, so libzstd's
|
||||
/// window never exceeds the block.
|
||||
#[cfg(any(feature = "bitshuffle", feature = "blosc"))]
|
||||
pub(crate) fn zstd_decode_into(
|
||||
decoder: &mut ruzstd::decoding::FrameDecoder,
|
||||
frames: &[u8],
|
||||
dst: &mut [u8],
|
||||
) -> Result<usize, FormatError> {
|
||||
decoder.set_max_window_size((2 * dst.len()).max(1 << 17) as u64);
|
||||
decoder
|
||||
.decode_all(frames, dst)
|
||||
.map_err(|e| FormatError::DecompressionError(format!("zstd: {e}")))
|
||||
|
||||
Reference in New Issue
Block a user