fix(format): bound what a VL read retains on a crafted global heap
VlResolver kept an owned copy of every object of every heap collection it parsed, for the whole read. Collections nested inside each other's object data, 32 bytes apart with each element pointing at a different one, made retained memory O(elements x file size): 1.58 GB for a 744 KB file (read_vl_strings did the same before VlResolver). Chaining every collection's objects into one shared run of tiny objects made parse time O(elements x objects) as well. libhdf5 refuses these files. - The cache records where each object lies (GlobalHeapCollection:: parse_index, new) instead of copying it, and is dropped past a 32 MiB budget. - A collection overlapping one already read is an error: libhdf5 gives every collection its own block, so only a crafted file has them. - parse and parse_index refuse a collection that runs past the end of the file and an object that runs past the end of its collection. tests/vl_heap_bounds.rs measures peak heap use with a counting allocator: 129 MB and 350 MB live before on its two crafted files (64 KB and 176 KB), 97 KB and 0.9 MB now. Conformance unchanged at 575 of 697. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
@@ -409,6 +409,32 @@ has produced more records than the file could physically hold.
|
||||
|
||||
---
|
||||
|
||||
## Crafted global heaps exhaust the variable-length reader's memory
|
||||
|
||||
**Status:** fixed on `feat/p2-vl-strings` (2026-09-26). Not a regression of
|
||||
that branch: every earlier release is affected through `read_vl_strings`.
|
||||
|
||||
Reading variable-length values kept an owned copy of every object of every
|
||||
global heap collection visited, for the whole read. A file whose collections
|
||||
nest inside one another's object data (32 bytes apart, each element pointing
|
||||
at a different one) made retained memory O(elements × file size): a 744 KB
|
||||
file reached 1.58 GB. Letting every collection's object chain jump to one
|
||||
shared run of tiny objects made the parse time O(elements × objects) too.
|
||||
libhdf5 refuses such files.
|
||||
|
||||
Now `VlResolver` caches where each object lies instead of a copy, drops its
|
||||
cache past a 32 MiB budget, and refuses a collection that overlaps one it
|
||||
has already read (libhdf5 gives each collection its own block, so only a
|
||||
crafted file has them). `GlobalHeapCollection::parse` (and the new
|
||||
`parse_index`) also refuse a collection that runs past the end of the file,
|
||||
or an object that runs past the end of its collection. Guarded by
|
||||
`crates/clawhdf5-format/tests/vl_heap_bounds.rs`, which measures peak heap
|
||||
use with a counting allocator. Still open: a file may point many elements
|
||||
at one large heap object, and a VL-*sequence* read then returns that
|
||||
object once per element, as h5py would.
|
||||
|
||||
---
|
||||
|
||||
## Extensible Array chunk indexes read back wrong data past the inline elements
|
||||
|
||||
**Status:** fixed on `main` (2026-09-20), after v2.6.0. **Every release up to
|
||||
|
||||
Reference in New Issue
Block a user