fix(format): bound B-tree v2 traversal against crafted files
Traversal recursed one frame per level with the depth taken from the file (a u16), and followed child addresses without asking whether they were shared. Two crafted inputs, both reproduced before fixing: - A node listing itself as its own child, under a header claiming 65 535 levels, overflowed the stack and aborted the process — SIGABRT, not an error a caller can handle — from under 100 bytes. - Levels whose children all point at one shared node below reached it fan-out^depth times: 29.5 million records in 8 s from ~5 KB, and one more level would exhaust memory. Depth is now capped at 64, as the fractal heap already was; no real tree approaches it, since even at the minimum fan-out of two that is over 2^64 records. And traversal stops once it has produced more records than the file has bytes to hold them — a valid tree stores each record once in its own bytes, so this bounds shared subtrees without trusting the header's own `total_records`. Both inputs now fail in under a millisecond. Every B-tree v2 user goes through this collector: dense attributes, v2 groups, shared messages and chunk indexes. To show the budget never refuses a real file, a new interop test has HDF5 2.0 write a depth-2 chunk index with 40 000 records and reads back all 160 000 values; it fails when the budget is deliberately made too tight. Also corrects `BM25Index::search`, which claimed to use Block-Max WAND. It scores exhaustively, and pruning would not help the store: `hybrid_search` needs every score because fusion normalises over them. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
@@ -1047,3 +1047,47 @@ with h5py.File("{path_str}", "r") as f:
|
||||
data[start..start + cols as usize]
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn h5py_deep_btree_v2_chunk_index_clawhdf5_reads() {
|
||||
// Two unlimited dimensions give a B-tree v2 chunk index, and 2x2 chunks
|
||||
// over 400x400 give 40 000 index records — enough for HDF5 to build a
|
||||
// tree of depth 2. Small h5py files only ever produce depth-0 trees, so
|
||||
// this is the one fixture that walks internal nodes: the path where the
|
||||
// traversal's record budget (the guard against crafted shared-subtree
|
||||
// trees) is spent, which must never refuse a real file.
|
||||
skip_if_no_python!();
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let path = dir.path().join("deep_btree.h5");
|
||||
let path_str = path.display().to_string();
|
||||
|
||||
let script = format!(
|
||||
r#"
|
||||
import h5py, numpy as np
|
||||
with h5py.File("{path_str}", "w", libver="latest") as f:
|
||||
d = f.create_dataset("x", shape=(400, 400), maxshape=(None, None),
|
||||
chunks=(2, 2), dtype="i4")
|
||||
d[...] = np.arange(160000, dtype="i4").reshape(400, 400)
|
||||
"#
|
||||
);
|
||||
run_python(&script);
|
||||
|
||||
// The fixture is only meaningful if HDF5 really built internal nodes.
|
||||
let bytes = std::fs::read(&path).unwrap();
|
||||
let at = bytes
|
||||
.windows(4)
|
||||
.position(|w| w == b"BTHD")
|
||||
.expect("expected a B-tree v2 chunk index");
|
||||
let depth = u16::from_le_bytes([bytes[at + 12], bytes[at + 13]]);
|
||||
assert!(
|
||||
depth >= 1,
|
||||
"fixture tree has depth {depth}; it tests nothing"
|
||||
);
|
||||
|
||||
let file = File::open(&path).unwrap();
|
||||
let values = file.dataset("x").unwrap().read_i32().unwrap();
|
||||
assert_eq!(values.len(), 160_000);
|
||||
for (i, &v) in values.iter().enumerate() {
|
||||
assert_eq!(v, i as i32, "element {i}");
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user