quantumclaw/clawhdf5
3
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

format: group walks go on past a failed node and hint what they read next

Browse Source 
Listing a large group over openUrl still took 6-11 passes (network round
trips) for the reviewer's 3000-dataset h5py file: each pass only found
the structures the walk reached before its first miss.

- The v1 and v2 B-tree collectors descend into every child of a node
  after one fails (they only read the siblings before, so a sibling's
  subtree came a pass later), then return the first error: results and
  errors unchanged. The v2 walk stops once its record budget is spent,
  so a shared-subtree tree still cannot multiply the work.
- Hints (`Storage::hint`, a no-op for every backend but the lazy one):
  a group B-tree node's and a symbol table node's body (read once their
  header gives a length, a round trip later when the body is in the
  next block), an object header's first chunk and its continuation
  chunks, the symbol table nodes a B-tree leaf names, a dense group's
  name index header and the heap's root block (both read right after
  the heap header). A listing also hints every child's object header as
  its entry is read, even after a failure, and every direct block of a
  dense group's heap (reading the indirect blocks, at most 4096 entries
  and 4 levels deep); a lookup does not.
- The fractal heap's indirect-block layout (entry sizes, where the first
  n entries end) is one helper used by the object reads and the hints.

Measured with tests/lazy.rs listing_cost_of_a_given_file on an h5py file
like the reviewer's (3000 datasets of 64 KiB, 198 MB), list('/'),
passes/requests/bytes, before -> after:
  earliest, 1 MiB:  6/73/192.5 MB -> 4/68/192.5 MB
  earliest, 64 KiB: 8/531/35.2 MB -> 5/530/35.3 MB
  latest,   1 MiB:  9/98/196.5 MB -> 5/86/196.5 MB
  latest,   64 KiB: 11/452/29.6 MB -> 6/454/30.5 MB
listing_a_large_group_takes_a_few_passes (512-byte blocks), budgets
tightened to the new counts: FileBuilder 600 children 5 -> 4 passes,
h5py 2000 children earliest 8 -> 5, latest 11 -> 6.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-27 22:47:44 -05:00
co-authored by Claude Opus 5.5
parent ca81c3ebfa
commit 6f5d14fd62
8 changed files with 443 additions and 111 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/clawhdf5-format/src/fractal_heap.rs
+166 -32
View File
@@ -10,7 +10,7 @@ use crate::addr::to_usize;
use crate::btree_v2::{BTreeV2Header, find_btree_v2_records_in};
use crate::error::FormatError;
use crate::filter_pipeline::FilterPipeline;
use crate::storage::{Storage, Window, len_usize, read_exact_at};
use crate::storage::{Storage, Window, len_usize, read_exact_at, read_upto};
/// Parsed fractal heap header (signature "FRHP").
#[derive(Debug, Clone)]
@@ -132,6 +132,9 @@ fn heap_id_type(first: u8) -> Result<u8, FormatError> {
const BTREE_HUGE_INDIRECT: u8 = 1;
const BTREE_HUGE_INDIRECT_FILTERED: u8 = 2;
/// Most child entries [`FractalHeapHeader::hint_managed_blocks`] walks.
const MAX_HINTED_BLOCKS: usize = 4096;
impl FractalHeapHeader {
/// Parse a fractal heap header at the given offset.
pub fn parse(
@@ -667,15 +670,8 @@ impl FractalHeapHeader {
"fractal heap: maximum recursion depth exceeded".into(),
));
}
let block_offset_bytes = (self.max_heap_size as usize).div_ceil(8);
let iblock_header = 5 + offset_size as usize + block_offset_bytes;
let nrows_usize = nrows as usize;
// Rows below max_direct_rows hold direct blocks; rows at/above hold
// child indirect blocks. (NOT the FRHP "starting rows" field.)
let start_indirect = self.max_direct_rows();
let max_direct_rows = nrows_usize.min(start_indirect);
// The block up to its last child entry. The walk below reads
// entries in order and stops at the one covering the target, which
// the geometry alone locates, so the first window ends there: a
@@ -685,32 +681,11 @@ impl FractalHeapHeader {
// over the whole block. Either window holds what it was asked for or
// ends at the end of the file, so its bounds checks are the
// whole-file ones.
let direct_entry = usize::from(offset_size)
+ if self.filter_pipeline.is_some() {
usize::from(self.length_size) + 4
} else {
0
};
let direct_entries = max_direct_rows.saturating_mul(usize::from(self.table_width));
let entries_len = |n: usize| {
n.min(direct_entries)
.saturating_mul(direct_entry)
.saturating_add(
n.saturating_sub(direct_entries)
.saturating_mul(usize::from(offset_size)),
)
};
let all_entries = direct_entries.saturating_add(
nrows_usize
.saturating_sub(start_indirect)
.saturating_mul(usize::from(self.table_width)),
);
let block_len = iblock_header.saturating_add(entries_len(all_entries));
let layout = self.indirect_layout(nrows_usize, offset_size);
let block_len = layout.len_upto(layout.all_entries);
let target_entry = self.indirect_entry_for(nrows_usize, iblock_heap_offset, target_offset);
let first_len = target_entry.map_or(block_len, |i| {
iblock_header
.saturating_add(entries_len(i.saturating_add(1)))
.min(block_len)
layout.len_upto(i.saturating_add(1)).min(block_len)
});
let mut next = self.walk_indirect_block(
&Window::read(file, iblock_addr as u64, first_len)?,
@@ -755,6 +730,140 @@ impl FractalHeapHeader {
}
}
/// Where the child entries of an indirect block of `nrows` rows are.
fn indirect_layout(&self, nrows: usize, offset_size: u8) -> IndirectLayout {
let block_offset_bytes = (self.max_heap_size as usize).div_ceil(8);
// Rows below max_direct_rows hold direct blocks; rows at/above hold
// child indirect blocks. (NOT the FRHP "starting rows" field.)
let start_indirect = self.max_direct_rows();
let direct_entries = nrows
.min(start_indirect)
.saturating_mul(usize::from(self.table_width));
IndirectLayout {
header: 5 + usize::from(offset_size) + block_offset_bytes,
direct_entry: usize::from(offset_size)
+ if self.filter_pipeline.is_some() {
usize::from(self.length_size) + 4
} else {
0
},
direct_entries,
indirect_entry: usize::from(offset_size),
all_entries: direct_entries.saturating_add(
nrows
.saturating_sub(start_indirect)
.saturating_mul(usize::from(self.table_width)),
),
}
}
/// Hint the heap's root block (see [`Storage::hint`]): every managed
/// object is read through it, and a storage that fetches between
/// attempts can fetch it along with whatever else the attempt missed
/// (the name index read before any object, say).
pub fn hint_root_block<S: Storage + ?Sized>(&self, file: &S) {
if is_undefined(self.root_block_address, self.offset_size) {
return;
}
let len = if self.current_rows_in_root_indirect_block == 0 {
if self.filter_pipeline.is_some() {
self.root_direct_block_filtered_size
} else {
self.starting_block_size
}
} else {
let layout = self.indirect_layout(
usize::from(self.current_rows_in_root_indirect_block),
self.offset_size,
);
layout.len_upto(layout.all_entries) as u64
};
file.hint(
self.root_block_address,
usize::try_from(len).unwrap_or(usize::MAX),
);
}
/// Hint every managed direct block of the heap (see
/// [`Storage::hint`]), for a caller about to read all of its objects (a
/// dense group's listing). The indirect blocks leading to them are read
/// here, as every object read goes through them, a few levels deep and
/// up to [`MAX_HINTED_BLOCKS`] entries; direct blocks are only hinted.
/// Nothing is returned and no error: a storage that has the file in
/// memory skips it, and the objects are read (and checked) as before.
pub fn hint_managed_blocks<S: Storage + ?Sized>(&self, file: &S) {
if file.as_contiguous().is_some()
|| is_undefined(self.root_block_address, self.offset_size)
|| self.current_rows_in_root_indirect_block == 0
{
// A direct root block is what `hint_root_block` hints.
return;
}
let mut budget = MAX_HINTED_BLOCKS;
self.hint_indirect_block(
file,
self.root_block_address,
usize::from(self.current_rows_in_root_indirect_block),
0,
&mut budget,
);
}
fn hint_indirect_block<S: Storage + ?Sized>(
&self,
file: &S,
addr: u64,
nrows: usize,
depth: usize,
budget: &mut usize,
) {
let os = self.offset_size;
let layout = self.indirect_layout(nrows, os);
let len = layout.len_upto(layout.all_entries);
if depth > 4 || len > 1 << 20 {
return;
}
let Ok(bytes) = read_upto(file, addr, len) else {
return;
};
if bytes.len() < len || bytes.get(..4) != Some(b"FHIB".as_slice()) {
return;
}
let start_indirect = self.max_direct_rows();
let mut pos = layout.header;
for row in 0..nrows {
for _ in 0..self.table_width {
let Some(left) = budget.checked_sub(1) else {
return;
};
*budget = left;
let Ok(child) = read_offset(&bytes, pos, os) else {
return;
};
if row < start_indirect {
let size = if self.filter_pipeline.is_some() {
match read_offset(&bytes, pos + usize::from(os), self.length_size) {
Ok(n) => n,
Err(_) => return,
}
} else {
self.block_size_for_row(row)
};
pos += layout.direct_entry;
if !is_undefined(child, os) {
file.hint(child, usize::try_from(size).unwrap_or(usize::MAX));
}
} else {
pos += layout.indirect_entry;
if !is_undefined(child, os) {
let rows = self.rows_for_size(self.block_size_for_row(row));
self.hint_indirect_block(file, child, usize::from(rows), depth + 1, budget);
}
}
}
}
}
/// Which child entry of an indirect block (numbered in walk order:
/// direct rows, then indirect rows) covers `target_offset`, from the
/// doubling-table geometry alone — the entry
@@ -939,6 +1048,31 @@ impl FractalHeapHeader {
}
}
/// Where an indirect block's child entries are: after its header, the
/// direct blocks' entries (address, and for a filtered heap the stored
/// size and filter mask), then the child indirect blocks' (address).
struct IndirectLayout {
header: usize,
direct_entry: usize,
direct_entries: usize,
indirect_entry: usize,
all_entries: usize,
}
impl IndirectLayout {
/// Bytes from the block's start to the end of its first `n` entries.
fn len_upto(&self, n: usize) -> usize {
let direct = n.min(self.direct_entries);
self.header
.saturating_add(direct.saturating_mul(self.direct_entry))
.saturating_add(
n.min(self.all_entries)
.saturating_sub(direct)
.saturating_mul(self.indirect_entry),
)
}
}
/// A managed direct block's location, extent and (for a filtered heap) its
/// stored size and filter mask.
/// The child of an indirect block that covers a heap offset.