fix(format): key the shared chunk cache by dataset

A File is Send + Sync and keeps one ChunkCache for all its datasets.
The cached readers bound that cache to "the current dataset" with
ensure_dataset(addr), then checked, built and read its index and its
decompressed chunks in separate lock acquisitions. Two threads reading
two chunked datasets interleaved those steps, so one could store its
chunk index under the other's binding, or get the other's decompressed
chunk for the same coordinate: wrong data, or an index-out-of-bounds
panic when the ranks differed (16 threads x 40 reads over 24 datasets
panicked on every run).

The cache now keeps per-dataset state keyed by chunk-index address:
the chunk index, ChunkIndex and ChunkLayout per dataset (held as Arcs,
built outside the lock, first writer wins), and decompressed chunks
keyed by (address, coordinate). The chunked readers use the new
addr-taking methods (chunks_for, chunk_layout_for, get/put_decompressed_in,
prefetch_hint_in) exclusively. Memory stays bounded: decompressed data by
the existing byte/slot budget across datasets, indexes by at most 64
datasets and 2^20 index entries in total, dropping the least recently
used dataset's index first. Switching datasets no longer throws away the
other datasets' cached chunks.

The address-less methods remain and act on the dataset last bound with
ensure_dataset; they are documented as not for concurrent readers.

Regression: threads_reading_different_datasets_get_their_own_chunks
(crates/clawhdf5/tests/concurrent_chunk_cache.rs), plus cache unit tests
datasets_sharing_coordinates_stay_separate, dataset_indexes_are_bounded
and concurrent_readers_of_different_datasets_see_their_own_chunks.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-25 21:12:52 -05:00
co-authored by Claude Opus 5.5
parent d074385944
commit 9066d34eaa
3 changed files with 660 additions and 351 deletions
@@ -0,0 +1,87 @@
//! A `File` is `Send + Sync` and keeps one chunk cache for all its datasets.
//! Threads reading different chunked datasets through the same `File` must
//! each get their own dataset's data.
use std::sync::Arc;
use clawhdf5::{File, FileBuilder};
const DATASETS: usize = 24;
const THREADS: usize = 16;
const ROUNDS: usize = 40;
/// Contents of dataset `k`: distinct from every other dataset's, element for
/// element, so any chunk served from the wrong dataset shows.
fn values(k: usize, n: usize) -> Vec<f64> {
(0..n).map(|i| (k * 100_000 + i) as f64).collect()
}
fn build() -> File {
let mut b = FileBuilder::new();
for k in 0..DATASETS {
let ds = b.create_dataset(&format!("d{k:02}"));
match k % 3 {
// 1-D, compressed: chunk offsets 0, 8, 16, ... in every dataset.
0 => {
ds.with_f64_data(&values(k, 64)).with_shape(&[64]);
ds.with_chunks(&[8]).with_deflate(1);
}
// 1-D, shuffle + compressed, a different length.
1 => {
ds.with_f64_data(&values(k, 40)).with_shape(&[40]);
ds.with_chunks(&[8]).with_shuffle().with_deflate(1);
}
// 2-D, compressed: coordinates (0,0), (0,4), (4,0), ... overlap
// the other datasets' in the first dimension.
_ => {
ds.with_f64_data(&values(k, 64)).with_shape(&[8, 8]);
ds.with_chunks(&[4, 4]).with_deflate(1);
}
}
}
File::from_bytes(b.finish().unwrap()).unwrap()
}
fn expected(k: usize) -> Vec<f64> {
values(k, if k % 3 == 1 { 40 } else { 64 })
}
#[test]
fn threads_reading_different_datasets_get_their_own_chunks() {
let file = Arc::new(build());
// Sequential sanity check first.
for k in 0..DATASETS {
let got = file.dataset(&format!("d{k:02}")).unwrap().read_f64();
assert_eq!(got.unwrap(), expected(k), "sequential d{k:02}");
}
let handles: Vec<_> = (0..THREADS)
.map(|t| {
let file = Arc::clone(&file);
std::thread::spawn(move || {
let mut wrong = Vec::new();
for round in 0..ROUNDS {
let k = (t * 7 + round * 5) % DATASETS;
let name = format!("d{k:02}");
match file.dataset(&name).unwrap().read_f64() {
Ok(v) if v == expected(k) => {}
Ok(v) => wrong.push(format!("{name}: wrong data, first {:?}", &v[..4])),
Err(e) => wrong.push(format!("{name}: {e}")),
}
}
wrong
})
})
.collect();
let failures: Vec<String> = handles
.into_iter()
.flat_map(|h| h.join().unwrap())
.collect();
assert!(
failures.is_empty(),
"{} of {} concurrent reads were wrong, e.g. {:?}",
failures.len(),
THREADS * ROUNDS,
&failures[..failures.len().min(5)]
);
}