quantumclaw/clawhdf5
3
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

perf(format): partial selection reads; out-of-range selections are errors

Browse Source 
read_raw_data_selection computed which chunks a selection intersects, threw
the answer away, decoded the entire dataset and picked elements out of it —
for contiguous layouts too. A 64x64 window of a 64 MB deflate dataset cost
105 ms, about half a full read; every selection cost the same whatever its
size.

New partial_read module: materialise only the selection's bounding box — the
overlapping rows of a contiguous dataset (straight from the file bytes) or the
overlapping chunks (only those are decompressed) — then run the existing
extractor over that buffer with the selection translated to the box origin, so
extraction semantics are exactly the full-read ones. It declines (falling back
to the old path) for All/None, compact/virtual/storage-less layouts, and boxes
covering more than half the dataset. That window now takes 0.39 ms, one row
2.7 ms, one column 5.2 ms.

Selections are validated against the dataset shape first. They were not: a
hyperslab past an edge came back padded with zeros and a point with an
out-of-range column wrapped into the next row, returning the wrong element
with no error. Now FormatError::SelectionOutOfBounds (also rank mismatch and
overlapping blocks); the facade's fill-aware path validates too.

Tests: equivalence against a reference extraction from a full read over 60
random hyperslabs/point lists per layout (contiguous, chunked, deflate) for
ranks 1-3. New read_harness bench binary with before/after in BENCHMARKS.md.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
This commit is contained in:
osobh
2026-09-19 13:57:14 -07:00
co-authored by Claude Fable 5.1
parent 3027380979
commit c6a7bbfc67
10 changed files with 842 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
BENCHMARKS.md
+63
View File
@@ -28,6 +28,69 @@
---
## Read harness
Produced by `cargo run --release -p clawhdf5-bench --bin read_harness`: a 4096 x
2048 `f64` dataset (64 MB) written three ways, read in full and through four
hyperslab selections, each from a fresh file handle. The last column is the
point: does a selection cost what the *selection* costs?
### Baseline (v2.4.0): every selection decodes the whole dataset
4096 x 2048 f64 (64 MB per dataset), chunks 256 x 256, file 129 MB
| layout | read | selected | time ms | MB/s of selection | vs full read |
|---|---|---:|---:|---:|---:|
| chunked + deflate | full (first) | 64 MB | 181.8 | 352 | |
| chunked + deflate | full (repeat) | 64 MB | 162.1 | 395 | 1.00x |
| chunked + deflate | 64 x 64 window (1 chunk) | 0.03 MB | 104.89 | 0 | 0.577x |
| chunked + deflate | 512 x 512 window (4-9 chunks) | 2.00 MB | 110.26 | 18 | 0.606x |
| chunked + deflate | one row | 0.02 MB | 105.81 | 0 | 0.582x |
| chunked + deflate | one column | 0.03 MB | 108.37 | 0 | 0.596x |
| chunked | full (first) | 64 MB | 97.4 | 657 | |
| chunked | full (repeat) | 64 MB | 86.7 | 738 | 1.00x |
| chunked | 64 x 64 window (1 chunk) | 0.03 MB | 40.97 | 1 | 0.420x |
| chunked | 512 x 512 window (4-9 chunks) | 2.00 MB | 44.66 | 45 | 0.458x |
| chunked | one row | 0.02 MB | 30.88 | 1 | 0.317x |
| chunked | one column | 0.03 MB | 30.27 | 1 | 0.311x |
| contiguous | full (first) | 64 MB | 57.6 | 1112 | |
| contiguous | full (repeat) | 64 MB | 53.5 | 1195 | 1.00x |
| contiguous | 64 x 64 window (1 chunk) | 0.03 MB | 30.97 | 1 | 0.538x |
| contiguous | 512 x 512 window (4-9 chunks) | 2.00 MB | 31.64 | 63 | 0.550x |
| contiguous | one row | 0.02 MB | 31.90 | 0 | 0.554x |
| contiguous | one column | 0.03 MB | 29.36 | 1 | 0.510x |
### After: partial reads
Only the rows of a contiguous dataset, or the chunks, that overlap the
selection's bounding box are read/decoded. A 64 x 64 window of the compressed
dataset: **105 -> 0.39 ms**; one row: **106 -> 2.7 ms**; one column:
**108 -> 5.2 ms**. (Absolute full-read times differ between the two runs
because the machine's speed drifted; compare the *vs full read* column.)
4096 x 2048 f64 (64 MB per dataset), chunks 256 x 256, file 129 MB
| layout | read | selected | time ms | MB/s of selection | vs full read |
|---|---|---:|---:|---:|---:|
| chunked + deflate | full (first) | 64 MB | 112.5 | 569 | |
| chunked + deflate | full (repeat) | 64 MB | 104.5 | 612 | 1.00x |
| chunked + deflate | 64 x 64 window (1 chunk) | 0.03 MB | 0.39 | 81 | 0.003x |
| chunked + deflate | 512 x 512 window (4-9 chunks) | 2.00 MB | 4.85 | 412 | 0.043x |
| chunked + deflate | one row | 0.02 MB | 2.69 | 6 | 0.024x |
| chunked + deflate | one column | 0.03 MB | 5.23 | 6 | 0.046x |
| chunked | full (first) | 64 MB | 70.0 | 915 | |
| chunked | full (repeat) | 64 MB | 61.7 | 1037 | 1.00x |
| chunked | 64 x 64 window (1 chunk) | 0.03 MB | 0.06 | 541 | 0.001x |
| chunked | 512 x 512 window (4-9 chunks) | 2.00 MB | 1.99 | 1005 | 0.028x |
| chunked | one row | 0.02 MB | 0.05 | 285 | 0.001x |
| chunked | one column | 0.03 MB | 0.45 | 69 | 0.006x |
| contiguous | full (first) | 64 MB | 60.3 | 1062 | |
| contiguous | full (repeat) | 64 MB | 56.4 | 1134 | 1.00x |
| contiguous | 64 x 64 window (1 chunk) | 0.03 MB | 0.08 | 396 | 0.001x |
| contiguous | 512 x 512 window (4-9 chunks) | 2.00 MB | 2.12 | 944 | 0.035x |
| contiguous | one row | 0.02 MB | 0.03 | 576 | 0.000x |
| contiguous | one column | 0.03 MB | 2.55 | 12 | 0.042x |
## Search harness baseline (v2.3.0)
Produced by `cargo run --release -p clawhdf5-bench --bin search_harness -- --full`