quantumclaw/clawhdf5
3
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

docs: chunked full reads decode in place; small pools no longer block

Browse Source 
CHANGELOG entry for the chunked read changes, and the known-issues entry
on concurrent chunked reads updated: both causes it names (per-read page
faults, readers waiting on a small pool) are fixed; the 16-thread
comparison with h5py stays open until re-measured on an idle machine.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-26 10:36:44 -05:00
co-authored by Claude Opus 5.5
parent 9e608b975c
commit 94b6df986c
2 changed files with 62 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/known-issues.md
+16 -5
View File
@@ -26,8 +26,9 @@ performance" below.
## Concurrent and contiguous read performance (measured 2026-09-26)
**Status:** open for chunked full reads (one cause fixed 2026-09-26); the
contiguous item is fixed (2026-09-26). Measured on
**Status:** open for chunked full reads at 16 threads until re-measured
(the causes identified below are fixed as of 2026-09-26); the contiguous
item is fixed (2026-09-26). Measured on
tank with `concurrent_read` against h5py 3.16 / HDF5 2.0 (`BENCHMARKS.md`,
"Concurrent reads"):
- **Partly fixed 2026-09-26.** Full reads of chunked datasets from several threads
@@ -48,9 +49,19 @@ tank with `concurrent_read` against h5py 3.16 / HDF5 2.0 (`BENCHMARKS.md`,
readers outside it still wait on its workers. Datasets larger than the
cache's budget were already read without inserting into it, and skipping
its lookups entirely gained only a few percent at 16 threads. Remaining
per-read overhead, not yet addressed: each full `read_f32` of a chunked
dataset faults in about three times its size in fresh pages (the output,
the `f32` copy of it, and a new buffer per decoded chunk).
per-read overhead: each full `read_f32` of a chunked dataset faults in
about three times its size in fresh pages (the output, the `f32` copy of
it, and a new buffer per decoded chunk).
**Fixed 2026-09-26** (both causes; see `CHANGELOG.md`, "Chunked full
reads"): chunks are decoded into buffers each thread reuses and copied
straight into the output, and the typed readers decode into the `Vec<T>`
they return, so a full read no longer faults in a buffer per chunk or a
second copy of its output; and the reading
thread decodes its own chunks with pool workers helping when free, so no
reader waits on a small or busy pool
(`crates/clawhdf5/tests/busy_decode_pool.rs`). The 16-thread comparison
with h5py processes has not been re-measured yet (tank was busy with
other work); this item stays open until it is.
- Contiguous datasets read 4x slower than h5py on one thread (2.5 vs
9.8 GB/s full, 0.12x for 256 x 256 hyperslabs).
**Fixed 2026-09-26** (re-measured on tank at `408f69e`: 13665 MB/s