diff --git a/CHANGELOG.md b/CHANGELOG.md index 29804fc..a877c30 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,52 @@ ## Unreleased +### Chunked full reads (2026-09-26) +- **Chunks are decoded straight into the output, into reused buffers.** A + full read of a chunked dataset faulted in about three times its size in + fresh pages: every chunk was decoded into a new buffer per filter stage + (the cached reader behind `read_*` decoded 128 chunks at a time before + placing any; the uncached one behind `MmapFile`, `LazyFile` and + `verify_provenance` decoded the whole dataset first), then assembled into + a byte buffer, which the typed readers copied once more. Now each chunk + is decoded into buffers the thread keeps between chunks and reads + (`clawhdf5_format::filters::DecodeScratch`, + `decompress_chunk_exact_with`: deflate inflates into a kept buffer with a + reset inflater, shuffle into the other one, Fletcher32 is checked in + place; other filters go through the registry as before) and copied + directly to its place in the output. Chunks still go into the file's + chunk cache when the whole dataset fits. Selection reads decode the + chunks they touch the same way. +- **Typed full reads of chunked data skip the byte buffer.** `read_f32`, + `read_f64`, `read_i32`, `read_i64` and `read_u64` (on `File`, `MmapFile` + and `LazyFile`) of a chunked dataset stored as that type in native byte + order decode every chunk into the returned `Vec` (huge-page backed when + large, like the byte readers' output); other types and byte orders + convert as before. New public + `clawhdf5_format::data_read::read_chunked_native`. +- **Reading threads no longer wait for a busy rayon pool.** A full read + handed its chunks to rayon and the calling thread slept until the pool had + decoded them, so with a small pool (2-4 threads) readers outside it queued + behind its workers. The calling thread now decodes too, and pool workers + join in only when free; a helper the pool starts after the read has + finished returns at once. A single read still spreads over the default + pool. This replaces the one-thread-pool special case below for full + reads. Chunks are placed from several threads only when the chunk index + puts them on the chunk grid at distinct places; a corrupt index is read + one chunk at a time, and the error reported is still the first failing + chunk's. New test `crates/clawhdf5/tests/busy_decode_pool.rs`. +- **Fixed:** in a filtered dataset, a chunk stored with every filter skipped + (filter mask) and shorter than a chunk read with zeros in place of its + missing part through `File`'s `read_*`; it is now an error naming the + chunk, as `MmapFile`/`LazyFile` already made it. +- New h5py comparison `crates/clawhdf5/tests/chunked_read_paths_interop.rs`: + every chunked read path (cached and uncached full reads, `MmapFile`, + `LazyFile`, small, strided and point selections, with and without the + `parallel` feature) for 1-8-byte integers and 2-8-byte floats in both + byte orders, through deflate, shuffle, Fletcher32, LZF, SZIP and Blosc, + with partial edge chunks, sparse datasets and fill values, and datasets + larger than the chunk cache. + ### Concurrent reads (2026-09-26) - **Full reads of chunked datasets scale with threads again when rayon's pool has one thread.** Each full read handed its chunks to rayon to diff --git a/docs/known-issues.md b/docs/known-issues.md index 2657cdf..0d2556b 100644 --- a/docs/known-issues.md +++ b/docs/known-issues.md @@ -26,8 +26,9 @@ performance" below. ## Concurrent and contiguous read performance (measured 2026-09-26) -**Status:** open for chunked full reads (one cause fixed 2026-09-26); the -contiguous item is fixed (2026-09-26). Measured on +**Status:** open for chunked full reads at 16 threads until re-measured +(the causes identified below are fixed as of 2026-09-26); the contiguous +item is fixed (2026-09-26). Measured on tank with `concurrent_read` against h5py 3.16 / HDF5 2.0 (`BENCHMARKS.md`, "Concurrent reads"): - **Partly fixed 2026-09-26.** Full reads of chunked datasets from several threads @@ -48,9 +49,19 @@ tank with `concurrent_read` against h5py 3.16 / HDF5 2.0 (`BENCHMARKS.md`, readers outside it still wait on its workers. Datasets larger than the cache's budget were already read without inserting into it, and skipping its lookups entirely gained only a few percent at 16 threads. Remaining - per-read overhead, not yet addressed: each full `read_f32` of a chunked - dataset faults in about three times its size in fresh pages (the output, - the `f32` copy of it, and a new buffer per decoded chunk). + per-read overhead: each full `read_f32` of a chunked dataset faults in + about three times its size in fresh pages (the output, the `f32` copy of + it, and a new buffer per decoded chunk). + **Fixed 2026-09-26** (both causes; see `CHANGELOG.md`, "Chunked full + reads"): chunks are decoded into buffers each thread reuses and copied + straight into the output, and the typed readers decode into the `Vec` + they return, so a full read no longer faults in a buffer per chunk or a + second copy of its output; and the reading + thread decodes its own chunks with pool workers helping when free, so no + reader waits on a small or busy pool + (`crates/clawhdf5/tests/busy_decode_pool.rs`). The 16-thread comparison + with h5py processes has not been re-measured yet (tank was busy with + other work); this item stays open until it is. - Contiguous datasets read 4x slower than h5py on one thread (2.5 vs 9.8 GB/s full, 0.12x for 256 x 256 hyperslabs). **Fixed 2026-09-26** (re-measured on tank at `408f69e`: 13665 MB/s