CHANGELOG.md

@@ -2,6 +2,52 @@
 
 ## Unreleased
 
+### Chunked full reads (2026-09-26)
+- **Chunks are decoded straight into the output, into reused buffers.** A
+  full read of a chunked dataset faulted in about three times its size in
+  fresh pages: every chunk was decoded into a new buffer per filter stage
+  (the cached reader behind `read_*` decoded 128 chunks at a time before
+  placing any; the uncached one behind `MmapFile`, `LazyFile` and
+  `verify_provenance` decoded the whole dataset first), then assembled into
+  a byte buffer, which the typed readers copied once more. Now each chunk
+  is decoded into buffers the thread keeps between chunks and reads
+  (`clawhdf5_format::filters::DecodeScratch`,
+  `decompress_chunk_exact_with`: deflate inflates into a kept buffer with a
+  reset inflater, shuffle into the other one, Fletcher32 is checked in
+  place; other filters go through the registry as before) and copied
+  directly to its place in the output. Chunks still go into the file's
+  chunk cache when the whole dataset fits. Selection reads decode the
+  chunks they touch the same way.
+- **Typed full reads of chunked data skip the byte buffer.** `read_f32`,
+  `read_f64`, `read_i32`, `read_i64` and `read_u64` (on `File`, `MmapFile`
+  and `LazyFile`) of a chunked dataset stored as that type in native byte
+  order decode every chunk into the returned `Vec` (huge-page backed when
+  large, like the byte readers' output); other types and byte orders
+  convert as before. New public
+  `clawhdf5_format::data_read::read_chunked_native`.
+- **Reading threads no longer wait for a busy rayon pool.** A full read
+  handed its chunks to rayon and the calling thread slept until the pool had
+  decoded them, so with a small pool (2-4 threads) readers outside it queued
+  behind its workers. The calling thread now decodes too, and pool workers
+  join in only when free; a helper the pool starts after the read has
+  finished returns at once. A single read still spreads over the default
+  pool. This replaces the one-thread-pool special case below for full
+  reads. Chunks are placed from several threads only when the chunk index
+  puts them on the chunk grid at distinct places; a corrupt index is read
+  one chunk at a time, and the error reported is still the first failing
+  chunk's. New test `crates/clawhdf5/tests/busy_decode_pool.rs`.
+- **Fixed:** in a filtered dataset, a chunk stored with every filter skipped
+  (filter mask) and shorter than a chunk read with zeros in place of its
+  missing part through `File`'s `read_*`; it is now an error naming the
+  chunk, as `MmapFile`/`LazyFile` already made it.
+- New h5py comparison `crates/clawhdf5/tests/chunked_read_paths_interop.rs`:
+  every chunked read path (cached and uncached full reads, `MmapFile`,
+  `LazyFile`, small, strided and point selections, with and without the
+  `parallel` feature) for 1-8-byte integers and 2-8-byte floats in both
+  byte orders, through deflate, shuffle, Fletcher32, LZF, SZIP and Blosc,
+  with partial edge chunks, sparse datasets and fill values, and datasets
+  larger than the chunk cache.
+
 ### Concurrent reads (2026-09-26)
 - **Full reads of chunked datasets scale with threads again when rayon's
   pool has one thread.** Each full read handed its chunks to rayon to

docs/known-issues.md

@@ -26,8 +26,9 @@ performance" below.
 
 ## Concurrent and contiguous read performance (measured 2026-09-26)
 
-**Status:** open for chunked full reads (one cause fixed 2026-09-26); the
+**Status:** open for chunked full reads at 16 threads until re-measured
-contiguous item is fixed (2026-09-26). Measured on
+(the causes identified below are fixed as of 2026-09-26); the contiguous
+item is fixed (2026-09-26). Measured on
 tank with `concurrent_read` against h5py 3.16 / HDF5 2.0 (`BENCHMARKS.md`,
 "Concurrent reads"):
 - **Partly fixed 2026-09-26.** Full reads of chunked datasets from several threads
@@ -48,9 +49,19 @@ tank with `concurrent_read` against h5py 3.16 / HDF5 2.0 (`BENCHMARKS.md`,
   readers outside it still wait on its workers. Datasets larger than the
   cache's budget were already read without inserting into it, and skipping
   its lookups entirely gained only a few percent at 16 threads. Remaining
-  per-read overhead, not yet addressed: each full `read_f32` of a chunked
+  per-read overhead: each full `read_f32` of a chunked dataset faults in
-  dataset faults in about three times its size in fresh pages (the output,
+  about three times its size in fresh pages (the output, the `f32` copy of
-  the `f32` copy of it, and a new buffer per decoded chunk).
+  it, and a new buffer per decoded chunk).
+  **Fixed 2026-09-26** (both causes; see `CHANGELOG.md`, "Chunked full
+  reads"): chunks are decoded into buffers each thread reuses and copied
+  straight into the output, and the typed readers decode into the `Vec<T>`
+  they return, so a full read no longer faults in a buffer per chunk or a
+  second copy of its output; and the reading
+  thread decodes its own chunks with pool workers helping when free, so no
+  reader waits on a small or busy pool
+  (`crates/clawhdf5/tests/busy_decode_pool.rs`). The 16-thread comparison
+  with h5py processes has not been re-measured yet (tank was busy with
+  other work); this item stays open until it is.
 - Contiguous datasets read 4x slower than h5py on one thread (2.5 vs
   9.8 GB/s full, 0.12x for 256 x 256 hyperslabs).
   **Fixed 2026-09-26** (re-measured on tank at `408f69e`: 13665 MB/s