fix(read): decode on the calling thread when rayon's pool has one thread

Full reads of chunked datasets handed their chunks to rayon. With a
one-thread pool (concurrent_read --decode-threads 1, RAYON_NUM_THREADS=1)
every thread reading through a File queued behind that single worker, so
16 readers decoded on one core: per-thread CPU time showed one thread
doing all the decoding and the readers almost none, and full reads
stopped at about 2x one thread. The cached full-read path and the
uncached reader behind verify_provenance now decode inline when the pool
cannot parallelise (parallel_read::pool_can_parallelise).

The File's chunk cache was the suspect but not the cause: datasets over
its budget were already read without inserting, and skipping its lookups
gained only a few percent at 16 threads.

The regression test keeps a one-thread global pool's worker busy and
requires a full read and verify_provenance to finish anyway; before the
fix both waited for the worker (timed out).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

crates/clawhdf5-format/src/chunked_read.rs

@@ -40,6 +40,7 @@ fn decompress_all_chunks(
     {
         if let Some(pl) = pipeline
             && parallel_read::should_use_parallel(chunks.len())
+            && parallel_read::pool_can_parallelise()
         {
             // Seed from the first chunk's address and count for determinism.
             let seed = chunks.first().map(|c| c.address).unwrap_or(0) ^ (chunks.len() as u64);
@@ -1056,7 +1057,9 @@ pub fn read_chunked_data_cached(
 
     // Decompress what the cache didn't have, a bounded batch at a time — in
     // parallel with the `parallel` feature (this path, the one the facade
-    // uses, was sequential; only the uncached reader was parallel). Chunks are
+    // uses, was sequential; only the uncached reader was parallel), unless the
+    // pool has one thread: then every reading thread would queue behind that
+    // one worker, so each decodes its own chunks instead. Chunks are
     // cached only when the whole dataset fits: pushing a larger dataset
     // through the cache just evicts each chunk moments after inserting it.
     let cache_them = total_bytes <= cache.max_bytes();
@@ -1073,12 +1076,13 @@ pub fn read_chunked_data_cached(
         };
         for batch in misses.chunks(DECODE_BATCH) {
             #[cfg(feature = "parallel")]
-            let decoded: Vec<Result<Vec<u8>, FormatError>> = if batch.len() >= 4 {
-                use rayon::prelude::*;
-                batch.par_iter().map(decode).collect()
-            } else {
-                batch.iter().map(decode).collect()
-            };
+            let decoded: Vec<Result<Vec<u8>, FormatError>> =
+                if batch.len() >= 4 && parallel_read::pool_can_parallelise() {
+                    use rayon::prelude::*;
+                    batch.par_iter().map(decode).collect()
+                } else {
+                    batch.iter().map(decode).collect()
+                };
             #[cfg(not(feature = "parallel"))]
             let decoded: Vec<Result<Vec<u8>, FormatError>> = batch.iter().map(decode).collect();
 

crates/clawhdf5-format/src/parallel_read.rs

@@ -27,6 +27,20 @@ pub fn should_use_parallel(chunk_count: usize) -> bool {
     chunk_count > PARALLEL_THRESHOLD
 }
 
+/// Whether handing a read's chunks to rayon can decode them faster than the
+/// calling thread would alone.
+///
+/// `false` when the pool the work would go to (the current pool inside a
+/// rayon worker, else the global one) has a single thread. Handing work to
+/// that pool is then worse than useless: the caller blocks while the one
+/// worker decodes, and every other thread reading at the same time queues
+/// behind the same worker, so N reader threads decode on one core. (That is
+/// how full reads with `--decode-threads 1` stopped scaling at about 2x in
+/// the `concurrent_read` benchmark.)
+pub fn pool_can_parallelise() -> bool {
+    rayon::current_num_threads() > 1
+}
+
 /// Decompress chunks in parallel using lane-partitioned assignment.
 ///
 /// Instead of naive `par_iter`, chunks are deterministically assigned to lanes