feat: add Pcodec lossless numerical compression filter (arXiv:2502.06112)

Implements Pcodec (filter ID 32023) via the `pco` 1.0.x crate as a new
optional compression codec. Pcodec achieves 30–94% better compression
ratio than Zstd for f32/f64 columnar data at 1–5 GiB/s decompression
speed, making it ideal for write-once/read-many embedding archives.

Write throughput at 512×512: 591 MiB/s (parity with Zstd-3 at 610 MiB/s).
For smaller chunks Zstd-3 remains faster due to Pcodec's fixed per-chunk
distributional analysis overhead.

- Add FILTER_PCODEC = 32023 constant to filter_pipeline.rs
- Add pcodec_compress/pcodec_decompress using pco::standalone API
- Wire into compress_chunk/decompress_chunk dispatch
- Add ChunkOptions.pcodec field and DatasetBuilder.with_pcodec() method
- Enable pcodec as highest-priority codec in build_pipeline()
- Add pco dep (optional, feature = "pcodec") to clawhdf5-format/clawhdf5
- Add write_2d_chunked_pcodec benchmark comparing pcodec vs zstd-3
- Document results in BENCHMARKS.md

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

This commit is contained in:

Omar Sobh

2026-07-01 01:16:08 +00:00

co-authored by

Claude Sonnet 4.6

parent e82b8f56bd

commit 5701e8045d

9 changed files with 184 additions and 7 deletions

									
										crates/clawhdf5-bench/Cargo.toml
									
		+1
		-1
	
												View File
												
				@@ -59,7 +59,7 @@ tempfile = "3"

				hdf5 = { version = "0.12", optional = true, package = "hdf5-metno" }

				[dev-dependencies]

				clawhdf5 = { path = "../clawhdf5", features = ["zstd"] }

				clawhdf5 = { path = "../clawhdf5", features = ["zstd", "pcodec"] }

				criterion = { version = "0.5", features = ["html_reports"] }

				[features]