Files

T

osobhandClaude Opus 5.5 c27a478e44 docs: fact-check the refreshed documentation against its sources

Numbers, API names, feature defaults and PR references checked against
CONFORMANCE.md, BENCHMARKS.md, CHANGELOG.md, the code and git history.

- int8 index figures (1.74x memory, 1.63x QPS) carry the dates git gives
  them (2026-09-19/20, machine not recorded, not re-run) instead of none;
  the Pi 5 1.18x carries 2026-09-21.
- BENCHMARKS headline: the libhdf5 chunked-write figure is the newest
  measurement (35x, 2026-09-23), not 45.3x (2026-08-03).
- Conformance counts follow the 2026-09-28 run (1 our-error, 2 ref-bug)
  in conformance/README.md, ROADMAP.md and CLAUDE.md, with a pointer to
  the bad_nbit_parms_walk.h5 flip.
- README: LZ4 is opt-in; the browser refuses reference/opaque/bitfield/
  time datasets too; zlib-rs byte-identity scoped to what was measured;
  macOS default links the system libz for inflate.
- Crate READMEs: system-zlib-decompress does something (macOS), SweepDetector
  lives in prefetch, checkpoint after more than 500 WAL entries, NetCDF-4
  unlimited-dimension size warning.
- agent-memory.md: string-dataset compression threshold, agents-md prints
  Markdown, float16 file sizes linked to their study.
- known-issues.md: contiguous selection reads, 1.21x vs h5py threads.
- docs/README.md, USE_CASES.md, ROADMAP.md, CLAUDE.md: range-read
  milestones M0-M5 and PRs #17-#19, missing README rows, CLI keygen/verify,
  dated figures, fast-math is not BLAS.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

2026-09-28 11:23:51 -05:00

2.6 KiB

Raw Blame History

clawhdf5-io

I/O building blocks under clawhdf5: the HDF5Read/HDF5ReadWrite traits with in-memory, borrowed, file and memory-mapped readers, plus several experimental modules (async reads, an HSDS client, a VOL-style trait, sub-filing, prefetch, and an MPI connector). The facade uses it for memory-mapped reads (MmapReader, and the private copy-on-write mapping that applies a metadata cache image).

Remote files are not read through this crate: HTTP(S) and object stores go through clawhdf5_format::storage::Storage and clawhdf5-remote.

Not on crates.io yet; depend on it from git:

[dependencies]
clawhdf5-io = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5", features = ["mmap"] }

Main items

Item	What
`HDF5Read`, `HDF5ReadWrite`	byte-level read/write traits; `MemoryReader`, `BorrowedReader`, `FileReader`, `FileWriter` implement them
`MmapReader`, `MmapReadWrite` (`mmap`)	memory-mapped files through `memmap2`; `HDF5Read::private_copy` gives a copy-on-write view
`prefetch::PrefetchReader`, `prefetch::SweepDetector`, `sweep`	read-ahead (`madvise(MADV_WILLNEED)` on mappings) and chunk-sweep prediction
`ParallelConfig`	lane partitioning for parallel chunk decoding
`vol::VirtualObjectLayer`, `vol::NativeVol`	a backend-agnostic object-layer trait (modelled on libhdf5's VOL)
`async_read` (`async`)	tokio-based `AsyncHDF5Read` and `AsyncHDF5File`
`hsds::HsdsClient` (`hsds`)	a REST client for an HSDS server
`subfiling`	splitting one logical file across several physical files
`mpi_vol::MpiVol` (`mpi-io`)	an MPI connector: see below

MPI (`mpi-io`)

MpiVol is not collective MPI-IO. Reads are root-read + broadcast (rank 0 reads the file with std::fs::read, parses the dataset and broadcasts the bytes); writes gather every rank's shard to rank 0, which writes the merged dataset. It does not call MPI_File_read_at_all or any other MPI-IO routine. Collective I/O is on the roadmap. clawhdf5-bench's mpi_io_bench binary exercises it.

Features

Feature	Default	What	Builds C
`mmap`	no (the `clawhdf5` facade turns it on)	`MmapReader`, `MmapReadWrite`	no
`async`	no	`async_read` (tokio)	no
`hsds`	no	`hsds` (reqwest, and `async`)	yes: reqwest's default TLS is native-tls (OpenSSL on Linux)
`mpi-io`	no	a real `MpiVol` (without it `MpiVol::new_world` returns an error)	yes: `mpi-sys` needs an MPI installation and libclang

License

MIT

2.6 KiB Raw Blame History