osobhandClaude Opus 5.5
c27a478e44
Numbers, API names, feature defaults and PR references checked against CONFORMANCE.md, BENCHMARKS.md, CHANGELOG.md, the code and git history. - int8 index figures (1.74x memory, 1.63x QPS) carry the dates git gives them (2026-09-19/20, machine not recorded, not re-run) instead of none; the Pi 5 1.18x carries 2026-09-21. - BENCHMARKS headline: the libhdf5 chunked-write figure is the newest measurement (35x, 2026-09-23), not 45.3x (2026-08-03). - Conformance counts follow the 2026-09-28 run (1 our-error, 2 ref-bug) in conformance/README.md, ROADMAP.md and CLAUDE.md, with a pointer to the bad_nbit_parms_walk.h5 flip. - README: LZ4 is opt-in; the browser refuses reference/opaque/bitfield/ time datasets too; zlib-rs byte-identity scoped to what was measured; macOS default links the system libz for inflate. - Crate READMEs: system-zlib-decompress does something (macOS), SweepDetector lives in prefetch, checkpoint after more than 500 WAL entries, NetCDF-4 unlimited-dimension size warning. - agent-memory.md: string-dataset compression threshold, agents-md prints Markdown, float16 file sizes linked to their study. - known-issues.md: contiguous selection reads, 1.21x vs h5py threads. - docs/README.md, USE_CASES.md, ROADMAP.md, CLAUDE.md: range-read milestones M0-M5 and PRs #17-#19, missing README rows, CLI keygen/verify, dated figures, fast-math is not BLAS. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
56 lines
2.6 KiB
Markdown
56 lines
2.6 KiB
Markdown
# clawhdf5-io
|
|
|
|
I/O building blocks under [`clawhdf5`](../clawhdf5/README.md): the
|
|
`HDF5Read`/`HDF5ReadWrite` traits with in-memory, borrowed, file and
|
|
memory-mapped readers, plus several experimental modules (async reads, an
|
|
HSDS client, a VOL-style trait, sub-filing, prefetch, and an MPI connector).
|
|
The facade uses it for memory-mapped reads (`MmapReader`, and the private
|
|
copy-on-write mapping that applies a metadata cache image).
|
|
|
|
Remote files are **not** read through this crate: HTTP(S) and object
|
|
stores go through `clawhdf5_format::storage::Storage` and
|
|
[`clawhdf5-remote`](../clawhdf5-remote/README.md).
|
|
|
|
Not on crates.io yet; depend on it from git:
|
|
|
|
```toml
|
|
[dependencies]
|
|
clawhdf5-io = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5", features = ["mmap"] }
|
|
```
|
|
|
|
## Main items
|
|
|
|
| Item | What |
|
|
|---|---|
|
|
| `HDF5Read`, `HDF5ReadWrite` | byte-level read/write traits; `MemoryReader`, `BorrowedReader`, `FileReader`, `FileWriter` implement them |
|
|
| `MmapReader`, `MmapReadWrite` (`mmap`) | memory-mapped files through `memmap2`; `HDF5Read::private_copy` gives a copy-on-write view |
|
|
| `prefetch::PrefetchReader`, `prefetch::SweepDetector`, `sweep` | read-ahead (`madvise(MADV_WILLNEED)` on mappings) and chunk-sweep prediction |
|
|
| `ParallelConfig` | lane partitioning for parallel chunk decoding |
|
|
| `vol::VirtualObjectLayer`, `vol::NativeVol` | a backend-agnostic object-layer trait (modelled on libhdf5's VOL) |
|
|
| `async_read` (`async`) | tokio-based `AsyncHDF5Read` and `AsyncHDF5File` |
|
|
| `hsds::HsdsClient` (`hsds`) | a REST client for an HSDS server |
|
|
| `subfiling` | splitting one logical file across several physical files |
|
|
| `mpi_vol::MpiVol` (`mpi-io`) | an MPI connector: see below |
|
|
|
|
### MPI (`mpi-io`)
|
|
|
|
`MpiVol` is **not collective MPI-IO**. Reads are root-read + broadcast
|
|
(rank 0 reads the file with `std::fs::read`, parses the dataset and
|
|
broadcasts the bytes); writes gather every rank's shard to rank 0, which
|
|
writes the merged dataset. It does not call `MPI_File_read_at_all` or any
|
|
other MPI-IO routine. Collective I/O is on the [roadmap](../../ROADMAP.md).
|
|
`clawhdf5-bench`'s `mpi_io_bench` binary exercises it.
|
|
|
|
## Features
|
|
|
|
| Feature | Default | What | Builds C |
|
|
|---|---|---|---|
|
|
| `mmap` | no (the `clawhdf5` facade turns it on) | `MmapReader`, `MmapReadWrite` | no |
|
|
| `async` | no | `async_read` (tokio) | no |
|
|
| `hsds` | no | `hsds` (reqwest, and `async`) | yes: reqwest's default TLS is native-tls (OpenSSL on Linux) |
|
|
| `mpi-io` | no | a real `MpiVol` (without it `MpiVol::new_world` returns an error) | yes: `mpi-sys` needs an MPI installation and libclang |
|
|
|
|
## License
|
|
|
|
MIT
|