# clawhdf5-io

I/O building blocks under [`clawhdf5`](../clawhdf5/README.md): the
`HDF5Read`/`HDF5ReadWrite` traits with in-memory, borrowed, file and
memory-mapped readers, plus several experimental modules (async reads, an
HSDS client, a VOL-style trait, sub-filing, prefetch, and an MPI connector).
The facade uses it for memory-mapped reads (`MmapReader`, and the private
copy-on-write mapping that applies a metadata cache image).

Remote files are **not** read through this crate: HTTP(S) and object
stores go through `clawhdf5_format::storage::Storage` and
[`clawhdf5-remote`](../clawhdf5-remote/README.md).

Not on crates.io yet; depend on it from git:

```toml
[dependencies]
clawhdf5-io = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5", features = ["mmap"] }
```

## Main items

| Item | What |
|---|---|
| `HDF5Read`, `HDF5ReadWrite` | byte-level read/write traits; `MemoryReader`, `BorrowedReader`, `FileReader`, `FileWriter` implement them |
| `MmapReader`, `MmapReadWrite` (`mmap`) | memory-mapped files through `memmap2`; `HDF5Read::private_copy` gives a copy-on-write view |
| `prefetch::PrefetchReader`, `sweep::SweepDetector` | read-ahead (`madvise(MADV_WILLNEED)` on mappings) and chunk-sweep prediction |
| `ParallelConfig` | lane partitioning for parallel chunk decoding |
| `vol::VirtualObjectLayer`, `vol::NativeVol` | a backend-agnostic object-layer trait (modelled on libhdf5's VOL) |
| `async_read` (`async`) | tokio-based `AsyncHDF5Read` and `AsyncHDF5File` |
| `hsds::HsdsClient` (`hsds`) | a REST client for an HSDS server |
| `subfiling` | splitting one logical file across several physical files |
| `mpi_vol::MpiVol` (`mpi-io`) | an MPI connector: see below |

### MPI (`mpi-io`)

`MpiVol` is **not collective MPI-IO**. Reads are root-read + broadcast
(rank 0 reads the file with `std::fs::read`, parses the dataset and
broadcasts the bytes); writes gather every rank's shard to rank 0, which
writes the merged dataset. It does not call `MPI_File_read_at_all` or any
other MPI-IO routine. Collective I/O is on the [roadmap](../../ROADMAP.md).
`clawhdf5-bench`'s `mpi_io_bench` binary exercises it.

## Features

| Feature | Default | What | Builds C |
|---|---|---|---|
| `mmap` | no (the `clawhdf5` facade turns it on) | `MmapReader`, `MmapReadWrite` | no |
| `async` | no | `async_read` (tokio) | no |
| `hsds` | no | `hsds` (reqwest, and `async`) | yes: reqwest's default TLS is native-tls (OpenSSL on Linux) |
| `mpi-io` | no | a real `MpiVol` (without it `MpiVol::new_world` returns an error) | yes: `mpi-sys` needs an MPI installation and libclang |

## License

MIT