osobhandClaude Opus 5.5
b55b24b7ba
Every crate under crates/ now has a README (android, bench, cli, napi and wasm had none), each saying what the crate is, its main types and functions (names checked against the code), its cargo features with defaults and which ones build C (checked with `cargo tree`), and links to the top-level docs. Corrections to the old stubs: - clawhdf5-derive: the derive is `H5Type`, not `HDF5Type`, and it needs clawhdf5-format as a dependency. - clawhdf5-filters: deflate backends only, and no library crate depends on it; the filter pipeline and every other codec are in -format. - clawhdf5-gpu: vector distance compute, not I/O; not used by HDF5Memory::search. - clawhdf5-io: MpiVol is root-read + broadcast, not collective MPI-IO. - clawhdf5-ann: from_hdf5/search(q, k) did not exist; load_from_hdf5 and search(q, k, ef). - clawhdf5-accel: checksum::crc32_simd did not exist; the SSE4 and wasm backends are reported but run the scalar kernels. - clawhdf5-gpu: the old example called l2_distances, which does not exist (l2_search). - clawhdf5-agent: it described a "vector store" with "GPU acceleration"; it now covers HDF5Memory, search options, WAL, signing, the graph. - crates.io/docs.rs badges removed and `cargo install <crate>` replaced: nothing is published; depend on git. - fuzz: the opt-in CLAWHDF5_FUZZ_SECONDS smoke run in ci-test.sh. - tools: the FileEditor interop tests that live in this crate. - remote, py: license, other front ends, limits, File.mode/flush/chunks. The Rust examples of the facade, format, filters, accel, ann, derive and agent READMEs were compiled and run as tests (netcdf4, gpu and remote compiled only) in a scratch crate; the CLI example was run. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
2.6 KiB
clawhdf5-io
I/O building blocks under clawhdf5: the
HDF5Read/HDF5ReadWrite traits with in-memory, borrowed, file and
memory-mapped readers, plus several experimental modules (async reads, an
HSDS client, a VOL-style trait, sub-filing, prefetch, and an MPI connector).
The facade uses it for memory-mapped reads (MmapReader, and the private
copy-on-write mapping that applies a metadata cache image).
Remote files are not read through this crate: HTTP(S) and object
stores go through clawhdf5_format::storage::Storage and
clawhdf5-remote.
Not on crates.io yet; depend on it from git:
[dependencies]
clawhdf5-io = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5", features = ["mmap"] }
Main items
| Item | What |
|---|---|
HDF5Read, HDF5ReadWrite |
byte-level read/write traits; MemoryReader, BorrowedReader, FileReader, FileWriter implement them |
MmapReader, MmapReadWrite (mmap) |
memory-mapped files through memmap2; HDF5Read::private_copy gives a copy-on-write view |
prefetch::PrefetchReader, sweep::SweepDetector |
read-ahead (madvise(MADV_WILLNEED) on mappings) and chunk-sweep prediction |
ParallelConfig |
lane partitioning for parallel chunk decoding |
vol::VirtualObjectLayer, vol::NativeVol |
a backend-agnostic object-layer trait (modelled on libhdf5's VOL) |
async_read (async) |
tokio-based AsyncHDF5Read and AsyncHDF5File |
hsds::HsdsClient (hsds) |
a REST client for an HSDS server |
subfiling |
splitting one logical file across several physical files |
mpi_vol::MpiVol (mpi-io) |
an MPI connector: see below |
MPI (mpi-io)
MpiVol is not collective MPI-IO. Reads are root-read + broadcast
(rank 0 reads the file with std::fs::read, parses the dataset and
broadcasts the bytes); writes gather every rank's shard to rank 0, which
writes the merged dataset. It does not call MPI_File_read_at_all or any
other MPI-IO routine. Collective I/O is on the roadmap.
clawhdf5-bench's mpi_io_bench binary exercises it.
Features
| Feature | Default | What | Builds C |
|---|---|---|---|
mmap |
no (the clawhdf5 facade turns it on) |
MmapReader, MmapReadWrite |
no |
async |
no | async_read (tokio) |
no |
hsds |
no | hsds (reqwest, and async) |
yes: reqwest's default TLS is native-tls (OpenSSL on Linux) |
mpi-io |
no | a real MpiVol (without it MpiVol::new_world returns an error) |
yes: mpi-sys needs an MPI installation and libclang |
License
MIT