osobhandClaude Opus 5.5
3100f0143b
- docs/README.md links the improvement logs and June plans where the refresh archived them (docs/archive/). - clawhdf5-py README: 'r+' creates and replaces attributes (compact or dense); only deleting them is unsupported. - scripts/run-benchmarks.sh benchmarked the pre-rename rustyhdf5-format and overwrote BENCHMARKS.md; nothing referenced it. Removed. - Cargo.toml descriptions no longer name rustyhdf5/edgehdf5; clawhdf5-gpu says it is not HDF5 I/O. - benchmarks/cross_platform.sh pointed at a ROADMAP section that no longer exists. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
clawhdf5-io
I/O building blocks under clawhdf5: the
HDF5Read/HDF5ReadWrite traits with in-memory, borrowed, file and
memory-mapped readers, plus several experimental modules (async reads, an
HSDS client, a VOL-style trait, sub-filing, prefetch, and an MPI connector).
The facade uses it for memory-mapped reads (MmapReader, and the private
copy-on-write mapping that applies a metadata cache image).
Remote files are not read through this crate: HTTP(S) and object
stores go through clawhdf5_format::storage::Storage and
clawhdf5-remote.
Not on crates.io yet; depend on it from git:
[dependencies]
clawhdf5-io = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5", features = ["mmap"] }
Main items
| Item | What |
|---|---|
HDF5Read, HDF5ReadWrite |
byte-level read/write traits; MemoryReader, BorrowedReader, FileReader, FileWriter implement them |
MmapReader, MmapReadWrite (mmap) |
memory-mapped files through memmap2; HDF5Read::private_copy gives a copy-on-write view |
prefetch::PrefetchReader, sweep::SweepDetector |
read-ahead (madvise(MADV_WILLNEED) on mappings) and chunk-sweep prediction |
ParallelConfig |
lane partitioning for parallel chunk decoding |
vol::VirtualObjectLayer, vol::NativeVol |
a backend-agnostic object-layer trait (modelled on libhdf5's VOL) |
async_read (async) |
tokio-based AsyncHDF5Read and AsyncHDF5File |
hsds::HsdsClient (hsds) |
a REST client for an HSDS server |
subfiling |
splitting one logical file across several physical files |
mpi_vol::MpiVol (mpi-io) |
an MPI connector: see below |
MPI (mpi-io)
MpiVol is not collective MPI-IO. Reads are root-read + broadcast
(rank 0 reads the file with std::fs::read, parses the dataset and
broadcasts the bytes); writes gather every rank's shard to rank 0, which
writes the merged dataset. It does not call MPI_File_read_at_all or any
other MPI-IO routine. Collective I/O is on the roadmap.
clawhdf5-bench's mpi_io_bench binary exercises it.
Features
| Feature | Default | What | Builds C |
|---|---|---|---|
mmap |
no (the clawhdf5 facade turns it on) |
MmapReader, MmapReadWrite |
no |
async |
no | async_read (tokio) |
no |
hsds |
no | hsds (reqwest, and async) |
yes: reqwest's default TLS is native-tls (OpenSSL on Linux) |
mpi-io |
no | a real MpiVol (without it MpiVol::new_world returns an error) |
yes: mpi-sys needs an MPI installation and libclang |
License
MIT