quantumclaw/clawhdf5
3
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

Merge pull request 'Range reads M0/M1 (indexed lookups, Storage trait), ZFP, in-place editing' (#17) from feat/p3-range-zfp-edit into main
CI / test-arm64 (push) Successful in 1m30s
Details
CI / test (push) Successful in 14m27s
Details

Browse Source 
Reviewed-on: #17
This commit was merged in pull request #17.
This commit is contained in:
osobh
2026-09-26 20:42:21 +00:00
parent 479d8b47e0 0645dcf173
commit 8f59b2e1c2
79 changed files with 13057 additions and 820 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+274
View File
@@ -2,6 +2,249 @@
## Unreleased
### Name lookups through the name index (2026-09-26)
- **Finding one link or attribute by name reads the name index, not every
entry.** In a dense group (links in a fractal heap) the v2 B-tree name
index (record type 5, lookup3 hash of the name) is descended to the
records with the name's hash and only their links are read — O(log n)
instead of all n. Path resolution (`File::dataset`, `resolve_path_any`,
soft-link targets) and `Group::dataset`/`Group::group` (on `File`,
`MmapFile` and `LazyFile`, which listed the whole group per call) use it;
names whose hashes collide are all compared, so the order libhdf5 gives
them does not matter. New `clawhdf5_format::group_v2::resolve_child`,
`btree_v2::find_btree_v2_records` (records in one key range), and a
`lookup-stats` feature counting heap objects read, for tests. Huge heap
objects are found through their index the same way.
- **`attr(name)`** on the facade's groups and datasets (all three file
types): one attribute, found in dense storage through its name index
(record type 8) instead of reading every attribute
(`clawhdf5_format::attribute::find_attribute_in_file`).
- **Two links of one name: the first wins everywhere.** A group cannot
validly hold two links of one name, but a damaged or hand-made one can.
The listing, `resolve_child` (`Group::dataset`/`group`) and path
resolution now all use only the first link of a name (header message
order in a compact group, name index order in a dense one) and ignore
the rest, even if the first dangles — libhdf5's rule for compact groups
(h5py fails to open a dangling first link although a later one
resolves). Before, the listing skipped a dangling first link and listed
the name via a later one that lookup did not follow, and path resolution
followed the last.
- **B-tree v2 internal nodes are checksum-verified.** Lookups prune
children by internal-node keys, so a corrupted internal node could hide
a name with no error; a mismatch is now `ChecksumMismatch`, as in
libhdf5, for lookups and listings alike.
- **`Group::entries()` and `File::group_at(address)`**: a listing's
`(name, address)` pairs, to open children without looking names up again.
- Test: `crates/clawhdf5/tests/indexed_lookup_interop.rs` — every child of
an h5py-written 35 001-link group (with colliding hashes) opened by name
reads at most two links per lookup (before: 35 001), matches h5py, and
every link kind (soft, relative, dangling, external) resolves as h5py
resolves it in dense and compact groups.
### Checked address conversion (2026-09-26)
- **No 64-bit file value is truncated on a 32-bit target.** All 119
truncating `u64 as usize` casts in `clawhdf5-format` that clippy's
`cast_possible_truncation` reports, under every feature the crate is
built with in CI except `szip` (115 with default features and
`plugin-filters`, 4 more behind `parallel`), are gone: file addresses,
lengths and counts go through `addr::to_usize`, which fails with
`FormatError::Overflow` where the value does not fit (wasm32 and other
32-bit targets; it used to wrap onto another part of the file), and
in-memory counts through `addr::saturating_usize`. On 64-bit targets
nothing changes. `scripts/check-32bit-casts.sh` (run by `ci-test.sh`)
lints the crate with no default features, with default features, and
with every optional feature but `szip` (for wasm32; the set with `zstd`,
which does not build for wasm32, for the host), and fails on any new
truncating cast. The facade, `clawhdf5-io` and `clawhdf5-ann` are not
covered.
### Range reads, milestone M1: the `Storage` trait (2026-09-26)
- **Breaking: `clawhdf5_format::error::FormatError` and `clawhdf5::Error`
are now `#[non_exhaustive]`.** An exhaustive `match` on either needs a
wildcard arm. `FormatError` has two new variants: `Storage(String)` (a
storage backend failed to serve a read) and
`ContiguousStorageRequired(&'static str)` (an operation not yet converted
to range reads was asked of a backend without the whole file in memory).
- New `clawhdf5_format::storage::Storage`, the synchronous, `no_std` read
interface of `docs/design/range-reads.md` option (a): `read_at(offset:
u64, len) -> Cow<[u8]>`, `read_ranges`, `len()` and an `as_contiguous()`
fast path; implemented for `[u8]`, `Vec<u8>`, and references, `Box`es
and (with `std`) `Arc`s of a `Storage`. Slices and `Vec`s serve borrowed
bytes, so parsing an in-memory file costs no copy.
- **The metadata parsers read through `Storage`.** Each converted parser has
an `*_in<S: Storage + ?Sized>(&S, ..)` core (a `&dyn Storage` works too),
and its `&[u8]` function is now a thin wrapper over it, so no caller
changes. The wrappers compile to a `[u8]` instance of the same code, so a
structure read in memory is a bounds check and a borrowed slice, with no
indirect call and no copy. Converted: the superblock
(`Superblock::parse_in`), its extension and cache image
(`read_superblock_extension_in`, `cache_image_state_in`), object headers
with their continuation chunks (`ObjectHeader::parse_in`), local and
global heaps, symbol-table nodes and the group B-tree (v1), fractal heaps,
fixed and extensible array chunk indexes, shared messages and the SOHM
table (`message_data_in`, `message_data_with_sohm_in`,
`load_sohm_table_in`, …), attributes (`extract_attributes_full_in`,
`extract_attributes_tolerant_in`, `AttributeMessage::parse_in_storage`),
fill values (`dataset_fill_value_from_storage`) and virtual-dataset
mappings (`DataLayout::resolve_vds_mappings_in`); also
`signature::find_signature_in`. Each structure is read with bounded reads
(a prefix, then the structure) instead of slicing the whole file; the
open-ended `&file_data[addr..]` slices in these modules are gone. Bounds
errors keep their values (absolute position, file length).
- Reads sized by untrusted fields are bounded by what the parser uses, so
a crafted size cannot turn one structure into a read of the rest of the
file on a range backend: local-heap names are read in growing pieces
(64 bytes first) rather than to the end of the data segment; a fractal
heap indirect block is read up to the entry covering the object (the
whole block only when that entry is unallocated); paged fixed and
extensible array data blocks over 1 MiB are read page by page, only the
pages in use; and a block under one checksum whose claimed size runs
past the end of the file fails its bounds check before any read (with
the `checksum` feature). An object header's prefix is one read (was
two).
- Structures still indexed by a v2 B-tree — dense attribute storage, a SOHM
B-tree index and huge fractal-heap objects found through their B-tree —
are not converted yet (the
v2 B-tree and dense groups come with milestone M3); over a backend without
the whole file in memory they are the clean `ContiguousStorageRequired`
error, never a partial result. Raw data, chunk B-tree (v1) indexes and VL
data are milestone M2.
- **No behaviour change**, checked three ways (2026-09-26, tank): every
existing test passes unchanged; the conformance sweep
(`conformance/run.sh --no-fetch`) gives a byte-identical `results.json`
at `f2ff2c4` and on this branch, and identical per-file probe output for
696 of the 697 files — the exception, `cve-2025-2310.h5`, reports one of
two errors depending on which parallel chunk decode fails first, at
`f2ff2c4` as on this branch; and a transcript of every converted `&[u8]`
function's result over the fixtures, the conformance corpus and the
h5py-written files below (748 files, 7 603 object headers) is
byte-identical between the two builds.
- **Speed on local files** (provisional: tank was shared with other jobs;
both builds linked into one binary and timed alternately, 200 rounds;
new Criterion bench `clawhdf5/benches/local_metadata_bench.rs` over a
400-group version-1 file, `clawhdf5-format/tests/fixtures/v1_groups_400.h5`):
against `f2ff2c4`, listing the file through the facade is 2.7% faster,
`ObjectHeader::parse` is within ±1%, symbol-table nodes and the group
B-tree walk are about 19% faster (their entry loops were tightened),
local-heap names and `resolve_group_children` 1.5–3% faster. The same
harness run on two copies of the old code differs by up to 2%. The
Criterion bench itself, old and new as separate binaries run alternately
(3 rounds), agrees except for `ObjectHeader::parse`, which it puts about
7% slower (25.9 vs 24.1 µs for 401 headers) while the listing that
parses those headers is 5–8% faster. **Rechecked on an idle tank
(2026-09-26, load 1.95 at the start, 3–4 during; Criterion, `main`
`479d8b4` vs this branch `b49ec39` as separate binaries, 2 alternating
rounds):** `ObjectHeader::parse` for 401 headers 24.0–24.1 µs → 25.5–25.8
µs (+6–7%, about 4 ns per header — real, not noise); symbol-table nodes
2.39 → 1.94–2.04 µs (−17%); group B-tree walk 405–411 → 322–351 ns
(−16%); listing the 400-group file through the facade 8.56–8.71 →
8.40–8.42 ms (−2.4%). Net, local metadata reads are slightly faster; the
per-header cost is a known, small regression. Data reads are unchanged:
`concurrent_read --decode-threads 1` (the `BENCHMARKS.md` "Concurrent
reads" workload), `main` and this branch alternating, 2 rounds each, on
the same idle start: every deflate and contiguous full-read row within
±2.4% of `main` (the cache-bound contiguous hyperslab rows vary by up to
±40% between `main`'s own rounds and are not comparable).
- New equivalence harness `clawhdf5-format/tests/storage_equivalence.rs`:
every converted parser runs over the file as a slice and over
`storage::CountingStorage` — a `Storage` that serves an in-memory buffer
through `read_at` only (`as_contiguous()` is `None`), copying what it
serves and counting reads — and must give identical results. It walks
the fixtures, files h5py writes for it (extensible arrays with super
blocks and paged data blocks, paged fixed arrays, large v1 and dense
groups, a user block, SOHM list and B-tree indexes, dense, shared and
committed-type attributes, fixed and extensible array data blocks over
1 MiB, whole and truncated), and with `CLAWHDF5_STORAGE_CORPUS=<dir>` a
corpus (all 653 HDF5 files of the conformance corpus pass). Only the
structures listed above as not converted may answer
`ContiguousStorageRequired`, and only in the checks that reach them; a
converted parser falling back to the whole file fails it. Milestones M2
and M3 extend it.
### ZFP (2026-09-26)
- **ZFP (filter 32013, H5Z-ZFP) reads, in pure Rust.** It was the last
filter in the conformance corpus that failed with `UnsupportedFilter`.
New feature `zfp` (`clawhdf5-format` and `clawhdf5`, included in
`plugin-filters`, no dependencies) ports the zfp 1.0.1 decoder and the
decompression half of H5Z-ZFP 1.1.1: every mode (fixed rate, fixed
precision, fixed accuracy, reversible, expert), int32, int64, float and
double, 1-4-D fields with partial blocks, and headers written by
big-endian machines (values byte-swapped as H5Z-ZFP does). Read only:
there is no ZFP encoder. The decoder is deterministic, so lossy modes have
one right answer, and clawhdf5 returns exactly libzfp's values.
- Tests: `crates/clawhdf5/tests/zfp_interop.rs` has h5py + hdf5plugin 7.1
(H5Z-ZFP 1.1.1, zfp 1.0.1) write 2205 datasets over 16 mode settings
(including expert parameters at their edges) x the four types x 1-4-D
shapes with partial edge chunks, partial blocks and unit chunk
dimensions x smooth, noisy, wide-range, zero and inf/NaN data; each must
read byte for byte as h5py reads it. `tests/zfp_alloc_bounds.rs` fuzzes
headers and streams under a counting allocator: no panics, and the output
is allocated only when it matches the chunk's size and the stream holds at
least a bit per block. A stream that ends before the decoder is done is an
error (libzfp reads past its buffer).
- Conformance on tank (2026-09-26, `conformance/run.sh --no-fetch`): 600 of
697 files ok (599 before); `h5ex_d_zfp.h5` now reads.
### In-place modification (2026-09-26)
- **`clawhdf5::FileEditor` modifies an existing file where it lies.**
`FileBuilder` builds whole files in memory; the editor opens a file
written by libhdf5 (any `libver`, including HDF5 2.0's own format) or by
clawhdf5 and changes only what an edit touches, recomputing the checksum
of every structure it changes. It takes an exclusive `flock` on the file
(the lock libhdf5 takes), so a second editor gets `Error::Locked`.
- `write_selection` / `write_all` / `write_values`: overwrite values of a
compact, contiguous (also never-written, late-allocated) or chunked
dataset, in its own datatype, under any selection. Chunks are decoded,
updated and re-encoded through the dataset's filters; a chunk that no
longer fits moves to the end of the file. New chunks are added to
version-1 B-tree (every chunked dataset of h5py's default `libver`),
Extensible Array, Fixed Array and single-chunk indexes — creating the
index, its data blocks, super blocks and pages, and splitting B-tree
nodes, as libhdf5 does: after the same sequence of writes the B-tree has
the same number of nodes per level and the Extensible Array header the
same block statistics as libhdf5's (tested). Filters run as libhdf5's
`H5Z_pipeline` runs them (new
`clawhdf5_format::filters::compress_chunk_masked`): an optional filter
that fails — LZF or Blosc output no smaller than the chunk — is skipped
and its filter-mask bit set, so the chunk is stored exactly as h5py
stores it; a mandatory filter that fails fails the edit. (Storing such
a chunk LZF-encoded at the raw size with a clear mask let a later
libhdf5 rewrite of it keep the stale mask, and h5py could no longer
read the dataset.)
- `resize`: grow a chunked dataset up to its maximum dimensions (h5py's
`Dataset.resize`).
- `set_attr`: add or replace an attribute in an object header, in free
space or in a new continuation chunk at the end of the file. A
version-2 header (h5py `libver='v110'` and later) without an Attribute
Info message gets one, as libhdf5's `H5O__attr_create` adds it: libhdf5
counts such a header's attributes through that message, and without it
h5py reported `len(obj.attrs) == 0` while listing them.
- Each edit is planned in memory and refused as a whole
(`Error::Unsupported`, file untouched) when any part is not supported:
new chunks in a version-2 B-tree index (two or more unlimited
dimensions) or an implicit index, shrinking, variable-length and
reference data, chunks through a filter this build cannot encode
(scale-offset, N-Bit, SZIP), attributes in dense storage, past an
object's compact limit or with tracked creation order, files with a
metadata cache
image, paged or persistent free space, or marked open by another
writer. New error variants `Error::Unsupported`,
`Error::InvalidArgument`, `Error::Locked`, and `clawhdf5::Error` is now
`#[non_exhaustive]` — a breaking change for code that matches it
exhaustively (the Python bindings map the new variants to
`NotImplementedError`, `ValueError` and `OSError`).
- Durability: the new space (chunks, index blocks) is written and synced
before any existing byte changes, then the metadata that links it in,
then a second sync. There is no journal: a crash during the second
phase can leave the file inconsistent (as with libhdf5 without SWMR).
Freed space is not reused (see `docs/known-issues.md`).
- Tests: `crates/clawhdf5-tools/tests/edit_interop.rs` (h5py `earliest`,
`v114` and `latest` files and clawhdf5 files; after every round h5py
reads the expected values, h5dump and `h5rs check --data` accept the
file, and h5py `r+` modifies it further; random operations against a
model) and `crates/clawhdf5/tests/edit_tests.rs`.
- `clawhdf5_format::type_builders::build_attr_message` is public.
### Chunked full reads (2026-09-26)
- **Chunks are decoded straight into the output, into reused buffers.** A
full read of a chunked dataset faulted in about three times its size in
@@ -1002,6 +1245,37 @@ and fails their objects (see below).
- CI keeps zlib-ng building and tested; the arm64 job no longer needs cmake.
### Correctness
- **`FileBuilder` stored LZF and Blosc chunks with filter mask 0 even when
the filter had not shrunk them** (fixed 2026-09-26). Latent in the
unreleased LZF/Blosc writer only (added 2026-09-26, "Plugin filters"):
no tagged release writes LZF or Blosc, so v2.7.0 and earlier are
unaffected. libhdf5 treats LZF and Blosc output no smaller than the chunk
as a filter failure and, both being optional filters, stores such a chunk
unfiltered with the filter's mask bit set. clawhdf5 stored the filter's
output with a clear mask. For an LZF chunk whose stream was exactly the
chunk's size (h5py stores `[182, 0, 0, 0, 0]` in a 5-byte chunk raw),
the first libhdf5 rewrite of that chunk stored the new data raw at the
same size and, the size being unchanged, kept the stale mask 0: h5py
then failed to read the dataset ("filter returned failure during read").
The whole-file writer now runs chunks through the pipeline as libhdf5
does (`clawhdf5_format::filters::compress_chunk_masked`, as `FileEditor`
already did) and records each chunk's real mask in every chunk index it
builds (single chunk, Fixed Array, Extensible Array, version-2 B-tree;
it builds no version-1 B-tree or implicit index), in the sequential and
`parallel` paths and `create_datasets_parallel`. Files whose chunks all
compress are byte-identical to before. `PrecompressedChunks::chunks` is
now `(raw size, stored bytes, filter mask)` (**breaking** for code that
reads it). Files written before the fix read correctly; rewrite them
(with this build or `h5repack`) before modifying them with libhdf5.
Tests: `plugin_filters_interop`
`skipped_optional_filters_are_masked_as_libhdf5_masks_them` (LZF,
shuffle+LZF+fletcher32 and Blosc, random, compressible and alternating
chunks, every index: masks equal an h5py-written twin's; after h5py r+
rewrites and extends the datasets, h5py, h5dump and our reader read every
value — before the fix 20 of 24 datasets had other masks than h5py's,
and h5py could not read the rewritten `[x, 0, 0, 0, 0]` datasets) and
`files_whose_chunks_all_compress_are_unchanged`; `chunked_write`
`skipped_lzf_chunks_are_masked_in_every_index`.
- **Scale-offset data read wrong values in every release that decoded it
(v2.2.0 to v2.7.0), silently, on ordinary h5py files** (fixed
2026-09-26). Of 1480 scale-offset datasets h5py writes across every
CLAUDE.md
+8 -1
View File
@@ -11,7 +11,7 @@ Cargo workspace with 18 crates under `crates/` (plus `libaec-sys`, an internal F
|-------|------|
| `clawhdf5-format` | HDF5 binary spec parser (superblock, B-tree, heap) — also holds shared type definitions and physical constants |
| `clawhdf5-io` | Read/write implementation |
| `clawhdf5-filters` | Deflate backends (zlib-rs, zlib-ng, Apple Compression); the HDF5 filter pipeline, the filter registry (`clawhdf5_format::filter_registry`) and the other codecs (LZ4, Zstd, SZIP, N-Bit, scale-offset, pcodec, and the pure-Rust plugin filters LZF, bitshuffle, bzip2, Blosc 1, and Blosc2 read-only) live in `clawhdf5-format`. No ZFP. |
| `clawhdf5-filters` | Deflate backends (zlib-rs, zlib-ng, Apple Compression); the HDF5 filter pipeline, the filter registry (`clawhdf5_format::filter_registry`) and the other codecs (LZ4, Zstd, SZIP, N-Bit, scale-offset, pcodec, and the pure-Rust plugin filters LZF, bitshuffle, bzip2, Blosc 1, and Blosc2 and ZFP read-only) live in `clawhdf5-format`. |
| `clawhdf5-derive` | Proc-macro derive for HDF5-serializable structs |
| `clawhdf5` | Main facade crate |
| `clawhdf5-netcdf4` | NetCDF-4 compatibility layer |
@@ -150,6 +150,13 @@ Cargo workspace with 18 crates under `crates/` (plus `libaec-sys`, an internal F
Alerts never block a save — drain them with `HDF5Memory::take_anomaly_alerts`.
`MemorySource` for this bookkeeping is inferred from the caller-supplied
`source_channel` string (a heuristic, not an authenticated trust boundary).
- In-place modification: `clawhdf5::FileEditor` (`crates/clawhdf5/src/edit/`)
overwrites values, grows chunked datasets and sets attributes in existing
files (h5py- or clawhdf5-written) without rewriting them; anything it
cannot do safely is `Error::Unsupported` before any write (limits in
`docs/known-issues.md`). Test changes with
`cargo test -p clawhdf5-tools --test edit_interop` (h5py, h5dump,
`h5rs check`).
- GPU-accelerated vector distance computation (`clawhdf5-gpu`, wgpu); HDF5 I/O itself is CPU-only
- Browser: `clawhdf5-wasm` (wasm-bindgen, read-only, file held in memory;
no Zstd/SZIP since they link C) and the `examples/wasm-viewer/` page.
CONFORMANCE.md
+6 -7
View File
@@ -13,15 +13,15 @@ fatal. This file is generated by `conformance/run.sh`; do not edit it by hand.
| | |
|---|---|
| date | 2026-09-26 17:10 UTC |
| clawhdf5 commit | `d0e3beb3aa8290aae523ce280b4380e75484b6bc` |
| date | 2026-09-26 20:06 UTC |
| clawhdf5 commit | `8fadb9f4242a35323262701328d380806d379140` |
| machine | `tank`: AMD Ryzen 7 7800X3D 8-Core Processor, 16 CPUs, 61 GiB, Linux 7.0.0-34-generic x86_64 |
| command | `conformance/run.sh --no-fetch --update-baseline` |
| rustc | rustc 1.98.1 (48a229cea 2026-09-01) |
| reference | h5py 3.16.0, HDF5 2.0.0, numpy 2.5.3, hdf5plugin 7.1.0, Python 3.14.4 |
| h5dump | Version 1.14.6 (CVE corpus only) |
| limits | 20 s timeout (SIGKILL), 4096 MiB address space, per process; 16 files in parallel |
| runtime | 24 s probing + comparing (0 s fetch/build before it) |
| runtime | 21 s probing + comparing (0 s fetch/build before it) |
## Results
@@ -38,16 +38,16 @@ A file's class is the first that applies:
| NCAS-CMS_pyfive | 33 | 32 | 0 | 1 | 0 | 0 | 0 | 0 | 0 |
| cve_hdf5 | 147 | 113 | 2 | 0 | 32 | 0 | 0 | 0 | 0 |
| h5py_data | 4 | 4 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| hdf5 | 466 | 403 | 2 | 1 | 60 | 0 | 0 | 0 | 0 |
| hdf5 | 466 | 404 | 1 | 1 | 60 | 0 | 0 | 0 | 0 |
| netcdf-c | 20 | 20 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| netcdf4-python | 18 | 18 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| usnistgov_h5wasm | 5 | 5 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| xarray-data | 4 | 4 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| **all** | **697** | **599** | **4** | **2** | **92** | **0** | **0** | **0** | **0** |
| **all** | **697** | **600** | **3** | **2** | **92** | **0** | **0** | **0** | **0** |
2 of the 2 mismatches are a known h5py bug, not ours (see *Known not-our-bug*).
3 of the 4 our-errors are corrupt data that HDF5 2.0 reads only through a bug and clawhdf5 refuses (see *Known not-our-bug*).
3 of the 3 our-errors are corrupt data that HDF5 2.0 reads only through a bug and clawhdf5 refuses (see *Known not-our-bug*).
Corpora (fetched by `conformance/fetch-corpus.sh` into the gitignored `conformance/.cache/`):
@@ -73,7 +73,6 @@ Grouped by normalised error message. *files* counts files whose class this cause
| files | objects | error | examples |
|---:|---:|---|---|
| 3 | 3 | `ChunkedReadError("…")` | `cve_hdf5/cvefiles/cve-2025-2308.h5`, `cve_hdf5/cvefiles/cve-2025-44904.h5`, `hdf5/test/testfiles/bad_nbit_parms_walk.h5` |
| 1 | 1 | `UnsupportedFilter(N)` | `hdf5/HDF5Examples/C/H5FLT/tfiles/h5ex_d_zfp.h5` |
## Mismatch root causes
README.md
+22 -5
View File
@@ -433,6 +433,23 @@ b.write("groups.h5")?;
A group holds at most 65 535 links; more is an error, as is a link over
65 515 bytes (a very long soft-link target) in a group of more than 8 links.
### Modifying an existing file
```rust
use clawhdf5::{AttrValue, FileEditor, Selection};
// A file from h5py or clawhdf5, dataset "x" chunked with maxshape=(None,).
let mut ed = FileEditor::open("data.h5")?; // exclusive lock, like libhdf5
ed.resize("x", &[1100])?; // h5py: ds.resize((1100,))
let sel = Selection::Hyperslab { start: vec![1000], stride: vec![1], count: vec![100], block: vec![1] };
ed.write_values("x", &sel, &[0.5f64; 100])?; // ds[1000:1100] = 0.5
ed.set_attr("x", "units", &AttrValue::String("m/s".into()))?;
```
Each call changes the file in place (no rewrite) and syncs it. What it
cannot change safely is refused before anything is written; see
[known issues](docs/known-issues.md) for the limits.
### Python
`crates/clawhdf5-py` is a Python package (PyO3 + numpy) that reads HDF5 with
@@ -780,14 +797,14 @@ stores keep their setting. Opt out with `float16 = false` or
| `bzip2` | no | bzip2 filter (id 307): read and write. Pure Rust (the `bzip2` crate's libbz2-rs-sys backend compiles no C) |
| `blosc` | no | Blosc 1 filter (id 32001): reads BloscLZ, LZ4/LZ4HC, Snappy, Zlib and Zstandard frames with byte or bit shuffle; writes LZ4, Snappy, Zlib or Zstandard (not BloscLZ). Pure Rust |
| `blosc2` | no | Blosc2 filter (id 32026), read only: hdf5plugin's frames and B2ND (n-D) chunks, BloscLZ, LZ4/LZ4HC, Zlib and Zstandard, with shuffle, bit shuffle, delta or truncated precision. Pure Rust |
| `plugin-filters` | no | All five above |
| `zfp` | no | ZFP filter (id 32013, H5Z-ZFP), read only: every mode (rate, precision, accuracy, reversible, expert) for int32, int64, float and double, 1-4-D, returning exactly libzfp's values. Pure Rust, no dependencies |
| `plugin-filters` | no | All six above |
ZFP (32013) is not implemented: reading it fails with `UnsupportedFilter`,
whose message names the filter. clawhdf5 cannot write Blosc2. Any other
clawhdf5 cannot write Blosc2 or ZFP. Any other
filter can be supplied at run time with `filter_registry::register_filter` (a
decoder closure, or a `FilterCodec` that also encodes). The facade
(`clawhdf5`) forwards `lzf`, `bitshuffle`, `bzip2`, `blosc`, `blosc2` and
`plugin-filters`. Write
(`clawhdf5`) forwards `lzf`, `bitshuffle`, `bzip2`, `blosc`, `blosc2`, `zfp`
and `plugin-filters`. Write
with `DatasetBuilder::with_lzf()`, `with_bitshuffle(..)`, `with_bzip2(..)`
and `with_blosc(..)`; h5py + hdf5plugin read the result (tested both ways in
`crates/clawhdf5/tests/plugin_filters_interop.rs`). The pure-Rust Zstandard
conformance/baseline.json
+8 -7
View File
@@ -1,15 +1,15 @@
{
"comment": "conformance/run.sh fails if the ok count drops below `ok` or a file in `ok_files` stops being ok. Regenerate with `conformance/run.sh --update-baseline` after an intended change.",
"commit": "d0e3beb3aa8290aae523ce280b4380e75484b6bc",
"date": "2026-09-26 17:10 UTC",
"commit": "8fadb9f4242a35323262701328d380806d379140",
"date": "2026-09-26 20:06 UTC",
"reference": "h5py 3.16.0 / HDF5 2.0.0",
"files": 697,
"ok": 599,
"ok": 600,
"counts": {
"h5py-cannot-read": 92,
"mismatch": 2,
"ok": 599,
"our-error": 4
"ok": 600,
"our-error": 3
},
"per_corpus": {
"NCAS-CMS_pyfive": {
@@ -27,8 +27,8 @@
"hdf5": {
"h5py-cannot-read": 60,
"mismatch": 1,
"ok": 403,
"our-error": 2
"ok": 404,
"our-error": 1
},
"netcdf-c": {
"ok": 20
@@ -202,6 +202,7 @@
"hdf5/HDF5Examples/C/H5FLT/tfiles/h5ex_d_jpeg.h5",
"hdf5/HDF5Examples/C/H5FLT/tfiles/h5ex_d_lz4.h5",
"hdf5/HDF5Examples/C/H5FLT/tfiles/h5ex_d_lzf.h5",
"hdf5/HDF5Examples/C/H5FLT/tfiles/h5ex_d_zfp.h5",
"hdf5/HDF5Examples/C/H5FLT/tfiles/h5ex_d_zstd.h5",
"hdf5/HDF5Examples/C/H5G/16/h5ex_g_iterate.h5",
"hdf5/HDF5Examples/C/H5G/16/h5ex_g_traverse.h5",
crates/clawhdf5-format/Cargo.toml
+7 -1
View File
@@ -78,8 +78,14 @@ bzip2 = ["dep:bzip2", "std"]
blosc = ["lz4_flex", "ruzstd", "snap", "deflate", "std"]
# Blosc2 (32026), read-only: frames, B2ND arrays, and the Blosc codecs above.
blosc2 = ["blosc"]
# ZFP (32013, H5Z-ZFP), read-only: every mode, for int32, int64, float and
# double fields of 1 to 4 dimensions.
zfp = []
# Every plugin filter above.
plugin-filters = ["lzf", "bitshuffle", "bzip2", "blosc", "blosc2"]
plugin-filters = ["lzf", "bitshuffle", "bzip2", "blosc", "blosc2", "zfp"]
# Test instrumentation: per-thread counts of heap objects read (see
# `lookup_stats`), so tests can bound the cost of a name lookup.
lookup-stats = ["std"]
[[bench]]
name = "parallel_decompress_bench"
crates/clawhdf5-format/src/addr.rs
+112
View File
@@ -0,0 +1,112 @@
//! File address and length → in-memory index conversion.
//!
//! HDF5 addresses and lengths are 64-bit; the file is parsed through a
//! `&[u8]` indexed by `usize`. On a 64-bit target every `u64` fits, but on a
//! 32-bit one (`wasm32`, `i686`, `thumbv7em`) an address past `usize::MAX`
//! used to be truncated by an `as usize` cast — silently pointing at another
//! part of the file — or to panic. [`to_usize`] is the one conversion the
//! parsers use instead: such an address is a clean
//! [`FormatError::Overflow`]. It cannot be inside the data anyway: no slice
//! is longer than `isize::MAX` bytes.
#[cfg(not(feature = "std"))]
use alloc::format;
use crate::error::FormatError;
/// A file address, offset or length from the file as a `usize` index.
///
/// Fails with [`FormatError::Overflow`] when the value does not fit this
/// platform's `usize` (only possible on targets narrower than 64 bits).
#[inline]
pub fn to_usize(value: u64) -> Result<usize, FormatError> {
to_index::<usize>(value)
}
/// [`to_usize`] for an index type of any width. `usize` is 64 bits wide on
/// the hosts CI tests on, where the error path cannot be reached through
/// `usize`; tests run the same code with `u32` in its place, as on a 32-bit
/// target.
#[inline]
fn to_index<T: TryFrom<u64>>(value: u64) -> Result<T, FormatError> {
T::try_from(value).map_err(|_| too_large(value))
}
/// A count or offset into an in-memory buffer (a codec's progress counter,
/// a size the writer computed from data it holds) as a `usize`, saturating
/// at `usize::MAX` instead of truncating.
///
/// For values that are bounded by the length of something in memory, so
/// always fit; if one ever did not, a saturated index fails its bounds check
/// or allocation instead of silently addressing the wrong bytes. A value
/// read from the file uses [`to_usize`].
#[inline]
pub fn saturating_usize(value: u64) -> usize {
saturating_index(value, usize::MAX)
}
/// [`saturating_usize`] for an index type of any width, whose largest
/// value is `max` (see [`to_index`]).
#[inline]
fn saturating_index<T: TryFrom<u64>>(value: u64, max: T) -> T {
T::try_from(value).unwrap_or(max)
}
#[cold]
#[inline(never)]
fn too_large(value: u64) -> FormatError {
FormatError::Overflow(format!(
"file address or length {value:#x} exceeds this platform's address space"
))
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn values_that_fit_convert_exactly() {
assert_eq!(to_usize(0), Ok(0));
assert_eq!(to_usize(0x1234), Ok(0x1234));
assert_eq!(to_usize(usize::MAX as u64), Ok(usize::MAX));
}
#[test]
fn saturating_conversion_never_wraps() {
assert_eq!(saturating_usize(0), 0);
assert_eq!(saturating_usize(0x1234), 0x1234);
assert_eq!(saturating_usize(usize::MAX as u64), usize::MAX);
// Past usize::MAX (32-bit targets) or at u64::MAX: saturates.
assert_eq!(saturating_usize(u64::MAX), usize::MAX);
}
#[test]
fn values_past_usize_max_are_an_error_not_truncated() {
// Reachable through `usize` only where it is narrower than u64 (no
// such target runs tests in CI), so the same conversion is run with
// u32 standing in for a 32-bit usize.
let max = u64::from(u32::MAX);
assert_eq!(to_index::<u32>(max), Ok(u32::MAX));
for past in [max + 1, max + 0x10, 0x1_0000_1234, u64::MAX] {
let err = to_index::<u32>(past).unwrap_err();
assert!(
matches!(err, FormatError::Overflow(_)),
"{past:#x}: {err:?}"
);
}
// Where an `as` cast would have wrapped to a small, valid-looking
// index, it is not returned.
assert_eq!(0x1_0000_1234_u64 as u32, 0x1234);
assert!(to_index::<u32>(0x1_0000_1234).is_err());
assert_eq!(saturating_index(max + 1, u32::MAX), u32::MAX);
assert_eq!(saturating_index(0x1_0000_1234, u32::MAX), u32::MAX);
assert_eq!(saturating_index(0x1234, u32::MAX), 0x1234);
// And through `usize` itself, whichever width it has here.
match (usize::MAX as u64).checked_add(1) {
Some(past) => assert!(matches!(to_usize(past), Err(FormatError::Overflow(_)))),
None => assert_eq!(to_usize(u64::MAX), Ok(usize::MAX)),
}
}
}
crates/clawhdf5-format/src/attribute.rs
+299 -59
View File
@@ -5,8 +5,10 @@ use alloc::{borrow::Cow, string::String, vec::Vec};
#[cfg(feature = "std")]
use std::borrow::Cow;
use crate::addr::to_usize;
use crate::attribute_info::AttributeInfoMessage;
use crate::btree_v2::{BTreeV2Header, collect_btree_v2_records};
use crate::btree_v2::{BTreeV2Header, collect_btree_v2_records, find_btree_v2_records};
use crate::checksum::jenkins_lookup3;
use crate::data_read;
use crate::dataspace::Dataspace;
use crate::datatype::Datatype;
@@ -15,6 +17,7 @@ use crate::fractal_heap::FractalHeapHeader;
use crate::message_type::MessageType;
use crate::object_header::ObjectHeader;
use crate::shared_message;
use crate::storage::{Storage, require_contiguous};
use crate::vl_data;
/// A parsed HDF5 attribute message.
@@ -50,7 +53,7 @@ impl AttributeMessage {
///
/// `length_size` is needed for dataspace dimension parsing.
pub fn parse(data: &[u8], length_size: u8) -> Result<AttributeMessage, FormatError> {
Self::parse_impl(data, length_size, None)
Self::parse_impl(data, length_size, None::<(&[u8], u8)>)
}
/// [`AttributeMessage::parse`] with access to the rest of the file, which
@@ -65,13 +68,24 @@ impl AttributeMessage {
offset_size: u8,
length_size: u8,
) -> Result<AttributeMessage, FormatError> {
Self::parse_impl(data, length_size, Some((file_data, offset_size)))
Self::parse_in_storage(data, file_data, offset_size, length_size)
}
fn parse_impl(
/// [`AttributeMessage::parse_in_file`] with the file behind any
/// [`Storage`].
pub fn parse_in_storage<S: Storage + ?Sized>(
data: &[u8],
file: &S,
offset_size: u8,
length_size: u8,
) -> Result<AttributeMessage, FormatError> {
Self::parse_impl(data, length_size, Some((file, offset_size)))
}
fn parse_impl<S: Storage + ?Sized>(
data: &[u8],
length_size: u8,
file: Option<(&[u8], u8)>,
file: Option<(&S, u8)>,
) -> Result<AttributeMessage, FormatError> {
ensure_len(data, 0, 2)?;
let version = data[0];
@@ -86,19 +100,19 @@ impl AttributeMessage {
/// The bytes of an embedded datatype/dataspace message, following the
/// shared-message reference when `shared` is set.
fn embedded_message<'a>(
fn embedded_message<'a, S: Storage + ?Sized>(
bytes: &'a [u8],
shared: bool,
msg_type: MessageType,
length_size: u8,
file: Option<(&[u8], u8)>,
file: Option<(&S, u8)>,
) -> Result<Cow<'a, [u8]>, FormatError> {
if !shared {
return Ok(Cow::Borrowed(bytes));
}
let (file_data, offset_size) = file.ok_or(FormatError::UnresolvedSharedMessage)?;
let shared_ref = shared_message::parse_shared_ref_sized(bytes, offset_size, length_size)?;
shared_message::resolve_shared_message(
shared_message::resolve_shared_message_in(
file_data,
&shared_ref,
msg_type,
@@ -143,10 +157,10 @@ impl AttributeMessage {
})
}
fn parse_v2(
fn parse_v2<S: Storage + ?Sized>(
data: &[u8],
length_size: u8,
file: Option<(&[u8], u8)>,
file: Option<(&S, u8)>,
) -> Result<AttributeMessage, FormatError> {
// Flags: bit 0 = datatype is shared, bit 1 = dataspace is shared.
let flags = data.get(1).copied().unwrap_or(0);
@@ -197,10 +211,10 @@ impl AttributeMessage {
})
}
fn parse_v3(
fn parse_v3<S: Storage + ?Sized>(
data: &[u8],
length_size: u8,
file: Option<(&[u8], u8)>,
file: Option<(&S, u8)>,
) -> Result<AttributeMessage, FormatError> {
// Flags: bit 0 = datatype is shared, bit 1 = dataspace is shared.
let flags = data.get(1).copied().unwrap_or(0);
@@ -341,7 +355,8 @@ fn compute_raw_data(
dataspace: &Dataspace,
datatype: &Datatype,
) -> Vec<u8> {
let num_elements = dataspace.num_elements() as usize;
// Saturating, like the product: the size is capped at what is there.
let num_elements = usize::try_from(dataspace.num_elements()).unwrap_or(usize::MAX);
let elem_size = datatype.type_size() as usize;
let expected_size = num_elements.saturating_mul(elem_size);
let available = data.len().saturating_sub(pos);
@@ -415,7 +430,20 @@ pub fn extract_attributes_full(
offset_size: u8,
length_size: u8,
) -> Result<Vec<AttributeMessage>, FormatError> {
extract_attributes_with(file_data, header, offset_size, length_size, &mut Err)
extract_attributes_full_in(file_data, header, offset_size, length_size)
}
/// [`extract_attributes_full`] over any [`Storage`]. Dense attribute
/// storage is indexed by a v2 B-tree, which is not read over [`Storage`]
/// yet: on a backend without the whole file in memory an object with dense
/// attributes is [`FormatError::ContiguousStorageRequired`].
pub fn extract_attributes_full_in<S: Storage + ?Sized>(
file: &S,
header: &ObjectHeader,
offset_size: u8,
length_size: u8,
) -> Result<Vec<AttributeMessage>, FormatError> {
extract_attributes_with(file, header, offset_size, length_size, &mut Err)
}
/// Like [`extract_attributes_full`], but an attribute that cannot be read
@@ -431,6 +459,17 @@ pub fn extract_attributes_tolerant(
header: &ObjectHeader,
offset_size: u8,
length_size: u8,
) -> Result<(Vec<AttributeMessage>, Vec<FormatError>), FormatError> {
extract_attributes_tolerant_in(file_data, header, offset_size, length_size)
}
/// [`extract_attributes_tolerant`] over any [`Storage`] (see
/// [`extract_attributes_full_in`] for dense storage).
pub fn extract_attributes_tolerant_in<S: Storage + ?Sized>(
file_data: &S,
header: &ObjectHeader,
offset_size: u8,
length_size: u8,
) -> Result<(Vec<AttributeMessage>, Vec<FormatError>), FormatError> {
let mut errors = Vec::new();
let attrs = extract_attributes_with(file_data, header, offset_size, length_size, &mut |e| {
@@ -442,8 +481,8 @@ pub fn extract_attributes_tolerant(
/// Read every attribute; each one that fails goes to `on_error`, which
/// either stops the read (returns the error) or skips that attribute.
fn extract_attributes_with(
file_data: &[u8],
fn extract_attributes_with<S: Storage + ?Sized>(
file_data: &S,
header: &ObjectHeader,
offset_size: u8,
length_size: u8,
@@ -453,42 +492,15 @@ fn extract_attributes_with(
// Each attribute's creation order, where the file records one.
let mut orders: Vec<u32> = Vec::new();
// Collect compact attributes (inline in OH)
for msg in &header.messages {
if msg.msg_type == MessageType::Attribute {
let attr = if shared_message::is_shared(msg.flags) {
// Shared attribute: resolve the reference to get actual attribute data
shared_message::parse_shared_ref_sized(&msg.data, offset_size, length_size)
.and_then(|shared_ref| {
shared_message::resolve_shared_message(
file_data,
&shared_ref,
MessageType::Attribute,
offset_size,
length_size,
)
})
.and_then(|resolved| {
AttributeMessage::parse_in_file(
&resolved,
file_data,
offset_size,
length_size,
)
})
} else {
AttributeMessage::parse_in_file(&msg.data, file_data, offset_size, length_size)
};
let attr = attr.and_then(|a| check_in_header(a, header));
match attr {
Ok(attr) => {
attrs.push(attr);
orders.push(msg.creation_order.map_or(0, u32::from));
}
Err(e) => on_error(e)?,
}
}
}
extract_compact_attributes(
file_data,
header,
offset_size,
length_size,
&mut attrs,
&mut orders,
on_error,
)?;
// Check for dense attributes via AttributeInfo message
let attr_info = find_attribute_info(header, offset_size)?;
@@ -519,6 +531,160 @@ fn extract_attributes_with(
Ok(attrs)
}
/// B-tree v2 record type of dense attribute storage's name index.
const ATTRIBUTE_NAME_INDEX: u8 = 8;
/// The attribute called `name` on the object with header `header`: the
/// first one [`extract_attributes_tolerant`] returns under that name, or
/// `None` if it returns none (an attribute that cannot be read is not
/// returned there either).
///
/// Compact attributes are in the header and are scanned. Dense attributes
/// are found through the name index (a v2 B-tree of lookup3 name hashes,
/// record type 8): only the attributes whose names hash like `name` are read
/// from the heap, O(log n) instead of all of them. Errors in the structures
/// that index the attributes fail the call, as they fail a listing.
pub fn find_attribute_in_file(
file_data: &[u8],
header: &ObjectHeader,
name: &str,
offset_size: u8,
length_size: u8,
) -> Result<Option<AttributeMessage>, FormatError> {
find_attribute_in(file_data, header, name, offset_size, length_size)
}
/// [`find_attribute_in_file`] over any [`Storage`] (see
/// [`extract_attributes_full_in`] for dense storage, whose name index still
/// needs the whole file in memory).
pub fn find_attribute_in<S: Storage + ?Sized>(
file_data: &S,
header: &ObjectHeader,
name: &str,
offset_size: u8,
length_size: u8,
) -> Result<Option<AttributeMessage>, FormatError> {
let attr_info = find_attribute_info(header, offset_size)?;
let dense = attr_info
.as_ref()
.and_then(|i| Some((i.fractal_heap_address?, i.btree_name_index_address?)));
let Some((fh_addr, btree_addr)) = dense else {
// Compact only (or dense storage without a name index, which a
// listing reports): as a listing finds it.
return Ok(
extract_attributes_tolerant_in(file_data, header, offset_size, length_size)?
.0
.into_iter()
.find(|a| a.name == name),
);
};
let contiguous = require_contiguous(file_data, "dense attribute storage (a v2 B-tree)")?;
let btree_hdr =
BTreeV2Header::parse(contiguous, to_usize(btree_addr)?, offset_size, length_size)?;
let fh = FractalHeapHeader::parse_in(file_data, fh_addr, offset_size, length_size)?;
if btree_hdr.tree_type != ATTRIBUTE_NAME_INDEX || btree_hdr.record_size < 4 {
return Ok(
extract_attributes_tolerant_in(file_data, header, offset_size, length_size)?
.0
.into_iter()
.find(|a| a.name == name),
);
}
// A listing has the compact attributes first.
let mut compact = Vec::new();
extract_compact_attributes(
file_data,
header,
offset_size,
length_size,
&mut compact,
&mut Vec::new(),
&mut |_| Ok(()),
)?;
if let Some(a) = compact.into_iter().find(|a| a.name == name) {
return Ok(Some(a));
}
// Record: heap ID + message flags(1) + creation order(4) + hash(4); the
// hash is the last field.
let hash = jenkins_lookup3(name.as_bytes());
let hash_at = usize::from(btree_hdr.record_size) - 4;
let records = find_btree_v2_records(contiguous, &btree_hdr, offset_size, &mut |r| match r
.get(hash_at..hash_at + 4)
{
Some(h) => u32::from_le_bytes([h[0], h[1], h[2], h[3]]).cmp(&hash),
None => core::cmp::Ordering::Less,
})?;
let id_len = usize::from(fh.heap_id_length);
for record in &records {
let Some(id_bytes) = record.data.get(..id_len) else {
continue;
};
let attr = fh
.read_managed_object_in(file_data, id_bytes, offset_size)
.and_then(|d| {
AttributeMessage::parse_in_storage(&d, file_data, offset_size, length_size)
});
// One that cannot be read is left out, as from a listing.
if let Ok(attr) = attr
&& attr.name == name
{
return Ok(Some(attr));
}
}
Ok(None)
}
/// The attributes stored in the object header itself (compact storage), and
/// each one's creation order into `orders`.
fn extract_compact_attributes<S: Storage + ?Sized>(
file_data: &S,
header: &ObjectHeader,
offset_size: u8,
length_size: u8,
attrs: &mut Vec<AttributeMessage>,
orders: &mut Vec<u32>,
on_error: &mut dyn FnMut(FormatError) -> Result<(), FormatError>,
) -> Result<(), FormatError> {
for msg in &header.messages {
if msg.msg_type == MessageType::Attribute {
let attr = if shared_message::is_shared(msg.flags) {
// Shared attribute: resolve the reference to get actual attribute data
shared_message::parse_shared_ref_sized(&msg.data, offset_size, length_size)
.and_then(|shared_ref| {
shared_message::resolve_shared_message_in(
file_data,
&shared_ref,
MessageType::Attribute,
offset_size,
length_size,
)
})
.and_then(|resolved| {
AttributeMessage::parse_in_storage(
&resolved,
file_data,
offset_size,
length_size,
)
})
} else {
AttributeMessage::parse_in_storage(&msg.data, file_data, offset_size, length_size)
};
let attr = attr.and_then(|a| check_in_header(a, header));
match attr {
Ok(attr) => {
attrs.push(attr);
orders.push(msg.creation_order.map_or(0, u32::from));
}
Err(e) => on_error(e)?,
}
}
}
Ok(())
}
/// Find and parse the Attribute Info message from an object header.
fn find_attribute_info(
header: &ObjectHeader,
@@ -536,8 +702,8 @@ fn find_attribute_info(
/// Extract attributes from dense storage (fractal heap + B-tree v2), and
/// each one's creation order into `orders`.
#[allow(clippy::too_many_arguments)]
fn extract_dense_attributes(
file_data: &[u8],
fn extract_dense_attributes<S: Storage + ?Sized>(
file_data: &S,
attr_info: &AttributeInfoMessage,
fh_addr: u64,
offset_size: u8,
@@ -547,7 +713,7 @@ fn extract_dense_attributes(
on_error: &mut dyn FnMut(FormatError) -> Result<(), FormatError>,
) -> Result<(), FormatError> {
// Parse fractal heap
let fh = FractalHeapHeader::parse(file_data, fh_addr as usize, offset_size, length_size)?;
let fh = FractalHeapHeader::parse_in(file_data, fh_addr, offset_size, length_size)?;
// Parse B-tree v2 for name index (type 8)
let btree_addr = attr_info
@@ -556,8 +722,10 @@ fn extract_dense_attributes(
expected: 1,
available: 0,
})?;
let btree_hdr = BTreeV2Header::parse(file_data, btree_addr as usize, offset_size, length_size)?;
let records = collect_btree_v2_records(file_data, &btree_hdr, offset_size, length_size)?;
let contiguous = require_contiguous(file_data, "dense attribute storage (a v2 B-tree)")?;
let btree_hdr =
BTreeV2Header::parse(contiguous, to_usize(btree_addr)?, offset_size, length_size)?;
let records = collect_btree_v2_records(contiguous, &btree_hdr, offset_size, length_size)?;
for record in &records {
// Per HDF5 spec, both type 8 and type 9 records start with heap_id:
@@ -574,9 +742,9 @@ fn extract_dense_attributes(
// The data in the heap is a complete attribute message
let attr = fh
.read_managed_object(file_data, id_bytes, offset_size)
.read_managed_object_in(file_data, id_bytes, offset_size)
.and_then(|attr_data| {
AttributeMessage::parse_in_file(&attr_data, file_data, offset_size, length_size)
AttributeMessage::parse_in_storage(&attr_data, file_data, offset_size, length_size)
});
match attr {
Ok(attr) => {
@@ -986,4 +1154,76 @@ mod tests {
let strs = attr.read_as_strings().unwrap();
assert_eq!(strs, vec!["abcd", "EFGH"]);
}
/// Every object's attributes in h5py-written files read identically
/// through a read_at-only CountingStorage — compact ones, shared ones
/// and those behind an Attribute Info message — except dense storage,
/// whose v2 B-tree index is not read over Storage yet: that is the clean
/// ContiguousStorageRequired error, never a partial list. Through a
/// slice as Storage every object matches.
#[test]
fn storage_reads_match_slice_reads() {
use crate::storage::CountingStorage;
let files: [(&str, &[u8]); 5] = [
("attrs", include_bytes!("../tests/fixtures/attrs.h5")),
(
"mixed_attrs",
include_bytes!("../tests/fixtures/mixed_attrs.h5"),
),
(
"dense_attrs",
include_bytes!("../tests/fixtures/dense_attrs.h5"),
),
(
"dense_attrs_root",
include_bytes!("../tests/fixtures/dense_attrs_root.h5"),
),
(
"shared_fill_value",
include_bytes!("../tests/fixtures/shared_fill_value.h5"),
),
];
let (mut same, mut dense, mut attrs) = (0, 0, 0);
for (name, file) in files {
let sb = crate::superblock::Superblock::parse(file, 0).unwrap();
let (os, ls) = (sb.offset_size, sb.length_size);
let mut addrs = vec![sb.root_group_address];
addrs.extend(
crate::group_v2::resolve_group_children(file, &sb, sb.root_group_address)
.unwrap()
.iter()
.map(|e| e.object_header_address),
);
let storage = CountingStorage::new(file.to_vec());
for addr in addrs {
let header = ObjectHeader::parse(file, addr as usize, os, ls).unwrap();
let want = extract_attributes_full(file, &header, os, ls);
let slice_storage = extract_attributes_full_in(&file, &header, os, ls);
assert_eq!(format!("{slice_storage:?}"), format!("{want:?}"));
let got = extract_attributes_full_in(&storage, &header, os, ls);
let got_t = extract_attributes_tolerant_in(&storage, &header, os, ls);
let is_dense = find_attribute_info(&header, os)
.unwrap()
.is_some_and(|i| i.fractal_heap_address.is_some());
if is_dense {
let e = FormatError::ContiguousStorageRequired(
"dense attribute storage (a v2 B-tree)",
);
assert_eq!(got.unwrap_err(), e, "{name}");
assert_eq!(got_t.unwrap_err(), e, "{name}");
dense += 1;
} else {
attrs += want.as_ref().map_or(0, Vec::len);
assert_eq!(format!("{got:?}"), format!("{want:?}"), "{name}");
let want_t = extract_attributes_tolerant(file, &header, os, ls);
assert_eq!(format!("{got_t:?}"), format!("{want_t:?}"), "{name}");
same += 1;
}
}
}
assert!(
same >= 5 && dense >= 2 && attrs >= 5,
"{same} {dense} {attrs}"
);
}
}
crates/clawhdf5-format/src/btree_v1.rs
+89 -20
View File
@@ -4,6 +4,7 @@
use alloc::vec::Vec;
use crate::error::FormatError;
use crate::storage::{Storage, read_exact_at};
/// A parsed B-tree v1 node.
#[derive(Debug, Clone)]
@@ -74,13 +75,28 @@ impl BTreeV1Node {
file_data: &[u8],
offset: usize,
offset_size: u8,
length_size: u8,
) -> Result<BTreeV1Node, FormatError> {
Self::parse_in(file_data, offset as u64, offset_size, length_size)
}
/// [`Self::parse`] over any [`Storage`]: one read of the node's header,
/// one of its keys and children.
pub fn parse_in<S: Storage + ?Sized>(
file: &S,
offset: u64,
offset_size: u8,
_length_size: u8,
) -> Result<BTreeV1Node, FormatError> {
// signature(4) + node_type(1) + node_level(1) + entries_used(2) = 8
// + left_sibling(offset_size) + right_sibling(offset_size)
let os = offset_size as usize;
let header_size = 8 + os * 2;
ensure_len(file_data, offset, header_size)?;
let header = read_exact_at(file, offset, header_size)?;
let file_data: &[u8] = &header;
// The header's read checked that `offset + header_size` fits.
let body_start = offset + header_size as u64;
let offset = 0usize;
if &file_data[offset..offset + 4] != b"TREE" {
return Err(FormatError::InvalidBTreeSignature);
@@ -102,31 +118,30 @@ impl BTreeV1Node {
} else {
Some(read_offset(file_data, pos, offset_size)?)
};
pos += os;
// For type 0: keys are offset_size bytes, children are offset_size bytes
// Layout: key[0], child[0], key[1], child[1], ..., key[N-1], child[N-1], key[N]
let eu = entries_used as usize;
let key_size = os; // For type 0, key = offset_size
let needed = eu * (key_size + os) + key_size; // eu children + (eu+1) keys
ensure_len(file_data, pos, needed)?;
let body = read_exact_at(file, body_start, needed)?;
let file_data: &[u8] = &body;
let mut keys = Vec::with_capacity(eu + 1);
let mut children = Vec::with_capacity(eu);
for _i in 0..eu {
// key[i]
let key = read_offset(file_data, pos, offset_size)?;
keys.push(key);
pos += key_size;
// child[i]
let child = read_offset(file_data, pos, offset_size)?;
children.push(child);
pos += os;
if os == 0 {
// What reading the first key reports (and keeps `chunks_exact`
// below from being given a zero size).
return Err(FormatError::InvalidOffsetSize(offset_size));
}
// final key
let key = read_offset(file_data, pos, offset_size)?;
keys.push(key);
// `needed` bytes: key[0], child[0], ..., child[eu - 1], key[eu].
let (pairs, last) = file_data.split_at(eu * (key_size + os));
for pair in pairs.chunks_exact(key_size + os) {
keys.push(read_offset(pair, 0, offset_size)?);
children.push(read_offset(pair, key_size, offset_size)?);
}
keys.push(read_offset(last, 0, offset_size)?);
Ok(BTreeV1Node {
node_type,
@@ -150,11 +165,21 @@ pub fn collect_symbol_table_nodes(
offset_size: u8,
length_size: u8,
) -> Result<Vec<u64>, FormatError> {
collect_symbol_table_nodes_inner(file_data, btree_address, offset_size, length_size, 0)
collect_symbol_table_nodes_in(file_data, btree_address, offset_size, length_size)
}
fn collect_symbol_table_nodes_inner(
file_data: &[u8],
/// [`collect_symbol_table_nodes`] over any [`Storage`]: two reads per node.
pub fn collect_symbol_table_nodes_in<S: Storage + ?Sized>(
file: &S,
btree_address: u64,
offset_size: u8,
length_size: u8,
) -> Result<Vec<u64>, FormatError> {
collect_symbol_table_nodes_inner(file, btree_address, offset_size, length_size, 0)
}
fn collect_symbol_table_nodes_inner<S: Storage + ?Sized>(
file: &S,
btree_address: u64,
offset_size: u8,
length_size: u8,
@@ -164,7 +189,7 @@ fn collect_symbol_table_nodes_inner(
return Err(FormatError::NestingDepthExceeded);
}
let node = BTreeV1Node::parse(file_data, btree_address as usize, offset_size, length_size)?;
let node = BTreeV1Node::parse_in(file, btree_address, offset_size, length_size)?;
if node.node_type != 0 {
return Err(FormatError::InvalidBTreeNodeType(node.node_type));
@@ -178,7 +203,7 @@ fn collect_symbol_table_nodes_inner(
let mut result = Vec::new();
for &child_addr in &node.children {
let child_snods = collect_symbol_table_nodes_inner(
file_data,
file,
child_addr,
offset_size,
length_size,
@@ -317,4 +342,48 @@ mod tests {
assert_eq!(node.entries_used, 1);
assert_eq!(node.children, vec![0x50]);
}
/// Nodes and trees, cut at every length, parse identically through a
/// `read_at`-only storage.
#[test]
fn storage_parse_matches_slice_parse() {
use crate::storage::CountingStorage;
let nodes = [
build_btree_node(0, 0, &[0, 5, 10], &[0x100, 0x200], None, None, 8),
build_btree_node(0, 0, &[0, 5], &[0x100], Some(0x40), Some(0x80), 4),
build_btree_node(1, 2, &[0, 5], &[0x100], None, Some(0x80), 8),
];
for (n, node) in nodes.iter().enumerate() {
let os = if n == 1 { 4 } else { 8 };
for cut in 0..=node.len() {
let f = &node[..cut];
let storage = CountingStorage::new(f.to_vec());
let want = BTreeV1Node::parse(f, 0, os, 8);
let got = BTreeV1Node::parse_in(&storage, 0, os, 8);
assert_eq!(format!("{got:?}"), format!("{want:?}"));
}
}
let leaf1 = build_btree_node(0, 0, &[0, 5], &[0xA00], None, None, 8);
let leaf2 = build_btree_node(0, 0, &[5, 10], &[0xB00], None, None, 8);
let internal = build_btree_node(0, 1, &[0, 5, 10], &[0, 256], None, None, 8);
let mut file = vec![0u8; 512 + internal.len()];
file[..leaf1.len()].copy_from_slice(&leaf1);
file[256..256 + leaf2.len()].copy_from_slice(&leaf2);
file[512..].copy_from_slice(&internal);
for cut in [file.len(), 300, 260, 100, 10] {
let mut f = file.clone();
if cut < 512 {
// Truncate the leaves, keep the root.
f[cut..512].fill(0);
}
let storage = CountingStorage::new(f.clone());
assert_eq!(
collect_symbol_table_nodes_in(&storage, 512, 8, 8),
collect_symbol_table_nodes(&f, 512, 8, 8)
);
}
let storage = CountingStorage::new(file);
collect_symbol_table_nodes_in(&storage, 512, 8, 8).unwrap();
assert_eq!(storage.reads(), 6);
}
}
crates/clawhdf5-format/src/btree_v2.rs
+198 -37
View File
@@ -2,10 +2,12 @@
#[cfg(not(feature = "std"))]
use alloc::vec::Vec;
use core::cmp::Ordering;
#[cfg(feature = "checksum")]
use byteorder::{ByteOrder, LittleEndian};
use crate::addr::to_usize;
use crate::error::FormatError;
/// Parsed B-tree v2 header (signature "BTHD").
@@ -216,7 +218,7 @@ pub fn collect_btree_v2_records(
// Root is a leaf
parse_leaf_records(
file_data,
header.root_node_address as usize,
to_usize(header.root_node_address)?,
header.num_records_in_root,
header.record_size,
)
@@ -225,7 +227,7 @@ pub fn collect_btree_v2_records(
let mut records = Vec::new();
collect_internal_records(
file_data,
header.root_node_address as usize,
to_usize(header.root_node_address)?,
header.num_records_in_root,
header.depth,
header.record_size,
@@ -289,9 +291,10 @@ fn parse_leaf_records(
Ok(records)
}
/// Recursively collect records from an internal node.
#[allow(clippy::too_many_arguments, clippy::only_used_in_recursion)]
fn collect_internal_records(
/// An internal node's layout: where its records start, and its children as
/// `(address, record count)`.
#[allow(clippy::too_many_arguments)]
fn read_internal_node(
file_data: &[u8],
offset: usize,
num_records: u16,
@@ -299,11 +302,8 @@ fn collect_internal_records(
record_size: u16,
node_size: u32,
offset_size: u8,
length_size: u8,
max_leaf_nrec: u64,
budget: &mut usize,
out: &mut Vec<BTreeV2Record>,
) -> Result<(), FormatError> {
) -> Result<(usize, Vec<(u64, u16)>), FormatError> {
// signature(4) + version(1) + type(1) = 6
ensure_len(file_data, offset, 6)?;
if &file_data[offset..offset + 4] != b"BTIN" {
@@ -314,7 +314,7 @@ fn collect_internal_records(
let rs = record_size as usize;
let mut pos = offset + 6;
// Read all records first
// Records first
let records_total = nr.checked_mul(rs).ok_or(FormatError::UnexpectedEof {
expected: usize::MAX,
available: file_data.len(),
@@ -346,7 +346,6 @@ fn collect_internal_records(
let child_ptr_size = offset_size as usize + nrec_width + total_nrec_width;
ensure_len(file_data, pos, num_children * child_ptr_size)?;
// Read child pointers
let mut children = Vec::with_capacity(num_children);
for _ in 0..num_children {
let addr = read_offset(file_data, pos, offset_size)?;
@@ -357,6 +356,78 @@ fn collect_internal_records(
children.push((addr, child_nrec));
}
// The checksum follows the child pointers and covers the node up to it.
// Lookups prune children by the keys in this node, so an unverified
// internal node could hide a record without any error: libhdf5 refuses
// a mismatch here, and so does this.
#[cfg(feature = "checksum")]
{
ensure_len(file_data, pos, 4)?;
let stored = LittleEndian::read_u32(&file_data[pos..pos + 4]);
let computed = crate::checksum::jenkins_lookup3(&file_data[offset..pos]);
if computed != stored {
return Err(FormatError::ChecksumMismatch {
expected: stored,
computed,
});
}
}
Ok((records_start, children))
}
/// Record `i` of an internal node whose records start at `records_start`.
fn internal_record(
file_data: &[u8],
records_start: usize,
i: usize,
rs: usize,
) -> Result<&[u8], FormatError> {
let overflow = || FormatError::UnexpectedEof {
expected: usize::MAX,
available: file_data.len(),
};
let rec_start = i
.checked_mul(rs)
.and_then(|o| records_start.checked_add(o))
.ok_or_else(overflow)?;
let rec_end = rec_start.checked_add(rs).ok_or_else(overflow)?;
file_data
.get(rec_start..rec_end)
.ok_or(FormatError::UnexpectedEof {
expected: rec_end,
available: file_data.len(),
})
}
/// Recursively collect records from an internal node.
#[allow(clippy::too_many_arguments, clippy::only_used_in_recursion)]
fn collect_internal_records(
file_data: &[u8],
offset: usize,
num_records: u16,
depth: u16,
record_size: u16,
node_size: u32,
offset_size: u8,
length_size: u8,
max_leaf_nrec: u64,
budget: &mut usize,
out: &mut Vec<BTreeV2Record>,
) -> Result<(), FormatError> {
let nr = num_records as usize;
let rs = record_size as usize;
let (records_start, children) = read_internal_node(
file_data,
offset,
num_records,
depth,
record_size,
node_size,
offset_size,
max_leaf_nrec,
)?;
let child_depth = depth - 1;
// Interleave: child[0], record[0], child[1], record[1], ..., child[nr]
// We collect child[0] records, then record[0], then child[1], etc.
for (i, &(child_addr, child_nrec)) in children.iter().enumerate() {
@@ -364,12 +435,12 @@ fn collect_internal_records(
// Before parsing, so a refused tree is not also a large allocation.
spend(budget, usize::from(child_nrec))?;
let leaf_recs =
parse_leaf_records(file_data, child_addr as usize, child_nrec, record_size)?;
parse_leaf_records(file_data, to_usize(child_addr)?, child_nrec, record_size)?;
out.extend(leaf_recs);
} else {
collect_internal_records(
file_data,
child_addr as usize,
to_usize(child_addr)?,
child_nrec,
child_depth,
record_size,
@@ -384,32 +455,10 @@ fn collect_internal_records(
// Add record[i] (except after the last child)
if i < nr {
let rec_offset = i.checked_mul(rs).ok_or(FormatError::UnexpectedEof {
expected: usize::MAX,
available: file_data.len(),
})?;
let rec_start =
records_start
.checked_add(rec_offset)
.ok_or(FormatError::UnexpectedEof {
expected: usize::MAX,
available: file_data.len(),
})?;
let rec_end = rec_start
.checked_add(rs)
.ok_or(FormatError::UnexpectedEof {
expected: usize::MAX,
available: file_data.len(),
})?;
if rec_end > file_data.len() {
return Err(FormatError::UnexpectedEof {
expected: rec_end,
available: file_data.len(),
});
}
let data = internal_record(file_data, records_start, i, rs)?;
spend(budget, 1)?;
out.push(BTreeV2Record {
data: file_data[rec_start..rec_end].to_vec(),
data: data.to_vec(),
});
}
}
@@ -417,6 +466,116 @@ fn collect_internal_records(
Ok(())
}
/// The records of a B-tree v2 that fall in one key range, found by
/// descending the tree instead of reading all of it.
///
/// `cmp` places a record relative to the range: `Less` if the record sorts
/// before it, `Greater` if after, `Equal` if the record is in it. The tree
/// must be ordered consistently with `cmp`, as libhdf5 orders it (a link or
/// attribute name index by name hash, so all records with one hash form a
/// range whatever order their names are in). Only the nodes whose key
/// interval overlaps the range are read: O(depth) nodes plus those holding
/// the matches. Matches come in tree order.
pub fn find_btree_v2_records(
file_data: &[u8],
header: &BTreeV2Header,
offset_size: u8,
cmp: &mut dyn FnMut(&[u8]) -> Ordering,
) -> Result<Vec<BTreeV2Record>, FormatError> {
if header.total_records == 0 || header.num_records_in_root == 0 {
return Ok(Vec::new());
}
if header.depth > MAX_DEPTH {
return Err(FormatError::NestingDepthExceeded);
}
// As in `collect_btree_v2_records`: a valid tree cannot hold more
// records than the file has room for, however its children are shared.
let mut budget = file_data.len() / usize::from(header.record_size.max(1));
let max_leaf_nrec = max_records_leaf(header.node_size, header.record_size);
let mut out = Vec::new();
find_in_node(
file_data,
header,
to_usize(header.root_node_address)?,
header.num_records_in_root,
header.depth,
offset_size,
max_leaf_nrec,
cmp,
&mut budget,
&mut out,
)?;
Ok(out)
}
#[allow(clippy::too_many_arguments)]
fn find_in_node(
file_data: &[u8],
header: &BTreeV2Header,
offset: usize,
num_records: u16,
depth: u16,
offset_size: u8,
max_leaf_nrec: u64,
cmp: &mut dyn FnMut(&[u8]) -> Ordering,
budget: &mut usize,
out: &mut Vec<BTreeV2Record>,
) -> Result<(), FormatError> {
spend(budget, usize::from(num_records))?;
if depth == 0 {
let records = parse_leaf_records(file_data, offset, num_records, header.record_size)?;
out.extend(
records
.into_iter()
.filter(|r| cmp(&r.data) == Ordering::Equal),
);
return Ok(());
}
let rs = usize::from(header.record_size);
let (records_start, children) = read_internal_node(
file_data,
offset,
num_records,
depth,
header.record_size,
header.node_size,
offset_size,
max_leaf_nrec,
)?;
let nr = usize::from(num_records);
let mut order = Vec::with_capacity(nr);
for i in 0..nr {
order.push(cmp(internal_record(file_data, records_start, i, rs)?));
}
// Child `i` holds the keys between record `i - 1` and record `i`: it can
// hold a match unless the record before it is already past the range or
// the record after it is still before it.
for (i, &(child_addr, child_nrec)) in children.iter().enumerate() {
let after_left = i == 0 || order[i - 1] != Ordering::Greater;
let before_right = i == nr || order[i] != Ordering::Less;
if after_left && before_right {
find_in_node(
file_data,
header,
to_usize(child_addr)?,
child_nrec,
depth - 1,
offset_size,
max_leaf_nrec,
cmp,
budget,
out,
)?;
}
if i < nr && order[i] == Ordering::Equal {
out.push(BTreeV2Record {
data: internal_record(file_data, records_start, i, rs)?.to_vec(),
});
}
}
Ok(())
}
/// Most records a subtree whose root is at `depth` can hold (libhdf5's
/// `cum_max_nrec`). See [`node_info`].
fn cum_max_records(
@@ -591,6 +750,8 @@ mod tests {
buf.extend_from_slice(&child_nrec.to_le_bytes()[..nrec_width]);
buf.resize(buf.len() + total_width, 0);
}
let sum = crate::checksum::jenkins_lookup3(&buf);
buf.extend_from_slice(&sum.to_le_bytes());
buf
}
crates/clawhdf5-format/src/btree_v2_write.rs
+58 -2
View File
@@ -6,6 +6,7 @@
//! libhdf5 uses (`H5B2__hdr_init`) and the reader decodes pointers with, so
//! the pointer widths the writer encodes are the ones every reader expects.
use crate::addr::saturating_usize;
#[cfg(not(feature = "std"))]
use alloc::{format, vec, vec::Vec};
@@ -105,7 +106,9 @@ pub(crate) fn build_btree_v2(
first_node: addr + hdr_len as u64,
nodes: Vec::new(),
};
let root = (n > 0).then(|| w.node(depth, 0, n as usize)).transpose()?;
let root = (n > 0)
.then(|| w.node(depth, 0, saturating_usize(n)))
.transpose()?;
let mut out = Vec::with_capacity(hdr_len + w.nodes.len() * p.node_size as usize);
out.extend_from_slice(b"BTHD");
@@ -201,7 +204,7 @@ impl TreeWriter<'_> {
"cannot spread {n} B-tree v2 records over {k} children at depth {depth}"
)));
}
let k = k as usize;
let k = saturating_usize(k);
let in_children = n - (k - 1);
let (base, extra) = (in_children / k, in_children % k);
@@ -385,6 +388,59 @@ mod tests {
assert!(nodes > 0);
}
/// Descending to a key range finds exactly the records a full read
/// holds in it — runs of equal keys that straddle node boundaries
/// included — at every depth, and nothing for keys not in the tree.
#[test]
fn a_key_range_search_matches_a_full_scan() {
use crate::btree_v2::find_btree_v2_records;
use core::cmp::Ordering;
let rs = 11usize;
// Keys 0, 0, 0, 2, 2, 2, 4, ...: runs of three, odd keys missing.
for n in [1usize, 45, 46, 1150, 30_000] {
let mut recs = Vec::with_capacity(n * rs);
for i in 0..n {
let mut r = vec![0u8; rs];
r[..8].copy_from_slice(&((i / 3 * 2) as u64).to_be_bytes());
r[8..].copy_from_slice(&[(i % 3) as u8, 0, 0]);
recs.extend_from_slice(&r);
}
let base = 4096u64;
let tree = build_btree_v2(params(512, 11), &recs, base, 8, 8).unwrap();
let mut file = vec![0u8; base as usize];
file.extend_from_slice(&tree);
let hdr = BTreeV2Header::parse(&file, base as usize, 8, 8).unwrap();
let all = collect_btree_v2_records(&file, &hdr, 8, 8).unwrap();
let key = |r: &[u8]| u64::from_be_bytes(r[..8].try_into().unwrap());
let last = key(&all[n - 1].data);
let probes = (0..=last + 1).step_by(if n > 1000 { 37 } else { 1 });
for k in probes.chain([last, last + 1, u64::MAX]) {
let found =
find_btree_v2_records(&file, &hdr, 8, &mut |r: &[u8]| key(r).cmp(&k)).unwrap();
let want: Vec<&[u8]> = all
.iter()
.map(|r| r.data.as_slice())
.filter(|r| key(r) == k)
.collect();
let got: Vec<&[u8]> = found.iter().map(|r| r.data.as_slice()).collect();
assert_eq!(got, want, "n {n} key {k}");
assert_eq!(
got.len(),
if k % 2 == 0 && k <= last {
want.len()
} else {
0
}
);
}
// Every record, or none, when the whole tree is in or out of range.
let every = find_btree_v2_records(&file, &hdr, 8, &mut |_| Ordering::Equal).unwrap();
assert_eq!(every.len(), n);
let none = find_btree_v2_records(&file, &hdr, 8, &mut |_| Ordering::Less).unwrap();
assert!(none.is_empty());
}
}
#[test]
fn a_node_too_small_or_too_big_is_an_error() {
assert!(build_btree_v2(params(16, 11), &records(1, 11), 0, 8, 8).is_err());
crates/clawhdf5-format/src/chunk_index.rs
+10 -1
View File
@@ -18,6 +18,7 @@ use alloc::collections::BTreeMap;
#[cfg(feature = "std")]
use std::collections::HashMap;
use crate::addr::to_usize;
use crate::chunk_cache::ChunkCoord;
use crate::chunked_read::ChunkInfo;
@@ -167,7 +168,15 @@ impl ChunkLayout {
for (_coord, ci) in index.iter() {
let coord: ChunkCoord = ci.offsets.iter().take(rank).copied().collect();
let chunk_offsets: Vec<usize> = coord.iter().map(|&o| o as usize).collect();
// `ds_dims` are `usize`: a chunk at an offset past `usize::MAX`
// (only on a 32-bit target) lies outside the dataset.
let Ok(chunk_offsets) = coord
.iter()
.map(|&o| to_usize(o))
.collect::<Result<Vec<usize>, _>>()
else {
continue;
};
let copies = if rank == 0 {
// Scalar dataset — single copy
crates/clawhdf5-format/src/chunked_read.rs
+32 -14
View File
@@ -6,6 +6,7 @@ extern crate alloc;
#[cfg(not(feature = "std"))]
use alloc::{format, vec, vec::Vec};
use crate::addr::to_usize;
#[cfg(feature = "std")]
use crate::chunk_cache::{CacheAlignedBuffer, ChunkCache};
use crate::data_layout::DataLayout;
@@ -265,7 +266,7 @@ fn fill_from_chunks(
)));
}
let offsets = &c.offsets[..rank];
let c_addr = c.address as usize;
let c_addr = to_usize(c.address)?;
let size = c.chunk_size as usize;
ensure_len(file_data, c_addr, size)?;
let raw = &file_data[c_addr..c_addr + size];
@@ -825,7 +826,7 @@ fn parse_chunk_node(
return Err(FormatError::NestingDepthExceeded);
}
let offset = btree_address as usize;
let offset = to_usize(btree_address)?;
let os = offset_size as usize;
// Parse B-tree v1 header
@@ -945,7 +946,8 @@ pub fn generate_implicit_chunks(
}
let total_chunks: u64 = num_chunks_per_dim.iter().product();
let mut chunks = Vec::with_capacity(total_chunks as usize);
// A capacity hint only (a count past `usize::MAX` could not be pushed).
let mut chunks = Vec::with_capacity(usize::try_from(total_chunks).unwrap_or(0));
for linear_idx in 0..total_chunks {
let mut offsets = vec![0u64; rank];
let mut remaining = linear_idx;
@@ -993,7 +995,7 @@ fn read_btree_v2_chunks(
use crate::btree_v2::{BTreeV2Header, collect_btree_v2_records};
let bad = |what: &str| FormatError::ChunkedReadError(format!("B-tree v2 chunk index: {what}"));
let header = BTreeV2Header::parse(file_data, addr as usize, offset_size, length_size)?;
let header = BTreeV2Header::parse(file_data, to_usize(addr)?, offset_size, length_size)?;
let rank = chunk_dims.len();
let os = offset_size as usize;
let record_size = header.record_size as usize;
@@ -1122,7 +1124,11 @@ pub fn list_chunks(
// Both v3 and v4 include element size as last dim (rank+1)
let (rank, chunk_dims) = chunk_geometry(chunk_dimensions, version, dataspace, elem_size)?;
let ds_dims: Vec<usize> = dataspace.dimensions.iter().map(|&d| d as usize).collect();
let ds_dims: Vec<usize> = dataspace
.dimensions
.iter()
.map(|&d| to_usize(d))
.collect::<Result<_, _>>()?;
// Collect chunks based on version and index type
let mut chunks = match (version, chunk_index_type) {
@@ -1158,7 +1164,7 @@ pub fn list_chunks(
// Fixed Array — use spatial chunk dims only
let spatial_chunk_dims: &[u32] = &chunk_dimensions[..rank];
let header =
FixedArrayHeader::parse(file_data, addr as usize, offset_size, length_size)?;
FixedArrayHeader::parse(file_data, to_usize(addr)?, offset_size, length_size)?;
read_fixed_array_chunks(
file_data,
&header,
@@ -1174,7 +1180,7 @@ pub fn list_chunks(
// Extensible Array — use spatial chunk dims only
let spatial_chunk_dims: &[u32] = &chunk_dimensions[..rank];
let header =
ExtensibleArrayHeader::parse(file_data, addr as usize, offset_size, length_size)?;
ExtensibleArrayHeader::parse(file_data, to_usize(addr)?, offset_size, length_size)?;
read_extensible_array_chunks(
file_data,
&header,
@@ -1349,7 +1355,11 @@ pub(crate) fn read_chunked_full<O>(
// dimension the total is 0 even if other dimensions are huge.
return Ok(output);
}
let ds_dims: Vec<usize> = dataspace.dimensions.iter().map(|&d| d as usize).collect();
let ds_dims: Vec<usize> = dataspace
.dimensions
.iter()
.map(|&d| to_usize(d))
.collect::<Result<_, _>>()?;
let placer = ChunkPlacer::new(&chunk_dims, &ds_dims, elem_size);
let chunk_total_bytes = checked_chunk_byte_len(&chunk_dims, elem_size)?;
// Chunks are cached only when the whole dataset fits: pushing a larger
@@ -1603,7 +1613,11 @@ pub fn read_chunked_data_sweep(
check_chunk_element_size(layout, datatype, offset_size)?;
let elem_size = datatype.type_size() as usize;
let (rank, chunk_dims) = chunk_geometry(chunk_dimensions, version, dataspace, elem_size)?;
let ds_dims: Vec<usize> = dataspace.dimensions.iter().map(|&d| d as usize).collect();
let ds_dims: Vec<usize> = dataspace
.dimensions
.iter()
.map(|&d| to_usize(d))
.collect::<Result<_, _>>()?;
// The per-file cache is shared across datasets (and threads); every
// lookup is keyed by this dataset's chunk-index address, so another
@@ -1659,7 +1673,7 @@ pub fn read_chunked_data_sweep(
cached
} else {
// Decompress from file
let c_addr = chunk_info.address as usize;
let c_addr = to_usize(chunk_info.address)?;
let size = chunk_info.chunk_size as usize;
ensure_len(file_data, c_addr, size)?;
let raw_chunk = &file_data[c_addr..c_addr + size];
@@ -1682,8 +1696,8 @@ pub fn read_chunked_data_sweep(
.offsets
.iter()
.take(rank)
.map(|&o| o as usize)
.collect();
.map(|&o| to_usize(o))
.collect::<Result<_, _>>()?;
if rank == 0 {
let copy_len = decompressed.len().min(output.len());
@@ -1743,7 +1757,11 @@ pub fn read_chunked_data_indexed(
check_chunk_element_size(layout, datatype, offset_size)?;
let elem_size = datatype.type_size() as usize;
let (rank, chunk_dims) = chunk_geometry(chunk_dimensions, version, dataspace, elem_size)?;
let ds_dims: Vec<usize> = dataspace.dimensions.iter().map(|&d| d as usize).collect();
let ds_dims: Vec<usize> = dataspace
.dimensions
.iter()
.map(|&d| to_usize(d))
.collect::<Result<_, _>>()?;
// Chunk index and assembly plan for this dataset, built on first access
// and kept per dataset (keyed by chunk-index address) in the shared cache.
@@ -1776,7 +1794,7 @@ pub fn read_chunked_data_indexed(
if let Some(cached) = cache.get_decompressed_in(addr, coord) {
chunk_buffers.push(cached);
} else {
let c_addr = *file_offset as usize;
let c_addr = to_usize(*file_offset)?;
let size = *file_size as usize;
ensure_len(file_data, c_addr, size)?;
let raw_chunk = &file_data[c_addr..c_addr + size];
crates/clawhdf5-format/src/chunked_write.rs
+123 -23
View File
@@ -3,6 +3,7 @@
#[cfg(not(feature = "std"))]
extern crate alloc;
use crate::addr::saturating_usize;
#[cfg(not(feature = "std"))]
use alloc::{format, vec, vec::Vec};
@@ -17,7 +18,7 @@ use crate::filter_pipeline::{
FILTER_LZF, FILTER_PCODEC, FILTER_PCODEC_NAME, FILTER_SHUFFLE, FILTER_ZSTD, FilterDescription,
FilterPipeline,
};
use crate::filters::compress_chunk;
use crate::filters::compress_chunk_masked;
/// Round a file offset up to the next cache-line boundary.
///
/// This ensures chunk data starts at an address that is a multiple of the
@@ -414,18 +415,18 @@ pub fn split_into_chunks(
// Dataset strides (row-major)
let mut ds_strides = vec![1usize; rank];
for i in (0..rank.saturating_sub(1)).rev() {
ds_strides[i] = ds_strides[i + 1] * shape[i + 1] as usize;
ds_strides[i] = ds_strides[i + 1] * saturating_usize(shape[i + 1]);
}
// Chunk strides
let mut chunk_strides = vec![1usize; rank];
for i in (0..rank.saturating_sub(1)).rev() {
chunk_strides[i] = chunk_strides[i + 1] * chunk_dims[i + 1] as usize;
chunk_strides[i] = chunk_strides[i + 1] * saturating_usize(chunk_dims[i + 1]);
}
let chunk_total_elements: usize = chunk_dims.iter().map(|&d| d as usize).product();
let chunk_total_elements: usize = chunk_dims.iter().map(|&d| saturating_usize(d)).product();
let mut result = Vec::with_capacity(total_chunks as usize);
let mut result = Vec::with_capacity(saturating_usize(total_chunks));
for linear_idx in 0..total_chunks {
// Convert linear index to chunk grid coordinates
@@ -453,8 +454,8 @@ pub fn split_into_chunks(
let coord_in_chunk = remaining_idx / chunk_strides[d];
remaining_idx %= chunk_strides[d];
let global_coord = offsets[d] as usize + coord_in_chunk;
if global_coord >= shape[d] as usize {
let global_coord = saturating_usize(offsets[d]) + coord_in_chunk;
if global_coord >= saturating_usize(shape[d]) {
out_of_bounds = true;
break;
}
@@ -489,7 +490,12 @@ pub fn split_into_chunks(
#[cfg(feature = "parallel")]
const PARALLEL_COMPRESS_THRESHOLD: usize = 2;
/// Compress all chunks, using parallel compression when beneficial.
/// Compress all chunks, using parallel compression when beneficial, and
/// return each chunk's stored bytes with its filter mask.
///
/// Chunks run through the pipeline as libhdf5 runs them
/// ([`compress_chunk_masked`]): an optional filter that fails — LZF or Blosc
/// output no smaller than its input — is skipped and its mask bit set.
///
/// With the `parallel` feature and more than [`PARALLEL_COMPRESS_THRESHOLD`]
/// filtered chunks, compression runs across rayon threads; otherwise it is
@@ -499,7 +505,7 @@ fn compress_all_chunks(
chunks: &[(Vec<u64>, Vec<u8>)],
pipeline: &Option<FilterPipeline>,
element_size: u32,
) -> Result<Vec<Vec<u8>>, FormatError> {
) -> Result<Vec<(Vec<u8>, u32)>, FormatError> {
#[cfg(feature = "parallel")]
{
if let Some(pl) = pipeline
@@ -508,7 +514,7 @@ fn compress_all_chunks(
use rayon::prelude::*;
return chunks
.par_iter()
.map(|(_offsets, chunk_bytes)| compress_chunk(chunk_bytes, pl, element_size))
.map(|(_offsets, chunk_bytes)| compress_chunk_masked(chunk_bytes, pl, element_size))
.collect();
}
}
@@ -518,9 +524,9 @@ fn compress_all_chunks(
.iter()
.map(|(_offsets, chunk_bytes)| {
if let Some(pl) = pipeline {
compress_chunk(chunk_bytes, pl, element_size)
compress_chunk_masked(chunk_bytes, pl, element_size)
} else {
Ok(chunk_bytes.clone())
Ok((chunk_bytes.clone(), 0))
}
})
.collect()
@@ -798,8 +804,10 @@ pub fn build_fixed_array_at(
/// writer passes eliminates the double-compression that the two-pass layout
/// algorithm previously performed.
pub struct PrecompressedChunks {
/// Per-chunk: (raw_size_bytes, compressed_bytes).
pub chunks: Vec<(u64, Vec<u8>)>,
/// Per-chunk: (raw_size_bytes, stored_bytes, filter_mask). Bit `i` of
/// the mask is set when filter `i` was skipped (an optional filter that
/// failed); 0 for every chunk of an unfiltered dataset.
pub chunks: Vec<(u64, Vec<u8>, u32)>,
pub has_filters: bool,
pub element_size: usize,
pub shape: Vec<u64>,
@@ -834,7 +842,7 @@ pub fn precompress_chunks(
let chunks = raw_chunks
.into_iter()
.zip(compressed)
.map(|((_offsets, raw_bytes), c)| (raw_bytes.len() as u64, c))
.map(|((_offsets, raw_bytes), (c, mask))| (raw_bytes.len() as u64, c, mask))
.collect();
Ok(PrecompressedChunks {
@@ -867,7 +875,7 @@ pub fn build_chunked_data_from_precompressed(
let mut data_buf = Vec::new();
let mut written_chunks = Vec::with_capacity(num_chunks);
for (raw_size, compressed) in &pre.chunks {
for (raw_size, compressed, filter_mask) in &pre.chunks {
let aligned_offset = align_to_cache_line(data_buf.len());
if aligned_offset > data_buf.len() {
data_buf.resize(aligned_offset, 0u8);
@@ -879,7 +887,7 @@ pub fn build_chunked_data_from_precompressed(
address,
compressed_size,
raw_size: *raw_size,
filter_mask: 0,
filter_mask: *filter_mask,
});
}
@@ -916,7 +924,7 @@ pub fn build_chunked_data_from_precompressed(
} else {
None
};
let filter_mask = if pre.has_filters { Some(0u32) } else { None };
let filter_mask = pre.has_filters.then_some(written_chunks[0].filter_mask);
serialize_v4_single_chunk(
&chunk_dims_u32,
chunk_addr,
@@ -1036,7 +1044,7 @@ impl ChunkIndexPlan {
Ok(Self::SingleChunk)
} else {
let grid = ChunkGrid::fixed_array(shape, Some(max), chunk_dims)?;
Ok(Self::FixedArray(grid, nslots as usize))
Ok(Self::FixedArray(grid, saturating_usize(nslots)))
}
}
1 => Ok(Self::ExtensibleArray(ChunkGrid::extensible_array(
@@ -1251,7 +1259,7 @@ pub fn write_selection_to_buffer(
let rank = dims.len();
let mut ds_strides = vec![1usize; rank];
for i in (0..rank.saturating_sub(1)).rev() {
ds_strides[i] = ds_strides[i + 1] * dims[i + 1] as usize;
ds_strides[i] = ds_strides[i + 1] * saturating_usize(dims[i + 1]);
}
let mut src_offset = 0usize;
@@ -1301,7 +1309,7 @@ pub fn write_selection_to_buffer(
buffer,
new_data,
src_offset,
current_ds_offset + coord as usize * ds_strides[d],
current_ds_offset + saturating_usize(coord) * ds_strides[d],
);
}
}
@@ -1328,14 +1336,14 @@ pub fn write_selection_to_buffer(
let rank = dims.len();
let mut ds_strides = vec![1usize; rank];
for i in (0..rank.saturating_sub(1)).rev() {
ds_strides[i] = ds_strides[i + 1] * dims[i + 1] as usize;
ds_strides[i] = ds_strides[i + 1] * saturating_usize(dims[i + 1]);
}
for (pi, pt) in pts.iter().enumerate() {
let flat: usize = pt
.iter()
.zip(ds_strides.iter())
.map(|(&p, &s)| p as usize * s)
.map(|(&p, &s)| saturating_usize(p) * s)
.sum();
let dst = flat * elem_size;
let src = pi * elem_size;
@@ -1943,6 +1951,98 @@ mod tests {
bytes_to_f64(&output)
}
/// Every chunk index the writer builds records each chunk's real filter
/// mask: LZF output no smaller than the chunk is skipped (bit 1, behind
/// shuffle) and the chunk stored shuffled only; compressible chunks keep
/// mask 0. The data reads back through both kinds of chunk.
#[cfg(feature = "lzf")]
#[test]
fn skipped_lzf_chunks_are_masked_in_every_index() {
let c = 64usize;
// Chunks alternate: random bytes (LZF cannot shrink them), then 7s.
let mut state = 0x1234_5678_u64;
let data: Vec<f64> = (0..4 * c)
.map(|i| {
if (i / c).is_multiple_of(2) {
state ^= state << 13;
state ^= state >> 7;
state ^= state << 17;
f64::from_bits(state)
} else {
7.0
}
})
.collect();
let raw = f64_to_bytes(&data);
let options = ChunkOptions {
plugin: Some(PluginFilter::Lzf),
..Default::default()
};
let c64 = c as u64;
#[allow(clippy::type_complexity)]
let cases: [(&[u64], &[u64], Option<&[u64]>, u8, &[u32]); 4] = [
(&[c64], &[c64], None, 1, &[2]),
(&[4 * c64], &[c64], None, 3, &[2, 0, 2, 0]),
(&[4 * c64], &[c64], Some(&[u64::MAX]), 4, &[2, 0, 2, 0]),
(
&[2, 2 * c64],
&[1, c64],
Some(&[u64::MAX, u64::MAX]),
5,
&[2, 0, 2, 0],
),
];
let base = 0x1000u64;
for (shape, chunks, maxshape, index_type, want_masks) in cases {
let n: u64 = shape.iter().product();
let raw = &raw[..n as usize * 8];
let result =
build_chunked_data_at_ext(raw, shape, chunks, 8, &options, base, maxshape).unwrap();
let mut file = vec![0u8; base as usize];
file.extend_from_slice(&result.data_bytes);
let layout = DataLayout::parse(&result.layout_message, 8, 8).unwrap();
assert!(
matches!(&layout, DataLayout::Chunked { chunk_index_type, .. }
if *chunk_index_type == Some(index_type)),
"{layout:?}"
);
let dataspace = Dataspace {
space_type: DataspaceType::Simple,
rank: shape.len() as u8,
dimensions: shape.to_vec(),
max_dimensions: maxshape.map(<[u64]>::to_vec),
};
let (mut infos, _) =
crate::chunked_read::list_chunks(&file, &layout, &dataspace, 8, 8, 8).unwrap();
infos.sort_by(|a, b| a.offsets.cmp(&b.offsets));
let masks: Vec<u32> = infos.iter().map(|i| i.filter_mask).collect();
assert_eq!(masks, want_masks, "index type {index_type}");
for info in &infos {
// Skipped chunks are stored at the chunk's size (shuffled).
assert_eq!(
info.chunk_size == (c * 8) as u32,
info.filter_mask != 0,
"{info:?}"
);
}
let pipeline = crate::filter_pipeline::FilterPipeline::parse(
result.pipeline_message.as_ref().unwrap(),
)
.unwrap();
let out = read_chunked_data(
&file,
&layout,
&dataspace,
&make_f64_type(),
Some(&pipeline),
8,
8,
)
.unwrap();
assert_eq!(out, raw, "index type {index_type}");
}
}
#[test]
fn ea_roundtrip_1d_inline_only() {
let values: Vec<f64> = (0..10).map(|i| i as f64).collect();
crates/clawhdf5-format/src/data_layout.rs
+55 -6
View File
@@ -6,7 +6,9 @@ use alloc::{format, string::String, vec::Vec};
#[cfg(feature = "std")]
use std::string::String;
use crate::addr::to_usize;
use crate::error::FormatError;
use crate::storage::Storage;
/// A single VDS (Virtual Dataset) source mapping.
///
@@ -207,7 +209,7 @@ pub fn parse_vds_mappings(
"VDS mapping shares a name with a later entry".into(),
));
}
Ok(idx as usize)
to_usize(idx)
};
let source_file = if flags & VDS_SOURCE_SAME_FILE != 0 {
@@ -309,6 +311,16 @@ impl DataLayout {
&mut self,
file_data: &[u8],
length_size: u8,
) -> Result<(), FormatError> {
self.resolve_vds_mappings_in(file_data, length_size)
}
/// [`Self::resolve_vds_mappings`] over any [`Storage`]: one read of the
/// global heap collection holding the mappings.
pub fn resolve_vds_mappings_in<S: Storage + ?Sized>(
&mut self,
file_data: &S,
length_size: u8,
) -> Result<(), FormatError> {
if let DataLayout::Virtual {
global_heap_address,
@@ -318,11 +330,8 @@ impl DataLayout {
} = self
&& let Some(addr) = *global_heap_address
{
let coll = crate::global_heap::GlobalHeapCollection::parse(
file_data,
addr as usize,
length_size,
)?;
let coll =
crate::global_heap::GlobalHeapCollection::parse_in(file_data, addr, length_size)?;
let obj = coll.get_object(*global_heap_index as u16).ok_or(
FormatError::GlobalHeapObjectNotFound {
collection_address: addr,
@@ -1305,4 +1314,44 @@ mod tests {
let blob = [0x01u8, 0, 0, 0, 0, 0, 0, 0, 0];
assert!(parse_vds_mappings(&blob, 8).unwrap().is_empty());
}
/// A virtual dataset's mappings resolve identically through a
/// read_at-only CountingStorage, in two reads of the global heap.
#[test]
fn vds_mappings_through_storage_match_slice() {
use crate::message_type::MessageType;
use crate::object_header::ObjectHeader;
use crate::storage::CountingStorage;
let file: &[u8] = include_bytes!("../tests/fixtures/vds_same_file.h5");
let sb = crate::superblock::Superblock::parse(file, 0).unwrap();
let (os, ls) = (sb.offset_size, sb.length_size);
let storage = CountingStorage::new(file.to_vec());
let mut virtuals = 0;
for child in
crate::group_v2::resolve_group_children(file, &sb, sb.root_group_address).unwrap()
{
let h =
ObjectHeader::parse(file, child.object_header_address as usize, os, ls).unwrap();
let Some(msg) = h
.messages
.iter()
.find(|m| m.msg_type == MessageType::DataLayout)
else {
continue;
};
let mut want = DataLayout::parse(&msg.data, os, ls).unwrap();
if !matches!(want, DataLayout::Virtual { .. }) {
continue;
}
let mut got = want.clone();
want.resolve_vds_mappings(file, ls).unwrap();
storage.reset();
got.resolve_vds_mappings_in(&storage, ls).unwrap();
assert_eq!(format!("{got:?}"), format!("{want:?}"));
assert!(matches!(&got, DataLayout::Virtual { mappings, .. } if !mappings.is_empty()));
assert_eq!(storage.reads(), 2);
virtuals += 1;
}
assert!(virtuals >= 1);
}
}
crates/clawhdf5-format/src/data_read.rs
+9 -8
View File
@@ -6,6 +6,7 @@ use alloc::{collections::BTreeMap, format, string::String, vec, vec::Vec};
#[cfg(feature = "std")]
use std::collections::BTreeMap;
use crate::addr::to_usize;
#[cfg(feature = "std")]
use crate::chunk_cache::ChunkCache;
use crate::chunked_read::read_chunked_data;
@@ -117,7 +118,7 @@ pub fn read_raw_data_zerocopy<'a>(
dataspace: &Dataspace,
datatype: &Datatype,
) -> Result<Option<&'a [u8]>, FormatError> {
let num_elements = dataspace.num_elements() as usize;
let num_elements = to_usize(dataspace.num_elements())?;
let elem_size = datatype.type_size() as usize;
let expected_size = num_elements.checked_mul(elem_size).ok_or_else(|| {
FormatError::Overflow(format!(
@@ -128,7 +129,7 @@ pub fn read_raw_data_zerocopy<'a>(
match layout {
DataLayout::Contiguous { address, size } => {
let addr = address.ok_or(FormatError::NoDataAllocated)?;
let addr = addr as usize;
let addr = to_usize(addr)?;
let sz = contiguous_read_len(*size, expected_size)?;
ensure_len(file_data, addr, sz)?;
Ok(Some(&file_data[addr..addr + sz]))
@@ -219,7 +220,7 @@ fn read_raw_data_full_impl(
length_size: u8,
resolver: Option<&VdsSourceResolver>,
) -> Result<Vec<u8>, FormatError> {
let num_elements = dataspace.num_elements() as usize;
let num_elements = to_usize(dataspace.num_elements())?;
let elem_size = datatype.type_size() as usize;
let expected_size = num_elements.checked_mul(elem_size).ok_or_else(|| {
FormatError::Overflow(format!(
@@ -239,7 +240,7 @@ fn read_raw_data_full_impl(
}
DataLayout::Contiguous { address, size } => {
let addr = address.ok_or(FormatError::NoDataAllocated)?;
let addr = addr as usize;
let addr = to_usize(addr)?;
let sz = contiguous_read_len(*size, expected_size)?;
ensure_len(file_data, addr, sz)?;
let mut out = crate::bulk_alloc::vec_for_bulk(sz);
@@ -582,7 +583,7 @@ pub fn extract_selection_from_buffer(
let rank = dims.len();
let mut ds_strides = vec![1usize; rank];
for i in (0..rank.saturating_sub(1)).rev() {
ds_strides[i] = ds_strides[i + 1] * dims[i + 1] as usize;
ds_strides[i] = ds_strides[i + 1] * to_usize(dims[i + 1])?;
}
let mut output = Vec::with_capacity(pts.len() * elem_size);
@@ -590,8 +591,8 @@ pub fn extract_selection_from_buffer(
let flat: usize = pt
.iter()
.zip(ds_strides.iter())
.map(|(&p, &s)| p as usize * s)
.sum();
.map(|(&p, &s)| Ok(to_usize(p)? * s))
.sum::<Result<usize, FormatError>>()?;
let src = flat * elem_size;
if src + elem_size <= full_data.len() {
output.extend_from_slice(&full_data[src..src + elem_size]);
@@ -1341,7 +1342,7 @@ pub fn read_compound_fields(
let mut fields = Vec::with_capacity(members.len());
for m in members {
let field_size = m.datatype.type_size() as usize;
let offset = m.byte_offset as usize;
let offset = to_usize(m.byte_offset)?;
if offset
.checked_add(field_size)
.is_none_or(|end| end > elem_size)
crates/clawhdf5-format/src/ea_writer.rs
+2 -1
View File
@@ -3,6 +3,7 @@
#[cfg(not(feature = "std"))]
extern crate alloc;
use crate::addr::saturating_usize;
#[cfg(not(feature = "std"))]
use alloc::{vec, vec::Vec};
@@ -247,7 +248,7 @@ pub fn build_extensible_array_at(
// Header (EAHD). The six statistics are, in order: super blocks, their
// bytes, data blocks, their bytes, max index set, elements realised.
let mut out = Vec::with_capacity((cursor - ea_base_address) as usize);
let mut out = Vec::with_capacity(saturating_usize(cursor - ea_base_address));
out.extend_from_slice(b"EAHD");
out.push(0); // version
out.push(client_id);
crates/clawhdf5-format/src/error.rs
+21
View File
@@ -12,7 +12,11 @@ use std::string::String;
use core::fmt;
/// Errors that can occur when parsing HDF5 binary format structures.
///
/// Non-exhaustive: new failure modes (new storage backends, new file
/// features) add variants, so a `match` needs a wildcard arm.
#[derive(Debug, Clone, PartialEq, Eq)]
#[non_exhaustive]
pub enum FormatError {
/// The HDF5 magic signature was not found at any valid offset.
SignatureNotFound,
@@ -243,6 +247,13 @@ pub enum FormatError {
/// A metadata cache image block libhdf5 refuses to load (the reason is
/// libhdf5's own error text).
InvalidCacheImage(&'static str),
/// The [`Storage`](crate::storage::Storage) backend failed to serve a
/// read (an I/O or network error, or a short read inside the file).
Storage(String),
/// The operation still needs the whole file as one slice and the
/// [`Storage`](crate::storage::Storage) backend has no contiguous view
/// (`as_contiguous()` is `None`); the text names the operation.
ContiguousStorageRequired(&'static str),
}
impl fmt::Display for FormatError {
@@ -529,6 +540,16 @@ impl fmt::Display for FormatError {
FormatError::InvalidCacheImage(why) => {
write!(f, "invalid metadata cache image: {why}")
}
FormatError::Storage(why) => {
write!(f, "storage read failed: {why}")
}
FormatError::ContiguousStorageRequired(what) => {
write!(
f,
"{what} needs the whole file in memory, which this storage backend does \
not provide"
)
}
}
}
}
crates/clawhdf5-format/src/extensible_array.rs
+217 -100
View File
@@ -9,19 +9,23 @@ extern crate alloc;
#[cfg(not(feature = "std"))]
use alloc::{format, vec, vec::Vec};
use crate::addr::to_usize;
use crate::chunk_grid::ChunkGrid;
use crate::chunked_read::ChunkInfo;
use crate::error::FormatError;
use crate::storage::{PAGED_BLOCK_ONE_READ_MAX, Storage, Window, read_exact_at};
/// Verify the Jenkins lookup3 checksum stored immediately after
/// `data[start..end]`, as every Extensible Array structure carries one.
/// `data[start..end]`, as every Extensible Array structure carries one. `w`
/// is a window of the file and `start`/`end` are relative to it.
///
/// A corrupt chunk index yields addresses pointing at the wrong bytes, so a
/// mismatch is an error: otherwise the damage surfaces as plausible data read
/// from the wrong chunk.
#[cfg(feature = "checksum")]
fn verify_checksum(data: &[u8], start: usize, end: usize) -> Result<(), FormatError> {
ensure_len(data, end, 4)?;
fn verify_checksum(w: &Window<'_>, start: usize, end: usize) -> Result<(), FormatError> {
w.ensure(end, 4)?;
let data: &[u8] = &w.bytes;
let stored = u32::from_le_bytes([data[end], data[end + 1], data[end + 2], data[end + 3]]);
let computed = crate::checksum::jenkins_lookup3(&data[start..end]);
if computed != stored {
@@ -34,7 +38,7 @@ fn verify_checksum(data: &[u8], start: usize, end: usize) -> Result<(), FormatEr
}
#[cfg(not(feature = "checksum"))]
fn verify_checksum(_data: &[u8], _start: usize, _end: usize) -> Result<(), FormatError> {
fn verify_checksum(_w: &Window<'_>, _start: usize, _end: usize) -> Result<(), FormatError> {
Ok(())
}
@@ -80,19 +84,6 @@ fn read_offset(data: &[u8], pos: usize, size: u8) -> Result<u64, FormatError> {
})
}
fn ensure_len(data: &[u8], offset: usize, needed: usize) -> Result<(), FormatError> {
if offset
.checked_add(needed)
.is_none_or(|end| end > data.len())
{
return Err(FormatError::UnexpectedEof {
expected: offset.saturating_add(needed),
available: data.len(),
});
}
Ok(())
}
fn is_undefined_addr(addr: u64, offset_size: u8) -> bool {
match offset_size {
2 => addr == 0xFFFF,
@@ -130,6 +121,16 @@ impl ExtensibleArrayHeader {
offset: usize,
offset_size: u8,
length_size: u8,
) -> Result<Self, FormatError> {
Self::parse_in(file_data, offset as u64, offset_size, length_size)
}
/// [`Self::parse`] over any [`Storage`]: one read of the header.
pub fn parse_in<S: Storage + ?Sized>(
file: &S,
offset: u64,
offset_size: u8,
length_size: u8,
) -> Result<Self, FormatError> {
// EAHD: signature(4) + version(1) + client_id(1) + element_size(1) +
// max_nelmts_bits(1) + idx_blk_elmts(1) + min_dblk_nelmts(1) +
@@ -137,9 +138,10 @@ impl ExtensibleArrayHeader {
// 6 stats fields (each length_size) + index_block_address(offset_size) + checksum(4)
let min_size =
4 + 1 + 1 + 1 + 1 + 1 + 1 + 1 + 1 + 6 * length_size as usize + offset_size as usize + 4;
ensure_len(file_data, offset, min_size)?;
let w = Window::read(file, offset, min_size)?;
w.ensure(0, min_size)?;
let d = &file_data[offset..];
let d: &[u8] = &w.bytes;
if &d[0..4] != b"EAHD" {
return Err(FormatError::ChunkedReadError(
"invalid Extensible Array header signature".into(),
@@ -172,7 +174,7 @@ impl ExtensibleArrayHeader {
pos += ls; // skip max_idx_set (6th stats field)
let index_block_address = read_offset(d, pos, offset_size)?;
pos += offset_size as usize;
verify_checksum(file_data, offset, offset + pos)?;
verify_checksum(&w, 0, pos)?;
Ok(ExtensibleArrayHeader {
client_id,
@@ -193,11 +195,11 @@ impl ExtensibleArrayHeader {
}
}
/// Read a single element from the extensible array element data.
/// Read a single element at offset `pos` of the window `w`.
/// Returns (chunk_info, bytes_consumed) or None if unallocated.
#[allow(clippy::too_many_arguments)]
fn read_element(
data: &[u8],
w: &Window<'_>,
pos: usize,
client_id: u8,
element_size: u8,
@@ -207,15 +209,11 @@ fn read_element(
grid: &ChunkGrid,
) -> Result<(Option<ChunkInfo>, usize), FormatError> {
let os = offset_size as usize;
let data: &[u8] = &w.bytes;
if client_id == 0 {
// Non-filtered: just address
if pos + os > data.len() {
return Err(FormatError::UnexpectedEof {
expected: pos + os,
available: data.len(),
});
}
w.ensure(pos, os)?;
if is_undefined(data, pos, offset_size) {
return Ok((None, os));
}
@@ -243,15 +241,7 @@ fn read_element(
}
let chunk_size_bytes = es - os - 4;
let elem_total = os + chunk_size_bytes + 4;
if pos
.checked_add(elem_total)
.is_none_or(|end| end > data.len())
{
return Err(FormatError::UnexpectedEof {
expected: pos.saturating_add(elem_total),
available: data.len(),
});
}
w.ensure(pos, elem_total)?;
if is_undefined(data, pos, offset_size) {
return Ok((None, elem_total));
}
@@ -315,9 +305,9 @@ fn page_nelmts(header: &ExtensibleArrayHeader) -> Option<usize> {
/// paged. The bitmap lives in the super block, not here — a paged data block
/// stores only its prefix, then one slot per page.
#[allow(clippy::too_many_arguments)]
fn read_data_block_elements(
file_data: &[u8],
db_offset: usize,
fn read_data_block_elements<S: Storage + ?Sized>(
file: &S,
db_offset: u64,
nelmts: usize,
header: &ExtensibleArrayHeader,
offset_size: u8,
@@ -330,21 +320,28 @@ fn read_data_block_elements(
// EADB: signature(4) + version(1) + client_id(1) + header_address(offset_size)
// + block offset(arr_off_size)
let db_header_size = 4 + 1 + 1 + offset_size as usize + arr_off_size(header);
ensure_len(file_data, db_offset, db_header_size)?;
let prefix = read_exact_at(file, db_offset, db_header_size)?;
if &file_data[db_offset..db_offset + 4] != b"EADB" {
if &prefix[0..4] != b"EADB" {
return Err(FormatError::ChunkedReadError(
"invalid Extensible Array data block signature".into(),
));
}
let mut pos = db_offset + db_header_size;
// Positions below are relative to the data block.
let mut pos = db_header_size;
let page = page_nelmts(header).ok_or_else(|| {
FormatError::Overflow("Extensible Array page element count overflows usize".into())
})?;
let elem_bytes = if header.client_id == 0 {
offset_size as usize
} else {
header.element_size as usize
};
let mut chunks = Vec::new();
let read_run = |from: usize,
let read_run = |w: &Window<'_>,
from: usize,
count: usize,
first_index: usize,
chunks: &mut Vec<ChunkInfo>|
@@ -352,7 +349,7 @@ fn read_data_block_elements(
let mut p = from;
for i in 0..count {
let (info, consumed) = read_element(
file_data,
w,
p,
header.client_id,
header.element_size,
@@ -370,18 +367,19 @@ fn read_data_block_elements(
};
if nelmts <= page {
// Prefix and elements are covered by one checksum.
let elem_bytes = if header.client_id == 0 {
offset_size as usize
} else {
header.element_size as usize
};
// Prefix and elements are covered by one checksum. One window holds
// all of it (or ends at the end of the file), so its bounds checks
// are the whole-file ones.
let end = nelmts
.checked_mul(elem_bytes)
.and_then(|b| pos.checked_add(b))
.ok_or_else(|| FormatError::Overflow("Extensible Array data block span".into()))?;
verify_checksum(file_data, db_offset, end)?;
read_run(pos, nelmts, start_index, &mut chunks)?;
// The checksum's bounds check comes first: make it before reading.
#[cfg(feature = "checksum")]
Window::check_extent(file, db_offset, end, 4)?;
let w = Window::read(file, db_offset, end.saturating_add(4))?;
verify_checksum(&w, 0, end)?;
read_run(&w, pos, nelmts, start_index, &mut chunks)?;
return Ok(chunks);
}
@@ -389,18 +387,32 @@ fn read_data_block_elements(
// each holding `page` elements followed by a checksum. Pages whose bit is
// clear were never written; their slot still occupies the file, so stride
// over it rather than reading zeros as addresses.
verify_checksum(file_data, db_offset, pos)?;
pos += 4;
let elem_bytes = if header.client_id == 0 {
offset_size as usize
let npages = nelmts.div_ceil(page);
// The whole data block in one window when it is small: every position
// checked below lies inside it (or past the end of the file). A larger
// block is read as its prefix, then each page in use on its own.
let block_len = pos
.saturating_add(4)
.saturating_add(npages.saturating_mul(page.saturating_mul(elem_bytes).saturating_add(4)));
let whole = if block_len <= PAGED_BLOCK_ONE_READ_MAX {
Some(Window::read(file, db_offset, block_len)?)
} else {
header.element_size as usize
None
};
let head_w;
let head = match &whole {
Some(w) => w,
None => {
head_w = Window::read(file, db_offset, pos + 4)?;
&head_w
}
};
verify_checksum(head, 0, pos)?;
pos += 4;
let page_stride = page
.checked_mul(elem_bytes)
.and_then(|b| b.checked_add(4))
.ok_or_else(|| FormatError::Overflow("Extensible Array page stride".into()))?;
let npages = nelmts.div_ceil(page);
for p in 0..npages {
// One bit per page across the whole super block, packed contiguously
// and MSB-first within each byte, as H5VM_bit_get reads it.
@@ -410,10 +422,20 @@ fn read_data_block_elements(
.is_some_and(|byte| byte & (0x80 >> (bit % 8)) != 0);
if initialised {
let count = core::cmp::min(page, nelmts - p * page);
// `w` holds the page from `base` on (positions below are
// relative to it, and `pos` to the data block).
let page_w;
let (w, base) = match &whole {
Some(w) => (w, 0),
None => {
page_w = Window::read(file, db_offset.saturating_add(pos as u64), page_stride)?;
(&page_w, pos)
}
};
// Each page carries its own checksum, over a full page's worth of
// slots even when the last one holds fewer live elements.
verify_checksum(file_data, pos, pos + page * elem_bytes)?;
read_run(pos, count, start_index + p * page, &mut chunks)?;
verify_checksum(w, pos - base, pos - base + page * elem_bytes)?;
read_run(w, pos - base, count, start_index + p * page, &mut chunks)?;
}
pos = pos
.checked_add(page_stride)
@@ -435,6 +457,32 @@ pub fn read_extensible_array_chunks(
chunk_dimensions: &[u32],
element_size: u32,
offset_size: u8,
length_size: u8,
) -> Result<Vec<ChunkInfo>, FormatError> {
read_extensible_array_chunks_in(
&file_data,
header,
dataset_dims,
max_dims,
chunk_dimensions,
element_size,
offset_size,
length_size,
)
}
/// [`read_extensible_array_chunks`] over any [`Storage`]: one read of the
/// index block's prefix, one of the whole index block, and the same for
/// every super block and data block it references.
#[allow(clippy::too_many_arguments)]
pub fn read_extensible_array_chunks_in<S: Storage + ?Sized>(
file: &S,
header: &ExtensibleArrayHeader,
dataset_dims: &[u64],
max_dims: Option<&[u64]>,
chunk_dimensions: &[u32],
element_size: u32,
offset_size: u8,
_length_size: u8,
) -> Result<Vec<ChunkInfo>, FormatError> {
let os = offset_size as usize;
@@ -451,19 +499,20 @@ pub fn read_extensible_array_chunks(
// Parse index block (EAIB): signature(4) + version(1) + client_id(1)
// + header address(offset_size), then the inline elements, then the
// direct data block addresses, then the super block addresses.
let ib_offset = header.index_block_address as usize;
// Positions below are relative to the index block.
let ib_offset = header.index_block_address;
let ib_header_size = 4 + 1 + 1 + os;
ensure_len(file_data, ib_offset, ib_header_size)?;
let prefix = read_exact_at(file, ib_offset, ib_header_size)?;
if &file_data[ib_offset..ib_offset + 4] != b"EAIB" {
if &prefix[0..4] != b"EAIB" {
return Err(FormatError::ChunkedReadError(
"invalid Extensible Array index block signature".into(),
));
}
let mut pos = ib_offset + ib_header_size;
let mut pos = ib_header_size;
let mut chunks = Vec::new();
let total_elements = header.num_elements as usize;
let total_elements = to_usize(header.num_elements)?;
let dmin = header.min_dblk_nelmts as usize;
if dmin == 0 || !dmin.is_power_of_two() {
@@ -520,13 +569,19 @@ pub fn read_extensible_array_chunks(
.and_then(|n| n.checked_mul(os).and_then(|b| p.checked_add(b)))
})
.ok_or_else(|| FormatError::Overflow("Extensible Array index block span".into()))?;
verify_checksum(file_data, ib_offset, ib_end)?;
// The whole index block in one window: every position read below is
// before `ib_end`.
// The checksum's bounds check comes first: make it before reading.
#[cfg(feature = "checksum")]
Window::check_extent(file, ib_offset, ib_end, 4)?;
let w = Window::read(file, ib_offset, ib_end.saturating_add(4))?;
verify_checksum(&w, 0, ib_end)?;
// 1. Elements stored inline in the index block.
let n_inline = (header.idx_blk_elmts as usize).min(total_elements);
for i in 0..n_inline {
let (info, consumed) = read_element(
file_data,
&w,
pos,
header.client_id,
header.element_size,
@@ -550,8 +605,8 @@ pub fn read_extensible_array_chunks(
if global_index >= total_elements {
return Ok(chunks);
}
ensure_len(file_data, pos, os)?;
let addr = read_offset(file_data, pos, offset_size)?;
w.ensure(pos, os)?;
let addr = read_offset(&w.bytes, pos, offset_size)?;
pos += os;
if !is_undefined_addr(addr, offset_size) {
if dblk_nelmts > page_nelmts(header).unwrap_or(usize::MAX) {
@@ -562,8 +617,8 @@ pub fn read_extensible_array_chunks(
));
}
chunks.extend(read_data_block_elements(
file_data,
addr as usize,
file,
addr,
dblk_nelmts,
header,
offset_size,
@@ -583,16 +638,16 @@ pub fn read_extensible_array_chunks(
if global_index >= total_elements {
break;
}
ensure_len(file_data, pos, os)?;
let sb_addr = read_offset(file_data, pos, offset_size)?;
w.ensure(pos, os)?;
let sb_addr = read_offset(&w.bytes, pos, offset_size)?;
pos += os;
let (ndblks, dblk_nelmts) = sblk_info(u, dmin).ok_or_else(|| {
FormatError::Overflow("Extensible Array super block layout overflows usize".into())
})?;
if !is_undefined_addr(sb_addr, offset_size) {
chunks.extend(read_super_block(
file_data,
sb_addr as usize,
file,
sb_addr,
ndblks,
dblk_nelmts,
header,
@@ -617,9 +672,9 @@ pub fn read_extensible_array_chunks(
/// + block offset + the page-init bitmap for every data block it owns
/// + one address per data block + checksum.
#[allow(clippy::too_many_arguments)]
fn read_super_block(
file_data: &[u8],
sb_offset: usize,
fn read_super_block<S: Storage + ?Sized>(
file: &S,
sb_offset: u64,
ndblks: usize,
dblk_nelmts: usize,
header: &ExtensibleArrayHeader,
@@ -630,9 +685,9 @@ fn read_super_block(
) -> Result<Vec<ChunkInfo>, FormatError> {
let os = offset_size as usize;
let sb_header_size = 4 + 1 + 1 + os + arr_off_size(header);
ensure_len(file_data, sb_offset, sb_header_size)?;
let prefix = read_exact_at(file, sb_offset, sb_header_size)?;
if &file_data[sb_offset..sb_offset + 4] != b"EASB" {
if &prefix[0..4] != b"EASB" {
return Err(FormatError::ChunkedReadError(
"invalid Extensible Array super block signature".into(),
));
@@ -654,29 +709,38 @@ fn read_super_block(
let bitmap_bytes = per_dblk_bitmap
.checked_mul(ndblks)
.ok_or_else(|| FormatError::Overflow("Extensible Array page bitmap size".into()))?;
let bitmap_start = sb_offset + sb_header_size;
ensure_len(file_data, bitmap_start, bitmap_bytes)?;
let bitmap = &file_data[bitmap_start..bitmap_start + bitmap_bytes];
// Positions below are relative to the super block, whose bytes (up to
// its checksum) are all in one window.
let bitmap_start = sb_header_size;
// The bitmap's bounds check, then (with checksums) the checksum's, come
// before anything else is read from the block: make them before reading
// it, so size fields stretching it past the end of the file cost no read.
Window::check_extent(file, sb_offset, bitmap_start, bitmap_bytes)?;
let mut pos = bitmap_start + bitmap_bytes;
let mut chunks = Vec::new();
let mut global_idx = start_index;
// One checksum covers the prefix, the bitmap and every data block address.
let sb_end = ndblks
.checked_mul(os)
.and_then(|b| pos.checked_add(b))
.ok_or_else(|| FormatError::Overflow("Extensible Array super block span".into()))?;
verify_checksum(file_data, sb_offset, sb_end)?;
#[cfg(feature = "checksum")]
Window::check_extent(file, sb_offset, sb_end, 4)?;
let w = Window::read(file, sb_offset, sb_end.saturating_add(4))?;
w.ensure(bitmap_start, bitmap_bytes)?;
let bitmap = &w.bytes[bitmap_start..bitmap_start + bitmap_bytes];
let mut chunks = Vec::new();
let mut global_idx = start_index;
verify_checksum(&w, 0, sb_end)?;
for i in 0..ndblks {
ensure_len(file_data, pos, os)?;
let addr = read_offset(file_data, pos, offset_size)?;
w.ensure(pos, os)?;
let addr = read_offset(&w.bytes, pos, offset_size)?;
pos += os;
if !is_undefined_addr(addr, offset_size) {
chunks.extend(read_data_block_elements(
file_data,
addr as usize,
file,
addr,
dblk_nelmts,
header,
offset_size,
@@ -887,11 +951,11 @@ mod tests {
assert_eq!(chunks[1].offsets, vec![20]);
}
/// Build a synthetic EA with inline elements + one direct data block.
#[test]
fn read_inline_plus_data_blocks() {
/// A synthetic EA with inline elements + one direct data block: the
/// file, with the header at 0x100 (8-byte offsets and lengths, 4 chunks
/// of 10 elements from 0x1000 on).
fn build_inline_plus_data_blocks() -> Vec<u8> {
let os: u8 = 8;
let ls: u8 = 8;
let osv = os as usize;
let chunk_byte_size = 10u64 * 8; // 10 elements × 8 bytes
let idx_blk_elmts = 2u8;
@@ -981,8 +1045,17 @@ mod tests {
dbpos += osv;
}
stamp_checksum(&mut file_data, aedb_offset, dbpos);
file_data
}
let header = ExtensibleArrayHeader::parse(&file_data, aehd_offset, os, ls).unwrap();
/// Build a synthetic EA with inline elements + one direct data block.
#[test]
fn read_inline_plus_data_blocks() {
let (os, ls) = (8u8, 8u8);
let chunk_byte_size = 10u64 * 8;
let base_addr = 0x1000u64;
let file_data = build_inline_plus_data_blocks();
let header = ExtensibleArrayHeader::parse(&file_data, 0x100, os, ls).unwrap();
let ds_dims = vec![40u64];
let chunk_dims = vec![10u32];
let chunks = read_extensible_array_chunks(
@@ -1018,7 +1091,8 @@ mod tests {
fn read_element_unallocated() {
let data = vec![0xFFu8; 16];
let grid = ChunkGrid::fixed_array(&[50], None, &[10]).unwrap();
let (info, consumed) = read_element(&data, 0, 0, 8, 8, 80, 0, &grid).unwrap();
let (info, consumed) =
read_element(&Window::whole(&data), 0, 0, 8, 8, 80, 0, &grid).unwrap();
assert!(info.is_none());
assert_eq!(consumed, 8);
}
@@ -1038,8 +1112,17 @@ mod tests {
data[12..16].copy_from_slice(&0u32.to_le_bytes());
let grid = ChunkGrid::fixed_array(&[50], None, &[10]).unwrap();
let (info, consumed) =
read_element(&data, 0, 1, elem_size as u8, os, 80, 2, &grid).unwrap();
let (info, consumed) = read_element(
&Window::whole(&data),
0,
1,
elem_size as u8,
os,
80,
2,
&grid,
)
.unwrap();
let ci = info.unwrap();
assert_eq!(ci.address, 0x2000);
assert_eq!(ci.chunk_size, 120);
@@ -1047,4 +1130,38 @@ mod tests {
assert_eq!(ci.offsets, vec![20]);
assert_eq!(consumed, elem_size);
}
/// The Storage path reads exactly what the slice path reads: the array
/// whole, cut at every length through its structures, and with a byte
/// damaged in each of them, through a read_at-only CountingStorage.
#[test]
fn storage_reads_match_slice_reads() {
use crate::storage::CountingStorage;
let full = build_inline_plus_data_blocks();
let mut files = Vec::new();
for cut in 0x100..0x340 {
files.push(full[..cut].to_vec());
}
for at in [0x104, 0x150, 0x204, 0x216, 0x230, 0x304, 0x318] {
let mut damaged = full.clone();
damaged[at] ^= 1;
files.push(damaged);
}
files.push(full);
let mut compared = 0;
for f in files {
let storage = CountingStorage::new(f.clone());
let want = ExtensibleArrayHeader::parse(&f, 0x100, 8, 8);
let got = ExtensibleArrayHeader::parse_in(&storage, 0x100, 8, 8);
assert_eq!(format!("{got:?}"), format!("{want:?}"));
let Ok(h) = want else { continue };
for dims in [&[40u64][..], &[25]] {
let want = read_extensible_array_chunks(&f, &h, dims, None, &[10], 8, 8, 8);
let got = read_extensible_array_chunks_in(&storage, &h, dims, None, &[10], 8, 8, 8);
assert_eq!(format!("{got:?}"), format!("{want:?}"), "{} bytes", f.len());
compared += 1;
}
}
assert!(compared > 100);
}
}
crates/clawhdf5-format/src/file_writer.rs
+11 -9
View File
@@ -3,6 +3,7 @@
//! Produces valid HDF5 files with v3 superblock, v2 object headers,
//! link messages, contiguous datasets, inline and dense attributes.
use crate::addr::saturating_usize;
#[cfg(not(feature = "std"))]
use alloc::{format, vec, vec::Vec};
@@ -336,7 +337,7 @@ pub(crate) fn build_single_block_fractal_heap(
// An object must fit one direct block: the writer has no huge-object
// path, and libhdf5 cannot read an object that overruns its block.
let max_managed = max_direct_block_size as usize - dblock_header_size;
let max_managed = saturating_usize(max_direct_block_size) - dblock_header_size;
if let Some(big) = serialized.iter().find(|s| s.len() > max_managed) {
return Err(FormatError::SerializationError(format!(
"a {}-byte message cannot go in dense storage: a fractal heap \
@@ -392,7 +393,7 @@ pub(crate) fn build_single_block_fractal_heap(
let dblock_addr = frhp_addr + frhp_size as u64;
let btree_addr = dblock_addr + starting_block_size;
let data_space = starting_block_size as usize - dblock_header_size;
let data_space = saturating_usize(starting_block_size) - dblock_header_size;
let free_space = data_space - total_data_size;
// Build fractal heap header
@@ -428,7 +429,7 @@ pub(crate) fn build_single_block_fractal_heap(
debug_assert_eq!(frhp.len(), frhp_size);
// Build direct block: header (with checksum) + data + padding
let mut dblock = Vec::with_capacity(starting_block_size as usize);
let mut dblock = Vec::with_capacity(saturating_usize(starting_block_size));
dblock.extend_from_slice(b"FHDB");
dblock.push(0); // version
write_offset(&mut dblock, frhp_addr, OFFSET_SIZE);
@@ -446,12 +447,12 @@ pub(crate) fn build_single_block_fractal_heap(
}
// Pad to full block size
dblock.resize(starting_block_size as usize, 0);
dblock.resize(saturating_usize(starting_block_size), 0);
// Checksum: computed over entire block with checksum field zeroed
let dblock_checksum = crate::checksum::jenkins_lookup3(&dblock);
dblock[cksum_pos..cksum_pos + 4].copy_from_slice(&dblock_checksum.to_le_bytes());
debug_assert_eq!(dblock.len(), starting_block_size as usize);
debug_assert_eq!(dblock.len(), saturating_usize(starting_block_size));
// Build heap IDs
let heap_ids: Vec<Vec<u8>> = obj_offsets
@@ -706,7 +707,7 @@ impl HeapIndirectBlock {
let cksum_pos = out.len();
out.extend_from_slice(&[0u8; 4]); // checksum placeholder
out.extend_from_slice(&b.data);
out.resize(d + b.size as usize, 0);
out.resize(d + saturating_usize(b.size), 0);
let cksum = crate::checksum::jenkins_lookup3(&out[d..]);
out[cksum_pos..cksum_pos + 4].copy_from_slice(&cksum.to_le_bytes());
child += b.size;
@@ -740,7 +741,7 @@ impl HeapPacker<'_> {
nrows: Option<usize>,
) -> Result<HeapIndirectBlock, FormatError> {
let geom = self.geom;
let width = geom.width as usize;
let width = saturating_usize(geom.width);
let mut slots = Vec::new();
let mut off = heap_offset;
let mut row = 0usize;
@@ -763,7 +764,8 @@ impl HeapPacker<'_> {
// A child whose biggest direct block cannot hold the
// next object is skipped whole, not walked.
let biggest = geom.row_size(child_rows.min(geom.max_direct_rows()) - 1);
if self.objects[self.next].len() > (biggest as usize - geom.dblock_header_size)
if self.objects[self.next].len()
> (saturating_usize(biggest) - geom.dblock_header_size)
{
slots.push(HeapSlot::Empty);
off += size;
@@ -794,7 +796,7 @@ impl HeapPacker<'_> {
/// objects as fit; leave it unallocated if not even the next one does.
fn fill_direct(&mut self, heap_offset: u64, size: u64) -> HeapSlot {
let header = self.geom.dblock_header_size;
let capacity = size as usize - header;
let capacity = saturating_usize(size) - header;
let mut data = Vec::new();
while let Some(obj) = self.objects.get(self.next) {
if data.len() + obj.len() > capacity {
crates/clawhdf5-format/src/fill_value.rs
+53 -3
View File
@@ -12,6 +12,7 @@
#[cfg(not(feature = "std"))]
use alloc::{format, vec, vec::Vec};
use crate::addr::to_usize;
use crate::chunked_read::{alloc_output, checked_byte_len, list_chunks};
use crate::data_layout::DataLayout;
use crate::dataspace::Dataspace;
@@ -116,9 +117,21 @@ pub fn dataset_fill_value_in(
messages: &[HeaderMessage],
offset_size: u8,
length_size: u8,
) -> Result<Option<Vec<u8>>, FormatError> {
dataset_fill_value_from_storage(&file_data, messages, offset_size, length_size)
}
/// [`dataset_fill_value_in`] with the file behind any
/// [`Storage`](crate::storage::Storage). (The trait is not imported here:
/// its `len` would shadow the slice method in this module.)
pub fn dataset_fill_value_from_storage(
file: &dyn crate::storage::Storage,
messages: &[HeaderMessage],
offset_size: u8,
length_size: u8,
) -> Result<Option<Vec<u8>>, FormatError> {
fill_value_from(messages, |msg| {
crate::shared_message::message_data_with_sohm(file_data, msg, offset_size, length_size)
crate::shared_message::message_data_with_sohm_in(file, msg, offset_size, length_size)
.map(|data| data.into_owned())
})
}
@@ -256,7 +269,11 @@ pub fn apply_to_unallocated_chunks(
length_size,
)?;
let rank = chunk_dims.len();
let ds_dims: Vec<usize> = dataspace.dimensions.iter().map(|&d| d as usize).collect();
let ds_dims: Vec<usize> = dataspace
.dimensions
.iter()
.map(|&d| to_usize(d))
.collect::<Result<_, _>>()?;
if rank == 0 || ds_dims.len() != rank || chunk_dims.contains(&0) {
return Ok(());
}
@@ -288,7 +305,7 @@ pub fn apply_to_unallocated_chunks(
let mut cell = 0usize;
let mut in_range = true;
for d in 0..rank {
let coord = chunk.offsets[d] as usize / chunk_dims[d];
let coord = to_usize(chunk.offsets[d])? / chunk_dims[d];
if coord >= grid[d] {
in_range = false;
break;
@@ -439,4 +456,37 @@ mod tests {
.collect();
assert_eq!(filled, [2, 3, 7, 8]);
}
/// Fill values, shared ones in the SOHM heap included, resolve
/// identically through a read_at-only CountingStorage.
#[test]
fn storage_reads_match_slice_reads() {
use crate::object_header::ObjectHeader;
use crate::storage::CountingStorage;
let file: &[u8] = include_bytes!("../tests/fixtures/shared_fill_value.h5");
let sb = crate::superblock::Superblock::parse(file, 0).unwrap();
let (os, ls) = (sb.offset_size, sb.length_size);
let storage = CountingStorage::new(file.to_vec());
let mut shared = 0;
let children =
crate::group_v2::resolve_group_children(file, &sb, sb.root_group_address).unwrap();
assert!(children.len() >= 3);
for child in children {
let h =
ObjectHeader::parse(file, child.object_header_address as usize, os, ls).unwrap();
shared += h
.messages
.iter()
.filter(|m| {
m.msg_type == MessageType::FillValue
&& crate::shared_message::is_shared(m.flags)
})
.count();
let want = dataset_fill_value_in(file, &h.messages, os, ls);
assert_eq!(want, Ok(Some((-7i32).to_le_bytes().to_vec())));
let got = dataset_fill_value_from_storage(&storage, &h.messages, os, ls);
assert_eq!(got, want, "{}", child.name);
}
assert!(shared >= 2);
}
}
crates/clawhdf5-format/src/filter_pipeline.rs
+2 -1
View File
@@ -27,7 +27,8 @@ pub const FILTER_LZF: u16 = 32000;
pub const FILTER_BLOSC: u16 = 32001;
/// Bitshuffle, optionally with LZ4 or Zstandard (hdf5plugin's `Bitshuffle`).
pub const FILTER_BITSHUFFLE: u16 = 32008;
/// ZFP lossy floating-point compression (hdf5plugin's `Zfp`). Not supported.
/// ZFP lossy (and lossless) compression of numeric arrays (H5Z-ZFP;
/// hdf5plugin's `Zfp`). Read-only, with the `zfp` feature.
pub const FILTER_ZFP: u16 = 32013;
/// Blosc 2 (hdf5plugin's `Blosc2`).
pub const FILTER_BLOSC2: u16 = 32026;
crates/clawhdf5-format/src/filter_registry.rs
+5 -3
View File
@@ -6,7 +6,7 @@
//! build: the HDF5 standard filters (deflate, shuffle, Fletcher32, szip,
//! N-Bit, scale-offset) and the plugin filters whose cargo features are
//! enabled (LZ4, Zstandard, pcodec, LZF, bitshuffle, bzip2, blosc,
//! blosc2).
//! blosc2, zfp).
//! [`builtin_filters`] lists them.
//! * **Registered filters** (`std` only) — codecs the application supplies
//! for any other ID with [`register_filter`] (a [`FilterCodec`], or just a
@@ -162,7 +162,7 @@ pub fn known_filter(id: u16) -> Option<(&'static str, Option<&'static str>)> {
32001 => ("Blosc", Some("blosc")),
32004 => ("LZ4", Some("lz4")),
32008 => ("bitshuffle", Some("bitshuffle")),
32013 => ("ZFP", None),
32013 => ("ZFP", Some("zfp")),
32015 => ("Zstandard", Some("zstd")),
32019 => ("JPEG", None),
32022 => ("BitGroom", None),
@@ -455,8 +455,10 @@ pub(crate) mod tests {
let msg = FormatError::UnsupportedFilter(32026).to_string();
assert!(msg.contains("Blosc2") && msg.contains("`blosc2`"), "{msg}");
let msg = FormatError::UnsupportedFilter(32013).to_string();
assert!(msg.contains("ZFP") && msg.contains("`zfp`"), "{msg}");
let msg = FormatError::UnsupportedFilter(32019).to_string();
assert!(
msg.contains("ZFP") && msg.contains("not implemented"),
msg.contains("JPEG") && msg.contains("not implemented"),
"{msg}"
);
let msg = FormatError::UnsupportedFilter(32000).to_string();
crates/clawhdf5-format/src/filters.rs
+140 -3
View File
@@ -3,6 +3,8 @@
#[cfg(not(feature = "std"))]
extern crate alloc;
#[cfg(feature = "deflate")]
use crate::addr::saturating_usize;
#[cfg(not(feature = "std"))]
use alloc::{boxed::Box, format, vec, vec::Vec};
@@ -346,6 +348,72 @@ pub fn compress_chunk(
Ok(result)
}
/// Filter flag bit 0: `H5Z_FLAG_OPTIONAL`.
const FILTER_FLAG_OPTIONAL: u16 = 0x0001;
/// Filters whose reference HDF5 filter (h5py's `lzf_filter.c`,
/// hdf5-blosc's `blosc_filter.c`) gives the encoder an output buffer only as
/// large as its input, so output that is not smaller than the input is a
/// failure there.
const FAIL_UNLESS_SMALLER: &[u16] = &[
crate::filter_pipeline::FILTER_LZF,
crate::filter_pipeline::FILTER_BLOSC,
];
/// Run a chunk through a filter pipeline for writing the way libhdf5's
/// `H5Z_pipeline` does, returning the bytes to store and the chunk's filter
/// mask (bit `i` set: filter `i` was skipped).
///
/// A filter that fails is skipped if the pipeline marks it optional
/// (`H5Z_FLAG_OPTIONAL`): its mask bit is set and the next filter gets the
/// same input. A mandatory filter that fails fails the write. Failure
/// includes what the reference filter counts as failure: LZF and Blosc
/// output that is not smaller than the input (h5py then stores the chunk
/// unfiltered with the bit set; storing it filtered with a clear mask can
/// leave a stale mask once libhdf5 rewrites the chunk at the same size).
///
/// A filter this build cannot encode is [`FormatError::UnsupportedFilter`]
/// even when optional: libhdf5 skips an optional filter only when its own
/// build lacks it, and every libhdf5 has the ones clawhdf5 cannot encode.
pub fn compress_chunk_masked(
data: &[u8],
pipeline: &FilterPipeline,
element_size: u32,
) -> Result<(Vec<u8>, u32), FormatError> {
if pipeline.filters.len() > 32 {
return Err(FormatError::CompressionError(
"more than 32 filters in a pipeline".into(),
));
}
let mut result = data.to_vec();
let mut mask = 0u32;
for (i, filter) in pipeline.filters.iter().enumerate() {
let ctx = FilterContext {
filter,
element_size: element_size as usize,
max_output: 0,
};
let out = match filter_registry::encode(&result, &ctx) {
Ok(out)
if FAIL_UNLESS_SMALLER.contains(&filter.filter_id) && out.len() >= result.len() =>
{
Err(FormatError::CompressionError(format!(
"filter {} did not shrink the chunk",
filter.filter_id
)))
}
r => r,
};
match out {
Ok(out) => result = out,
Err(e @ FormatError::UnsupportedFilter(_)) => return Err(e),
Err(_) if filter.flags & FILTER_FLAG_OPTIONAL != 0 => mask |= 1 << i,
Err(e) => return Err(e),
}
}
Ok((result, mask))
}
/// The filters compiled into this build, sorted by ID (see
/// [`crate::filter_registry`]). A filter whose cargo feature is off is left
/// out, so it fails as [`FormatError::UnsupportedFilter`] like any unknown ID.
@@ -432,6 +500,13 @@ pub(crate) static BUILTIN_FILTERS: &[BuiltinFilter] = &[
decode: crate::filters_bitshuffle::bitshuffle_decode,
encode: Some(crate::filters_bitshuffle::bitshuffle_encode),
},
#[cfg(feature = "zfp")]
BuiltinFilter {
id: crate::filter_pipeline::FILTER_ZFP,
name: "zfp",
decode: crate::filters_zfp::zfp_decode,
encode: None,
},
#[cfg(feature = "zstd")]
BuiltinFilter {
id: FILTER_ZSTD,
@@ -1114,7 +1189,11 @@ fn inflate_bounded_into(
loop {
let (in_before, out_before) = (inflater.total_in(), inflater.total_out());
let status = inflater
.decompress_vec(&data[in_before as usize..], out, FlushDecompress::Finish)
.decompress_vec(
&data[saturating_usize(in_before)..],
out,
FlushDecompress::Finish,
)
.map_err(|e| format!("deflate: {e}"))?;
if out.len() > limit {
return Err("deflate: output exceeds size limit".into());
@@ -1132,7 +1211,7 @@ fn inflate_bounded_into(
}
Status::Ok | Status::BufError => {
// Room left, so the decoder stopped for want of input.
if inflater.total_in() as usize >= data.len()
if saturating_usize(inflater.total_in()) >= data.len()
|| (inflater.total_in(), inflater.total_out()) == (in_before, out_before)
{
return Err("deflate: truncated stream".into());
@@ -1232,7 +1311,11 @@ pub(crate) fn deflate_bounded(data: &[u8], level: u32) -> Result<Vec<u8>, String
loop {
let (in_before, out_before) = (deflater.total_in(), deflater.total_out());
let status = deflater
.compress_vec(&data[in_before as usize..], &mut out, FlushCompress::Finish)
.compress_vec(
&data[saturating_usize(in_before)..],
&mut out,
FlushCompress::Finish,
)
.map_err(|e| format!("deflate: {e}"))?;
match status {
Status::StreamEnd => return Ok(out),
@@ -2099,6 +2182,60 @@ mod tests {
}
}
/// `compress_chunk_masked` follows `H5Z_pipeline`: an optional LZF that
/// does not shrink the chunk is skipped with its mask bit set (h5py
/// stores `[182, 0, 0, 0, 0]` raw with mask 1), a mandatory one fails,
/// and filters that grow the data (deflate) are kept, as libhdf5 keeps
/// them.
#[test]
#[cfg(all(feature = "lzf", feature = "deflate"))]
fn masked_compression_skips_optional_filters_that_fail() {
use crate::filter_pipeline::FILTER_LZF;
let opt = |id: u16| FilterDescription {
flags: FILTER_FLAG_OPTIONAL,
..filter(id)
};
let pl = |filters: Vec<FilterDescription>| FilterPipeline {
version: 2,
filters,
};
let raw = [182u8, 0, 0, 0, 0];
let (out, mask) = compress_chunk_masked(&raw, &pl(vec![opt(FILTER_LZF)]), 1).unwrap();
assert_eq!((out.as_slice(), mask), (&raw[..], 1));
let (out, mask) = compress_chunk_masked(
&raw,
&pl(vec![
opt(FILTER_SHUFFLE),
opt(FILTER_LZF),
filter(FILTER_FLETCHER32),
]),
1,
)
.unwrap();
assert_eq!((out.len(), mask), (raw.len() + 4, 2));
assert_eq!(
decompress_chunk_masked(
&out,
&pl(vec![
opt(FILTER_SHUFFLE),
opt(FILTER_LZF),
filter(FILTER_FLETCHER32)
]),
raw.len(),
1,
mask
)
.unwrap(),
raw
);
assert!(compress_chunk_masked(&raw, &pl(vec![filter(FILTER_LZF)]), 1).is_err());
let zeros = [0u8; 256];
let (out, mask) = compress_chunk_masked(&zeros, &pl(vec![opt(FILTER_LZF)]), 1).unwrap();
assert!(out.len() < zeros.len() && mask == 0);
let (out, mask) = compress_chunk_masked(&raw, &pl(vec![opt(FILTER_DEFLATE)]), 1).unwrap();
assert!(out.len() > raw.len() && mask == 0);
}
#[test]
#[cfg(feature = "deflate")]
fn filter_mask_skips_only_the_masked_filters() {
crates/clawhdf5-format/src/filters_blosc2.rs
+2 -1
View File
@@ -46,6 +46,7 @@
//! variable-length blocks, dictionaries, lazy chunks, user-defined codecs
//! and registered filters (e.g. bytedelta), sparse frames.
use crate::addr::saturating_usize;
use crate::error::FormatError;
use crate::filter_registry::FilterContext;
use crate::filters_bitshuffle::bitunshuffle_block;
@@ -546,7 +547,7 @@ fn parse_frame(buf: &[u8], limit: usize) -> Result<Frame<'_>, FormatError> {
return Err(err("negative size in frame header"));
}
let header_len = header_len as usize;
let buf = &buf[..frame_len as usize];
let buf = &buf[..saturating_usize(frame_len)];
let cbytes = usize::try_from(cbytes).map_err(|_| err("bad compressed size"))?;
let data_end = header_len
.checked_add(cbytes)
crates/clawhdf5-format/src/filters_bzip2.rs
+4 -3
View File
@@ -4,6 +4,7 @@
//! the compression level). Decoded with the `bzip2` crate's default backend,
//! `libbz2-rs-sys`, a pure-Rust port of libbzip2.
use crate::addr::saturating_usize;
use crate::error::FormatError;
use crate::filter_registry::FilterContext;
@@ -28,7 +29,7 @@ pub(crate) fn bzip2_decode(input: &[u8], ctx: &FilterContext<'_>) -> Result<Vec<
loop {
let (in_before, out_before) = (dec.total_in(), dec.total_out());
let status = dec
.decompress_vec(&input[in_before as usize..], &mut out)
.decompress_vec(&input[saturating_usize(in_before)..], &mut out)
.map_err(|e| err(&e.to_string()))?;
if out.len() > limit {
return Err(err("output exceeds the chunk size"));
@@ -43,7 +44,7 @@ pub(crate) fn bzip2_decode(input: &[u8], ctx: &FilterContext<'_>) -> Result<Vec<
.max(1);
out.try_reserve_exact(grow)
.map_err(|_| err("cannot allocate the output buffer"))?;
} else if dec.total_in() as usize >= input.len()
} else if saturating_usize(dec.total_in()) >= input.len()
|| (dec.total_in(), dec.total_out()) == (in_before, out_before)
{
return Err(err("truncated stream"));
@@ -61,7 +62,7 @@ pub(crate) fn bzip2_encode(input: &[u8], ctx: &FilterContext<'_>) -> Result<Vec<
// bzip2's worst case is about 1% + 600 bytes over the input.
let mut out = Vec::with_capacity(input.len() + input.len() / 100 + 600);
loop {
let consumed = enc.total_in() as usize;
let consumed = saturating_usize(enc.total_in());
let status = enc
.compress_vec(&input[consumed..], &mut out, Action::Finish)
.map_err(|e| cerr(e.to_string()))?;
crates/clawhdf5-format/src/filters_zfp.rs
+1083
View File
File diff suppressed because it is too large Load Diff
crates/clawhdf5-format/src/fixed_array.rs
+255 -74
View File
@@ -6,19 +6,23 @@ extern crate alloc;
#[cfg(not(feature = "std"))]
use alloc::{format, vec, vec::Vec};
use crate::addr::to_usize;
use crate::chunk_grid::ChunkGrid;
use crate::chunked_read::ChunkInfo;
use crate::error::FormatError;
use crate::storage::{PAGED_BLOCK_ONE_READ_MAX, Storage, Window, len_usize, read_exact_at};
/// Verify the Jenkins lookup3 checksum stored immediately after
/// `data[start..end]`, as every Fixed Array structure carries one.
/// `data[start..end]`, as every Fixed Array structure carries one. `w` is
/// a window of the file and `start`/`end` are relative to it.
///
/// A corrupt chunk index silently yields addresses pointing at the wrong
/// bytes, so a mismatch has to be an error rather than a shrug: without this
/// the damage surfaces as plausible-looking data from the wrong chunk.
#[cfg(feature = "checksum")]
fn verify_checksum(data: &[u8], start: usize, end: usize) -> Result<(), FormatError> {
ensure_len(data, end, 4)?;
fn verify_checksum(w: &Window<'_>, start: usize, end: usize) -> Result<(), FormatError> {
w.ensure(end, 4)?;
let data = &w.bytes;
let stored = u32::from_le_bytes([data[end], data[end + 1], data[end + 2], data[end + 3]]);
let computed = crate::checksum::jenkins_lookup3(&data[start..end]);
if computed != stored {
@@ -31,7 +35,7 @@ fn verify_checksum(data: &[u8], start: usize, end: usize) -> Result<(), FormatEr
}
#[cfg(not(feature = "checksum"))]
fn verify_checksum(_data: &[u8], _start: usize, _end: usize) -> Result<(), FormatError> {
fn verify_checksum(_w: &Window<'_>, _start: usize, _end: usize) -> Result<(), FormatError> {
Ok(())
}
@@ -73,19 +77,6 @@ fn read_length(data: &[u8], pos: usize, size: u8) -> Result<u64, FormatError> {
read_offset(data, pos, size)
}
fn ensure_len(data: &[u8], offset: usize, needed: usize) -> Result<(), FormatError> {
if offset
.checked_add(needed)
.is_none_or(|end| end > data.len())
{
return Err(FormatError::UnexpectedEof {
expected: offset.saturating_add(needed),
available: data.len(),
});
}
Ok(())
}
fn is_undefined(data: &[u8], pos: usize, size: u8) -> bool {
let s = size as usize;
if pos + s > data.len() {
@@ -101,13 +92,24 @@ impl FixedArrayHeader {
offset: usize,
offset_size: u8,
length_size: u8,
) -> Result<Self, FormatError> {
Self::parse_in(file_data, offset as u64, offset_size, length_size)
}
/// [`Self::parse`] over any [`Storage`]: one read of the header.
pub fn parse_in<S: Storage + ?Sized>(
file: &S,
offset: u64,
offset_size: u8,
length_size: u8,
) -> Result<Self, FormatError> {
// FAHD signature(4) + version(1) + client_id(1) + element_size(1) +
// max_nelmts_bits(1) + num_elements(length_size) + data_block_addr(offset_size) + checksum(4)
let min_size = 4 + 1 + 1 + 1 + 1 + length_size as usize + offset_size as usize + 4;
ensure_len(file_data, offset, min_size)?;
let w = Window::read(file, offset, min_size)?;
w.ensure(0, min_size)?;
let d = &file_data[offset..];
let d: &[u8] = &w.bytes;
if &d[0..4] != b"FAHD" {
return Err(FormatError::ChunkedReadError(
"invalid Fixed Array header signature".into(),
@@ -130,7 +132,7 @@ impl FixedArrayHeader {
pos += length_size as usize;
let data_block_address = read_offset(d, pos, offset_size)?;
pos += offset_size as usize;
verify_checksum(file_data, offset, offset + pos)?;
verify_checksum(&w, 0, pos)?;
Ok(FixedArrayHeader {
client_id,
@@ -156,15 +158,39 @@ pub fn read_fixed_array_chunks(
chunk_dimensions: &[u32],
element_size: u32,
offset_size: u8,
length_size: u8,
) -> Result<Vec<ChunkInfo>, FormatError> {
read_fixed_array_chunks_in(
&file_data,
header,
dataset_dims,
max_dims,
chunk_dimensions,
element_size,
offset_size,
length_size,
)
}
/// [`read_fixed_array_chunks`] over any [`Storage`]: one read of the data
/// block's prefix, one of the whole data block (pages included).
#[allow(clippy::too_many_arguments)]
pub fn read_fixed_array_chunks_in<S: Storage + ?Sized>(
file: &S,
header: &FixedArrayHeader,
dataset_dims: &[u64],
max_dims: Option<&[u64]>,
chunk_dimensions: &[u32],
element_size: u32,
offset_size: u8,
_length_size: u8,
) -> Result<Vec<ChunkInfo>, FormatError> {
let db_offset = header.data_block_address as usize;
let file_len = len_usize(file);
let db_offset = to_usize(header.data_block_address)?;
// Parse data block header: FADB(4) + version(1) + client_id(1) + header_address(offset_size)
let db_header_size = 4 + 1 + 1 + offset_size as usize;
ensure_len(file_data, db_offset, db_header_size)?;
let d = &file_data[db_offset..];
let d = read_exact_at(file, db_offset as u64, db_header_size)?;
if &d[0..4] != b"FADB" {
return Err(FormatError::ChunkedReadError(
"invalid Fixed Array data block signature".into(),
@@ -174,11 +200,11 @@ pub fn read_fixed_array_chunks(
// Elements start immediately after the data block prefix.
let elements_start = db_offset + db_header_size;
let num_elements = header.num_elements as usize;
let num_elements = to_usize(header.num_elements)?;
// A chunk index cannot describe more elements than the file has bytes (each
// element occupies at least `offset_size` bytes). Reject a corrupt count
// before it can drive a huge loop or overflow an offset computation.
if num_elements > file_data.len() {
if num_elements > file_len {
return Err(FormatError::ChunkedReadError(
"Fixed Array element count exceeds file size".into(),
));
@@ -208,30 +234,34 @@ pub fn read_fixed_array_chunks(
chunk_dimensions.iter().map(|&d| d as u64).product::<u64>() * element_size as u64;
let mut chunks = Vec::new();
let push_element =
|i: usize, abs: usize, chunks: &mut Vec<ChunkInfo>| -> Result<(), FormatError> {
if let Some((address, chunk_size, filter_mask)) = parse_fa_element(
file_data,
abs,
header.client_id,
offset_size,
header.element_size,
chunk_byte_size,
)? {
// A slot beyond the current extent is ignored, as the
// library does.
let Some(offsets) = grid.offsets(i as u64) else {
return Ok(());
};
chunks.push(ChunkInfo {
chunk_size,
filter_mask,
offsets,
address,
});
}
Ok(())
};
// `rel` is relative to the data block, whose bytes are in `w`.
let push_element = |w: &Window<'_>,
i: usize,
rel: usize,
chunks: &mut Vec<ChunkInfo>|
-> Result<(), FormatError> {
if let Some((address, chunk_size, filter_mask)) = parse_fa_element(
w,
rel,
header.client_id,
offset_size,
header.element_size,
chunk_byte_size,
)? {
// A slot beyond the current extent is ignored, as the
// library does.
let Some(offsets) = grid.offsets(i as u64) else {
return Ok(());
};
chunks.push(ChunkInfo {
chunk_size,
filter_mask,
offsets,
address,
});
}
Ok(())
};
// A data block is paged when it holds more elements than fit in one page.
// `max_nelmts_bits` is an untrusted u8; a shift >= the pointer width would
@@ -246,10 +276,16 @@ pub fn read_fixed_array_chunks(
if !is_paged {
// Non-paged: prefix, then `num_elements` elements packed directly,
// then a checksum over both.
verify_checksum(file_data, db_offset, elem_at(elements_start, num_elements)?)?;
// then a checksum over both. One window holds all of it (or ends at
// the end of the file), so its bounds checks are the whole-file ones.
let end = elem_at(elements_start, num_elements)?;
// The checksum's bounds check comes first: make it before reading.
#[cfg(feature = "checksum")]
Window::check_extent(file, db_offset as u64, end - db_offset, 4)?;
let w = Window::read(file, db_offset as u64, end.saturating_add(4) - db_offset)?;
verify_checksum(&w, 0, end - db_offset)?;
for i in 0..num_elements {
push_element(i, elem_at(elements_start, i)?, &mut chunks)?;
push_element(&w, i, elem_at(elements_start, i)? - db_offset, &mut chunks)?;
}
return Ok(chunks);
}
@@ -272,22 +308,40 @@ pub fn read_fixed_array_chunks(
.and_then(|x| x.checked_add(4))
.ok_or_else(stride_overflow)?;
if bitmap_start + bitmap_size > file_data.len() {
if bitmap_start + bitmap_size > file_len {
return Err(FormatError::UnexpectedEof {
expected: bitmap_start + bitmap_size,
available: file_data.len(),
available: file_len,
});
}
// The whole data block in one window when it is small: every page slot
// is at most `page_stride` bytes, so every position checked below lies
// inside it (or past the end of the file). A larger block is read as its
// prefix and bitmap, then each page in use on its own.
let block_len = (pages_start - db_offset).saturating_add(npages.saturating_mul(page_stride));
let whole = if block_len <= PAGED_BLOCK_ONE_READ_MAX {
Some(Window::read(file, db_offset as u64, block_len)?)
} else {
None
};
let head_w;
let head = match &whole {
Some(w) => w,
None => {
head_w = Window::read(file, db_offset as u64, pages_start - db_offset)?;
&head_w
}
};
// The prefix and page bitmap are covered by their own checksum, and each
// initialised page by one of its own.
verify_checksum(file_data, db_offset, bitmap_start + bitmap_size)?;
verify_checksum(head, 0, bitmap_start + bitmap_size - db_offset)?;
for p in 0..npages {
let page_first = p * page_nelmts; // < num_elements, cannot overflow
let page_count = core::cmp::min(page_nelmts, num_elements - page_first);
// Check the page-init bit (MSB-first within each byte).
let bit_byte = file_data[bitmap_start + p / 8];
let bit_byte = head.bytes[bitmap_start + p / 8 - db_offset];
let bit_mask = 1u8 << (7 - (p % 8));
if bit_byte & bit_mask == 0 {
continue; // entire page unallocated
@@ -297,21 +351,33 @@ pub fn read_fixed_array_chunks(
.checked_mul(page_stride)
.and_then(|o| pages_start.checked_add(o))
.ok_or_else(stride_overflow)?;
verify_checksum(file_data, page_off, elem_at(page_off, page_count)?)?;
let page_end = elem_at(page_off, page_count)?;
// `w` holds the page from `base` on (positions below are relative
// to it).
let page_w;
let (w, base) = match &whole {
Some(w) => (w, db_offset),
None => {
page_w =
Window::read(file, page_off as u64, page_end.saturating_add(4) - page_off)?;
(&page_w, page_off)
}
};
verify_checksum(w, page_off - base, page_end - base)?;
for e in 0..page_count {
push_element(page_first + e, elem_at(page_off, e)?, &mut chunks)?;
push_element(w, page_first + e, elem_at(page_off, e)? - base, &mut chunks)?;
}
}
Ok(chunks)
}
/// Parse a single Fixed Array element at absolute file offset `abs`.
/// Parse a single Fixed Array element at offset `abs` of the window `w`.
///
/// Returns `Some((address, chunk_size, filter_mask))` for an allocated chunk, or
/// `None` if the element is undefined (an unallocated chunk, address all-`0xFF`).
fn parse_fa_element(
file_data: &[u8],
w: &Window<'_>,
abs: usize,
client_id: u8,
offset_size: u8,
@@ -321,12 +387,8 @@ fn parse_fa_element(
let os = offset_size as usize;
if client_id == 0 {
// Non-filtered: element is just the chunk address.
if abs + os > file_data.len() {
return Err(FormatError::UnexpectedEof {
expected: abs + os,
available: file_data.len(),
});
}
w.ensure(abs, os)?;
let file_data: &[u8] = &w.bytes;
if is_undefined(file_data, abs, offset_size) {
return Ok(None);
}
@@ -341,17 +403,14 @@ fn parse_fa_element(
));
}
let chunk_size_bytes = es - os - 4;
if abs + es > file_data.len() {
return Err(FormatError::UnexpectedEof {
expected: abs + es,
available: file_data.len(),
});
}
w.ensure(abs, es)?;
let file_data: &[u8] = &w.bytes;
if is_undefined(file_data, abs, offset_size) {
return Ok(None);
}
let address = read_offset(file_data, abs, offset_size)?;
let chunk_size = read_variable_length(&file_data[abs + os..], chunk_size_bytes)?;
let chunk_size =
read_variable_length(&file_data[abs + os..abs + es - 4], chunk_size_bytes)?;
let fm_off = abs + os + chunk_size_bytes;
let filter_mask = u32::from_le_bytes([
file_data[fm_off],
@@ -813,4 +872,126 @@ mod tests {
.collect();
assert_eq!(got, expect);
}
/// A fixed array (header at 0x100, data block at 0x200) of `n` chunks,
/// filtered or not, paged when `n` exceeds `1 << page_bits`; every
/// page initialised except page 1.
fn build_fixed_array(n: usize, filtered: bool, page_bits: u8) -> Vec<u8> {
let os = 8usize;
let es = if filtered { os + 4 + 4 } else { os };
let (fahd, db) = (0x100usize, 0x200usize);
let mut f = vec![0u8; 0x2000];
f[fahd..fahd + 4].copy_from_slice(b"FAHD");
f[fahd + 5] = u8::from(filtered);
f[fahd + 6] = es as u8;
f[fahd + 7] = page_bits;
f[fahd + 8..fahd + 16].copy_from_slice(&(n as u64).to_le_bytes());
f[fahd + 16..fahd + 24].copy_from_slice(&(db as u64).to_le_bytes());
stamp_checksum(&mut f, fahd, fahd + 24);
f[db..db + 4].copy_from_slice(b"FADB");
f[db + 5] = u8::from(filtered);
f[db + 6..db + 14].copy_from_slice(&(fahd as u64).to_le_bytes());
let elems = db + 6 + os;
let write = |f: &mut Vec<u8>, at: usize, i: usize| {
let addr = if i == 2 {
u64::MAX
} else {
0x1000 + i as u64 * 0x100
};
f[at..at + os].copy_from_slice(&addr.to_le_bytes());
if filtered {
f[at + os..at + os + 4].copy_from_slice(&(100 + i as u32).to_le_bytes());
f[at + os + 4..at + os + 8].copy_from_slice(&(i as u32 & 1).to_le_bytes());
}
};
let page = 1usize << page_bits;
if n <= page {
for i in 0..n {
write(&mut f, elems + i * es, i);
}
stamp_checksum(&mut f, db, elems + n * es);
} else {
let npages = n.div_ceil(page);
let bitmap = npages.div_ceil(8);
for p in 0..npages {
if p != 1 {
f[elems + p / 8] |= 0x80 >> (p % 8);
}
}
stamp_checksum(&mut f, db, elems + bitmap);
let pages_start = elems + bitmap + 4;
for p in (0..npages).filter(|&p| p != 1) {
let at = pages_start + p * (page * es + 4);
let count = page.min(n - p * page);
for e in 0..count {
write(&mut f, at + e * es, p * page + e);
}
stamp_checksum(&mut f, at, at + count * es);
}
}
f
}
/// Non-paged and paged, filtered and unfiltered arrays, cut at every
/// length through the data block and with a damaged byte, read
/// identically through a `read_at`-only storage.
#[test]
fn storage_reads_match_slice_reads() {
use crate::storage::CountingStorage;
for (n, filtered, bits) in [(3, false, 10), (3, true, 10), (11, false, 2), (11, true, 2)] {
let full = build_fixed_array(n, filtered, bits);
let es = if filtered { 16 } else { 8 };
let dims = [n as u64 * 20];
let h = FixedArrayHeader::parse(&full, 0x100, 8, 8).unwrap();
let chunks = read_fixed_array_chunks(&full, &h, &dims, None, &[20], 8, 8, 8).unwrap();
// Chunk 2 is unallocated, and so is page 1 of a paged array.
let expect = if n > 4 { n - 1 - 4 } else { n - 1 };
assert_eq!(chunks.len(), expect);
let mut files = Vec::new();
for cut in (0x100..0x200 + 40 + n * (es + 4) + 16).step_by(3) {
files.push(full[..cut].to_vec());
}
for at in [0x104, 0x210, 0x21a, 0x230] {
let mut damaged = full.clone();
damaged[at] ^= 1;
files.push(damaged);
}
files.push(full);
for f in files {
let storage = CountingStorage::new(f.clone());
let want = FixedArrayHeader::parse(&f, 0x100, 8, 8);
let got = FixedArrayHeader::parse_in(&storage, 0x100, 8, 8);
assert_eq!(format!("{got:?}"), format!("{want:?}"));
let Ok(h) = want else { continue };
let want = read_fixed_array_chunks(&f, &h, &dims, None, &[20], 8, 8, 8);
let got = read_fixed_array_chunks_in(&storage, &h, &dims, None, &[20], 8, 8, 8);
assert_eq!(format!("{got:?}"), format!("{want:?}"), "{} bytes", f.len());
}
}
}
/// A header whose element count stretches its data block (one checksum
/// over the whole block) far past the end of a 16 MiB file: the
/// checksum's bounds check fails before the block is read, with the
/// slice read's error.
#[cfg(feature = "checksum")]
#[test]
fn oversized_block_fails_before_reading() {
use crate::storage::CountingStorage;
let mut f = build_fixed_array(3, false, 10);
f.resize(16 << 20, 0);
let mut h = FixedArrayHeader::parse(&f, 0x100, 8, 8).unwrap();
h.max_nelmts_bits = 30;
h.num_elements = 4 << 20;
let dims = [h.num_elements * 20];
let want = read_fixed_array_chunks(&f, &h, &dims, None, &[20], 8, 8, 8);
assert!(
matches!(want, Err(FormatError::UnexpectedEof { .. })),
"{want:?}"
);
let storage = CountingStorage::new(f);
let got = read_fixed_array_chunks_in(&storage, &h, &dims, None, &[20], 8, 8, 8);
assert_eq!(format!("{got:?}"), format!("{want:?}"));
assert!(storage.bytes_read() < 64, "{} bytes", storage.bytes_read());
}
}
crates/clawhdf5-format/src/fractal_heap.rs
+411 -71
View File
@@ -6,9 +6,11 @@ use alloc::{format, vec::Vec};
#[cfg(feature = "checksum")]
use byteorder::{ByteOrder, LittleEndian};
use crate::btree_v2::{BTreeV2Header, collect_btree_v2_records};
use crate::addr::to_usize;
use crate::btree_v2::{BTreeV2Header, find_btree_v2_records};
use crate::error::FormatError;
use crate::filter_pipeline::FilterPipeline;
use crate::storage::{Storage, Window, len_usize, read_exact_at, require_contiguous};
/// Parsed fractal heap header (signature "FRHP").
#[derive(Debug, Clone)]
@@ -138,6 +140,36 @@ impl FractalHeapHeader {
offset_size: u8,
length_size: u8,
) -> Result<FractalHeapHeader, FormatError> {
Self::parse_in(file_data, offset as u64, offset_size, length_size)
}
/// [`Self::parse`] over any [`Storage`]: one read of the header (two
/// when it holds an I/O filter pipeline).
pub fn parse_in<S: Storage + ?Sized>(
file: &S,
offset: u64,
offset_size: u8,
length_size: u8,
) -> Result<FractalHeapHeader, FormatError> {
// Every field up to the checksum, without and with the filter
// information; the window holds all of it (or ends at the end of
// the file), so its bounds checks are the whole-file ones.
let (os, ls) = (usize::from(offset_size), usize::from(length_size));
let unfiltered_len = 26 + 12 * ls + 3 * os;
let mut w = Window::read(file, offset, unfiltered_len)?;
if w.bytes.len() == unfiltered_len {
let filter_len = usize::from(u16::from_le_bytes([w.bytes[7], w.bytes[8]]));
if filter_len > 0 {
w = Window::read(file, offset, unfiltered_len + ls + 4 + filter_len)?;
}
}
let ensure_len = |_: &[u8], pos: usize, needed: usize| w.ensure(pos, needed);
let read_offset = |_: &[u8], pos: usize, size: u8| {
w.ensure(pos, usize::from(size))?;
read_offset(&w.bytes, pos, size)
};
let file_data: &[u8] = &w.bytes;
let offset = 0usize;
ensure_len(file_data, offset, 5)?;
if &file_data[offset..offset + 4] != b"FRHP" {
return Err(FormatError::InvalidFractalHeapSignature);
@@ -148,9 +180,6 @@ impl FractalHeapHeader {
return Err(FormatError::InvalidFractalHeapVersion(version));
}
let os = offset_size as usize;
let ls = length_size as usize;
let mut pos = offset + 5;
ensure_len(file_data, pos, 2)?;
let heap_id_length = u16::from_le_bytes([file_data[pos], file_data[pos + 1]]);
@@ -354,6 +383,19 @@ impl FractalHeapHeader {
id_bytes: &[u8],
offset_size: u8,
) -> Result<Vec<u8>, FormatError> {
self.read_managed_object_in(file_data, id_bytes, offset_size)
}
/// [`Self::read_managed_object`] over any [`Storage`]. A huge object
/// found through the huge-object v2 B-tree still needs the whole file
/// in memory ([`FormatError::ContiguousStorageRequired`] otherwise).
pub fn read_managed_object_in<S: Storage + ?Sized>(
&self,
file_data: &S,
id_bytes: &[u8],
offset_size: u8,
) -> Result<Vec<u8>, FormatError> {
crate::lookup_stats::heap_object_read();
let Some(&first) = id_bytes.first() else {
return Err(FormatError::UnexpectedEof {
expected: 1,
@@ -383,7 +425,11 @@ impl FractalHeapHeader {
}
/// Read a huge object (heap ID type 1).
fn read_huge_object(&self, file_data: &[u8], id: &[u8]) -> Result<Vec<u8>, FormatError> {
fn read_huge_object<S: Storage + ?Sized>(
&self,
file: &S,
id: &[u8],
) -> Result<Vec<u8>, FormatError> {
let os = usize::from(self.offset_size);
let ls = usize::from(self.length_size);
// (address, stored length, filter mask, decoded length); the last two
@@ -414,18 +460,17 @@ impl FractalHeapHeader {
let key_len = (usize::from(self.heap_id_length).saturating_sub(1)).min(8);
ensure_len(id, 1, key_len)?;
let key = le_uint(&id[1..1 + key_len]);
self.find_huge_record(file_data, key)?
self.find_huge_record(file, key)?
};
let start = usize::try_from(addr).map_err(|_| heap_error("huge object address"))?;
let len = usize::try_from(stored_len).map_err(|_| heap_error("huge object length"))?;
ensure_len(file_data, start, len)?;
let stored = &file_data[start..start + len];
let stored = read_exact_at(file, start as u64, len)?;
match &self.filter_pipeline {
None => Ok(stored.to_vec()),
None => Ok(stored.into_owned()),
Some(pipeline) => {
let mem = usize::try_from(mem_len).map_err(|_| heap_error("huge object size"))?;
let out = crate::filters::decompress_chunk_masked(stored, pipeline, mem, 1, mask)?;
let out = crate::filters::decompress_chunk_masked(&stored, pipeline, mem, 1, mask)?;
if out.len() != mem {
return Err(heap_error("filtered huge object decoded to the wrong size"));
}
@@ -436,9 +481,9 @@ impl FractalHeapHeader {
/// Look up huge object `key` in the huge-object v2 B-tree, returning
/// (address, stored length, filter mask, decoded length).
fn find_huge_record(
fn find_huge_record<S: Storage + ?Sized>(
&self,
file_data: &[u8],
file: &S,
key: u64,
) -> Result<(u64, u64, u32, u64), FormatError> {
if is_undefined(self.huge_btree_address, self.offset_size) {
@@ -446,9 +491,11 @@ impl FractalHeapHeader {
"huge object ID but the heap has no huge-object index",
));
}
// The v2 B-tree is read from a slice until it is converted.
let file_data = require_contiguous(file, "a huge fractal-heap object's B-tree")?;
let hdr = BTreeV2Header::parse(
file_data,
self.huge_btree_address as usize,
to_usize(self.huge_btree_address)?,
self.offset_size,
self.length_size,
)?;
@@ -463,8 +510,12 @@ impl FractalHeapHeader {
if hdr.tree_type != expected_type || usize::from(hdr.record_size) < rec_len {
return Err(heap_error("unexpected huge-object B-tree record type"));
}
let records =
collect_btree_v2_records(file_data, &hdr, self.offset_size, self.length_size)?;
// Records are ordered by ID (the last field): descend to the ones
// equal to `key` instead of reading the whole index.
let id_at = rec_len - ls;
let records = find_btree_v2_records(file_data, &hdr, self.offset_size, &mut |r| {
le_uint(&r[id_at..id_at + ls]).cmp(&key)
})?;
for rec in &records {
let d = &rec.data;
if d.len() < rec_len {
@@ -513,9 +564,9 @@ impl FractalHeapHeader {
}
/// Read a managed object (heap ID type 0).
fn read_heap_managed(
fn read_heap_managed<S: Storage + ?Sized>(
&self,
file_data: &[u8],
file_data: &S,
id_bytes: &[u8],
offset_size: u8,
) -> Result<Vec<u8>, FormatError> {
@@ -533,24 +584,24 @@ impl FractalHeapHeader {
self.read_from_direct_block(
file_data,
DirectBlock {
addr: self.root_block_address as usize,
addr: to_usize(self.root_block_address)?,
size: self.starting_block_size,
heap_offset: 0,
filtered_size: self.root_direct_block_filtered_size,
filter_mask: self.root_direct_block_filter_mask,
},
heap_offset,
obj_len as usize,
to_usize(obj_len)?,
)
} else {
// Root is an indirect block — limit recursion to 64 levels
self.read_from_indirect_block(
file_data,
self.root_block_address as usize,
to_usize(self.root_block_address)?,
self.current_rows_in_root_indirect_block,
0, // block offset
heap_offset,
obj_len as usize,
to_usize(obj_len)?,
offset_size,
64, // max recursion depth
)
@@ -563,27 +614,27 @@ impl FractalHeapHeader {
/// header), so we just add it to the block address minus the block's heap
/// offset. A filtered heap stores each direct block (header included)
/// through its filter pipeline, so the block is decoded first.
fn read_from_direct_block(
fn read_from_direct_block<S: Storage + ?Sized>(
&self,
file_data: &[u8],
file: &S,
block: DirectBlock,
target_offset: u64,
length: usize,
) -> Result<Vec<u8>, FormatError> {
if target_offset < block.heap_offset {
return Err(FormatError::UnexpectedEof {
expected: block.heap_offset as usize,
available: target_offset as usize,
expected: to_usize(block.heap_offset)?,
available: to_usize(target_offset)?,
});
}
let local_offset = (target_offset - block.heap_offset) as usize;
let local_offset = to_usize(target_offset - block.heap_offset)?;
if let Some(pipeline) = &self.filter_pipeline {
let stored_len = usize::try_from(block.filtered_size)
.map_err(|_| heap_error("direct block size"))?;
let size = usize::try_from(block.size).map_err(|_| heap_error("direct block size"))?;
ensure_len(file_data, block.addr, stored_len)?;
let stored = read_exact_at(file, block.addr as u64, stored_len)?;
let decoded = crate::filters::decompress_chunk_masked(
&file_data[block.addr..block.addr + stored_len],
&stored,
pipeline,
size,
1,
@@ -597,17 +648,16 @@ impl FractalHeapHeader {
.checked_add(local_offset)
.ok_or(FormatError::UnexpectedEof {
expected: usize::MAX,
available: file_data.len(),
available: len_usize(file),
})?;
ensure_len(file_data, pos, length)?;
Ok(file_data[pos..pos + length].to_vec())
Ok(read_exact_at(file, pos as u64, length)?.into_owned())
}
/// Read an object by traversing an indirect block to find the right direct block.
#[allow(clippy::too_many_arguments)]
fn read_from_indirect_block(
fn read_from_indirect_block<S: Storage + ?Sized>(
&self,
file_data: &[u8],
file: &S,
iblock_addr: usize,
nrows: u16,
iblock_heap_offset: u64,
@@ -621,29 +671,169 @@ impl FractalHeapHeader {
"fractal heap: maximum recursion depth exceeded".into(),
));
}
// Parse indirect block header
ensure_len(file_data, iblock_addr, 4)?;
if &file_data[iblock_addr..iblock_addr + 4] != b"FHIB" {
return Err(FormatError::InvalidFractalHeapSignature);
}
let block_offset_bytes = (self.max_heap_size as usize).div_ceil(8);
let iblock_header = 5 + offset_size as usize + block_offset_bytes;
let mut pos = iblock_addr + iblock_header;
let tw = self.table_width as u64;
let nrows_usize = nrows as usize;
let mut current_heap_offset = iblock_heap_offset;
// Rows below max_direct_rows hold direct blocks; rows at/above hold
// child indirect blocks. (NOT the FRHP "starting rows" field.)
let start_indirect = self.max_direct_rows();
let max_direct_rows = nrows_usize.min(start_indirect);
// The block up to its last child entry. The walk below reads
// entries in order and stops at the one covering the target, which
// the geometry alone locates, so the first window ends there: a
// header claiming a huge table costs a read of the entries in front
// of the target, not of the rest of the file. Only when that entry
// is unallocated (or none covers the target) does the walk go on,
// over the whole block. Either window holds what it was asked for or
// ends at the end of the file, so its bounds checks are the
// whole-file ones.
let direct_entry = usize::from(offset_size)
+ if self.filter_pipeline.is_some() {
usize::from(self.length_size) + 4
} else {
0
};
let direct_entries = max_direct_rows.saturating_mul(usize::from(self.table_width));
let entries_len = |n: usize| {
n.min(direct_entries)
.saturating_mul(direct_entry)
.saturating_add(
n.saturating_sub(direct_entries)
.saturating_mul(usize::from(offset_size)),
)
};
let all_entries = direct_entries.saturating_add(
nrows_usize
.saturating_sub(start_indirect)
.saturating_mul(usize::from(self.table_width)),
);
let block_len = iblock_header.saturating_add(entries_len(all_entries));
let target_entry = self.indirect_entry_for(nrows_usize, iblock_heap_offset, target_offset);
let first_len = target_entry.map_or(block_len, |i| {
iblock_header
.saturating_add(entries_len(i.saturating_add(1)))
.min(block_len)
});
let mut next = self.walk_indirect_block(
&Window::read(file, iblock_addr as u64, first_len)?,
nrows_usize,
iblock_heap_offset,
target_offset,
offset_size,
target_entry.map_or(usize::MAX, |i| i.saturating_add(1)),
)?;
if next.is_none() && first_len < block_len {
next = self.walk_indirect_block(
&Window::read(file, iblock_addr as u64, block_len)?,
nrows_usize,
iblock_heap_offset,
target_offset,
offset_size,
usize::MAX,
)?;
}
match next {
Some(IndirectChild::Direct(block)) => {
self.read_from_direct_block(file, block, target_offset, length)
}
Some(IndirectChild::Indirect {
addr,
nrows,
heap_offset,
}) => self.read_from_indirect_block(
file,
addr,
nrows,
heap_offset,
target_offset,
length,
offset_size,
depth_remaining - 1,
),
None => Err(FormatError::UnexpectedEof {
expected: to_usize(target_offset)?.saturating_add(length),
available: len_usize(file),
}),
}
}
/// Which child entry of an indirect block (numbered in walk order:
/// direct rows, then indirect rows) covers `target_offset`, from the
/// doubling-table geometry alone — the entry
/// [`Self::walk_indirect_block`] stops at if it is allocated. `None`
/// when no entry does.
fn indirect_entry_for(&self, nrows: usize, heap_offset: u64, target: u64) -> Option<usize> {
// The walk adds block sizes with saturation; in u128 the same test
// is `cur <= target < cur + size` without it (a target of u64::MAX
// is never inside a saturated range).
if target == u64::MAX {
return None;
}
let (tw, target) = (u128::from(self.table_width), u128::from(target));
let mut cur = u128::from(heap_offset);
let mut before = 0usize;
for row in 0..nrows {
if target < cur {
return None;
}
// Direct and indirect rows alike span this row's block size per
// entry.
let size = u128::from(self.block_size_for_row(row));
let span = size * tw;
if size > 0 && target < cur + span {
let col = usize::try_from((target - cur) / size).ok()?;
return before.checked_add(col);
}
cur += span;
before = before.saturating_add(self.table_width as usize);
}
None
}
/// Walk an indirect block's child entries in order, in the window `w`
/// (the block from its signature on), and return the allocated child
/// covering `target_offset`, or `None` when no entry among the first
/// `limit` does.
fn walk_indirect_block(
&self,
w: &Window<'_>,
nrows: usize,
iblock_heap_offset: u64,
target_offset: u64,
offset_size: u8,
limit: usize,
) -> Result<Option<IndirectChild>, FormatError> {
let ensure_len = |_: &[u8], pos: usize, needed: usize| w.ensure(pos, needed);
let read_offset = |_: &[u8], pos: usize, size: u8| {
w.ensure(pos, usize::from(size))?;
read_offset(&w.bytes, pos, size)
};
let file_data: &[u8] = &w.bytes;
let block_offset_bytes = (self.max_heap_size as usize).div_ceil(8);
let iblock_header = 5 + offset_size as usize + block_offset_bytes;
let tw = self.table_width as u64;
let mut current_heap_offset = iblock_heap_offset;
let start_indirect = self.max_direct_rows();
let max_direct_rows = nrows.min(start_indirect);
let mut walked = 0usize;
// Parse indirect block header
ensure_len(file_data, 0, 4)?;
if &file_data[..4] != b"FHIB" {
return Err(FormatError::InvalidFractalHeapSignature);
}
let mut pos = iblock_header;
for row in 0..max_direct_rows {
let block_size = self.block_size_for_row(row);
for _col in 0..tw {
if walked == limit {
return Ok(None);
}
walked += 1;
let child_addr = read_offset(file_data, pos, offset_size)?;
pos += offset_size as usize;
@@ -670,18 +860,13 @@ impl FractalHeapHeader {
&& target_offset >= current_heap_offset
&& target_offset < block_end
{
return self.read_from_direct_block(
file_data,
DirectBlock {
addr: child_addr as usize,
size: block_size,
heap_offset: current_heap_offset,
filtered_size,
filter_mask,
},
target_offset,
length,
);
return Ok(Some(IndirectChild::Direct(DirectBlock {
addr: to_usize(child_addr)?,
size: block_size,
heap_offset: current_heap_offset,
filtered_size,
filter_mask,
})));
}
current_heap_offset = block_end;
}
@@ -690,11 +875,15 @@ impl FractalHeapHeader {
// Rows at and above `start_indirect` hold child indirect blocks. A
// child in row r spans exactly that row's block size of heap space,
// so it has as many rows as a table of that total size needs.
for row in start_indirect..nrows_usize {
for row in start_indirect..nrows {
let child_space = self.block_size_for_row(row);
let child_nrows = self.rows_for_size(child_space);
for _col in 0..tw {
if walked == limit {
return Ok(None);
}
walked += 1;
let child_addr = read_offset(file_data, pos, offset_size)?;
pos += offset_size as usize;
@@ -703,25 +892,16 @@ impl FractalHeapHeader {
&& target_offset >= current_heap_offset
&& target_offset < block_end
{
return self.read_from_indirect_block(
file_data,
child_addr as usize,
child_nrows,
current_heap_offset,
target_offset,
length,
offset_size,
depth_remaining - 1,
);
return Ok(Some(IndirectChild::Indirect {
addr: to_usize(child_addr)?,
nrows: child_nrows,
heap_offset: current_heap_offset,
}));
}
current_heap_offset = block_end;
}
}
Err(FormatError::UnexpectedEof {
expected: target_offset as usize + length,
available: file_data.len(),
})
Ok(None)
}
/// Number of rows in the doubling table whose block size is at most the
@@ -765,6 +945,16 @@ impl FractalHeapHeader {
/// A managed direct block's location, extent and (for a filtered heap) its
/// stored size and filter mask.
/// The child of an indirect block that covers a heap offset.
enum IndirectChild {
Direct(DirectBlock),
Indirect {
addr: usize,
nrows: u16,
heap_offset: u64,
},
}
struct DirectBlock {
addr: usize,
size: u64,
@@ -1024,4 +1214,154 @@ mod tests {
let id = [0x40u8, 0, 0, 0, 0, 0, 0];
assert!(hdr.read_managed_object(&file_data, &id, 8).is_err());
}
/// Headers, and managed (in a direct root and through an indirect
/// root), huge and tiny objects read identically through a
/// `read_at`-only storage, for every truncation of the file.
#[test]
fn storage_reads_match_slice_reads() {
use crate::storage::CountingStorage;
let (mut file, header_end) = build_simple_heap(8, 8);
// An indirect root block at 600: row 0 holds the direct block at
// 256, then three undefined blocks.
file[600..604].copy_from_slice(b"FHIB");
let mut at = 600 + 5 + 8 + 2;
for addr in [256u64, u64::MAX, u64::MAX, u64::MAX] {
file[at..at + 8].copy_from_slice(&addr.to_le_bytes());
at += 8;
}
file[900..905].copy_from_slice(b"huge!");
let managed_id = |offset: u64, len: u64| {
let payload = offset | (len << 16);
let mut id = vec![0u8];
id.extend_from_slice(&payload.to_le_bytes()[..6]);
id
};
let mut huge = vec![0x10u8];
huge.extend_from_slice(&900u64.to_le_bytes());
huge.extend_from_slice(&5u64.to_le_bytes());
let ids = [
managed_id(15, 13),
managed_id(15, 200),
managed_id(130, 4),
huge,
vec![0x22, b'a', b'b', b'c', 0, 0, 0],
];
let mut cuts: Vec<usize> = (0..=header_end + 1).collect();
cuts.extend([256, 260, 271, 280, 600, 610, 620, 640, 900, 903, file.len()]);
for cut in cuts {
let f = &file[..cut];
let storage = CountingStorage::new(f.to_vec());
let want = FractalHeapHeader::parse(f, 0, 8, 8);
let got = FractalHeapHeader::parse_in(&storage, 0, 8, 8);
assert_eq!(format!("{got:?}"), format!("{want:?}"), "cut {cut}");
let Ok(direct) = want else { continue };
let mut indirect = direct.clone();
indirect.root_block_address = 600;
indirect.current_rows_in_root_indirect_block = 1;
let mut huge_ids = direct.clone();
huge_ids.heap_id_length = 17;
for hdr in [&direct, &indirect, &huge_ids] {
for id in &ids {
assert_eq!(
hdr.read_managed_object_in(&storage, id, 8),
hdr.read_managed_object(f, id, 8),
"cut {cut}"
);
}
}
}
}
/// A header claiming a huge doubling table (width 0xFFFF, 0xFFFF rows in
/// the root indirect block) in a 16 MiB file: reading an object from the
/// table's first block reads the entries up to it, not the rest of the
/// file, and gives what the slice read gives. When the covering entry is
/// unallocated the walk goes on over the whole block, still identically.
#[test]
fn huge_table_claims_read_only_what_the_walk_needs() {
use crate::storage::CountingStorage;
let (mut file, _) = build_simple_heap(8, 8);
file.resize(16 << 20, 0);
file[600..604].copy_from_slice(b"FHIB");
let first_entry = 600 + 5 + 8 + 2;
file[first_entry..first_entry + 8].copy_from_slice(&256u64.to_le_bytes());
let mut hdr = FractalHeapHeader::parse(&file, 0, 8, 8).unwrap();
hdr.table_width = 0xFFFF;
hdr.root_block_address = 600;
hdr.current_rows_in_root_indirect_block = 0xFFFF;
let managed_id = |offset: u64, len: u64| {
let payload = offset | (len << 16);
let mut id = vec![0u8];
id.extend_from_slice(&payload.to_le_bytes()[..6]);
id
};
let storage = CountingStorage::new(file.clone());
let id = managed_id(15, 13);
let want = hdr.read_managed_object(&file, &id, 8);
assert!(want.is_ok(), "{want:?}");
storage.reset();
assert_eq!(hdr.read_managed_object_in(&storage, &id, 8), want);
assert!(
storage.bytes_read() < 1024,
"{} bytes in {} reads",
storage.bytes_read(),
storage.reads()
);
// The second entry (heap offsets 128..256) is unallocated (zero is
// not the undefined address, so make it all ones).
file[first_entry + 8..first_entry + 16].fill(0xFF);
let storage = CountingStorage::new(file.clone());
let id = managed_id(130, 4);
let want = hdr.read_managed_object(&file, &id, 8);
assert_eq!(hdr.read_managed_object_in(&storage, &id, 8), want);
}
/// A huge object found through the huge-object B-tree needs the whole
/// file in memory until the B-tree reader is converted: a clean error
/// on other storage.
#[test]
fn huge_object_btree_needs_contiguous_storage() {
use crate::storage::CountingStorage;
let (file, _) = build_simple_heap(8, 8);
let mut hdr = FractalHeapHeader::parse(&file, 0, 8, 8).unwrap();
hdr.huge_btree_address = 700;
let storage = CountingStorage::new(file);
assert_eq!(
hdr.read_managed_object_in(&storage, &[0x10, 1, 0, 0, 0, 0, 0], 8),
Err(FormatError::ContiguousStorageRequired(
"a huge fractal-heap object's B-tree"
))
);
}
/// A header with an I/O filter pipeline (read in a second, longer
/// window) parses identically through a `read_at`-only storage, for
/// every truncation.
#[test]
fn filtered_header_parses_identically_through_storage() {
use crate::storage::CountingStorage;
let (simple, header_end) = build_simple_heap(8, 8);
let pipeline = [2u8, 1, 1, 0, 0, 0, 1, 0, 6, 0, 0, 0]; // deflate, level 6
let mut header = simple[..header_end - 4].to_vec();
header[7..9].copy_from_slice(&(pipeline.len() as u16).to_le_bytes());
header.extend_from_slice(&100u64.to_le_bytes()); // root block's stored size
header.extend_from_slice(&0u32.to_le_bytes()); // its filter mask
header.extend_from_slice(&pipeline);
let sum = crate::checksum::jenkins_lookup3(&header);
header.extend_from_slice(&sum.to_le_bytes());
let mut file = header.clone();
file.resize(256, 0);
let hdr = FractalHeapHeader::parse(&file, 0, 8, 8).unwrap();
assert!(hdr.filter_pipeline.is_some());
for cut in 0..=file.len() {
let f = &file[..cut];
let storage = CountingStorage::new(f.to_vec());
assert_eq!(
format!("{:?}", FractalHeapHeader::parse_in(&storage, 0, 8, 8)),
format!("{:?}", FractalHeapHeader::parse(f, 0, 8, 8)),
"cut {cut}"
);
}
}
}
crates/clawhdf5-format/src/global_heap.rs
+113 -26
View File
@@ -1,9 +1,12 @@
//! HDF5 Global Heap collection parsing.
#[cfg(not(feature = "std"))]
use alloc::{format, string::String, vec::Vec};
use alloc::{borrow::Cow, format, string::String, vec::Vec};
#[cfg(feature = "std")]
use std::borrow::Cow;
use crate::error::FormatError;
use crate::storage::{Storage, len_usize, read_exact_at};
/// Magic signature for global heap collections.
const GCOL_SIGNATURE: [u8; 4] = *b"GCOL";
@@ -28,19 +31,20 @@ pub struct GlobalHeapObject {
pub data: Vec<u8>,
}
fn ensure_len(data: &[u8], offset: usize, needed: usize) -> Result<(), FormatError> {
/// Checks that `[offset, offset + needed)` ends by `data_len`.
fn ensure_len(data_len: usize, offset: usize, needed: usize) -> Result<(), FormatError> {
match offset.checked_add(needed) {
Some(end) if end <= data.len() => Ok(()),
Some(end) if end <= data_len => Ok(()),
_ => Err(FormatError::UnexpectedEof {
expected: offset.saturating_add(needed),
available: data.len(),
available: data_len,
}),
}
}
fn read_length(data: &[u8], offset: usize, length_size: u8) -> Result<u64, FormatError> {
let s = length_size as usize;
ensure_len(data, offset, s)?;
ensure_len(data.len(), offset, s)?;
let slice = &data[offset..offset + s];
Ok(match length_size {
2 => u16::from_le_bytes([slice[0], slice[1]]) as u64,
@@ -95,7 +99,17 @@ impl GlobalHeapCollection {
offset: usize,
length_size: u8,
) -> Result<GlobalHeapCollection, FormatError> {
let index = Self::parse_index(file_data, offset, length_size)?;
Self::parse_in(file_data, offset as u64, length_size)
}
/// [`Self::parse`] over any [`Storage`]: one read of the header, one of
/// the collection.
pub fn parse_in<S: Storage + ?Sized>(
file: &S,
offset: u64,
length_size: u8,
) -> Result<GlobalHeapCollection, FormatError> {
let (bytes, base, index) = Self::read_collection(file, offset, length_size)?;
Ok(GlobalHeapCollection {
collection_size: index.collection_size,
objects: index
@@ -104,7 +118,7 @@ impl GlobalHeapCollection {
.map(|o| GlobalHeapObject {
index: o.index,
reference_count: o.reference_count,
data: file_data[o.offset..o.offset + o.size].to_vec(),
data: bytes[o.offset - base..o.offset - base + o.size].to_vec(),
})
.collect(),
})
@@ -122,43 +136,72 @@ impl GlobalHeapCollection {
offset: usize,
length_size: u8,
) -> Result<GlobalHeapIndex, FormatError> {
Self::parse_index_in(file_data, offset as u64, length_size)
}
/// [`Self::parse_index`] over any [`Storage`]: one read of the header,
/// one of the collection. The object offsets are file offsets.
pub fn parse_index_in<S: Storage + ?Sized>(
file: &S,
offset: u64,
length_size: u8,
) -> Result<GlobalHeapIndex, FormatError> {
Ok(Self::read_collection(file, offset, length_size)?.2)
}
/// Read the collection at `offset` and index its objects: the
/// collection's bytes, its offset as a `usize`, and the index (with
/// file offsets).
fn read_collection<S: Storage + ?Sized>(
file: &S,
offset: u64,
length_size: u8,
) -> Result<(Cow<'_, [u8]>, usize, GlobalHeapIndex), FormatError> {
let file_len = len_usize(file);
// signature(4) + version(1) + reserved(3) + collection_size(length_size),
// padded to a multiple of 8 as libhdf5 lays it out (`H5HG_SIZEOF_HDR`).
// With 8-byte lengths the padding is 0; with 4-byte lengths it is 4,
// and reading without it put every object 4 bytes early.
let header_size = pad8(8 + length_size as usize);
ensure_len(file_data, offset, header_size)?;
let header = read_exact_at(file, offset, header_size)?;
let offset = usize::try_from(offset).map_err(|_| FormatError::UnexpectedEof {
expected: usize::MAX,
available: file_len,
})?;
if file_data[offset..offset + 4] != GCOL_SIGNATURE {
if header[..4] != GCOL_SIGNATURE {
return Err(FormatError::InvalidGlobalHeapSignature);
}
let version = file_data[offset + 4];
let version = header[4];
if version != 1 {
return Err(FormatError::InvalidGlobalHeapVersion(version));
}
let collection_size = read_length(file_data, offset + 8, length_size)?;
let collection_size = read_length(&header, 8, length_size)?;
let collection_end = usize::try_from(collection_size)
.ok()
.and_then(|size| offset.checked_add(size))
.ok_or(FormatError::UnexpectedEof {
expected: usize::MAX,
available: file_data.len(),
available: file_len,
})?;
if collection_end > file_data.len() {
if collection_end > file_len {
return Err(FormatError::UnexpectedEof {
expected: collection_end,
available: file_data.len(),
available: file_len,
});
}
let collection = read_exact_at(file, offset as u64, collection_end - offset)?;
// Positions below are file offsets; `file_data(p)` is the byte at `p`.
let file_data = |p: usize| collection[p - offset];
let mut pos = offset + header_size;
let mut objects = Vec::new();
// Parse objects until we hit index 0 (free space) or run out of space
while pos + 2 <= collection_end {
let object_index = u16::from_le_bytes([file_data[pos], file_data[pos + 1]]);
let object_index = u16::from_le_bytes([file_data(pos), file_data(pos + 1)]);
if object_index == 0 {
// Free space marker — done
@@ -168,11 +211,12 @@ impl GlobalHeapCollection {
// object_index(2) + reference_count(2) + reserved(4) +
// object_size(length_size), padded to 8 (`H5HG_SIZEOF_OBJHDR`).
let obj_header_size = pad8(8 + length_size as usize);
ensure_len(&file_data[..collection_end], pos, obj_header_size)?;
ensure_len(collection_end, pos, obj_header_size)?;
let reference_count = u16::from_le_bytes([file_data[pos + 2], file_data[pos + 3]]);
let object_size = usize::try_from(read_length(file_data, pos + 8, length_size)?)
.map_err(|_| FormatError::Overflow("global heap object size".into()))?;
let reference_count = u16::from_le_bytes([file_data(pos + 2), file_data(pos + 3)]);
let object_size =
usize::try_from(read_length(&collection[pos - offset..], 8, length_size)?)
.map_err(|_| FormatError::Overflow("global heap object size".into()))?;
pos += obj_header_size;
if pos
@@ -197,10 +241,11 @@ impl GlobalHeapCollection {
pos = pos.saturating_add(pad8(object_size));
}
Ok(GlobalHeapIndex {
let index = GlobalHeapIndex {
collection_size,
objects,
})
};
Ok((collection, offset, index))
}
/// Get an object by its index.
@@ -226,7 +271,7 @@ mod tests {
let mut obj_size_total = 0usize;
for (_, _, data) in objects {
let obj_header = pad8(8 + ls);
obj_size_total += obj_header + pad8(data.len());
obj_size_total += obj_header + pad8(<[u8]>::len(data));
}
// Free space marker (2 bytes for index 0)
obj_size_total += 2;
@@ -251,15 +296,17 @@ mod tests {
buf.extend_from_slice(&ref_count.to_le_bytes());
buf.extend_from_slice(&[0u8; 4]); // reserved
match length_size {
4 => buf.extend_from_slice(&(data.len() as u32).to_le_bytes()),
8 => buf.extend_from_slice(&(data.len() as u64).to_le_bytes()),
// `<[u8]>::len`: with `Storage` in scope `data.len()` on a
// `&&[u8]` resolves to `Storage::len` (a `u64`).
4 => buf.extend_from_slice(&(<[u8]>::len(data) as u32).to_le_bytes()),
8 => buf.extend_from_slice(&(<[u8]>::len(data) as u64).to_le_bytes()),
_ => panic!("unsupported"),
}
buf.resize(buf.len() + (pad8(8 + ls) - (8 + ls)), 0);
buf.extend_from_slice(data);
// Pad to 8 bytes
let padded = pad8(data.len());
buf.resize(buf.len() + (padded - data.len()), 0);
let padded = pad8(<[u8]>::len(data));
buf.resize(buf.len() + (padded - <[u8]>::len(data)), 0);
}
// Free space marker
@@ -327,4 +374,44 @@ mod tests {
assert_eq!(coll.objects.len(), 1);
assert_eq!(coll.objects[0].data, b"test");
}
/// Collections, and every truncation of them, index and parse
/// identically through a `read_at`-only storage: two reads each.
#[test]
fn storage_parse_matches_slice_parse() {
use crate::storage::CountingStorage;
let objs: &[(u16, u16, &[u8])] = &[(1, 1, b"hello"), (2, 3, b"a longer object")];
for ls in [4u8, 8] {
let coll = build_collection(objs, ls);
let mut corrupt = coll.clone();
corrupt[8] = 200; // collection size past the end of the file
let mut overrun = coll.clone();
let size_at = pad8(8 + ls as usize) + 8;
overrun[size_at] = 250; // first object runs past the collection
for full in [coll, corrupt, overrun] {
for at in [0usize, 5] {
for cut in 0..=full.len() {
let mut f = vec![0u8; at];
f.extend_from_slice(&full[..cut]);
let storage = CountingStorage::new(f.clone());
let want = GlobalHeapCollection::parse(&f, at, ls);
let got = GlobalHeapCollection::parse_in(&storage, at as u64, ls);
assert_eq!(format!("{got:?}"), format!("{want:?}"));
let want = GlobalHeapCollection::parse_index(&f, at, ls);
let got = GlobalHeapCollection::parse_index_in(&storage, at as u64, ls);
assert_eq!(format!("{got:?}"), format!("{want:?}"));
}
}
}
}
let storage = CountingStorage::new(build_collection(objs, 8));
assert_eq!(
GlobalHeapCollection::parse_in(&storage, 0, 8)
.unwrap()
.objects
.len(),
2
);
assert_eq!(storage.reads(), 2);
}
}
crates/clawhdf5-format/src/group_v1.rs
+6 -5
View File
@@ -3,6 +3,7 @@
#[cfg(not(feature = "std"))]
use alloc::{string::String, vec::Vec};
use crate::addr::to_usize;
use crate::btree_v1::collect_symbol_table_nodes;
use crate::error::FormatError;
use crate::local_heap::LocalHeap;
@@ -54,7 +55,7 @@ pub(crate) fn v1_group_entries(
// Parse local heap
let heap = LocalHeap::parse(
file_data,
sym_table_msg.local_heap_address as usize,
to_usize(sym_table_msg.local_heap_address)?,
offset_size,
length_size,
)?;
@@ -70,7 +71,7 @@ pub(crate) fn v1_group_entries(
let mut entries = Vec::new();
let mut heap_checked = false;
for snod_addr in snod_addrs {
let snod = SymbolTableNode::parse(file_data, snod_addr as usize, offset_size)?;
let snod = SymbolTableNode::parse(file_data, to_usize(snod_addr)?, offset_size)?;
for entry in &snod.entries {
// Like libhdf5, look at the heap's free list only once a name is
// needed: an empty group with a damaged heap still lists.
@@ -152,7 +153,7 @@ fn for_each_v1_soft_link(
) -> Result<(), FormatError> {
let heap = LocalHeap::parse(
file_data,
sym_table_msg.local_heap_address as usize,
to_usize(sym_table_msg.local_heap_address)?,
offset_size,
length_size,
)?;
@@ -164,7 +165,7 @@ fn for_each_v1_soft_link(
)?;
let mut heap_checked = false;
for snod_addr in snod_addrs {
let snod = SymbolTableNode::parse(file_data, snod_addr as usize, offset_size)?;
let snod = SymbolTableNode::parse(file_data, to_usize(snod_addr)?, offset_size)?;
for entry in &snod.entries {
if entry.cache_type != CACHE_TYPE_SOFT_LINK {
continue;
@@ -242,7 +243,7 @@ pub fn resolve_path(
// Not last — must be a group, parse its object header to get symbol table
let obj_header = ObjectHeader::parse(
file_data,
entry.object_header_address as usize,
to_usize(entry.object_header_address)?,
offset_size,
length_size,
)?;
crates/clawhdf5-format/src/group_v2.rs
+231 -60
View File
@@ -6,7 +6,14 @@
#[cfg(not(feature = "std"))]
use alloc::{string::String, vec::Vec};
use crate::btree_v2::{BTreeV2Header, collect_btree_v2_records};
#[cfg(not(feature = "std"))]
use alloc::collections::BTreeSet;
#[cfg(feature = "std")]
use std::collections::BTreeSet;
use crate::addr::to_usize;
use crate::btree_v2::{BTreeV2Header, collect_btree_v2_records, find_btree_v2_records};
use crate::checksum::jenkins_lookup3;
use crate::error::FormatError;
use crate::fractal_heap::FractalHeapHeader;
use crate::group_v1::{self, GroupEntry};
@@ -93,13 +100,14 @@ fn for_each_dense_link(
mut visit: impl FnMut(LinkMessage),
) -> Result<(), FormatError> {
// Parse fractal heap
let fh = FractalHeapHeader::parse(file_data, fh_addr as usize, offset_size, length_size)?;
let fh = FractalHeapHeader::parse(file_data, to_usize(fh_addr)?, offset_size, length_size)?;
// Parse B-tree v2 for name index
let btree_addr = link_info
.btree_name_index_address
.ok_or_else(|| FormatError::PathNotFound(String::from("no B-tree v2 name index")))?;
let btree_hdr = BTreeV2Header::parse(file_data, btree_addr as usize, offset_size, length_size)?;
let btree_hdr =
BTreeV2Header::parse(file_data, to_usize(btree_addr)?, offset_size, length_size)?;
let records = collect_btree_v2_records(file_data, &btree_hdr, offset_size, length_size)?;
for record in &records {
@@ -156,35 +164,69 @@ fn resolve_dense_entries(
Ok(entries)
}
/// The soft or external link called `name` in this group, if there is one.
/// Hard links are what `resolve_group_entries` returns; this is consulted only
/// when a path component isn't among them.
fn find_symbolic_link(
/// The soft link called `name` in a v1 (symbol table) group, if there is
/// one. Hard links are what `resolve_group_entries` returns; this is
/// consulted only when a path component isn't among them.
fn find_v1_symbolic_link(
file_data: &[u8],
object_header: &ObjectHeader,
name: &str,
offset_size: u8,
length_size: u8,
) -> Result<Option<LinkTarget>, FormatError> {
if is_v1_group(object_header) {
let Some(sym_msg) = object_header
.messages
.iter()
.find(|m| m.msg_type == MessageType::SymbolTable)
else {
return Ok(None);
};
let stm = SymbolTableMessage::parse(&sym_msg.data, offset_size)?;
return group_v1::find_v1_soft_link(file_data, &stm, name, offset_size, length_size)
.map(|target| target.map(|target_path| LinkTarget::Soft { target_path }));
}
if !is_v2_group(object_header) {
let Some(sym_msg) = object_header
.messages
.iter()
.find(|m| m.msg_type == MessageType::SymbolTable)
else {
return Ok(None);
}
let is_symbolic = |t: &LinkTarget| !matches!(t, LinkTarget::Hard { .. });
};
let stm = SymbolTableMessage::parse(&sym_msg.data, offset_size)?;
group_v1::find_v1_soft_link(file_data, &stm, name, offset_size, length_size)
.map(|target| target.map(|target_path| LinkTarget::Soft { target_path }))
}
/// B-tree v2 record type of a dense group's link name index.
const LINK_NAME_INDEX: u8 = 5;
/// The links called `name` in a v2 group (a valid group has at most one),
/// in storage order: header message order for a compact group, name index
/// order for a dense one.
///
/// In dense storage the link name index (a v2 B-tree of lookup3 name
/// hashes, record type 5) is descended to the records with the name's hash,
/// and only their links are read from the heap — O(log n) instead of every
/// link. libhdf5 orders records with equal hashes by name; all of them are
/// read and compared here, so that order does not matter. An index of
/// another type is scanned in full.
fn links_named(
file_data: &[u8],
object_header: &ObjectHeader,
name: &str,
offset_size: u8,
length_size: u8,
) -> Result<Vec<LinkMessage>, FormatError> {
let mut found = Vec::new();
let link_info = find_link_info(object_header, offset_size)?;
let mut found = None;
if let Some(fh_addr) = link_info.fractal_heap_address {
let Some(fh_addr) = link_info.fractal_heap_address else {
for msg in &object_header.messages {
if msg.msg_type == MessageType::Link
&& let Some(link) = parse_link(&msg.data, offset_size)?
&& link.name == name
{
found.push(link);
}
}
return Ok(found);
};
let fh = FractalHeapHeader::parse(file_data, to_usize(fh_addr)?, offset_size, length_size)?;
let btree_addr = link_info
.btree_name_index_address
.ok_or_else(|| FormatError::PathNotFound(String::from("no B-tree v2 name index")))?;
let btree_hdr =
BTreeV2Header::parse(file_data, to_usize(btree_addr)?, offset_size, length_size)?;
if btree_hdr.tree_type != LINK_NAME_INDEX {
for_each_dense_link(
file_data,
&link_info,
@@ -192,26 +234,151 @@ fn find_symbolic_link(
offset_size,
length_size,
|link| {
if link.name == name && is_symbolic(&link.link_target) {
found = Some(link.link_target);
if link.name == name {
found.push(link);
}
},
)?;
} else {
for msg in &object_header.messages {
if msg.msg_type == MessageType::Link {
let Some(link) = parse_link(&msg.data, offset_size)? else {
continue;
};
if link.name == name && is_symbolic(&link.link_target) {
found = Some(link.link_target);
}
}
return Ok(found);
}
// Record: hash(4) + heap ID.
let hash = jenkins_lookup3(name.as_bytes());
let records = find_btree_v2_records(file_data, &btree_hdr, offset_size, &mut |r| {
match r.get(..4) {
Some(h) => u32::from_le_bytes([h[0], h[1], h[2], h[3]]).cmp(&hash),
// Too short to hold a hash (a corrupt record size): never a match.
None => core::cmp::Ordering::Less,
}
})?;
let id_len = usize::from(fh.heap_id_length);
for record in &records {
let Some(id_bytes) = record.data.get(4..4 + id_len) else {
continue;
};
let link_data = fh.read_managed_object(file_data, id_bytes, offset_size)?;
if let Some(link) = parse_link(&link_data, offset_size)?
&& link.name == name
{
found.push(link);
}
}
Ok(found)
}
/// The link called `name` in a v2 group, if any.
///
/// A valid group has at most one; libhdf5 cannot create two. If a damaged
/// or hand-made group has several, the first wins and the rest are
/// ignored, whatever their kind and even if the first cannot be followed.
/// That is libhdf5's rule for a compact group (`H5G__compact_lookup` stops
/// at the first Link message of that name; h5py then fails to open a
/// dangling first link although a later one resolves). For a dense group
/// "first" is first in name index order; libhdf5 binary-searches the index
/// and may land on another of several exact duplicates. The listing
/// ([`resolve_group_children`]), [`resolve_child`] and path resolution all
/// apply this rule, so they agree.
fn first_link_named(
file_data: &[u8],
object_header: &ObjectHeader,
name: &str,
offset_size: u8,
length_size: u8,
) -> Result<Option<LinkMessage>, FormatError> {
Ok(
links_named(file_data, object_header, name, offset_size, length_size)?
.into_iter()
.next(),
)
}
/// The link [`resolve_path_any`] follows for one path component `name` of
/// the group with header `object_header`: a hard link (as `Hard`), else a
/// soft or external link of that name, else `None`. Fails with
/// `PathNotFound` if the object is not a group.
fn lookup_link(
file_data: &[u8],
object_header: &ObjectHeader,
name: &str,
offset_size: u8,
length_size: u8,
) -> Result<Option<LinkTarget>, FormatError> {
if is_v1_group(object_header) {
let entries = resolve_group_entries(file_data, object_header, offset_size, length_size)?;
if let Some(e) = entries
.iter()
.find(|e| e.name == name && e.object_header_address != u64::MAX)
{
return Ok(Some(LinkTarget::Hard {
object_header_address: e.object_header_address,
}));
}
return find_v1_symbolic_link(file_data, object_header, name, offset_size, length_size);
}
if !is_v2_group(object_header) {
return Err(FormatError::PathNotFound(String::from(
"object header is not a group",
)));
}
Ok(
first_link_named(file_data, object_header, name, offset_size, length_size)?
.map(|link| link.link_target)
.filter(|t| {
!matches!(
t,
LinkTarget::Hard {
object_header_address: u64::MAX
}
)
}),
)
}
/// The object header address of the child called `name` of the group at
/// `group_address`: the address [`resolve_group_children`] lists under that
/// name, or `PathNotFound` if it lists none.
///
/// A dense group's child is found through its link name index (see
/// [`links_named`]) and only the named link is read and, if it is a soft
/// link, followed — not every link in the group. A v1 group is listed.
pub fn resolve_child(
file_data: &[u8],
superblock: &Superblock,
group_address: u64,
name: &str,
) -> Result<u64, FormatError> {
let os = superblock.offset_size;
let ls = superblock.length_size;
let not_found = || FormatError::PathNotFound(String::from(name));
let header = ObjectHeader::parse(file_data, to_usize(group_address)?, os, ls)?;
if !is_v2_group(&header) || is_v1_group(&header) {
return resolve_group_children(file_data, superblock, group_address)?
.into_iter()
.find(|e| e.name == name)
.map(|e| e.object_header_address)
.ok_or_else(not_found);
}
// The first link of that name only, as the listing (see
// `first_link_named`).
match first_link_named(file_data, &header, name, os, ls)?.map(|l| l.link_target) {
Some(LinkTarget::Hard {
object_header_address,
}) => Ok(object_header_address),
Some(LinkTarget::Soft { target_path }) => {
match resolve_path_from(file_data, superblock, group_address, &target_path) {
// Left out of the listing: dangling, cyclic, or in another file.
Err(
FormatError::PathNotFound(_)
| FormatError::NestingDepthExceeded
| FormatError::ExternalLinkUnsupported { .. },
) => Err(not_found()),
other => other,
}
}
Some(LinkTarget::External { .. }) | None => Err(not_found()),
}
}
/// Find and parse the Link Info message from an object header.
fn find_link_info(
object_header: &ObjectHeader,
@@ -298,7 +465,7 @@ pub fn resolve_group_children(
) -> Result<Vec<GroupEntry>, FormatError> {
let os = superblock.offset_size;
let ls = superblock.length_size;
let header = ObjectHeader::parse(file_data, group_address as usize, os, ls)?;
let header = ObjectHeader::parse(file_data, to_usize(group_address)?, os, ls)?;
let mut entries = Vec::new();
let mut soft = Vec::new();
@@ -315,16 +482,23 @@ pub fn resolve_group_children(
}
entries.extend(all.into_iter().filter(|e| !group_v1::is_v1_soft_link(e)));
} else if is_v2_group(&header) {
let mut visit = |link: LinkMessage| match link.link_target {
LinkTarget::Hard {
object_header_address,
} => entries.push(GroupEntry {
name: link.name,
object_header_address,
cache_type: 0,
}),
LinkTarget::Soft { target_path } => soft.push((link.name, target_path)),
LinkTarget::External { .. } => {}
// Only the first link of each name counts (see `first_link_named`).
let mut seen = BTreeSet::new();
let mut visit = |link: LinkMessage| {
if !seen.insert(link.name.clone()) {
return;
}
match link.link_target {
LinkTarget::Hard {
object_header_address,
} => entries.push(GroupEntry {
name: link.name,
object_header_address,
cache_type: 0,
}),
LinkTarget::Soft { target_path } => soft.push((link.name, target_path)),
LinkTarget::External { .. } => {}
}
};
let link_info = find_link_info(&header, os)?;
if let Some(fh_addr) = link_info.fractal_heap_address {
@@ -383,24 +557,21 @@ fn resolve_path_following_links(
let ls = superblock.length_size;
let mut current_addr = start;
let mut current_header = ObjectHeader::parse(file_data, start as usize, os, ls)?;
let mut current_header = ObjectHeader::parse(file_data, to_usize(start)?, os, ls)?;
for (i, component) in components.iter().enumerate() {
let entries = resolve_group_entries(file_data, &current_header, os, ls)?;
let found = entries
.iter()
.find(|e| e.name == *component && e.object_header_address != u64::MAX);
match found {
Some(entry) => {
match lookup_link(file_data, &current_header, component, os, ls)? {
Some(LinkTarget::Hard {
object_header_address,
}) => {
if i == components.len() - 1 {
return Ok(entry.object_header_address);
return Ok(object_header_address);
}
current_addr = entry.object_header_address;
current_header = ObjectHeader::parse(file_data, current_addr as usize, os, ls)?;
current_addr = object_header_address;
current_header = ObjectHeader::parse(file_data, to_usize(current_addr)?, os, ls)?;
}
None => {
return match find_symbolic_link(file_data, &current_header, component, os, ls)? {
found => {
return match found {
Some(LinkTarget::Soft { target_path }) => {
if depth >= MAX_SOFT_LINK_DEPTH {
return Err(FormatError::NestingDepthExceeded);
crates/clawhdf5-format/src/lane_partition.rs
+2 -1
View File
@@ -112,7 +112,8 @@ pub fn partition(
for idx in 0..num_items {
let h = fxhash_combine(seed, idx as u64);
let lane = (h % num_lanes as u64) as usize;
// Below `num_lanes`, so it fits.
let lane = crate::addr::saturating_usize(h % num_lanes as u64);
lanes[lane].push(idx);
}
crates/clawhdf5-format/src/lib.rs
+5
View File
@@ -57,6 +57,7 @@
#[cfg(not(feature = "std"))]
extern crate alloc;
pub mod addr;
pub mod attribute;
pub mod attribute_info;
pub mod btree_v1;
@@ -94,6 +95,8 @@ mod filters_bzip2;
#[cfg(feature = "lzf")]
pub mod filters_lzf;
mod filters_szip;
#[cfg(feature = "zfp")]
pub mod filters_zfp;
pub mod fixed_array;
pub mod float16;
pub mod fractal_heap;
@@ -107,6 +110,7 @@ pub mod lane_partition;
pub mod link_info;
pub mod link_message;
pub mod local_heap;
pub mod lookup_stats;
pub mod message_type;
pub mod metadata_cache;
pub mod metadata_index;
@@ -120,6 +124,7 @@ pub mod property_list;
pub mod selection;
pub mod shared_message;
pub mod signature;
pub mod storage;
pub mod superblock;
pub mod superblock_ext;
pub mod symbol_table;
crates/clawhdf5-format/src/link_message.rs
+2 -1
View File
@@ -3,6 +3,7 @@
#[cfg(not(feature = "std"))]
use alloc::{string::String, vec::Vec};
use crate::addr::to_usize;
use crate::datatype::CharacterSet;
use crate::error::FormatError;
@@ -247,7 +248,7 @@ impl LinkMessage {
};
// Link name length
let name_len = read_offset(data, pos, name_size_field_width)? as usize;
let name_len = to_usize(read_offset(data, pos, name_size_field_width)?)?;
pos += name_size_field_width as usize;
// Link name
crates/clawhdf5-format/src/local_heap.rs
+148 -39
View File
@@ -3,7 +3,9 @@
#[cfg(not(feature = "std"))]
use alloc::string::String;
use crate::addr::to_usize;
use crate::error::FormatError;
use crate::storage::{Storage, len_usize, read_exact_at};
/// Parsed HDF5 Local Heap header.
#[derive(Debug, Clone)]
@@ -16,21 +18,6 @@ pub struct LocalHeap {
pub data_segment_address: u64,
}
/// Checks that `[offset, offset + needed)` fits within `data`, guarding the
/// addition against `usize` overflow from a crafted near-`usize::MAX` offset.
fn ensure_len(data: &[u8], offset: usize, needed: usize) -> Result<(), FormatError> {
if offset
.checked_add(needed)
.is_none_or(|end| end > data.len())
{
return Err(FormatError::UnexpectedEof {
expected: offset.saturating_add(needed),
available: data.len(),
});
}
Ok(())
}
fn read_offset(data: &[u8], pos: usize, size: u8) -> Result<u64, FormatError> {
let s = size as usize;
if pos.checked_add(s).is_none_or(|end| end > data.len()) {
@@ -50,6 +37,10 @@ fn read_offset(data: &[u8], pos: usize, size: u8) -> Result<u64, FormatError> {
})
}
/// First read of a name on a backend without the file in memory: most link
/// names are shorter than this.
const NAME_READ_START: usize = 64;
impl LocalHeap {
/// Parse a local heap header at the given offset in the file data.
pub fn parse(
@@ -57,12 +48,24 @@ impl LocalHeap {
offset: usize,
offset_size: u8,
length_size: u8,
) -> Result<LocalHeap, FormatError> {
Self::parse_in(file_data, offset as u64, offset_size, length_size)
}
/// [`Self::parse`] over any [`Storage`]: one read of the header.
pub fn parse_in<S: Storage + ?Sized>(
file: &S,
offset: u64,
offset_size: u8,
length_size: u8,
) -> Result<LocalHeap, FormatError> {
// signature(4) + version(1) + reserved(3) = 8, then length_size*2 + offset_size
let ls = length_size as usize;
let os = offset_size as usize;
let total = 8 + ls * 2 + os;
ensure_len(file_data, offset, total)?;
let header = read_exact_at(file, offset, total)?;
let file_data: &[u8] = &header;
let offset = 0usize;
if &file_data[offset..offset + 4] != b"HEAP" {
return Err(FormatError::InvalidLocalHeapSignature);
@@ -99,6 +102,16 @@ impl LocalHeap {
/// The end of the list is `H5HL_FREE_NULL` (1); an all-ones value (the
/// undefined address) is accepted as "no free list" too.
pub fn validate_free_list(&self, file_data: &[u8], length_size: u8) -> Result<(), FormatError> {
self.validate_free_list_in(file_data, length_size)
}
/// [`Self::validate_free_list`] over any [`Storage`]: two small reads
/// per free block.
pub fn validate_free_list_in<S: Storage + ?Sized>(
&self,
file: &S,
length_size: u8,
) -> Result<(), FormatError> {
const FREE_NULL: u64 = 1;
let ls = length_size as usize;
let undefined = if ls >= 8 {
@@ -123,11 +136,12 @@ impl LocalHeap {
.and_then(|a| usize::try_from(a).ok())
.ok_or(FormatError::InvalidLocalHeapFreeList)?;
let block_offset = next;
next = read_offset(file_data, at, length_size)?;
next = read_offset(&read_exact_at(file, at as u64, ls)?, 0, length_size)?;
if next == 0 {
return Err(FormatError::InvalidLocalHeapFreeList);
}
let block_size = read_offset(file_data, at + ls, length_size)?;
let block_size =
read_offset(&read_exact_at(file, (at + ls) as u64, ls)?, 0, length_size)?;
if block_offset
.checked_add(block_size)
.is_none_or(|end| end > size)
@@ -140,43 +154,65 @@ impl LocalHeap {
/// Read a null-terminated string from the heap's data segment at the given byte offset.
pub fn read_string(&self, file_data: &[u8], string_offset: u64) -> Result<String, FormatError> {
let seg_addr = self.data_segment_address as usize;
self.read_string_in(file_data, string_offset)
}
/// [`Self::read_string`] over any [`Storage`]: one read of up to 64
/// bytes for a short name, more (each four times the last) up to the end
/// of the data segment for a longer one.
pub fn read_string_in<S: Storage + ?Sized>(
&self,
file: &S,
string_offset: u64,
) -> Result<String, FormatError> {
let file_len = len_usize(file);
let seg_addr = to_usize(self.data_segment_address)?;
let str_start =
seg_addr
.checked_add(string_offset as usize)
.checked_add(to_usize(string_offset)?)
.ok_or(FormatError::Overflow(
"local heap seg_addr + string_offset overflow".into(),
))?;
let seg_end = seg_addr
.checked_add(self.data_segment_size as usize)
.checked_add(to_usize(self.data_segment_size)?)
.ok_or(FormatError::Overflow(
"local heap seg_addr + data_segment_size overflow".into(),
))?;
if str_start >= file_data.len() || str_start >= seg_end {
if str_start >= file_len || str_start >= seg_end {
return Err(FormatError::UnexpectedEof {
expected: str_start + 1,
available: file_data.len(),
available: file_len,
});
}
// Find null terminator
let search_end = seg_end.min(file_data.len());
let mut end = str_start;
while end < search_end && file_data[end] != 0 {
end += 1;
// Find the null terminator, which lies before the end of the data
// segment (or of the file). In memory that is one borrowed slice;
// otherwise the bytes are read in growing pieces, so a name costs a
// read of about its own length, not of the rest of the segment
// (whose size is an untrusted header field).
let search_end = seg_end.min(file_len);
let total = search_end - str_start;
let mut want = if file.as_contiguous().is_some() {
total
} else {
total.min(NAME_READ_START)
};
loop {
let rest = read_exact_at(file, str_start as u64, want)?;
if let Some(len) = rest.iter().position(|&b| b == 0) {
let s = core::str::from_utf8(&rest[..len])
.map_err(|_| FormatError::InvalidLocalHeapSignature)?;
return Ok(String::from(s));
}
if want == total {
return Err(FormatError::UnexpectedEof {
expected: search_end + 1,
available: search_end,
});
}
want = want.saturating_mul(4).min(total);
}
if end >= search_end {
return Err(FormatError::UnexpectedEof {
expected: end + 1,
available: search_end,
});
}
let s = core::str::from_utf8(&file_data[str_start..end])
.map_err(|_| FormatError::InvalidLocalHeapSignature)?;
Ok(String::from(s))
}
}
@@ -345,4 +381,77 @@ mod tests {
let err = LocalHeap::parse(&file, 0, 8, 8).unwrap_err();
assert_eq!(err, FormatError::InvalidLocalHeapVersion(1));
}
/// Header, free list and strings read identically through a
/// `read_at`-only storage, for every truncation of the file.
#[test]
fn storage_reads_match_slice_reads() {
use crate::storage::CountingStorage;
let plain = build_heap_file(0, 64, &["", "alpha", "beta"], 8, 8);
// A free block of 16 bytes at segment offset 12, ending the list.
let mut free = build_heap_file(0, 64, &["", "alpha", "beta", &"x".repeat(20)], 8, 8);
free[16..24].copy_from_slice(&12u64.to_le_bytes());
free[64 + 12..64 + 20].copy_from_slice(&1u64.to_le_bytes());
free[64 + 20..64 + 28].copy_from_slice(&16u64.to_le_bytes());
let mut bad_free = free.clone();
bad_free[64 + 20..64 + 28].copy_from_slice(&99u64.to_le_bytes());
for full in [plain, free, bad_free] {
for cut in 0..=full.len() {
let f = &full[..cut];
let storage = CountingStorage::new(f.to_vec());
let want = LocalHeap::parse(f, 0, 8, 8);
let got = LocalHeap::parse_in(&storage, 0, 8, 8);
assert_eq!(format!("{got:?}"), format!("{want:?}"));
let Ok(heap) = want else { continue };
assert_eq!(
heap.validate_free_list_in(&storage, 8),
heap.validate_free_list(f, 8)
);
for off in [0u64, 1, 2, 6, 7, 11, 100] {
assert_eq!(heap.read_string_in(&storage, off), heap.read_string(f, off));
}
}
}
}
/// Names of every length around the first read's size, and one with no
/// terminator, read identically through a `read_at`-only storage; a
/// short name in a heap whose header claims a huge data segment costs
/// one small read, not a read of the rest of the file.
#[test]
fn long_names_and_hostile_segment_sizes() {
use crate::storage::CountingStorage;
let names: Vec<String> = [0usize, 1, 63, 64, 65, 255, 256, 257, 1000, 5000]
.iter()
.map(|&n| "n".repeat(n))
.collect();
let refs: Vec<&str> = names.iter().map(String::as_str).collect();
let mut file = build_heap_file(0, 64, &refs, 8, 8);
let heap = LocalHeap::parse(&file, 0, 8, 8).unwrap();
let storage = CountingStorage::new(file.clone());
let mut off = 0u64;
for name in &names {
let got = heap.read_string_in(&storage, off);
assert_eq!(got, heap.read_string(&file, off));
assert_eq!(got.unwrap(), *name);
off += name.len() as u64 + 1;
}
// The last name loses its terminator: both report the same error.
let seg_end = 64 + heap.data_segment_size as usize;
file[seg_end - 1] = b'n';
let storage = CountingStorage::new(file.clone());
let last = off - names[names.len() - 1].len() as u64 - 1;
let want = heap.read_string(&file, last);
assert!(want.is_err());
assert_eq!(heap.read_string_in(&storage, last), want);
// A 64 MiB file whose heap claims a data segment reaching its end.
let mut big = build_heap_file(0, 64, &["short", "names"], 8, 8);
big.resize(64 << 20, 0);
big[8..16].copy_from_slice(&((64u64 << 20) - 64).to_le_bytes());
let heap = LocalHeap::parse(&big, 0, 8, 8).unwrap();
let storage = CountingStorage::new(big.clone());
assert_eq!(heap.read_string_in(&storage, 6).unwrap(), "names");
assert_eq!((storage.reads(), storage.bytes_read()), (1, 64));
}
}
crates/clawhdf5-format/src/lookup_stats.rs
+31
View File
@@ -0,0 +1,31 @@
//! Work counters for tests of lookup cost (feature `lookup-stats`).
//!
//! Counts fractal-heap objects read — each is one link or attribute message
//! decoded out of a dense group or dense attribute storage — so a test can
//! check that finding one name reads a handful of them, not the whole group.
//! Per thread, so tests running in parallel do not see each other's reads.
//! Without the feature the counting compiles to nothing.
#[cfg(feature = "lookup-stats")]
std::thread_local! {
static HEAP_OBJECTS: core::cell::Cell<u64> = const { core::cell::Cell::new(0) };
}
/// Record one heap object read.
#[inline(always)]
pub(crate) fn heap_object_read() {
#[cfg(feature = "lookup-stats")]
HEAP_OBJECTS.with(|c| c.set(c.get() + 1));
}
/// Heap objects read on this thread since the last [`reset`].
#[cfg(feature = "lookup-stats")]
pub fn heap_objects_read() -> u64 {
HEAP_OBJECTS.with(core::cell::Cell::get)
}
/// Zero this thread's counters.
#[cfg(feature = "lookup-stats")]
pub fn reset() {
HEAP_OBJECTS.with(|c| c.set(0));
}
crates/clawhdf5-format/src/object_header.rs
+144 -47
View File
@@ -5,8 +5,10 @@ use alloc::vec::Vec;
use byteorder::{ByteOrder, LittleEndian};
use crate::addr::to_usize;
use crate::error::FormatError;
use crate::message_type::MessageType;
use crate::storage::{Storage, Window, len_usize, read_exact_at};
/// OHDR signature for v2 object headers.
const OHDR_SIGNATURE: [u8; 4] = *b"OHDR";
@@ -118,32 +120,52 @@ impl ObjectHeader {
offset_size: u8,
length_size: u8,
) -> Result<ObjectHeader, FormatError> {
ensure_len(data, offset, 4)?;
if data[offset..offset + 4] == OHDR_SIGNATURE {
Self::parse_v2(data, offset, offset_size, length_size)
Self::parse_in(data, offset as u64, offset_size, length_size)
}
/// [`Self::parse`] over any [`Storage`].
///
/// Reads the prefix (at most [`V2_PREFIX_MAX`] bytes, signature
/// included), then each chunk as one bounded read, continuation chunks
/// included.
pub fn parse_in<S: Storage + ?Sized>(
file: &S,
offset: u64,
offset_size: u8,
length_size: u8,
) -> Result<ObjectHeader, FormatError> {
// The longest prefix of either version, in one read. It holds the
// whole prefix or ends at the end of the file, so its bounds checks
// are the whole-file ones.
let prefix = Window::read(file, offset, V2_PREFIX_MAX)?;
prefix.ensure(0, 4)?;
if prefix.bytes[..4] == OHDR_SIGNATURE {
Self::parse_v2(file, offset, &prefix, offset_size, length_size)
} else {
Self::parse_v1(data, offset, offset_size, length_size)
Self::parse_v1(file, offset, &prefix, offset_size, length_size)
}
}
fn parse_v1(
data: &[u8],
offset: usize,
fn parse_v1<S: Storage + ?Sized>(
file: &S,
offset: u64,
prefix: &Window<'_>,
offset_size: u8,
length_size: u8,
) -> Result<ObjectHeader, FormatError> {
// version(1) + reserved(1) + num_messages(2) + ref_count(4) + header_size(4) = 12
// then pad to 8-byte alignment from start of header
ensure_len(data, offset, 12)?;
prefix.ensure(0, 12)?;
let prefix = &prefix.bytes[..12];
let version = data[offset];
let version = prefix[0];
if version != 1 {
return Err(FormatError::InvalidObjectHeaderVersion(version));
}
let num_messages = LittleEndian::read_u16(&data[offset + 2..offset + 4]) as usize;
let reference_count = LittleEndian::read_u32(&data[offset + 4..offset + 8]);
let header_data_size = LittleEndian::read_u32(&data[offset + 8..offset + 12]) as usize;
let num_messages = LittleEndian::read_u16(&prefix[2..4]) as usize;
let reference_count = LittleEndian::read_u32(&prefix[4..8]);
let header_data_size = LittleEndian::read_u32(&prefix[8..12]) as usize;
// libhdf5 (H5O__prefix_deserialize): a header with messages needs room
// for at least one message header, and one without has an empty chunk.
@@ -161,14 +183,15 @@ impl ObjectHeader {
.checked_add(12 + padding)
.ok_or(FormatError::UnexpectedEof {
expected: usize::MAX,
available: data.len(),
available: len_usize(file),
})?;
ensure_len(data, msg_start, header_data_size)?;
let mut messages = Vec::new();
// parse_v1_chunk reads the chunk, with the bounds check that was here.
// The prefix's count (NIL messages included, capped: it is untrusted)
// sizes the list once instead of growing it message by message.
let mut messages = Vec::with_capacity(num_messages.min(64));
let chunk0_count = Self::parse_v1_chunk(
data,
file,
msg_start,
header_data_size,
offset_size,
@@ -208,9 +231,9 @@ impl ObjectHeader {
/// end of the chunk, or leftover bytes too few for a message header (a
/// "gap", which only version 2 allows).
#[allow(clippy::too_many_arguments)]
fn parse_v1_chunk(
data: &[u8],
offset: usize,
fn parse_v1_chunk<S: Storage + ?Sized>(
file: &S,
offset: u64,
length: usize,
offset_size: u8,
length_size: u8,
@@ -220,9 +243,10 @@ impl ObjectHeader {
if depth_remaining == 0 {
return Err(FormatError::NestingDepthExceeded);
}
ensure_len(data, offset, length)?;
let end = offset + length;
let mut pos = offset;
let chunk = read_exact_at(file, offset, length)?;
let data: &[u8] = &chunk;
let end = length;
let mut pos = 0usize;
let mut count = 0usize;
while pos < end {
@@ -264,11 +288,11 @@ impl ObjectHeader {
// Follow continuations (v1 continuation chunks are just raw
// messages, no signature); check_message has checked the body.
if msg_type == MessageType::ObjectHeaderContinuation {
let cont_offset = read_offset(body, 0, offset_size)? as usize;
let cont_length = read_offset(body, offset_size as usize, length_size)? as usize;
let cont_offset = to_usize(read_offset(body, 0, offset_size)?)?;
let cont_length = to_usize(read_offset(body, offset_size as usize, length_size)?)?;
Self::parse_v1_chunk(
data,
cont_offset,
file,
cont_offset as u64,
cont_length,
offset_size,
length_size,
@@ -281,12 +305,22 @@ impl ObjectHeader {
Ok(count)
}
fn parse_v2(
data: &[u8],
offset: usize,
fn parse_v2<S: Storage + ?Sized>(
file: &S,
offset: u64,
prefix: &Window<'_>,
offset_size: u8,
length_size: u8,
) -> Result<ObjectHeader, FormatError> {
// `ensure_len` checks positions relative to the header against the
// prefix window and reports them as the whole-file check did, with
// absolute positions and the file's length.
let data: &[u8] = &prefix.bytes;
let file_len = len_usize(file);
let base = usize::try_from(offset).unwrap_or(usize::MAX);
let abs = |rel: usize| base.saturating_add(rel);
let ensure_len = |_: &[u8], rel: usize, needed: usize| prefix.ensure(rel, needed);
let offset = 0usize;
// signature(4) + version(1) + flags(1) = 6
ensure_len(data, offset, 6)?;
@@ -339,7 +373,7 @@ impl ObjectHeader {
_ => unreachable!(),
};
ensure_len(data, pos, chunk_size_width as usize)?;
let chunk0_size = read_offset(data, pos, chunk_size_width)? as usize;
let chunk0_size = to_usize(read_offset(data, pos, chunk_size_width)?)?;
pos += chunk_size_width as usize;
// Bit 2: attribute creation order tracked → messages include creation order field
let has_creation_order = flags & 0x04 != 0;
@@ -351,15 +385,20 @@ impl ObjectHeader {
}
let chunk0_msg_start = pos;
let chunk0_msg_end = pos
.checked_add(chunk0_size)
.ok_or(FormatError::UnexpectedEof {
let Some(chunk0_abs_end) = abs(pos).checked_add(chunk0_size) else {
return Err(FormatError::UnexpectedEof {
expected: usize::MAX,
available: data.len(),
})?;
available: file_len,
});
};
let chunk0_msg_end = chunk0_abs_end - base;
// The whole first chunk, prefix to checksum, in one read (its
// bounds check is the one on the checksum's 4 bytes).
let chunk0 = read_exact_at(file, base as u64, chunk0_msg_end.saturating_add(4))?;
let data: &[u8] = &chunk0;
// Validate checksum: from OHDR signature through all messages (before checksum)
ensure_len(data, chunk0_msg_end, 4)?;
#[cfg(feature = "checksum")]
{
let stored = LittleEndian::read_u32(&data[chunk0_msg_end..chunk0_msg_end + 4]);
@@ -394,8 +433,8 @@ impl ObjectHeader {
}
cont_remaining -= 1;
Self::parse_v2_continuation(
data,
cont_offset,
file,
cont_offset as u64,
cont_length,
has_creation_order,
offset_size,
@@ -472,8 +511,8 @@ impl ObjectHeader {
let msg_type = MessageType::from_u16(msg_type_raw);
if msg_type == MessageType::ObjectHeaderContinuation {
// check_message has checked the body holds both fields.
let cont_off = read_offset(body, 0, offset_size)? as usize;
let cont_len = read_offset(body, offset_size as usize, length_size)? as usize;
let cont_off = to_usize(read_offset(body, 0, offset_size)?)?;
let cont_len = to_usize(read_offset(body, offset_size as usize, length_size)?)?;
continuations.push((cont_off, cont_len));
} else if msg_type == MessageType::Nil {
null_count += 1;
@@ -494,9 +533,9 @@ impl ObjectHeader {
}
#[allow(clippy::too_many_arguments)]
fn parse_v2_continuation(
data: &[u8],
offset: usize,
fn parse_v2_continuation<S: Storage + ?Sized>(
file: &S,
offset: u64,
length: usize,
has_creation_order: bool,
offset_size: u8,
@@ -505,7 +544,9 @@ impl ObjectHeader {
continuations: &mut Vec<(usize, usize)>,
) -> Result<(), FormatError> {
// OCHK signature(4) + messages + checksum(4)
ensure_len(data, offset, length)?;
let chunk = read_exact_at(file, offset, length)?;
let data: &[u8] = &chunk;
let offset = 0usize;
if length < 8 {
return Err(FormatError::UnexpectedEof {
expected: 8,
@@ -546,6 +587,10 @@ impl ObjectHeader {
}
}
/// Longest version-2 object header prefix: signature(4) + version(1) +
/// flags(1) + times(16) + attribute phase change(4) + chunk-0 size(8).
const V2_PREFIX_MAX: usize = 34;
/// Size of a version-1 message header: type(2) + size(2) + flags(1) + reserved(3).
const V1_MSG_HEADER_SIZE: usize = 8;
@@ -754,13 +799,13 @@ mod tests {
let mut msg_bytes = Vec::new();
for (mtype, mdata, mflags) in messages {
// v1 message sizes are multiples of 8 (the data is zero-padded).
let padded = mdata.len().div_ceil(8) * 8;
let padded = <[u8]>::len(mdata).div_ceil(8) * 8;
msg_bytes.extend_from_slice(&mtype.to_le_bytes()); // type(2)
msg_bytes.extend_from_slice(&(padded as u16).to_le_bytes()); // size(2)
msg_bytes.push(*mflags); // flags(1)
msg_bytes.extend_from_slice(&[0u8; 3]); // reserved(3)
msg_bytes.extend_from_slice(mdata); // data
msg_bytes.resize(msg_bytes.len() + padded - mdata.len(), 0);
msg_bytes.resize(msg_bytes.len() + padded - <[u8]>::len(mdata), 0);
}
let mut buf = Vec::new();
@@ -1280,4 +1325,56 @@ mod tests {
let err = ObjectHeader::parse(&data, 0, 8, 8).unwrap_err();
assert!(matches!(err, FormatError::UnexpectedEof { .. }));
}
/// Every header, and every truncation of it, parses to the same result
/// (or the same error) through a `read_at`-only storage as from a slice;
/// a header in one chunk takes two reads (prefix, chunk).
#[test]
fn parse_in_matches_slice_parse() {
use crate::storage::CountingStorage;
let mut headers = vec![
build_v1_header(&[], 8, 8),
build_v1_header(&[(0x0001, &[1, 2, 3], 0), (0x0003, &[9; 8], 0)], 8, 8),
build_v2_header(0x00, &[(0x01, &[42], 0)], None),
build_v2_header(0x03, &[(0x01, &[1, 2], 0), (0x03, &[3], 0)], None),
build_v2_header(0x24, &[(0x01, &[1], 0)], Some((1, 2, 3, 4))),
build_v2_header(0x35, &[(0x01, &[1], 0)], Some((5, 6, 7, 8))),
];
// A v2 header with a continuation chunk at 256.
let mut ochk = OCHK_SIGNATURE.to_vec();
ochk.extend_from_slice(&[0x03, 2, 0, 0, 0xDE, 0xAD]);
let sum = crate::checksum::jenkins_lookup3(&ochk);
ochk.extend_from_slice(&sum.to_le_bytes());
let mut cont = 256u64.to_le_bytes().to_vec();
cont.extend_from_slice(&(ochk.len() as u64).to_le_bytes());
let main = build_v2_header(0x00, &[(0x01, &[42], 0), (0x10, &cont, 0)], None);
let mut with_cont = vec![0u8; 256 + ochk.len()];
with_cont[..main.len()].copy_from_slice(&main);
with_cont[256..].copy_from_slice(&ochk);
headers.push(with_cont);
for h in headers {
for at in [0usize, 3] {
for cut in 0..=h.len() {
let mut f = vec![0u8; at];
f.extend_from_slice(&h[..cut]);
if at == 0 && cut == h.len() {
f.resize(f.len() + 64, 0);
}
let want = ObjectHeader::parse(&f, at, 8, 8);
let storage = CountingStorage::new(f.clone());
let got = ObjectHeader::parse_in(&storage, at as u64, 8, 8);
assert_eq!(
format!("{got:?}"),
format!("{want:?}"),
"at {at}, cut {cut}"
);
}
}
}
let one_chunk = build_v2_header(0x00, &[(0x01, &[42], 0)], None);
let storage = CountingStorage::new(one_chunk);
ObjectHeader::parse_in(&storage, 0, 8, 8).unwrap();
assert_eq!(storage.reads(), 2);
}
}
crates/clawhdf5-format/src/parallel_read.rs
+4 -3
View File
@@ -7,6 +7,7 @@
//! The lane assignment is seeded by dataset metadata so repeated reads of
//! the same region produce identical partitions (cache-friendly, reproducible).
use crate::addr::to_usize;
use crate::chunked_read::ChunkInfo;
use crate::error::FormatError;
use crate::filter_pipeline::FilterPipeline;
@@ -210,7 +211,7 @@ pub fn decompress_chunks_lane_partitioned(
for &index in &indices {
let chunk_info = &chunks[index];
let c_addr = chunk_info.address as usize;
let c_addr = to_usize(chunk_info.address)?;
let size = chunk_info.chunk_size as usize;
if c_addr
@@ -288,7 +289,7 @@ pub fn decompress_chunks_parallel(
.par_iter()
.enumerate()
.map(|(index, chunk_info)| {
let c_addr = chunk_info.address as usize;
let c_addr = to_usize(chunk_info.address)?;
let size = chunk_info.chunk_size as usize;
if c_addr
.checked_add(size)
@@ -332,7 +333,7 @@ pub fn decompress_chunks_sequential(
) -> Result<Vec<Vec<u8>>, FormatError> {
let mut result = Vec::with_capacity(chunks.len());
for chunk_info in chunks {
let c_addr = chunk_info.address as usize;
let c_addr = to_usize(chunk_info.address)?;
let size = chunk_info.chunk_size as usize;
if c_addr
.checked_add(size)
crates/clawhdf5-format/src/partial_read.rs
+12 -3
View File
@@ -203,7 +203,12 @@ fn copy_overlap(
};
let (src_strides, out_strides) = (strides(src_shape), strides(box_extent));
let last = rank - 1;
let run = ((hi[last] - lo[last]) as usize) * elem_size;
// Byte offsets into the in-memory buffers; one that does not fit `usize`
// (a 32-bit target) is out of both buffers, like one past their ends.
let bytes = |elements: u64| usize::try_from(elements).ok()?.checked_mul(elem_size);
let Some(run) = bytes(hi[last] - lo[last]) else {
return;
};
let mut idx = lo.clone();
loop {
@@ -213,8 +218,12 @@ fn copy_overlap(
let out_at: u64 = (0..rank)
.map(|d| (idx[d] - box_start[d]) * out_strides[d])
.sum();
let (s, o) = (src_at as usize * elem_size, out_at as usize * elem_size);
if let (Some(from), Some(to)) = (src.get(s..s + run), out.get_mut(o..o + run)) {
if let (Some(s), Some(o)) = (bytes(src_at), bytes(out_at))
&& let (Some(from), Some(to)) = (
src.get(s..s.saturating_add(run)),
out.get_mut(o..o.saturating_add(run)),
)
{
to.copy_from_slice(from);
}
// Advance over every dimension but the last.
crates/clawhdf5-format/src/selection.rs
+5 -4
View File
@@ -19,6 +19,7 @@ use alloc::{vec, vec::Vec};
use core::ops::Range;
use crate::addr::to_usize;
use crate::error::FormatError;
/// A selection describing which elements of a dataset to access.
@@ -562,7 +563,7 @@ fn decode_hyperslab(r: &mut SelReader, version: u64) -> Result<SerializedSelecti
if !matches!(enc_size, 2 | 4 | 8) {
return Err(sel_err("unsupported hyperslab coordinate encoding size"));
}
let rank = r.uint(4)? as usize;
let rank = to_usize(r.uint(4)?)?;
// HDF5 caps dataspace rank at 32 (H5S_MAX_RANK). Reject anything else so a
// corrupt rank can't drive a huge allocation or read loop.
if rank == 0 || rank > 32 {
@@ -625,11 +626,11 @@ fn decode_hyperslab(r: &mut SelReader, version: u64) -> Result<SerializedSelecti
return Err(FormatError::UnexpectedEof {
expected: r
.pos
.saturating_add(nblocks.saturating_mul(per_block) as usize),
.saturating_add(to_usize(nblocks.saturating_mul(per_block))?),
available: r.data.len(),
});
}
let n = nblocks as usize * rank;
let n = to_usize(nblocks)? * rank;
let (mut starts, mut ends) = (Vec::with_capacity(n), Vec::with_capacity(n));
for _ in 0..nblocks {
for _ in 0..rank {
@@ -662,7 +663,7 @@ fn blocks_union_coords(
.filter(|&t| t <= MAX_EXPANDED_POINTS)
.ok_or_else(|| sel_err("irregular hyperslab selection is too large to expand"))?;
}
let mut out = Vec::with_capacity(total as usize);
let mut out = Vec::with_capacity(to_usize(total)?);
for (s, e) in starts.chunks_exact(rank).zip(ends.chunks_exact(rank)) {
let mut cur = s.to_vec();
'block: loop {
crates/clawhdf5-format/src/shared_message.rs
+227 -29
View File
@@ -28,6 +28,7 @@ use crate::error::FormatError;
use crate::fractal_heap::FractalHeapHeader;
use crate::message_type::MessageType;
use crate::object_header::ObjectHeader;
use crate::storage::{Storage, Window, read_exact_at, require_contiguous};
/// Fractal heap ID length for SOHM entries (fixed at 8 bytes).
const FHEAP_ID_LEN: usize = 8;
@@ -253,17 +254,31 @@ pub fn parse_sohm_table(
nindexes: u8,
offset_size: u8,
) -> Result<SohmTable, FormatError> {
ensure_len(file_data, table_addr, 4)?;
if &file_data[table_addr..table_addr + 4] != b"SMTB" {
parse_sohm_table_in(file_data, table_addr as u64, nindexes, offset_size)
}
/// [`parse_sohm_table`] over any [`Storage`]: one read of the signature,
/// one of every index entry.
pub fn parse_sohm_table_in<S: Storage + ?Sized>(
file: &S,
table_addr: u64,
nindexes: u8,
offset_size: u8,
) -> Result<SohmTable, FormatError> {
let sig = read_exact_at(file, table_addr, 4)?;
if *sig != *b"SMTB" {
return Err(FormatError::InvalidSohmTableSignature);
}
let mut pos = table_addr + 4;
let os = offset_size as usize;
let entry_size = 1 + 1 + 2 + 4 + 2 + 2 + 2 + os + os; // 14 + 2*offset_size
// Positions below are relative to the table.
let w = Window::read(file, table_addr, 4 + nindexes as usize * entry_size)?;
let file_data: &[u8] = &w.bytes;
let mut pos = 4;
let mut indexes = Vec::with_capacity(nindexes as usize);
for _ in 0..nindexes {
ensure_len(file_data, pos, entry_size)?;
w.ensure(pos, entry_size)?;
let version = file_data[pos];
if version != 0 {
return Err(FormatError::InvalidSohmTableVersion(version));
@@ -369,16 +384,29 @@ pub fn parse_sohm_list(
num_messages: u16,
offset_size: u8,
) -> Result<Vec<SohmEntry>, FormatError> {
ensure_len(file_data, list_addr, 4)?;
if &file_data[list_addr..list_addr + 4] != b"SMLI" {
parse_sohm_list_in(file_data, list_addr as u64, num_messages, offset_size)
}
/// [`parse_sohm_list`] over any [`Storage`]: one read of the signature, one
/// of every entry.
pub fn parse_sohm_list_in<S: Storage + ?Sized>(
file: &S,
list_addr: u64,
num_messages: u16,
offset_size: u8,
) -> Result<Vec<SohmEntry>, FormatError> {
let sig = read_exact_at(file, list_addr, 4)?;
if *sig != *b"SMLI" {
return Err(FormatError::InvalidSohmListSignature);
}
let entry_sz = sohm_entry_size(offset_size);
let mut pos = list_addr + 4;
// Positions below are relative to the list.
let w = Window::read(file, list_addr, 4 + num_messages as usize * entry_sz)?;
let mut pos = 4;
let mut entries = Vec::with_capacity(num_messages as usize);
for _ in 0..num_messages {
ensure_len(file_data, pos, entry_sz)?;
let entry = parse_sohm_entry(&file_data[pos..], offset_size)?;
w.ensure(pos, entry_sz)?;
let entry = parse_sohm_entry(&w.bytes[pos..], offset_size)?;
entries.push(entry);
pos += entry_sz;
}
@@ -392,6 +420,20 @@ pub fn parse_sohm_btree_entries(
offset_size: u8,
length_size: u8,
) -> Result<Vec<SohmEntry>, FormatError> {
parse_sohm_btree_entries_in(file_data, btree_addr as u64, offset_size, length_size)
}
/// [`parse_sohm_btree_entries`] over any [`Storage`]. The v2 B-tree is not
/// read over [`Storage`] yet, so this needs the whole file in memory
/// ([`FormatError::ContiguousStorageRequired`] otherwise).
pub fn parse_sohm_btree_entries_in<S: Storage + ?Sized>(
file: &S,
btree_addr: u64,
offset_size: u8,
length_size: u8,
) -> Result<Vec<SohmEntry>, FormatError> {
let file_data = require_contiguous(file, "a shared-message B-tree index")?;
let btree_addr = usize::try_from(btree_addr).unwrap_or(usize::MAX);
let header = BTreeV2Header::parse(file_data, btree_addr, offset_size, length_size)?;
let records = collect_btree_v2_records(file_data, &header, offset_size, length_size)?;
let mut entries = Vec::with_capacity(records.len());
@@ -413,15 +455,24 @@ pub fn load_sohm_table(
offset_size: u8,
length_size: u8,
) -> Result<Option<SohmTable>, FormatError> {
let sig = crate::signature::find_signature(file_data)?;
let sb = crate::superblock::Superblock::parse(file_data, sig)?;
load_sohm_table_in(file_data, offset_size, length_size)
}
/// [`load_sohm_table`] over any [`Storage`].
pub fn load_sohm_table_in<S: Storage + ?Sized>(
file_data: &S,
offset_size: u8,
length_size: u8,
) -> Result<Option<SohmTable>, FormatError> {
let sig = crate::signature::find_signature_in(file_data)?;
let sb = crate::superblock::Superblock::parse_in(file_data, sig)?;
let Some(ext_addr) = sb
.superblock_extension_address
.filter(|&a| !is_undefined(a, offset_size))
else {
return Ok(None);
};
let ext = ObjectHeader::parse(file_data, ext_addr as usize, offset_size, length_size)?;
let ext = ObjectHeader::parse_in(file_data, ext_addr, offset_size, length_size)?;
let Some(msg) = ext
.messages
.iter()
@@ -430,9 +481,9 @@ pub fn load_sohm_table(
return Ok(None);
};
let table_msg = parse_sohm_table_message(&msg.data, offset_size)?;
parse_sohm_table(
parse_sohm_table_in(
file_data,
table_msg.table_address as usize,
table_msg.table_address,
table_msg.nindexes,
offset_size,
)
@@ -446,17 +497,27 @@ pub fn message_data_with_sohm<'a>(
msg: &'a crate::object_header::HeaderMessage,
offset_size: u8,
length_size: u8,
) -> Result<Cow<'a, [u8]>, FormatError> {
message_data_with_sohm_in(file_data, msg, offset_size, length_size)
}
/// [`message_data_with_sohm`] over any [`Storage`].
pub fn message_data_with_sohm_in<'a, S: Storage + ?Sized>(
file_data: &S,
msg: &'a crate::object_header::HeaderMessage,
offset_size: u8,
length_size: u8,
) -> Result<Cow<'a, [u8]>, FormatError> {
if !is_shared(msg.flags) {
return Ok(Cow::Borrowed(&msg.data));
}
let shared_ref = parse_shared_ref_sized(&msg.data, offset_size, length_size)?;
let table = if shared_ref.heap_id.is_some() {
load_sohm_table(file_data, offset_size, length_size)?
load_sohm_table_in(file_data, offset_size, length_size)?
} else {
None
};
resolve_shared_message_with_sohm(
resolve_shared_message_with_sohm_in(
file_data,
&shared_ref,
msg.msg_type,
@@ -495,6 +556,25 @@ pub fn resolve_sohm_message(
target_msg_type: MessageType,
offset_size: u8,
length_size: u8,
) -> Result<Vec<u8>, FormatError> {
resolve_sohm_message_in(
&file_data,
heap_id,
sohm_table,
target_msg_type,
offset_size,
length_size,
)
}
/// [`resolve_sohm_message`] over any [`Storage`].
pub fn resolve_sohm_message_in<S: Storage + ?Sized>(
file_data: &S,
heap_id: &[u8; FHEAP_ID_LEN],
sohm_table: &SohmTable,
target_msg_type: MessageType,
offset_size: u8,
length_size: u8,
) -> Result<Vec<u8>, FormatError> {
let index = find_index_for_msg_type(sohm_table, target_msg_type)
.ok_or(FormatError::InvalidSharedMessageVersion(2))?;
@@ -503,13 +583,9 @@ pub fn resolve_sohm_message(
return Err(FormatError::InvalidSharedMessageVersion(2));
}
let fh_header = FractalHeapHeader::parse(
file_data,
index.heap_addr as usize,
offset_size,
length_size,
)?;
fh_header.read_managed_object(file_data, heap_id, offset_size)
let fh_header =
FractalHeapHeader::parse_in(file_data, index.heap_addr, offset_size, length_size)?;
fh_header.read_managed_object_in(file_data, heap_id, offset_size)
}
/// The payload of an object-header message, following the indirection if the
@@ -526,12 +602,22 @@ pub fn message_data<'a>(
msg: &'a crate::object_header::HeaderMessage,
offset_size: u8,
length_size: u8,
) -> Result<Cow<'a, [u8]>, FormatError> {
message_data_in(file_data, msg, offset_size, length_size)
}
/// [`message_data`] over any [`Storage`].
pub fn message_data_in<'a, S: Storage + ?Sized>(
file_data: &S,
msg: &'a crate::object_header::HeaderMessage,
offset_size: u8,
length_size: u8,
) -> Result<Cow<'a, [u8]>, FormatError> {
if !is_shared(msg.flags) {
return Ok(Cow::Borrowed(&msg.data));
}
let shared_ref = parse_shared_ref_sized(&msg.data, offset_size, length_size)?;
resolve_shared_message(
resolve_shared_message_in(
file_data,
&shared_ref,
msg.msg_type,
@@ -553,13 +639,30 @@ pub fn resolve_shared_message(
target_msg_type: MessageType,
offset_size: u8,
length_size: u8,
) -> Result<Vec<u8>, FormatError> {
resolve_shared_message_in(
&file_data,
shared_ref,
target_msg_type,
offset_size,
length_size,
)
}
/// [`resolve_shared_message`] over any [`Storage`].
pub fn resolve_shared_message_in<S: Storage + ?Sized>(
file_data: &S,
shared_ref: &SharedMessageRef,
target_msg_type: MessageType,
offset_size: u8,
length_size: u8,
) -> Result<Vec<u8>, FormatError> {
let table = if shared_ref.heap_id.is_some() {
load_sohm_table(file_data, offset_size, length_size)?
load_sohm_table_in(file_data, offset_size, length_size)?
} else {
None
};
resolve_shared_message_with_sohm(
resolve_shared_message_with_sohm_in(
file_data,
shared_ref,
target_msg_type,
@@ -577,6 +680,25 @@ pub fn resolve_shared_message_with_sohm(
offset_size: u8,
length_size: u8,
sohm_table: Option<&SohmTable>,
) -> Result<Vec<u8>, FormatError> {
resolve_shared_message_with_sohm_in(
&file_data,
shared_ref,
target_msg_type,
offset_size,
length_size,
sohm_table,
)
}
/// [`resolve_shared_message_with_sohm`] over any [`Storage`].
pub fn resolve_shared_message_with_sohm_in<S: Storage + ?Sized>(
file_data: &S,
shared_ref: &SharedMessageRef,
target_msg_type: MessageType,
offset_size: u8,
length_size: u8,
sohm_table: Option<&SohmTable>,
) -> Result<Vec<u8>, FormatError> {
// Dispatch on what the reference carries rather than on `ref_type`: v1/v2
// references are always an object-header address whatever their type
@@ -586,8 +708,7 @@ pub fn resolve_shared_message_with_sohm(
shared_ref.heap_id.as_ref(),
) {
(Some(addr), _) => {
let target_header =
ObjectHeader::parse(file_data, addr as usize, offset_size, length_size)?;
let target_header = ObjectHeader::parse_in(file_data, addr, offset_size, length_size)?;
for msg in &target_header.messages {
if msg.msg_type == target_msg_type && !is_shared(msg.flags) {
return Ok(msg.data.clone());
@@ -614,7 +735,7 @@ pub fn resolve_shared_message_with_sohm(
}
(None, Some(heap_id)) => {
let table = sohm_table.ok_or(FormatError::InvalidSharedMessageVersion(2))?;
resolve_sohm_message(
resolve_sohm_message_in(
file_data,
heap_id,
table,
@@ -1052,4 +1173,81 @@ mod tests {
// With 2-byte offsets: OH=2+2=4, heap=12, entry=1+4+12=17
assert_eq!(sohm_entry_size(2), 17);
}
/// SOHM tables and lists parse identically through a read_at-only
/// CountingStorage: at two offsets, with 4- and 8-byte offsets, cut at
/// every length and with a bad signature.
#[test]
fn storage_reads_match_slice_reads() {
use crate::storage::CountingStorage;
let idx = |t: u8, n: u16| SohmIndex {
index_type: t,
mesg_types: 0x0008,
min_mesg_size: 50,
list_max: 50,
btree_min: 40,
num_messages: n,
index_addr: 0x3000,
heap_addr: 0x4000,
};
let heap_entry = |h: u32| SohmEntry {
location: 0,
hash: h,
heap_id: Some([1, 2, 3, 4, 5, 6, 7, h as u8]),
ref_count: Some(h),
mesg_index: None,
oh_addr: None,
};
let oh_entry = SohmEntry {
location: 1,
hash: 9,
heap_id: None,
ref_count: None,
mesg_index: Some(3),
oh_addr: Some(0x7000),
};
let mut compared = 0;
for os in [4u8, 8] {
let smtb = build_smtb(&[idx(0, 2), idx(1, 7)], os);
let smli = build_smli(&[heap_entry(1), oh_entry.clone(), heap_entry(2)], os);
for (body, n) in [(smtb, 2u16), (smli, 3)] {
let is_table = &body[..4] == b"SMTB";
for at in [0usize, 0x40] {
let mut full = vec![0u8; at];
full.extend_from_slice(&body);
let mut files = Vec::new();
for cut in at..=full.len() {
files.push(full[..cut].to_vec());
}
let mut bad = full.clone();
bad[at] = b'X';
files.push(bad);
for f in files {
let st = CountingStorage::new(f.clone());
let (want, got) = if is_table {
(
format!("{:?}", parse_sohm_table(&f, at, n as u8, os)),
format!("{:?}", parse_sohm_table_in(&st, at as u64, n as u8, os)),
)
} else {
(
format!("{:?}", parse_sohm_list(&f, at, n, os)),
format!("{:?}", parse_sohm_list_in(&st, at as u64, n, os)),
)
};
assert_eq!(got, want, "{} bytes", f.len());
assert!(st.reads() <= 2);
compared += 1;
}
}
}
}
assert!(compared > 200);
// The B-tree index is not read over Storage yet: a clean error.
let st = CountingStorage::new(vec![0u8; 64]);
assert_eq!(
parse_sohm_btree_entries_in(&st, 0, 8, 8).unwrap_err(),
FormatError::ContiguousStorageRequired("a shared-message B-tree index")
);
}
}
crates/clawhdf5-format/src/signature.rs
+37
View File
@@ -1,6 +1,7 @@
//! HDF5 file signature (magic bytes) detection.
use crate::error::FormatError;
use crate::storage::{Storage, read_exact_at};
/// The 8-byte HDF5 magic signature.
pub const HDF5_SIGNATURE: [u8; 8] = [0x89, b'H', b'D', b'F', b'\r', b'\n', 0x1A, b'\n'];
@@ -39,6 +40,20 @@ pub fn find_signature(data: &[u8]) -> Result<usize, FormatError> {
Err(FormatError::SignatureNotFound)
}
/// [`find_signature`] over any [`Storage`]: one 8-byte read per candidate
/// offset.
pub fn find_signature_in<S: Storage + ?Sized>(file: &S) -> Result<u64, FormatError> {
let len = file.len();
let mut offset = 0u64;
while offset.checked_add(8).is_some_and(|end| end <= len) {
if *read_exact_at(file, offset, 8)? == HDF5_SIGNATURE {
return Ok(offset);
}
offset = if offset == 0 { 512 } else { offset * 2 };
}
Err(FormatError::SignatureNotFound)
}
/// Split a file into its user block and its HDF5 bytes.
///
/// Returns `(user_block, hdf5)`: `user_block` is everything before the
@@ -132,4 +147,26 @@ mod tests {
data[512..520].copy_from_slice(&HDF5_SIGNATURE);
assert_eq!(find_signature(&data), Ok(0));
}
#[test]
fn find_signature_in_matches_slice_search() {
use crate::storage::CountingStorage;
for (len, at) in [
(0, None),
(7, None),
(8, Some(0)),
(600, Some(512)),
(5000, Some(4096)),
(3000, Some(2048)),
(3000, None),
] {
let mut data = vec![0u8; len];
if let Some(at) = at {
data[at..at + 8].copy_from_slice(&HDF5_SIGNATURE);
}
let want = find_signature(&data).map(|o| o as u64);
let got = find_signature_in(&CountingStorage::new(data));
assert_eq!(got, want, "{len} {at:?}");
}
}
}
crates/clawhdf5-format/src/storage.rs
+466
View File
@@ -0,0 +1,466 @@
//! Where the parsers read the file from: the [`Storage`] trait.
//!
//! Every parser used to take the whole file as one `&[u8]`. [`Storage`] is
//! the abstraction that replaces it (see `docs/design/range-reads.md`,
//! option (a)): a parser asks for the bytes it needs, `[offset, offset +
//! len)`, with 64-bit offsets, and gets them back as a [`Cow`] — borrowed
//! when the backend holds the file in memory (a `Vec`, an mmap), owned when
//! it had to fetch them (a range request, a block cache).
//!
//! `impl Storage for [u8]` serves the in-memory case with no copy, and
//! [`Storage::as_contiguous`] lets a hot loop borrow the whole file at once
//! when the backend has it. Modules are converted one at a time: a converted
//! parser has an `*_in<S: Storage + ?Sized>(file: &S, ..)` core and keeps
//! its old `&[u8]` signature as a thin wrapper, so callers do not change.
//!
//! The cores are generic rather than taking `&dyn Storage` so that the
//! wrappers monomorphise for `[u8]`: the bounds check of each structure read
//! inlines to what the slice code did, with no indirect call and no copy,
//! which keeps local files as fast as before the migration. A `&dyn Storage`
//! still works (`S = dyn Storage`), and a remote backend pays one indirect
//! call per structure read.
//!
//! The trait is synchronous and `no_std`: parsing is CPU work, and a remote
//! backend bridges to its own I/O.
#[cfg(not(feature = "std"))]
use alloc::{borrow::Cow, boxed::Box, vec::Vec};
#[cfg(feature = "std")]
use std::{borrow::Cow, boxed::Box, vec::Vec};
use core::ops::Range;
use crate::error::FormatError;
/// A random-access source of file bytes.
///
/// Offsets are relative to the start of the HDF5 data (the superblock), like
/// every address in the file.
pub trait Storage {
/// Bytes `[offset, offset + len)`.
///
/// The result is shorter than `len` only when the range runs past the
/// end of the storage (and empty when `offset` is at or past the end);
/// a backend that cannot serve a range returns an error instead of a
/// short read.
fn read_at(&self, offset: u64, len: usize) -> Result<Cow<'_, [u8]>, FormatError>;
/// Current length of the storage in bytes.
fn len(&self) -> u64;
/// Whether the storage holds no bytes.
fn is_empty(&self) -> bool {
self.len() == 0
}
/// Several reads at once, in the order given. Backends that talk to a
/// remote store coalesce and parallelise these; the default reads them
/// one by one with [`Storage::read_at`].
fn read_ranges(&self, ranges: &[Range<u64>]) -> Result<Vec<Cow<'_, [u8]>>, FormatError> {
ranges
.iter()
.map(|r| {
let len = usize::try_from(r.end.saturating_sub(r.start)).map_err(|_| {
FormatError::Overflow("read range longer than the address space".into())
})?;
self.read_at(r.start, len)
})
.collect()
}
/// The whole storage as one slice, when the backend has it in memory
/// (a `Vec`, an mmap). Hot loops use this to keep their zero-copy path;
/// `None` means every byte has to go through [`Storage::read_at`].
fn as_contiguous(&self) -> Option<&[u8]> {
None
}
}
impl Storage for [u8] {
#[inline]
fn read_at(&self, offset: u64, len: usize) -> Result<Cow<'_, [u8]>, FormatError> {
let n = self.len();
let start = usize::try_from(offset).map_or(n, |o| o.min(n));
let end = start.saturating_add(len).min(n);
Ok(Cow::Borrowed(&self[start..end]))
}
#[inline]
fn len(&self) -> u64 {
<[u8]>::len(self) as u64
}
#[inline]
fn as_contiguous(&self) -> Option<&[u8]> {
Some(self)
}
}
impl Storage for Vec<u8> {
#[inline]
fn read_at(&self, offset: u64, len: usize) -> Result<Cow<'_, [u8]>, FormatError> {
self.as_slice().read_at(offset, len)
}
#[inline]
fn len(&self) -> u64 {
Vec::len(self) as u64
}
#[inline]
fn as_contiguous(&self) -> Option<&[u8]> {
Some(self.as_slice())
}
}
impl<T: Storage + ?Sized> Storage for &T {
#[inline]
fn read_at(&self, offset: u64, len: usize) -> Result<Cow<'_, [u8]>, FormatError> {
(**self).read_at(offset, len)
}
#[inline]
fn len(&self) -> u64 {
(**self).len()
}
#[inline]
fn read_ranges(&self, ranges: &[Range<u64>]) -> Result<Vec<Cow<'_, [u8]>>, FormatError> {
(**self).read_ranges(ranges)
}
#[inline]
fn as_contiguous(&self) -> Option<&[u8]> {
(**self).as_contiguous()
}
}
impl<T: Storage + ?Sized> Storage for Box<T> {
#[inline]
fn read_at(&self, offset: u64, len: usize) -> Result<Cow<'_, [u8]>, FormatError> {
(**self).read_at(offset, len)
}
#[inline]
fn len(&self) -> u64 {
(**self).len()
}
#[inline]
fn read_ranges(&self, ranges: &[Range<u64>]) -> Result<Vec<Cow<'_, [u8]>>, FormatError> {
(**self).read_ranges(ranges)
}
#[inline]
fn as_contiguous(&self) -> Option<&[u8]> {
(**self).as_contiguous()
}
}
#[cfg(feature = "std")]
impl<T: Storage + ?Sized> Storage for std::sync::Arc<T> {
#[inline]
fn read_at(&self, offset: u64, len: usize) -> Result<Cow<'_, [u8]>, FormatError> {
(**self).read_at(offset, len)
}
#[inline]
fn len(&self) -> u64 {
(**self).len()
}
#[inline]
fn read_ranges(&self, ranges: &[Range<u64>]) -> Result<Vec<Cow<'_, [u8]>>, FormatError> {
(**self).read_ranges(ranges)
}
#[inline]
fn as_contiguous(&self) -> Option<&[u8]> {
(**self).as_contiguous()
}
}
/// `storage.len()` as the `usize` the parsers' end-of-file errors report
/// (saturating on targets where the file is larger than the address space).
#[inline]
pub(crate) fn len_usize<S: Storage + ?Sized>(file: &S) -> usize {
usize::try_from(file.len()).unwrap_or(usize::MAX)
}
/// Bytes `[offset, offset + len)`, all of them.
///
/// A range that runs past the end of the storage is
/// [`FormatError::UnexpectedEof`] with `expected = offset + len` and
/// `available = storage length` — the error the `&[u8]` parsers give for
/// the same bounds check (`offset + len > file_data.len()`).
#[inline]
pub fn read_exact_at<S: Storage + ?Sized>(
file: &S,
offset: u64,
len: usize,
) -> Result<Cow<'_, [u8]>, FormatError> {
let eof = || FormatError::UnexpectedEof {
expected: usize::try_from(offset)
.unwrap_or(usize::MAX)
.saturating_add(len),
available: len_usize(file),
};
// In-memory fast path: plain slicing (for `S = [u8]` this inlines to
// the slice code's bounds check).
if let Some(all) = file.as_contiguous() {
return usize::try_from(offset)
.ok()
.and_then(|start| all.get(start..start.checked_add(len)?))
.map(Cow::Borrowed)
.ok_or_else(eof);
}
match offset.checked_add(len as u64) {
Some(end) if end <= file.len() => {}
_ => return Err(eof()),
}
let bytes = file.read_at(offset, len)?;
if bytes.len() < len {
// The storage shrank or the backend served a short read inside the
// file: never parse a partial structure.
return Err(short_read());
}
Ok(bytes)
}
#[cold]
#[inline(never)]
fn short_read() -> FormatError {
FormatError::Storage(
"short read inside the file (the storage shrank or the backend failed)".into(),
)
}
/// Largest paged data block (fixed or extensible array) read in one piece.
/// A bigger one is read as its prefix and then page by page, only the pages
/// in use, so a block whose size fields claim more than the file holds
/// costs no more than the pages it really has.
pub(crate) const PAGED_BLOCK_ONE_READ_MAX: usize = 1 << 20;
/// A window of the file: up to `max` bytes read at `base`, fewer only at
/// the end of the file. Its [`Window::ensure`] reports a bounds failure
/// exactly as the whole-file check `ensure_len(file_data, base + rel, n)`
/// did — with the absolute position and the file's length — as long as
/// every position checked lies within the `max` bytes the window was asked
/// for: then a position past the window is past the end of the file.
pub(crate) struct Window<'a> {
/// The bytes, from `base` on.
pub bytes: Cow<'a, [u8]>,
base: usize,
file_len: usize,
}
impl<'a> Window<'a> {
/// Read up to `max` bytes at `base`.
pub fn read<S: Storage + ?Sized>(
file: &'a S,
base: u64,
max: usize,
) -> Result<Self, FormatError> {
Ok(Window {
bytes: read_upto(file, base, max)?,
base: usize::try_from(base).unwrap_or(usize::MAX),
file_len: len_usize(file),
})
}
/// A whole in-memory file as one window (base 0).
#[cfg(test)]
pub fn whole(bytes: &'a [u8]) -> Self {
Window {
bytes: Cow::Borrowed(bytes),
base: 0,
file_len: bytes.len(),
}
}
/// [`Window::ensure`] for a window at `base` that has not been read:
/// whether `[rel, rel + needed)` lies in the file, with the same error.
/// Lets a parser whose first step is to check a structure's whole extent
/// (a checksum at its end) fail before reading a structure that a
/// hostile size field has stretched past the end of the file.
pub fn check_extent<S: Storage + ?Sized>(
file: &S,
base: u64,
rel: usize,
needed: usize,
) -> Result<(), FormatError> {
let base = usize::try_from(base).unwrap_or(usize::MAX);
let file_len = len_usize(file);
match base.checked_add(rel).and_then(|p| p.checked_add(needed)) {
Some(end) if end <= file_len => Ok(()),
_ => Err(FormatError::UnexpectedEof {
expected: base.saturating_add(rel).saturating_add(needed),
available: file_len,
}),
}
}
/// Check that `[rel, rel + needed)` (relative to `base`) is in the file.
#[inline]
pub fn ensure(&self, rel: usize, needed: usize) -> Result<(), FormatError> {
match rel.checked_add(needed) {
Some(end) if end <= self.bytes.len() => Ok(()),
_ => Err(FormatError::UnexpectedEof {
expected: self.base.saturating_add(rel).saturating_add(needed),
available: self.file_len,
}),
}
}
}
/// Up to `max` bytes from `offset` on: fewer only at the end of the
/// storage. For structures whose size is only known once their prefix has
/// been parsed and whose parsers bound-check what they are given.
#[inline]
pub fn read_upto<S: Storage + ?Sized>(
file: &S,
offset: u64,
max: usize,
) -> Result<Cow<'_, [u8]>, FormatError> {
if let Some(all) = file.as_contiguous() {
let start = usize::try_from(offset).map_or(all.len(), |o| o.min(all.len()));
let end = start.saturating_add(max).min(all.len());
return Ok(Cow::Borrowed(&all[start..end]));
}
let avail = file.len().saturating_sub(offset);
let len = usize::try_from(avail).map_or(max, |a| a.min(max));
let bytes = file.read_at(offset, len)?;
if bytes.len() < len {
return Err(short_read());
}
Ok(bytes)
}
/// Borrow the whole file for a code path that has not been converted to
/// [`Storage`] yet. On a backend without a contiguous view this is the
/// clean [`FormatError::ContiguousStorageRequired`] error, never a guess.
#[inline]
pub fn require_contiguous<'a, S: Storage + ?Sized>(
file: &'a S,
what: &'static str,
) -> Result<&'a [u8], FormatError> {
file.as_contiguous()
.ok_or(FormatError::ContiguousStorageRequired(what))
}
/// A [`Storage`] over an in-memory buffer that serves every byte through
/// [`Storage::read_at`] (its [`Storage::as_contiguous`] is `None`, so no
/// parser can take the whole-slice shortcut), copies what it serves (as a
/// remote backend would), and counts the reads and bytes.
///
/// It is the equivalence harness of the range-read migration: parsing a
/// file through it must give exactly what parsing the `&[u8]` gives, and
/// the counters are the request counts a cacheless range reader would make.
#[derive(Debug)]
pub struct CountingStorage {
data: Vec<u8>,
reads: portable_atomic::AtomicU64,
bytes: portable_atomic::AtomicU64,
}
impl CountingStorage {
/// Serve `data` (the file from the superblock on).
pub fn new(data: Vec<u8>) -> Self {
CountingStorage {
data,
reads: portable_atomic::AtomicU64::new(0),
bytes: portable_atomic::AtomicU64::new(0),
}
}
/// Number of `read_at` calls served so far.
pub fn reads(&self) -> u64 {
self.reads.load(portable_atomic::Ordering::Relaxed)
}
/// Number of bytes served so far.
pub fn bytes_read(&self) -> u64 {
self.bytes.load(portable_atomic::Ordering::Relaxed)
}
/// Reset both counters.
pub fn reset(&self) {
self.reads.store(0, portable_atomic::Ordering::Relaxed);
self.bytes.store(0, portable_atomic::Ordering::Relaxed);
}
}
impl Storage for CountingStorage {
fn read_at(&self, offset: u64, len: usize) -> Result<Cow<'_, [u8]>, FormatError> {
let got = self.data.as_slice().read_at(offset, len)?;
self.reads.fetch_add(1, portable_atomic::Ordering::Relaxed);
self.bytes
.fetch_add(got.len() as u64, portable_atomic::Ordering::Relaxed);
Ok(Cow::Owned(got.into_owned()))
}
fn len(&self) -> u64 {
self.data.len() as u64
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn slice_reads_are_borrowed_and_clamped() {
let data: Vec<u8> = (0u8..10).collect();
let s: &[u8] = &data;
let dynamic: &dyn Storage = &s;
assert_eq!(dynamic.len(), 10);
let r = dynamic.read_at(2, 3).unwrap();
assert!(matches!(r, Cow::Borrowed(_)));
assert_eq!(&*r, &[2, 3, 4]);
assert_eq!(&*dynamic.read_at(8, 5).unwrap(), &[8, 9]);
assert!(dynamic.read_at(10, 5).unwrap().is_empty());
assert!(dynamic.read_at(u64::MAX, 5).unwrap().is_empty());
assert_eq!(dynamic.as_contiguous(), Some(&data[..]));
let v: &dyn Storage = &data;
assert_eq!(v.as_contiguous(), Some(&data[..]));
}
#[test]
fn read_exact_matches_slice_bounds_errors() {
let data = [0u8; 10];
let s: &[u8] = &data;
assert_eq!(&*read_exact_at(&s, 4, 6).unwrap(), &[0; 6]);
assert_eq!(
read_exact_at(&s, 4, 7).unwrap_err(),
FormatError::UnexpectedEof {
expected: 11,
available: 10
}
);
assert!(read_exact_at(&s, u64::MAX, 1).is_err());
assert_eq!(read_upto(&s, 7, 100).unwrap().len(), 3);
assert_eq!(read_upto(&s, 70, 100).unwrap().len(), 0);
}
#[test]
fn counting_storage_counts_and_hides_the_slice() {
let c = CountingStorage::new((0u8..10).collect());
assert!(c.as_contiguous().is_none());
let r = c.read_at(3, 4).unwrap();
assert!(matches!(r, Cow::Owned(_)));
assert_eq!(&*r, &[3, 4, 5, 6]);
c.read_at(8, 4).unwrap();
assert_eq!((c.reads(), c.bytes_read()), (2, 6));
c.reset();
assert_eq!((c.reads(), c.bytes_read()), (0, 0));
}
#[test]
fn read_ranges_default_loops() {
let data: Vec<u8> = (0u8..10).collect();
let s: &[u8] = &data;
let got = s.read_ranges(&[1..3, 5..9]).unwrap();
assert_eq!(&*got[0], &[1, 2]);
assert_eq!(&*got[1], &[5, 6, 7, 8]);
}
}
crates/clawhdf5-format/src/superblock.rs
+60 -8
View File
@@ -7,6 +7,11 @@ use byteorder::{ByteOrder, LittleEndian};
use crate::error::FormatError;
use crate::signature::HDF5_SIGNATURE;
use crate::storage::{Storage, read_upto};
/// Bytes read to parse a superblock: more than the largest one (version 1
/// with 8-byte offsets and lengths, 100 bytes).
const SUPERBLOCK_READ_LEN: usize = 128;
/// Parsed HDF5 superblock (all versions).
#[derive(Debug, Clone, PartialEq, Eq)]
@@ -161,7 +166,16 @@ impl Superblock {
file_data: &[u8],
signature_offset: usize,
) -> Result<u64, FormatError> {
let refreshed = Superblock::parse(file_data, signature_offset)?;
self.refresh_eof_in(file_data, signature_offset as u64)
}
/// [`Self::refresh_eof`] over any [`Storage`].
pub fn refresh_eof_in<S: Storage + ?Sized>(
&mut self,
file: &S,
signature_offset: u64,
) -> Result<u64, FormatError> {
let refreshed = Superblock::parse_in(file, signature_offset)?;
self.eof_address = refreshed.eof_address;
self.consistency_flags = refreshed.consistency_flags;
Ok(self.eof_address)
@@ -219,15 +233,23 @@ impl Superblock {
/// [`FormatError::UserBlockNotStripped`] because the addresses in the
/// returned superblock would otherwise be applied to the wrong bytes.
pub fn parse(data: &[u8], signature_offset: usize) -> Result<Superblock, FormatError> {
Self::parse_in(data, signature_offset as u64)
}
/// [`Self::parse`] over any [`Storage`]: one read of the first
/// [`SUPERBLOCK_READ_LEN`] bytes (fewer when the file is shorter, which
/// is then refused with the same end-of-file errors as a short slice).
pub fn parse_in<S: Storage + ?Sized>(
file: &S,
signature_offset: u64,
) -> Result<Superblock, FormatError> {
if signature_offset != 0 {
return Err(FormatError::UserBlockNotStripped(signature_offset as u64));
return Err(FormatError::UserBlockNotStripped(signature_offset));
}
let d = data
.get(signature_offset..)
.ok_or(FormatError::UnexpectedEof {
expected: signature_offset + 1,
available: data.len(),
})?;
// Every bounds check below needs at most 100 bytes, so on a longer
// file none of them can fail and the window's length does not show.
let window = read_upto(file, 0, SUPERBLOCK_READ_LEN)?;
let d: &[u8] = &window;
ensure_len(d, 9)?; // signature(8) + version(1)
// Verify signature
@@ -894,4 +916,34 @@ mod tests {
assert_eq!(parsed.version, 3);
assert_eq!(parsed.page_size, None);
}
/// Through a storage that serves only `read_at`, every version parses
/// to the same superblock, and every truncation to the same error, as
/// from a slice — in one read.
#[test]
fn parse_in_matches_slice_parse() {
use crate::storage::CountingStorage;
let mut files = vec![
build_v0_bytes(8),
build_v0_bytes(4),
build_v1_bytes(8),
build_v1_bytes(4),
build_v2_bytes(8, 2),
build_v2_bytes(4, 3),
];
for f in files.clone() {
let mut long = f.clone();
long.resize(4096, 0xAB);
files.push(long);
for cut in [0, 5, 9, 13, 20, 30, f.len() - 1] {
files.push(f[..cut.min(f.len())].to_vec());
}
}
for f in files {
let want = Superblock::parse(&f, 0);
let storage = CountingStorage::new(f.clone());
assert_eq!(Superblock::parse_in(&storage, 0), want, "{} bytes", f.len());
assert_eq!(storage.reads(), 1);
}
}
}
crates/clawhdf5-format/src/superblock_ext.rs
+105 -12
View File
@@ -21,13 +21,14 @@
//! file is never copied whole.
#[cfg(not(feature = "std"))]
use alloc::{collections::BTreeSet, vec::Vec};
use alloc::{borrow::Cow, collections::BTreeSet, vec::Vec};
#[cfg(feature = "std")]
use std::collections::BTreeSet;
use std::{borrow::Cow, collections::BTreeSet};
use crate::error::FormatError;
use crate::message_type::MessageType;
use crate::object_header::ObjectHeader;
use crate::storage::{Storage, read_exact_at};
use crate::superblock::Superblock;
/// Message type of the File Space Info message.
@@ -169,6 +170,15 @@ impl<'a> Cursor<'a> {
pub fn read_superblock_extension(
data: &[u8],
sb: &Superblock,
) -> Result<Option<SuperblockExtension>, FormatError> {
read_superblock_extension_in(data, sb)
}
/// [`read_superblock_extension`] over any [`Storage`]; its length is the
/// end of file.
pub fn read_superblock_extension_in<S: Storage + ?Sized>(
file: &S,
sb: &Superblock,
) -> Result<Option<SuperblockExtension>, FormatError> {
let os = sb.offset_size;
let ls = sb.length_size;
@@ -181,8 +191,8 @@ pub fn read_superblock_extension(
return Ok(None);
};
let addr = usize::try_from(addr).map_err(|_| ext_err("address out of range"))?;
let header = ObjectHeader::parse(data, addr, os, ls)?;
let eoa = data.len() as u64;
let header = ObjectHeader::parse_in(file, addr as u64, os, ls)?;
let eoa = file.len();
let mut ext = SuperblockExtension::default();
for msg in &header.messages {
@@ -340,12 +350,21 @@ impl CacheImage {
data: &[u8],
location: CacheImageLocation,
sb: &Superblock,
) -> Result<Self, FormatError> {
Self::decode_in(data, location, sb)
}
/// [`Self::decode`] over any [`Storage`]: one read of the image block.
pub fn decode_in<S: Storage + ?Sized>(
file: &S,
location: CacheImageLocation,
sb: &Superblock,
) -> Result<Self, FormatError> {
let (offset_size, length_size) = (sb.offset_size, sb.length_size);
let bad = FormatError::InvalidCacheImage;
let block = image_block(data, location)?;
let eoa = data.len() as u64;
let mut c = Cursor::new(block, bad(RAN_OFF));
let block = image_block_in(file, location)?;
let eoa = file.len();
let mut c = Cursor::new(&block, bad(RAN_OFF));
// Header: signature, version, flags, image data length, entry count.
if c.take(4)? != MDCI_SIGNATURE {
@@ -463,6 +482,14 @@ impl CacheImage {
image_block(data, self.location)
}
/// [`Self::block`] over any [`Storage`].
pub fn block_in<'a, S: Storage + ?Sized>(
&self,
file: &'a S,
) -> Result<Cow<'a, [u8]>, FormatError> {
image_block_in(file, self.location)
}
/// Write every entry over `dst`, the file's bytes from the superblock
/// on (as long as the `data` the image was decoded from), taking the
/// entries from `block` (the image block, see [`Self::block`]). `block`
@@ -483,13 +510,32 @@ impl CacheImage {
}
fn image_block(data: &[u8], location: CacheImageLocation) -> Result<&[u8], FormatError> {
let (start, len) = image_block_range(data.len() as u64, location)?;
let start = crate::addr::to_usize(start)?;
Ok(&data[start..start + len])
}
fn image_block_in<S: Storage + ?Sized>(
file: &S,
location: CacheImageLocation,
) -> Result<Cow<'_, [u8]>, FormatError> {
let (start, len) = image_block_range(file.len(), location)?;
read_exact_at(file, start, len)
}
/// Where the image block is, checked against a file of `file_len` bytes.
fn image_block_range(
file_len: u64,
location: CacheImageLocation,
) -> Result<(u64, usize), FormatError> {
let bad = FormatError::InvalidCacheImage;
let start = usize::try_from(location.address).map_err(|_| bad("address out of range"))?;
let len = usize::try_from(location.length).map_err(|_| bad("length out of range"))?;
start
.checked_add(len)
.and_then(|end| data.get(start..end))
.ok_or(bad("image block extends past the end of the file"))
.filter(|&end| end as u64 <= file_len)
.ok_or(bad("image block extends past the end of the file"))?;
Ok((start as u64, len))
}
/// What an opener must do before reading a file's metadata: check the
@@ -498,11 +544,19 @@ fn image_block(data: &[u8], location: CacheImageLocation) -> Result<&[u8], Forma
/// ([`CacheImage::decode`]). `data` is the file from the superblock on, up
/// to its recorded end of file.
pub fn cache_image_state(data: &[u8], sb: &Superblock) -> Result<CacheImageState, FormatError> {
match read_superblock_extension(data, sb)? {
cache_image_state_in(data, sb)
}
/// [`cache_image_state`] over any [`Storage`].
pub fn cache_image_state_in<S: Storage + ?Sized>(
file: &S,
sb: &Superblock,
) -> Result<CacheImageState, FormatError> {
match read_superblock_extension_in(file, sb)? {
Some(SuperblockExtension {
cache_image: Some(location),
..
}) => Ok(match CacheImage::decode(data, location, sb) {
}) => Ok(match CacheImage::decode_in(file, location, sb) {
Ok(image) => CacheImageState::Loaded(image),
Err(e) => CacheImageState::Unloadable(e),
}),
@@ -566,7 +620,7 @@ mod tests {
/// holds the given messages, padded to `len` bytes.
fn file_with_ext(messages: &[(u16, &[u8])], len: usize) -> Vec<u8> {
let mut body = Vec::new();
for (t, d) in messages {
for &(t, d) in messages {
let padded = d.len().div_ceil(8) * 8;
body.extend_from_slice(&t.to_le_bytes());
body.extend_from_slice(&(padded as u16).to_le_bytes());
@@ -809,4 +863,43 @@ mod tests {
// An entry cannot be its own parent.
assert!(load(image_with_deps(&[(40, b"C", 1, Some(40))])).is_err());
}
/// The extension and cache image decode identically through a
/// `read_at`-only storage, errors included.
#[test]
fn storage_parse_matches_slice_parse() {
use crate::storage::CountingStorage;
let img = image(&[(16, b"HEADER"), (40, b"NODE")]);
let mut with_image = file_with_ext(&[(MSG_MDCI, &mdci(256, img.len() as u64))], 256);
with_image.extend_from_slice(&img);
let mut bad_image = with_image.clone();
bad_image[256] = b'X';
let files = [
file_with_ext(&[(MSG_FSINFO, &fsinfo_v1(4096, false, 0))], 256),
file_with_ext(&[(MSG_FSINFO, &fsinfo_v1(256, false, 0))], 256),
file_with_ext(&[(MSG_MDCI, &mdci(128, 64))], 192),
file_with_ext(&[(MSG_MDCI, &mdci(0x10100, 0x1000_0000))], 2565),
file_with_ext(&[(MSG_FSINFO, &fsinfo_v1(4096, true, 12))], 60),
with_image,
bad_image,
];
for f in files {
let storage = CountingStorage::new(f.clone());
let sb = sb_v2(48);
assert_eq!(
read_superblock_extension_in(&storage, &sb),
read_superblock_extension(&f, &sb)
);
assert_eq!(
cache_image_state_in(&storage, &sb),
cache_image_state(&f, &sb)
);
if let Ok(CacheImageState::Loaded(image)) = cache_image_state(&f, &sb) {
assert_eq!(
&*image.block_in(&storage).unwrap(),
image.block(&f).unwrap()
);
}
}
}
}
crates/clawhdf5-format/src/symbol_table.rs
+52 -43
View File
@@ -4,6 +4,7 @@
use alloc::vec::Vec;
use crate::error::FormatError;
use crate::storage::{Storage, read_exact_at};
/// Symbol Table message (type 0x0011) found in v1 group object headers.
#[derive(Debug, Clone, PartialEq)]
@@ -79,65 +80,49 @@ impl SymbolTableNode {
offset: usize,
offset_size: u8,
) -> Result<SymbolTableNode, FormatError> {
// signature(4) + version(1) + reserved(1) + number_of_symbols(2) = 8
if offset
.checked_add(8)
.is_none_or(|end| end > file_data.len())
{
return Err(FormatError::UnexpectedEof {
expected: offset.saturating_add(8),
available: file_data.len(),
});
}
Self::parse_in(file_data, offset as u64, offset_size)
}
if &file_data[offset..offset + 4] != b"SNOD" {
/// [`Self::parse`] over any [`Storage`]: one read of the node's header,
/// one of its entries.
pub fn parse_in<S: Storage + ?Sized>(
file: &S,
offset: u64,
offset_size: u8,
) -> Result<SymbolTableNode, FormatError> {
// signature(4) + version(1) + reserved(1) + number_of_symbols(2) = 8
let header = read_exact_at(file, offset, 8)?;
if &header[..4] != b"SNOD" {
return Err(FormatError::InvalidSymbolTableNodeSignature);
}
let version = file_data[offset + 4];
let version = header[4];
if version != 1 {
return Err(FormatError::InvalidSymbolTableNodeVersion(version));
}
let num_symbols =
u16::from_le_bytes([file_data[offset + 6], file_data[offset + 7]]) as usize;
let num_symbols = u16::from_le_bytes([header[6], header[7]]) as usize;
let os = offset_size as usize;
// Each entry: link_name_offset(os) + obj_hdr_addr(os) + cache_type(4) + reserved(4) + scratch(16)
let entry_size = os + os + 4 + 4 + 16;
let entries_start = offset + 8;
let needed = entries_start.checked_add(num_symbols * entry_size).ok_or(
FormatError::UnexpectedEof {
expected: usize::MAX,
available: file_data.len(),
},
)?;
if needed > file_data.len() {
return Err(FormatError::UnexpectedEof {
expected: needed,
available: file_data.len(),
});
}
// `offset + 8` fits: the header's read checked it. The entries'
// read is the bounds check (`offset + 8 + entries > file length`,
// which cannot overflow: at most 65535 entries of 40 bytes).
let body = read_exact_at(file, offset + 8, num_symbols * entry_size)?;
let file_data: &[u8] = &body;
let mut entries = Vec::with_capacity(num_symbols);
let mut pos = entries_start;
for _ in 0..num_symbols {
let link_name_offset = read_offset(file_data, pos, offset_size)?;
pos += os;
let object_header_address = read_offset(file_data, pos, offset_size)?;
pos += os;
let cache_type = u32::from_le_bytes([
file_data[pos],
file_data[pos + 1],
file_data[pos + 2],
file_data[pos + 3],
]);
pos += 4;
for entry in file_data.chunks_exact(entry_size) {
let link_name_offset = read_offset(entry, 0, offset_size)?;
let object_header_address = read_offset(entry, os, offset_size)?;
let pos = 2 * os;
let cache_type =
u32::from_le_bytes([entry[pos], entry[pos + 1], entry[pos + 2], entry[pos + 3]]);
// reserved 4 bytes
pos += 4;
let mut scratch_pad = [0u8; 16];
scratch_pad.copy_from_slice(&file_data[pos..pos + 16]);
pos += 16;
scratch_pad.copy_from_slice(&entry[pos + 8..pos + 24]);
entries.push(SymbolTableEntry {
link_name_offset,
@@ -256,4 +241,28 @@ mod tests {
let result = SymbolTableNode::parse(&data, usize::MAX / 2, 8);
assert!(result.is_err());
}
/// Nodes, cut at every length and at an offset, parse identically
/// through a `read_at`-only storage.
#[test]
fn storage_parse_matches_slice_parse() {
use crate::storage::CountingStorage;
for os in [4u8, 8] {
let node = build_snod(&[(0, 0x100, 0), (8, 0x200, 1), (16, 0x300, 2)], os);
let mut bad = node.clone();
bad[4] = 2;
for full in [node, bad] {
for at in [0usize, 7] {
for cut in 0..=full.len() {
let mut f = vec![0u8; at];
f.extend_from_slice(&full[..cut]);
let storage = CountingStorage::new(f.clone());
let want = SymbolTableNode::parse(&f, at, os);
let got = SymbolTableNode::parse_in(&storage, at as u64, os);
assert_eq!(format!("{got:?}"), format!("{want:?}"));
}
}
}
}
}
}
crates/clawhdf5-format/src/type_builders.rs
+2 -1
View File
@@ -296,7 +296,8 @@ impl EnumTypeBuilder {
// ---- Attribute helper ----
pub(crate) fn build_attr_message(name: &str, value: &AttrValue) -> AttributeMessage {
/// The attribute message the writers store for `value` under `name`.
pub fn build_attr_message(name: &str, value: &AttrValue) -> AttributeMessage {
match value {
AttrValue::F64(v) => AttributeMessage {
name: name.to_string(),
crates/clawhdf5-format/src/vds.rs
+7 -6
View File
@@ -15,6 +15,7 @@
#[cfg(not(feature = "std"))]
use alloc::{format, string::String, vec, vec::Vec};
use crate::addr::to_usize;
use crate::data_layout::{DataLayout, VdsMapping, parse_vds_mappings};
use crate::dataspace::Dataspace;
use crate::datatype::Datatype;
@@ -208,7 +209,7 @@ fn load_mappings(
return Ok(Vec::new());
};
let coll =
crate::global_heap::GlobalHeapCollection::parse(file_data, addr as usize, length_size)?;
crate::global_heap::GlobalHeapCollection::parse(file_data, to_usize(addr)?, length_size)?;
let index = u16::try_from(*global_heap_index)
.map_err(|_| vds_err("VDS mapping heap index out of range"))?;
let obj = coll
@@ -611,12 +612,12 @@ fn scatter(
return Err(vds_err("virtual/source selection element counts differ"));
}
for (&v, &s) in vidx.iter().zip(sidx) {
let (vo, so) = (v as usize * elem_size, s as usize * elem_size);
let (vo, so) = (to_usize(v)? * elem_size, to_usize(s)? * elem_size);
if vo + elem_size > out.len() || so + elem_size > src.len() {
return Err(vds_err("virtual dataset selection out of bounds"));
}
out[vo..vo + elem_size].copy_from_slice(&src[so..so + elem_size]);
mapped[v as usize] = true;
mapped[to_usize(v)?] = true;
}
Ok(())
}
@@ -747,7 +748,7 @@ fn selection_indices(
return Err(vds_err("VDS selection blocks overlap"));
}
}
let mut out = Vec::with_capacity(volume as usize);
let mut out = Vec::with_capacity(to_usize(volume)?);
for (s, e) in starts.chunks_exact(rank).zip(ends.chunks_exact(rank)) {
let mut cur = s.to_vec();
'block: loop {
@@ -868,7 +869,7 @@ fn load_source_file(whole: &mut [u8]) -> Result<(), FormatError> {
// read as before, up to its length.
let end = sb
.data_end(base as u64, whole.len() as u64)
.map_or(whole.len(), |e| base + e as usize);
.map_or(Ok(whole.len()), |e| to_usize(e).map(|e| base + e))?;
crate::superblock_ext::apply_cache_image_in_place(&mut whole[base..end], &sb)
}
@@ -921,7 +922,7 @@ fn open_source(file_data: &[u8], path: &str) -> Result<Option<OpenSource>, Forma
Err(FormatError::PathNotFound(_)) => return Ok(None),
Err(e) => return Err(e),
};
let header = crate::object_header::ObjectHeader::parse(file_data, addr as usize, os, ls)?;
let header = crate::object_header::ObjectHeader::parse(file_data, to_usize(addr)?, os, ls)?;
let mut src = OpenSource {
offset_size: os,
length_size: ls,
crates/clawhdf5-format/src/vl_data.rs
+4 -3
View File
@@ -9,6 +9,7 @@ use alloc::{collections::BTreeMap, format, string::String, vec, vec::Vec};
#[cfg(feature = "std")]
use std::collections::BTreeMap;
use crate::addr::to_usize;
use crate::error::FormatError;
use crate::global_heap::{GlobalHeapCollection, GlobalHeapIndex};
@@ -55,7 +56,7 @@ pub fn parse_vl_references(
) -> Result<Vec<VlElement>, FormatError> {
let elem_size = 4 + offset_size as usize + 4; // length + address + index
let total =
(num_elements as usize)
to_usize(num_elements)?
.checked_mul(elem_size)
.ok_or(FormatError::UnexpectedEof {
expected: usize::MAX,
@@ -68,7 +69,7 @@ pub fn parse_vl_references(
});
}
let mut elements = Vec::with_capacity(num_elements as usize);
let mut elements = Vec::with_capacity(to_usize(num_elements)?);
let mut pos = 0;
for _ in 0..num_elements {
@@ -406,7 +407,7 @@ impl<'a> VlResolver<'a> {
let index =
GlobalHeapCollection::parse_index(self.file_data, offset, self.length_size)?;
// parse_index checked that the collection lies in the file.
let end = offset + index.collection_size as usize;
let end = offset + to_usize(index.collection_size)?;
self.check_overlap(offset, end)?;
let coll = CachedCollection::new(index);
if self.cached_bytes.saturating_add(coll.cost()) > self.budget {
crates/clawhdf5-format/tests/fixtures/v1_groups_400.h5
Vendored
BIN
View File
Binary file not shown.
crates/clawhdf5-format/tests/storage_equivalence.rs
+701
View File
@@ -0,0 +1,701 @@
//! Equivalence harness for the range-read migration
//! (`docs/design/range-reads.md`, milestone M1).
//!
//! Every metadata parser converted to [`Storage`] must give exactly what its
//! `&[u8]` form gives. This walks real files — the fixtures, files h5py
//! writes to exercise the less common structures, and optionally the
//! conformance corpus — and, for every object, runs each converted parser
//! twice: over the file as a slice, and over a [`CountingStorage`] that
//! serves the same bytes through `read_at` only (`as_contiguous()` is
//! `None`, so no parser can fall back to the whole slice). The results must
//! be identical, value for value and error for error.
//!
//! The one allowed difference is [`FormatError::ContiguousStorageRequired`]
//! from the storage path, and only from the structures still indexed by a v2
//! B-tree (dense attributes, a SOHM B-tree index, huge fractal-heap objects;
//! see `CONTIGUOUS_REQUIRED`), which fail cleanly instead of reading the
//! whole file. Those are counted; the error from any other site or check
//! fails the harness.
//!
//! Milestones M2/M3 extend `check_object` with the raw-data and group
//! parsers as they are converted.
//!
//! - `CLAWHDF5_STORAGE_CORPUS=dir[:dir...]` adds every `.h5`/`.hdf5`/`.he5`/
//! `.nc`/`.h5ad` file under those directories (the conformance corpus is
//! `conformance/.cache/corpus`); `CLAWHDF5_STORAGE_REPORT=1` prints the
//! per-file read counts.
//! - The h5py-written files honour `CLAWHDF5_PYTHON` and
//! `CLAWHDF5_REQUIRE_INTEROP` like the facade's interop tests.
use std::collections::{HashSet, VecDeque};
use std::fmt::Debug;
use std::path::{Path, PathBuf};
use std::process::Command;
use clawhdf5_format::attribute::{
extract_attributes_full, extract_attributes_full_in, extract_attributes_tolerant,
extract_attributes_tolerant_in,
};
use clawhdf5_format::attribute_info::AttributeInfoMessage;
use clawhdf5_format::btree_v1::{collect_symbol_table_nodes, collect_symbol_table_nodes_in};
use clawhdf5_format::btree_v2::{BTreeV2Header, collect_btree_v2_records};
use clawhdf5_format::data_layout::DataLayout;
use clawhdf5_format::dataspace::Dataspace;
use clawhdf5_format::datatype::Datatype;
use clawhdf5_format::error::FormatError;
use clawhdf5_format::extensible_array::{
ExtensibleArrayHeader, read_extensible_array_chunks, read_extensible_array_chunks_in,
};
use clawhdf5_format::fill_value::{dataset_fill_value_from_storage, dataset_fill_value_in};
use clawhdf5_format::fixed_array::{
FixedArrayHeader, read_fixed_array_chunks, read_fixed_array_chunks_in,
};
use clawhdf5_format::fractal_heap::FractalHeapHeader;
use clawhdf5_format::link_info::LinkInfoMessage;
use clawhdf5_format::local_heap::LocalHeap;
use clawhdf5_format::message_type::MessageType;
use clawhdf5_format::object_header::ObjectHeader;
use clawhdf5_format::shared_message::{
self, load_sohm_table, load_sohm_table_in, message_data_with_sohm, message_data_with_sohm_in,
parse_sohm_btree_entries, parse_sohm_btree_entries_in, parse_sohm_list, parse_sohm_list_in,
};
use clawhdf5_format::signature::split_user_block;
use clawhdf5_format::storage::{CountingStorage, Storage};
use clawhdf5_format::superblock::Superblock;
use clawhdf5_format::superblock_ext::{
cache_image_state, cache_image_state_in, read_superblock_extension,
read_superblock_extension_in,
};
use clawhdf5_format::symbol_table::{SymbolTableMessage, SymbolTableNode};
/// Objects visited per file, heap objects read per heap: enough to cover
/// every structure kind while keeping a 35 000-group file fast.
const MAX_OBJECTS: usize = 1500;
const MAX_HEAP_IDS: usize = 200;
/// The structures that still need the whole file in memory, because they
/// are found through a version-2 B-tree (not converted yet), and the checks
/// that can reach each of them. Anything else answering
/// [`FormatError::ContiguousStorageRequired`] is a converted parser falling
/// back to the whole file, and fails the harness.
const CONTIGUOUS_REQUIRED: &[(&str, &[&str])] = &[
(
"dense attribute storage (a v2 B-tree)",
&["attributes", "attributes (tolerant)"],
),
(
"a shared-message B-tree index",
&[
"SOHM B-tree",
"shared message",
"fill value",
"attributes",
"attributes (tolerant)",
],
),
(
"a huge fractal-heap object's B-tree",
&["heap object", "attributes", "attributes (tolerant)"],
),
];
fn may_require_contiguous(check: &str, site: &str) -> bool {
CONTIGUOUS_REQUIRED
.iter()
.any(|(s, checks)| *s == site && checks.contains(&check))
}
#[derive(Default, Debug)]
struct Tally {
files: usize,
objects: usize,
checks: usize,
contiguous_required: usize,
reads: u64,
bytes: u64,
/// Chunk indexes (fixed and extensible arrays) read, and the most bytes
/// one of them took through the storage.
chunk_indexes: usize,
max_chunk_index_bytes: u64,
}
struct Walk<'a> {
slice: &'a [u8],
storage: &'a CountingStorage,
name: String,
tally: &'a mut Tally,
}
impl Walk<'_> {
/// The storage result must equal the slice result, or be the clean
/// "needs the whole file" error.
fn same<T: Debug>(
&mut self,
what: &str,
want: &Result<T, FormatError>,
got: &Result<T, FormatError>,
) {
self.tally.checks += 1;
if let Err(FormatError::ContiguousStorageRequired(site)) = got {
assert!(
may_require_contiguous(what, site),
"{}: {what} fell back to the whole file ({site}), which only the \
v2-B-tree-indexed structures may do",
self.name
);
self.tally.contiguous_required += 1;
return;
}
let (w, g) = (format!("{want:?}"), format!("{got:?}"));
assert!(
w == g,
"{}: {what} differs\n slice: {}\n storage: {}",
self.name,
&w[..w.len().min(600)],
&g[..g.len().min(600)]
);
}
fn index_read(&mut self, bytes_before: u64) {
self.tally.chunk_indexes += 1;
let bytes = self.storage.bytes_read() - bytes_before;
self.tally.max_chunk_index_bytes = self.tally.max_chunk_index_bytes.max(bytes);
}
fn st(&self) -> &dyn Storage {
self.storage
}
fn run(&mut self) {
let slice = self.slice;
let sb = Superblock::parse(slice, 0);
self.same("superblock", &sb, &Superblock::parse_in(self.st(), 0));
let Ok(sb) = sb else { return };
let (os, ls) = (sb.offset_size, sb.length_size);
let want = read_superblock_extension(slice, &sb);
self.same(
"superblock extension",
&want,
&read_superblock_extension_in(self.st(), &sb),
);
let want = cache_image_state(slice, &sb);
self.same("cache image", &want, &cache_image_state_in(self.st(), &sb));
let table = load_sohm_table(slice, os, ls);
self.same("SOHM table", &table, &load_sohm_table_in(self.st(), os, ls));
if let Ok(Some(table)) = &table {
for idx in &table.indexes {
if idx.index_type == 0 {
let want =
parse_sohm_list(slice, idx.index_addr as usize, idx.num_messages, os);
let got = parse_sohm_list_in(self.st(), idx.index_addr, idx.num_messages, os);
self.same("SOHM list", &want, &got);
} else {
let want = parse_sohm_btree_entries(slice, idx.index_addr as usize, os, ls);
let got = parse_sohm_btree_entries_in(self.st(), idx.index_addr, os, ls);
self.same("SOHM B-tree", &want, &got);
}
}
}
let mut seen = HashSet::new();
let mut queue = VecDeque::from([sb.root_group_address]);
while let Some(addr) = queue.pop_front() {
if seen.len() >= MAX_OBJECTS || !seen.insert(addr) {
continue;
}
self.tally.objects += 1;
self.check_object(&sb, addr);
// Traversal only (group lookups are milestone M0/M3 work).
if let Ok(children) =
clawhdf5_format::group_v2::resolve_group_children(slice, &sb, addr)
{
queue.extend(children.iter().map(|c| c.object_header_address));
}
}
}
fn check_object(&mut self, sb: &Superblock, addr: u64) {
let slice = self.slice;
let (os, ls) = (sb.offset_size, sb.length_size);
let header = ObjectHeader::parse(slice, addr as usize, os, ls);
self.same(
"object header",
&header,
&ObjectHeader::parse_in(self.st(), addr, os, ls),
);
let Ok(header) = header else { return };
let want = extract_attributes_full(slice, &header, os, ls);
self.same(
"attributes",
&want,
&extract_attributes_full_in(self.st(), &header, os, ls),
);
let want = extract_attributes_tolerant(slice, &header, os, ls);
let got = extract_attributes_tolerant_in(self.st(), &header, os, ls);
self.same("attributes (tolerant)", &want, &got);
let want = dataset_fill_value_in(slice, &header.messages, os, ls);
self.same(
"fill value",
&want,
&dataset_fill_value_from_storage(self.st(), &header.messages, os, ls),
);
for msg in &header.messages {
if shared_message::is_shared(msg.flags) {
let want = message_data_with_sohm(slice, msg, os, ls);
let got = message_data_with_sohm_in(self.st(), msg, os, ls);
self.same("shared message", &want, &got);
}
match msg.msg_type {
MessageType::SymbolTable => {
if let Ok(stm) = SymbolTableMessage::parse(&msg.data, os) {
self.check_v1_group(&stm, os, ls);
}
}
MessageType::LinkInfo => {
if let Ok(li) = LinkInfoMessage::parse(&msg.data, os) {
self.check_heap(
li.fractal_heap_address,
li.btree_name_index_address,
4,
os,
ls,
);
}
}
MessageType::AttributeInfo => {
if let Ok(ai) = AttributeInfoMessage::parse(&msg.data, os) {
self.check_heap(
ai.fractal_heap_address,
ai.btree_name_index_address,
0,
os,
ls,
);
}
}
_ => {}
}
}
self.check_layout(&header, os, ls);
}
/// A symbol-table group: its local heap, B-tree, nodes and names.
fn check_v1_group(&mut self, stm: &SymbolTableMessage, os: u8, ls: u8) {
let slice = self.slice;
let heap = LocalHeap::parse(slice, stm.local_heap_address as usize, os, ls);
self.same(
"local heap",
&heap,
&LocalHeap::parse_in(self.st(), stm.local_heap_address, os, ls),
);
let nodes = collect_symbol_table_nodes(slice, stm.btree_address, os, ls);
let got = collect_symbol_table_nodes_in(self.st(), stm.btree_address, os, ls);
self.same("group B-tree", &nodes, &got);
let Ok(heap) = heap else { return };
let want = heap.validate_free_list(slice, ls);
self.same(
"local heap free list",
&want,
&heap.validate_free_list_in(self.st(), ls),
);
let Ok(nodes) = nodes else { return };
for &node in nodes.iter().take(MAX_HEAP_IDS) {
let snod = SymbolTableNode::parse(slice, node as usize, os);
self.same(
"symbol table node",
&snod,
&SymbolTableNode::parse_in(self.st(), node, os),
);
let Ok(snod) = snod else { continue };
for e in &snod.entries {
let want = heap.read_string(slice, e.link_name_offset);
self.same(
"link name",
&want,
&heap.read_string_in(self.st(), e.link_name_offset),
);
}
}
}
/// A dense group's or dense attributes' fractal heap: the header, and
/// the objects its name index points at. `id_at` is where the heap ID
/// starts in a name-index record (after the hash for links).
fn check_heap(&mut self, heap: Option<u64>, index: Option<u64>, id_at: usize, os: u8, ls: u8) {
let slice = self.slice;
let Some(heap_addr) = heap else { return };
let fh = FractalHeapHeader::parse(slice, heap_addr as usize, os, ls);
self.same(
"fractal heap",
&fh,
&FractalHeapHeader::parse_in(self.st(), heap_addr, os, ls),
);
let (Ok(fh), Some(index)) = (fh, index) else {
return;
};
let Ok(bt) = BTreeV2Header::parse(slice, index as usize, os, ls) else {
return;
};
let Ok(records) = collect_btree_v2_records(slice, &bt, os, ls) else {
return;
};
let id_len = fh.heap_id_length as usize;
for rec in records.iter().take(MAX_HEAP_IDS) {
let Some(id) = rec.data.get(id_at..id_at + id_len) else {
continue;
};
let want = fh.read_managed_object(slice, id, os);
self.same(
"heap object",
&want,
&fh.read_managed_object_in(self.st(), id, os),
);
}
}
/// A dataset's layout: VDS mappings, and fixed/extensible array chunk
/// indexes.
fn check_layout(&mut self, header: &ObjectHeader, os: u8, ls: u8) {
let slice = self.slice;
let find = |t: MessageType| {
header
.messages
.iter()
.find(|m| m.msg_type == t)
.and_then(|m| shared_message::message_data_with_sohm(slice, m, os, ls).ok())
};
let Some(layout) = find(MessageType::DataLayout) else {
return;
};
let Ok(layout) = DataLayout::parse(&layout, os, ls) else {
return;
};
match &layout {
DataLayout::Virtual { .. } => {
let (mut want, mut got) = (layout.clone(), layout.clone());
let w = want.resolve_vds_mappings(slice, ls).map(|()| want);
let g = got.resolve_vds_mappings_in(self.st(), ls).map(|()| got);
self.same("VDS mappings", &w, &g);
}
DataLayout::Chunked {
chunk_dimensions,
btree_address: Some(addr),
version: 4,
chunk_index_type: Some(kind @ (3 | 4)),
..
} => {
let (Some(ds), Some(dt)) =
(find(MessageType::Dataspace), find(MessageType::Datatype))
else {
return;
};
let (Ok(ds), Ok((dt, _))) = (Dataspace::parse(&ds, ls), Datatype::parse(&dt))
else {
return;
};
let rank = ds.dimensions.len();
if chunk_dimensions.len() < rank {
return;
}
let dims = &chunk_dimensions[..rank];
let max = ds.max_dimensions.as_deref();
let es = dt.type_size();
if *kind == 3 {
let h = FixedArrayHeader::parse(slice, *addr as usize, os, ls);
self.same(
"fixed array header",
&h,
&FixedArrayHeader::parse_in(self.st(), *addr, os, ls),
);
let Ok(h) = h else { return };
let want =
read_fixed_array_chunks(slice, &h, &ds.dimensions, max, dims, es, os, ls);
let before = self.storage.bytes_read();
let got = read_fixed_array_chunks_in(
self.st(),
&h,
&ds.dimensions,
max,
dims,
es,
os,
ls,
);
self.index_read(before);
self.same("fixed array chunks", &want, &got);
} else {
let h = ExtensibleArrayHeader::parse(slice, *addr as usize, os, ls);
let got = ExtensibleArrayHeader::parse_in(self.st(), *addr, os, ls);
self.same("extensible array header", &h, &got);
let Ok(h) = h else { return };
let want = read_extensible_array_chunks(
slice,
&h,
&ds.dimensions,
max,
dims,
es,
os,
ls,
);
let before = self.storage.bytes_read();
let got = read_extensible_array_chunks_in(
self.st(),
&h,
&ds.dimensions,
max,
dims,
es,
os,
ls,
);
self.index_read(before);
self.same("extensible array chunks", &want, &got);
}
}
_ => {}
}
}
}
fn check_file(path: &Path, tally: &mut Tally) {
let Ok(bytes) = std::fs::read(path) else {
return;
};
check_bytes(&path.display().to_string(), &bytes, tally);
}
fn check_bytes(name: &str, bytes: &[u8], tally: &mut Tally) {
let Ok((_, hdf5)) = split_user_block(bytes) else {
return;
};
let storage = CountingStorage::new(hdf5.to_vec());
let before = (tally.checks, tally.objects);
let mut walk = Walk {
slice: hdf5,
storage: &storage,
name: name.to_string(),
tally,
};
walk.run();
tally.files += 1;
tally.reads += storage.reads();
tally.bytes += storage.bytes_read();
if std::env::var("CLAWHDF5_STORAGE_REPORT").is_ok_and(|v| v == "1") {
eprintln!(
"{:>6} objects {:>7} checks {:>8} reads {:>12} bytes {}",
tally.objects - before.1,
tally.checks - before.0,
storage.reads(),
storage.bytes_read(),
name
);
}
}
fn hdf5_files(dir: &Path, out: &mut Vec<PathBuf>) {
let Ok(entries) = std::fs::read_dir(dir) else {
return;
};
for e in entries.flatten() {
let p = e.path();
if p.is_dir() {
hdf5_files(&p, out);
} else if p
.extension()
.and_then(|x| x.to_str())
.is_some_and(|x| matches!(x, "h5" | "hdf5" | "he5" | "nc" | "h5ad" | "hdf"))
{
out.push(p);
}
}
}
#[test]
fn fixtures_parse_identically_through_storage() {
let mut files = Vec::new();
hdf5_files(
&Path::new(env!("CARGO_MANIFEST_DIR")).join("tests/fixtures"),
&mut files,
);
files.sort();
assert!(files.len() >= 40, "{} fixtures", files.len());
let mut tally = Tally::default();
for f in &files {
check_file(f, &mut tally);
}
eprintln!("fixtures: {tally:?}");
assert!(tally.objects >= 150 && tally.checks >= 1000, "{tally:?}");
// Reads really went through read_at.
assert!(tally.reads > tally.objects as u64);
}
#[test]
fn corpus_parses_identically_through_storage() {
let Ok(dirs) = std::env::var("CLAWHDF5_STORAGE_CORPUS") else {
eprintln!("CLAWHDF5_STORAGE_CORPUS not set; skipping the corpus");
return;
};
let mut files = Vec::new();
for d in std::env::split_paths(&dirs) {
hdf5_files(&d, &mut files);
}
files.sort();
let mut tally = Tally::default();
for f in &files {
check_file(f, &mut tally);
}
eprintln!("corpus: {tally:?}");
assert!(tally.files > 0);
}
fn python() -> String {
std::env::var("CLAWHDF5_PYTHON").unwrap_or_else(|_| "python3".to_string())
}
fn interop_required() -> bool {
std::env::var("CLAWHDF5_REQUIRE_INTEROP").is_ok_and(|v| v == "1")
}
/// Files h5py writes to cover what the fixtures do not: extensible arrays
/// deep enough for super blocks and paged data blocks, paged fixed arrays,
/// big symbol-table and dense groups, dense and shared attributes, a SOHM
/// list and a SOHM B-tree, committed datatypes, and a user block.
const GENERATE: &str = r#"
import ctypes, glob, os, sys, h5py, numpy as np
out = sys.argv[1]
def p(n): return os.path.join(out, n)
with h5py.File(p('ea.h5'), 'w', libver='latest') as f:
# One unlimited dimension: extensible array. 3000 chunks reach super
# blocks; 2-element chunks of int8 keep data small.
d = f.create_dataset('ea', shape=(6000,), maxshape=(None,), chunks=(2,), dtype='i1')
d[:] = np.arange(6000) % 100
d2 = f.create_dataset('ea_deflate', shape=(4000, 3), maxshape=(None, 3), chunks=(2, 3),
dtype='f4', compression='gzip')
d2[:] = np.random.default_rng(1).random((4000, 3))
d3 = f.create_dataset('ea_sparse', shape=(100000,), maxshape=(None,), chunks=(4,), dtype='i2')
d3[0:8] = 1; d3[50000:50004] = 2; d3[99996:] = 3
fa = f.create_dataset('fa_paged', shape=(5000,), chunks=(1,), dtype='u1')
fa[::3] = 7
fa2 = f.create_dataset('fa', shape=(40, 40), chunks=(8, 8), dtype='f8', compression='gzip')
fa2[:] = 1.5
for i in range(30):
f.attrs[f'a{i}'] = np.arange(i + 1)
g = f.create_group('dense')
for i in range(300):
g.create_group(f'child{i:04d}').attrs['i'] = i
f['committed'] = np.dtype([('x', 'i4'), ('y', 'f8')])
f.create_dataset('uses_committed', shape=(3,), dtype=f['committed'])
f['uses_committed'].attrs.create('ta', data=np.zeros(2, dtype=f['committed'].dtype), dtype=f['committed'])
f.attrs['vl'] = ['alpha', 'beta', 'gamma']
with h5py.File(p('v1_groups.h5'), 'w', libver='earliest', userblock_size=512) as f:
for i in range(400):
g = f.create_group(f'g{i:04d}')
g.attrs['n'] = i
f.create_dataset('x', data=np.arange(10))
# Paged chunk indexes whose data blocks are bigger than the storage reads
# in one piece (1 MiB), with two chunks written: a fixed array of 300 000
# chunks (a 2.4 MB data block) and an extensible array grown to 1.2e9 (its
# last data block holds 131 072 chunks: over 1 MiB).
with h5py.File(p('big_paged.h5'), 'w', libver='latest') as f:
d = f.create_dataset('fa', shape=(300000,), chunks=(1,), dtype='u1')
d[5] = 1; d[250000] = 2
e = f.create_dataset('ea', shape=(1,), maxshape=(None,), chunks=(1,), dtype='u1')
e.resize((1200000000,)); e[10] = 1; e[1100000000] = 3
libs = glob.glob(os.path.join(os.path.dirname(h5py.__file__), '..', 'h5py.libs', 'libhdf5-*.so*'))
if libs:
lib = ctypes.CDLL(libs[0])
lib.H5Pset_shared_mesg_nindexes.argtypes = [ctypes.c_int64, ctypes.c_uint]
lib.H5Pset_shared_mesg_index.argtypes = [ctypes.c_int64, ctypes.c_uint, ctypes.c_uint, ctypes.c_uint]
lib.H5Pset_shared_mesg_phase_change.argtypes = [ctypes.c_int64, ctypes.c_uint, ctypes.c_uint]
for name, list_max in [('sohm_list.h5', 50), ('sohm_btree.h5', 0)]:
fcpl = h5py.h5p.create(h5py.h5p.FILE_CREATE)
assert lib.H5Pset_shared_mesg_nindexes(fcpl.id, 1) >= 0
# datatype, dataspace, fill value, filter pipeline, attribute
assert lib.H5Pset_shared_mesg_index(fcpl.id, 0, 0x02 | 0x04 | 0x08 | 0x10 | 0x20, 1) >= 0
assert lib.H5Pset_shared_mesg_phase_change(fcpl.id, list_max, 0) >= 0
fapl = h5py.h5p.create(h5py.h5p.FILE_ACCESS)
fapl.set_libver_bounds(h5py.h5f.LIBVER_LATEST, h5py.h5f.LIBVER_LATEST)
fid = h5py.h5f.create(p(name).encode(), h5py.h5f.ACC_TRUNC, fcpl=fcpl, fapl=fapl)
with h5py.File(fid) as f:
for i in range(20):
d = f.create_dataset(f'd{i}', shape=(10, i + 1), dtype='f4', fillvalue=-1.0,
chunks=(5, 1), compression='gzip')
d.attrs['units'] = 'metres per second, a long enough string to share'
d.attrs['scale'] = np.arange(20, dtype='f8')
print('ok')
"#;
#[test]
fn h5py_files_parse_identically_through_storage() {
let dir = Path::new(env!("CARGO_TARGET_TMPDIR")).join("storage_equivalence");
let _ = std::fs::remove_dir_all(&dir);
std::fs::create_dir_all(&dir).unwrap();
let out = Command::new(python())
.args(["-c", GENERATE, dir.to_str().unwrap()])
.output();
match out {
Ok(o) if o.status.success() => {}
Ok(o) if interop_required() => panic!(
"h5py generation failed:\n{}\n{}",
String::from_utf8_lossy(&o.stdout),
String::from_utf8_lossy(&o.stderr)
),
Err(e) if interop_required() => panic!("python not available: {e}"),
_ => {
eprintln!("python3 with h5py unavailable; skipping");
return;
}
}
let mut files = Vec::new();
hdf5_files(&dir, &mut files);
files.sort();
let names: Vec<_> = files
.iter()
.map(|f| f.file_name().unwrap().to_string_lossy().into_owned())
.collect();
for want in ["ea.h5", "v1_groups.h5", "big_paged.h5"] {
assert!(names.iter().any(|n| n == want), "{names:?}");
}
if interop_required() {
assert!(names.iter().any(|n| n == "sohm_btree.h5"), "{names:?}");
}
let mut tally = Tally::default();
for f in &files {
if f.ends_with("big_paged.h5") {
// Only the pages in use are read, not the whole data blocks:
// read in one piece, the fixed array took 2.4 MB and the
// extensible array 1.2 MB (its super block's page bitmap and
// block addresses are most of what remains).
let mut big = Tally::default();
check_file(f, &mut big);
eprintln!("big paged blocks: {big:?}");
assert_eq!(big.chunk_indexes, 2, "{big:?}");
assert!(big.max_chunk_index_bytes < 256 << 10, "{big:?}");
// Truncated anywhere, the page-by-page reads still agree with
// the slice reads (errors included).
let bytes = std::fs::read(f).unwrap();
for cut in (0..bytes.len()).step_by(bytes.len() / 97) {
check_bytes(
&format!("big_paged.h5 cut at {cut}"),
&bytes[..cut],
&mut big,
);
}
eprintln!("big paged blocks, truncated: {big:?}");
assert!(big.chunk_indexes > 100, "{big:?}");
}
check_file(f, &mut tally);
}
eprintln!("h5py files: {tally:?}");
assert!(tally.objects >= 700, "{tally:?}");
// Dense attributes and the SOHM B-tree are the known clean errors.
assert!(tally.contiguous_required > 0, "{tally:?}");
}
crates/clawhdf5-format/tests/zfp_alloc_bounds.rs
+274
View File
@@ -0,0 +1,274 @@
//! Crafted ZFP filter parameters and streams cannot make the decoder panic
//! or allocate out of proportion to the chunk it decodes.
//!
//! The field size comes from the filter's `cd_values` (up to 2^48 values),
//! not from the chunk: the decoder allocates the output only when it matches
//! the chunk's size (or, when that is unknown, is within the 256 MiB
//! ceiling), and only when the stream is long enough to hold a bit per
//! block. Peak heap use is measured with a counting global allocator; the
//! tests share it, so each holds `SERIAL` for its whole run.
#![cfg(feature = "zfp")]
use std::alloc::{GlobalAlloc, Layout, System};
use std::sync::Mutex;
use std::sync::atomic::{AtomicUsize, Ordering};
use clawhdf5_format::filters_zfp::zfp_decompress;
struct Counting;
static CURRENT: AtomicUsize = AtomicUsize::new(0);
static PEAK: AtomicUsize = AtomicUsize::new(0);
static SERIAL: Mutex<()> = Mutex::new(());
unsafe impl GlobalAlloc for Counting {
unsafe fn alloc(&self, layout: Layout) -> *mut u8 {
let p = unsafe { System.alloc(layout) };
if !p.is_null() {
let now = CURRENT.fetch_add(layout.size(), Ordering::Relaxed) + layout.size();
PEAK.fetch_max(now, Ordering::Relaxed);
}
p
}
unsafe fn alloc_zeroed(&self, layout: Layout) -> *mut u8 {
let p = unsafe { System.alloc_zeroed(layout) };
if !p.is_null() {
let now = CURRENT.fetch_add(layout.size(), Ordering::Relaxed) + layout.size();
PEAK.fetch_max(now, Ordering::Relaxed);
}
p
}
unsafe fn dealloc(&self, ptr: *mut u8, layout: Layout) {
unsafe { System.dealloc(ptr, layout) };
CURRENT.fetch_sub(layout.size(), Ordering::Relaxed);
}
}
#[global_allocator]
static ALLOC: Counting = Counting;
/// Bytes allocated at the peak of `f`, above what was live when it started.
fn peak_during<T>(f: impl FnOnce() -> T) -> (T, usize) {
let base = CURRENT.load(Ordering::Relaxed);
PEAK.store(base, Ordering::Relaxed);
let out = f();
(out, PEAK.load(Ordering::Relaxed).saturating_sub(base))
}
fn lock() -> std::sync::MutexGuard<'static, ()> {
SERIAL.lock().unwrap_or_else(|e| e.into_inner())
}
/// What decoding may hold: the output (at most the limit, and at most a
/// 4-D block of doubles, 2 KiB, per bit of input), and a little more.
fn bound(max_output: usize, input: &[u8]) -> usize {
let limit = if max_output == 0 {
256 << 20
} else {
max_output
};
limit.min(input.len() * 8 * 2048) + 4096
}
/// An LSB-first bit writer.
#[derive(Default)]
struct Bits {
v: Vec<u8>,
n: usize,
}
impl Bits {
fn put(&mut self, x: u64, bits: usize) {
for i in 0..bits {
if self.n.is_multiple_of(8) {
self.v.push(0);
}
if (x >> i) & 1 == 1 {
*self.v.last_mut().unwrap() |= 1 << (self.n % 8);
}
self.n += 1;
}
}
}
/// H5Z-ZFP `cd_values`: a version word and a zfp header for a field of
/// `ztype` (0 int32, 1 int64, 2 float, 3 double) and sizes `n` (fastest
/// first), with `mode` (12 bits, or 64 when `long`).
fn cd_values(ztype: u64, n: &[u64], mode: u64, long: bool) -> Vec<u32> {
let mut b = Bits::default();
for c in b"zfp" {
b.put(*c as u64, 8);
}
b.put(5, 8);
let dims = n.len();
let mut meta = 0u64;
let bits = [48, 24, 16, 12][dims - 1];
for &v in n.iter().rev() {
meta = (meta << bits) + v - 1;
}
meta = (meta << 2) + dims as u64 - 1;
meta = (meta << 2) + ztype;
b.put(meta, 52);
b.put(mode, if long { 64 } else { 12 });
b.v.resize(b.v.len().div_ceil(4) * 4, 0);
let mut cd = vec![0x1001_1111u32];
cd.extend(
b.v.chunks(4)
.map(|w| u32::from_le_bytes(w.try_into().unwrap())),
);
cd
}
fn elem(ztype: u64) -> usize {
if ztype & 1 == 0 { 4 } else { 8 }
}
/// A 1-D field of 2^32 doubles (32 GiB) in a 1-byte chunk: refused for
/// its size, with or without the chunk size known, before anything is
/// allocated.
#[test]
fn huge_fields_are_refused_without_allocating() {
let _g = lock();
for (n, ztype) in [
(vec![1u64 << 32], 3),
(vec![1 << 24, 1 << 24], 3),
(vec![4096; 4], 1),
] {
let cd = cd_values(ztype, &n, 2176, false);
for max_output in [0usize, 1 << 20] {
let (r, peak) = peak_during(|| zfp_decompress(&[0xff], &cd, max_output));
assert!(r.is_err(), "{n:?}: decoded {:?} bytes", r.map(|v| v.len()));
assert!(peak < 4096, "{n:?}: peak {peak} bytes");
}
}
}
/// A field of the chunk's size whose stream is too short for its blocks
/// is refused before the output is allocated.
#[test]
fn short_streams_are_refused_before_allocating() {
let _g = lock();
let n = [1u64 << 18];
let cd = cd_values(2, &n, 2176, false);
let size = (1 << 18) * 4;
let (r, peak) = peak_during(|| zfp_decompress(&[0u8; 100], &cd, size));
assert!(r.is_err());
assert!(peak < 4096, "peak {peak} bytes");
// A stream with a bit per block: all-zero blocks, which decode.
let input = vec![0u8; (1 << 16) / 8];
let out = zfp_decompress(&input, &cd, size).unwrap();
assert_eq!(out, vec![0u8; size]);
}
/// xorshift64*: deterministic, so a failure reproduces.
struct Rng(u64);
impl Rng {
fn next(&mut self) -> u64 {
let mut x = self.0;
x ^= x >> 12;
x ^= x << 25;
x ^= x >> 27;
self.0 = x;
x.wrapping_mul(0x2545_F491_4F6C_DD1D)
}
fn below(&mut self, n: u64) -> u64 {
self.next() % n.max(1)
}
}
/// A mode word: one of the four short forms, or the 64-bit expert form
/// with parameters at and past their edges (maxbits below a float block's
/// exponent, minbits past maxbits, precision 0, minexp at the reversible
/// boundary).
fn mode(rng: &mut Rng) -> (u64, bool) {
match rng.below(6) {
0 => (rng.below(2048), false),
1 => (2048 + rng.below(128), false),
2 => (2176, false),
3 => (2177 + rng.below(4094 - 2177 + 1), false),
_ => {
fn pick(rng: &mut Rng, v: [u64; 6]) -> u64 {
v[rng.below(6) as usize]
}
let r = [rng.below(2000), rng.below(0x8000), rng.below(40)];
let minbits = pick(rng, [0, 1, 11, r[0], r[1], 1]);
let r = [rng.below(0x8000), rng.below(40)];
let maxbits = pick(rng, [minbits, minbits + r[1], 7, 11, r[0], 0x7fff]);
let maxprec = rng.below(0x80);
let r = [rng.below(0x8000), rng.below(400)];
let minexp = pick(
rng,
[
r[0],
16495 - 1074,
16495 - 1075,
16495 + r[1] - 200,
16495 - 1074,
r[0],
],
);
let m = ((((minexp << 7) + maxprec) << 15) + maxbits) << 15;
((m + minbits) << 12 | 0xfff, true)
}
}
}
/// Random fields, modes and streams (random bytes, runs of ones, mostly
/// zeros; of every length), decoded with the chunk size known and not,
/// and with header words that are random too.
#[test]
fn fuzzed_headers_and_streams_stay_within_the_allocation_bound() {
let _g = lock();
let mut rng = Rng(0x2f9);
let mut decoded = 0;
for i in 0..20_000 {
let ztype = rng.below(4);
let dims = 1 + rng.below(4) as usize;
let max = [300, 40, 14, 8][dims - 1];
let n: Vec<u64> = (0..dims).map(|_| 1 + rng.below(max)).collect();
let (m, long) = mode(&mut rng);
let mut cd = cd_values(ztype, &n, m, long);
if rng.below(10) == 0 {
let at = rng.below(cd.len() as u64) as usize;
cd[at] ^= 1 << rng.below(32);
}
if rng.below(20) == 0 {
cd.truncate(rng.below(cd.len() as u64 + 1) as usize);
}
let len = match rng.below(4) {
0 => rng.below(8),
1 => rng.below(300),
_ => rng.below(20_000),
} as usize;
let input: Vec<u8> = match rng.below(3) {
0 => (0..len).map(|_| rng.next() as u8).collect(),
1 => (0..len)
.map(|_| [0, 0xff, rng.next() as u8][rng.below(3) as usize])
.collect(),
_ => (0..len)
.map(|_| [0, 0, 0, 1, 0x80, rng.next() as u8][rng.below(6) as usize])
.collect(),
};
let size = n.iter().product::<u64>() as usize * elem(ztype);
let max_output = if rng.below(4) == 0 { 0 } else { size };
let (r, peak) = peak_during(|| zfp_decompress(&input, &cd, max_output));
if let Ok(out) = &r {
decoded += 1;
if max_output != 0 {
assert_eq!(out.len(), max_output, "iteration {i}");
}
}
assert!(
peak <= bound(max_output, &input),
"iteration {i}: peak {peak} bytes for {n:?} from {} bytes ({:?})",
input.len(),
r.map(|v| v.len())
);
}
// Most inputs are streams zfp decodes without running out.
assert!(decoded > 5_000, "only {decoded} decoded");
}
crates/clawhdf5-py/src/lib.rs
+9 -2
View File
@@ -66,7 +66,9 @@ fn _panic_for_test() -> PyResult<()> {
/// - I/O errors -> `PyIOError`
/// - Format/parsing errors -> `PyValueError`
/// - Missing dataset/path errors -> `PyKeyError`
/// - Other errors -> `PyOSError`
/// - Invalid arguments -> `PyValueError`
/// - Unsupported operations -> `PyNotImplementedError`
/// - Other errors (a locked file, ...) -> `PyOSError`
pub(crate) fn to_py_err(e: clawhdf5_rs::Error) -> PyErr {
use clawhdf5_rs::Error;
match &e {
@@ -79,9 +81,14 @@ pub(crate) fn to_py_err(e: clawhdf5_rs::Error) -> PyErr {
| Error::ZeroCopyNotContiguous
| Error::ZeroCopyNonNativeEndian
| Error::ZeroCopyTypeMismatch { .. }
| Error::ZeroCopyUnaligned { .. } => {
| Error::ZeroCopyUnaligned { .. }
| Error::InvalidArgument(_) => {
PyErr::new::<pyo3::exceptions::PyValueError, _>(e.to_string())
}
Error::Unsupported(_) => {
PyErr::new::<pyo3::exceptions::PyNotImplementedError, _>(e.to_string())
}
_ => PyErr::new::<pyo3::exceptions::PyOSError, _>(e.to_string()),
}
}
crates/clawhdf5-tools/tests/edit_interop.rs
+1437
View File
File diff suppressed because it is too large Load Diff
crates/clawhdf5/Cargo.toml
+9 -2
View File
@@ -19,7 +19,8 @@ rayon = { version = "1", optional = true }
tempfile = { workspace = true }
criterion = { workspace = true }
clawhdf5-io = { path = "../clawhdf5-io", version = "2.7.0", features = ["mmap"] }
clawhdf5-format = { path = "../clawhdf5-format", version = "2.7.0", features = ["parallel", "fast-checksum"] }
clawhdf5-format = { path = "../clawhdf5-format", version = "2.7.0", features = ["parallel", "fast-checksum", "lookup-stats"] }
serde_json = "1"
clawhdf5-filters = { path = "../clawhdf5-filters", version = "2.7.0" }
[[bench]]
@@ -30,6 +31,10 @@ harness = false
name = "parallel_bench"
harness = false
[[bench]]
name = "local_metadata_bench"
harness = false
[features]
default = ["mmap", "provenance", "lzf"]
mmap = ["clawhdf5-io/mmap"]
@@ -48,8 +53,10 @@ bitshuffle = ["clawhdf5-format/bitshuffle"]
bzip2 = ["clawhdf5-format/bzip2"]
blosc = ["clawhdf5-format/blosc"]
blosc2 = ["clawhdf5-format/blosc2"]
# ZFP (32013), read-only.
zfp = ["clawhdf5-format/zfp"]
# Every plugin filter.
plugin-filters = ["lzf", "bitshuffle", "bzip2", "blosc", "blosc2"]
plugin-filters = ["lzf", "bitshuffle", "bzip2", "blosc", "blosc2", "zfp"]
# Dataset::verify_provenance() — recompute a dataset's SHA-256 and compare
# against its stored _provenance_sha256 attribute. On by default, matching
# clawhdf5-format's own default-on `provenance` feature.
crates/clawhdf5/benches/local_metadata_bench.rs
+91
View File
@@ -0,0 +1,91 @@
//! Metadata parsing over an in-memory file: the local fast path that the
//! range-read `Storage` migration must not slow down
//! (`docs/design/range-reads.md`, "Keeping the local fast path").
//!
//! Only the `&[u8]` APIs are used, so the same file builds against older
//! revisions for an A/B comparison. The input is a version-1 (symbol table)
//! file with 400 groups, written by h5py with `libver='earliest'` and a
//! 512-byte user block (`clawhdf5-format/tests/fixtures/v1_groups_400.h5`).
use clawhdf5::{File, Group};
use clawhdf5_format::btree_v1::collect_symbol_table_nodes;
use clawhdf5_format::message_type::MessageType;
use clawhdf5_format::object_header::ObjectHeader;
use clawhdf5_format::superblock::Superblock;
use clawhdf5_format::symbol_table::{SymbolTableMessage, SymbolTableNode};
use criterion::{Criterion, criterion_group, criterion_main};
use std::hint::black_box;
const FIXTURE: &str = concat!(
env!("CARGO_MANIFEST_DIR"),
"/../clawhdf5-format/tests/fixtures/v1_groups_400.h5"
);
fn walk(g: &Group<'_>, objs: &mut usize) {
for name in g.datasets().unwrap_or_default() {
*objs += 1;
if let Ok(ds) = g.dataset(&name) {
let _ = black_box(ds.shape());
let _ = black_box(ds.dtype());
let _ = black_box(ds.attrs());
}
}
for name in g.groups().unwrap_or_default() {
*objs += 1;
if let Ok(sub) = g.group(&name) {
walk(&sub, objs);
}
}
}
fn bench_local_metadata(c: &mut Criterion) {
let bytes = std::fs::read(FIXTURE).unwrap();
let (_, f) = clawhdf5_format::signature::split_user_block(&bytes).unwrap();
let sb = Superblock::parse(f, 0).unwrap();
let (os, ls) = (sb.offset_size, sb.length_size);
let root = ObjectHeader::parse(f, sb.root_group_address as usize, os, ls).unwrap();
let stm = root
.messages
.iter()
.find(|m| m.msg_type == MessageType::SymbolTable)
.map(|m| SymbolTableMessage::parse(&m.data, os).unwrap())
.unwrap();
let nodes = collect_symbol_table_nodes(f, stm.btree_address, os, ls).unwrap();
let headers: Vec<u64> = nodes
.iter()
.flat_map(|&a| SymbolTableNode::parse(f, a as usize, os).unwrap().entries)
.map(|e| e.object_header_address)
.collect();
assert_eq!(headers.len(), 401);
let mut g = c.benchmark_group("local_metadata");
g.bench_function("object_header_parse_x401", |b| {
b.iter(|| {
for &a in &headers {
black_box(ObjectHeader::parse(f, a as usize, os, ls).unwrap());
}
})
});
g.bench_function("snod_parse_all", |b| {
b.iter(|| {
for &a in &nodes {
black_box(SymbolTableNode::parse(f, a as usize, os).unwrap());
}
})
});
g.bench_function("btree_v1_walk", |b| {
b.iter(|| black_box(collect_symbol_table_nodes(f, stm.btree_address, os, ls).unwrap()))
});
let file = File::open(FIXTURE).unwrap();
g.bench_function("facade_list_400_groups", |b| {
b.iter(|| {
let mut n = 0;
walk(&file.root(), &mut n);
assert_eq!(n, 401);
})
});
g.finish();
}
criterion_group!(benches, bench_local_metadata);
criterion_main!(benches);
crates/clawhdf5/src/edit/btree1.rs
+403
View File
@@ -0,0 +1,403 @@
//! Inserting into (and updating) a version-1 B-tree chunk index (node type
//! 1; layout versions 1-3), as libhdf5's `H5B_insert` does:
//!
//! - keys compare lexicographically over the chunk offsets *and* the
//! element-size coordinate (0 in a chunk's own key), so a node's final
//! ("right") key after an append is the last chunk's offsets with the
//! element-size coordinate set to the element size — the smallest key
//! greater than that chunk, which is what libhdf5 writes;
//! - a full node (2K children) splits before the insertion: the right-most
//! node of a level keeps 90% of its children, the left-most 10%, any other
//! half (libhdf5's default split ratios); siblings are relinked;
//! - a full root splits by moving its left half to a new node, so the root
//! keeps its address (the layout message never changes).
//!
//! Version-1 B-tree nodes carry no checksum.
use std::cmp::Ordering;
use crate::edit::image::{Image, get_uint, put_uint, undef};
use crate::error::Error;
#[derive(Debug, Clone, PartialEq, Eq)]
pub(crate) struct Key {
pub(crate) size: u32,
pub(crate) mask: u32,
/// Offsets in every dimension, the element-size one last.
pub(crate) offs: Vec<u64>,
}
fn cmp(a: &[u64], b: &[u64]) -> Ordering {
a.cmp(b)
}
#[derive(Debug, Clone)]
struct Node {
addr: u64,
level: u8,
left: u64,
right: u64,
/// `children.len() + 1` keys.
keys: Vec<Key>,
children: Vec<u64>,
}
pub(crate) struct BTree1 {
root: u64,
/// Children per node at most (2K).
two_k: usize,
ndims: usize,
elem_size: u64,
}
fn bad(why: &str) -> Error {
Error::Format(clawhdf5_format::error::FormatError::ChunkedReadError(
format!("chunk B-tree: {why}"),
))
}
enum Ins {
Done,
/// The node split; the new right sibling and its first key.
Split(Key, u64),
}
impl BTree1 {
/// `k` is the file's chunk B-tree K (children per node are 2K);
/// `ndims` counts the element-size dimension.
pub(crate) fn new(root: u64, k: u16, ndims: usize, elem_size: u64) -> Result<Self, Error> {
if k == 0 || ndims < 2 {
return Err(bad("bad parameters"));
}
Ok(Self {
root,
two_k: 2 * k as usize,
ndims,
elem_size,
})
}
fn key_size(&self) -> usize {
8 + 8 * self.ndims
}
fn node_size(&self, os: u8) -> usize {
let os = os as usize;
8 + 2 * os + (self.two_k + 1) * self.key_size() + self.two_k * os
}
fn read(&self, img: &Image<'_>, addr: u64) -> Result<Node, Error> {
let os = img.os;
let osz = os as usize;
let d = img.read(addr, 8 + 2 * osz)?;
if &d[0..4] != b"TREE" || d[4] != 1 {
return Err(bad("not a chunk B-tree node"));
}
let level = d[5];
let n = u16::from_le_bytes([d[6], d[7]]) as usize;
if n > self.two_k {
return Err(bad("node holds more children than 2K"));
}
let left = get_uint(&d[8..], os);
let right = get_uint(&d[8 + osz..], os);
let ks = self.key_size();
let body = img.read(addr + 8 + 2 * osz as u64, (n + 1) * ks + n * osz)?;
let mut keys = Vec::with_capacity(n + 1);
let mut children = Vec::with_capacity(n);
let mut p = 0;
for i in 0..=n {
let k = &body[p..p + ks];
keys.push(Key {
size: u32::from_le_bytes([k[0], k[1], k[2], k[3]]),
mask: u32::from_le_bytes([k[4], k[5], k[6], k[7]]),
offs: (0..self.ndims)
.map(|d| {
u64::from_le_bytes(k[8 + 8 * d..16 + 8 * d].try_into().unwrap_or([0; 8]))
})
.collect(),
});
p += ks;
if i < n {
children.push(get_uint(&body[p..], os));
p += osz;
}
}
Ok(Node {
addr,
level,
left,
right,
keys,
children,
})
}
fn write(&self, img: &mut Image<'_>, node: &Node) -> Result<(), Error> {
let os = img.os;
let osz = os as usize;
let mut d = vec![0u8; self.node_size(os)];
d[0..4].copy_from_slice(b"TREE");
d[4] = 1;
d[5] = node.level;
d[6..8].copy_from_slice(&(node.children.len() as u16).to_le_bytes());
put_uint(&mut d[8..], node.left, os);
put_uint(&mut d[8 + osz..], node.right, os);
let ks = self.key_size();
let mut p = 8 + 2 * osz;
for (i, k) in node.keys.iter().enumerate() {
d[p..p + 4].copy_from_slice(&k.size.to_le_bytes());
d[p + 4..p + 8].copy_from_slice(&k.mask.to_le_bytes());
for (j, o) in k.offs.iter().enumerate() {
d[p + 8 + 8 * j..p + 16 + 8 * j].copy_from_slice(&o.to_le_bytes());
}
p += ks;
if i < node.children.len() {
put_uint(&mut d[p..], node.children[i], os);
p += osz;
}
}
// Unused key/child slots stay zero, as libhdf5 leaves them.
img.write(node.addr, &d)
}
/// Create a tree holding one chunk; returns it (its root is a new leaf).
pub(crate) fn create(
img: &mut Image<'_>,
k: u16,
ndims: usize,
elem_size: u64,
key: Key,
addr: u64,
) -> Result<Self, Error> {
let mut t = Self::new(0, k, ndims, elem_size)?;
let root = img.alloc(t.node_size(img.os) as u64)?;
t.root = root;
let right = t.right_key_after(&key);
let node = Node {
addr: root,
level: 0,
left: undef(img.os),
right: undef(img.os),
keys: vec![key, right],
children: vec![addr],
};
t.write(img, &node)?;
Ok(t)
}
pub(crate) fn root(&self) -> u64 {
self.root
}
/// The smallest key above chunk `key`: its offsets with the element-size
/// coordinate one element in (what libhdf5 writes as a right key).
fn right_key_after(&self, key: &Key) -> Key {
let mut offs = key.offs.clone();
if let Some(last) = offs.last_mut() {
*last = self.elem_size;
}
Key {
size: 0,
mask: 0,
offs,
}
}
/// Insert chunk `key` at address `addr`, or update it when the tree
/// already has a chunk at those offsets.
pub(crate) fn insert(&mut self, img: &mut Image<'_>, key: Key, addr: u64) -> Result<(), Error> {
if key.offs.len() != self.ndims || key.offs[self.ndims - 1] != 0 {
return Err(bad("bad chunk key"));
}
let root = self.read(img, self.root)?;
if let Ins::Split(mid, right_addr) = self.insert_at(img, root, &key, addr, 64)? {
// The root split: move its (left) half to a new node so the root
// keeps its address, then make the root the parent of both.
let old = self.read(img, self.root)?;
let right = self.read(img, right_addr)?;
let new_left = img.alloc(self.node_size(img.os) as u64)?;
let mut moved = old.clone();
moved.addr = new_left;
self.write(img, &moved)?;
let mut right = right;
right.left = new_left;
self.write(img, &right)?;
let first = old.keys[0].clone();
let last = right
.keys
.last()
.cloned()
.ok_or_else(|| bad("empty node"))?;
let new_root = Node {
addr: self.root,
level: old.level + 1,
left: undef(img.os),
right: undef(img.os),
keys: vec![first, mid, last],
children: vec![new_left, right_addr],
};
self.write(img, &new_root)?;
}
Ok(())
}
fn insert_at(
&self,
img: &mut Image<'_>,
mut node: Node,
key: &Key,
addr: u64,
depth: u8,
) -> Result<Ins, Error> {
if depth == 0 {
return Err(bad("tree too deep"));
}
let n = node.children.len();
if n == 0 {
return Err(bad("empty node"));
}
// The child whose range holds the key: the last i with
// keys[i] <= key (the first child when the key is below them all).
let mut i = node
.keys
.iter()
.take(n)
.rposition(|k| cmp(&k.offs, &key.offs) != Ordering::Greater)
.unwrap_or(0);
if node.level == 0 {
if node.keys[i].offs == key.offs {
node.keys[i].size = key.size;
node.keys[i].mask = key.mask;
node.children[i] = addr;
self.write(img, &node)?;
return Ok(Ins::Done);
}
// Insert after child i unless the key is below every child.
let pos = if cmp(&key.offs, &node.keys[0].offs) == Ordering::Less {
0
} else {
i + 1
};
return self.add_child(img, node, pos, key.clone(), addr);
}
let child = self.read(img, node.children[i])?;
if child.level + 1 != node.level {
return Err(bad("inconsistent node levels"));
}
let ins = self.insert_at(img, child, key, addr, depth - 1)?;
let mut changed = false;
if cmp(&key.offs, &node.keys[0].offs) == Ordering::Less && i == 0 {
node.keys[0] = key.clone();
changed = true;
}
if cmp(&key.offs, &node.keys[n].offs) != Ordering::Less {
node.keys[n] = self.right_key_after(key);
changed = true;
}
match ins {
Ins::Done => {
if changed {
self.write(img, &node)?;
}
Ok(Ins::Done)
}
Ins::Split(mid, right) => {
i += 1;
self.add_child(img, node, i, mid, right)
}
}
}
/// Insert child `addr` with left key `key` at position `pos` of `node`
/// (splitting it first when full), and write what changed.
fn add_child(
&self,
img: &mut Image<'_>,
mut node: Node,
pos: usize,
key: Key,
addr: u64,
) -> Result<Ins, Error> {
let n = node.children.len();
if n < self.two_k {
Self::insert_child(self, &mut node, pos, key, addr);
self.write(img, &node)?;
return Ok(Ins::Done);
}
// Split first (H5B__split): how many children stay left.
let undefined = undef(img.os);
let mut nleft = if node.right == undefined {
(self.two_k as f64 * 0.9) as usize
} else if node.left == undefined {
(self.two_k as f64 * 0.1) as usize
} else {
self.two_k / 2
};
if pos < nleft && nleft == self.two_k {
nleft -= 1;
} else if pos >= nleft && nleft == 0 {
nleft += 1;
}
let right_addr = img.alloc(self.node_size(img.os) as u64)?;
let mut right = Node {
addr: right_addr,
level: node.level,
left: node.addr,
right: node.right,
keys: node.keys[nleft..].to_vec(),
children: node.children[nleft..].to_vec(),
};
if node.right != undefined {
let mut sib = self.read(img, node.right)?;
sib.left = right_addr;
self.write(img, &sib)?;
}
node.keys.truncate(nleft + 1);
node.children.truncate(nleft);
node.right = right_addr;
if pos <= nleft && !(pos == nleft && nleft < n && self.goes_right(&key, &right)) {
self.insert_child(&mut node, pos, key, addr);
} else {
self.insert_child(&mut right, pos - nleft, key, addr);
}
self.write(img, &node)?;
self.write(img, &right)?;
let mid = right.keys[0].clone();
Ok(Ins::Split(mid, right_addr))
}
/// For an insertion exactly at the split point: whether the key belongs
/// to the right half (it is not below the right half's first key).
fn goes_right(&self, key: &Key, right: &Node) -> bool {
cmp(&key.offs, &right.keys[0].offs) != Ordering::Less
}
fn insert_child(&self, node: &mut Node, pos: usize, key: Key, addr: u64) {
let n = node.children.len();
if node.level == 0 {
// A leaf: the new chunk's key goes at `pos`. At the end, the
// node's right key moves up to stay above the new chunk.
if pos == n {
let right = self.right_key_after(&key);
let last = node.keys.len() - 1;
if cmp(&node.keys[last].offs, &right.offs) == Ordering::Less {
node.keys[last] = right;
}
node.keys.insert(n, key);
} else {
node.keys.insert(pos, key);
}
} else {
// An internal node: `key` is the new child's left key, taking
// position `pos` (the child's range starts there).
if pos == n {
// A child split off the last child: its right key is the
// parent's right key already.
node.keys.insert(n, key);
} else {
node.keys.insert(pos, key);
}
}
node.children.insert(pos, addr);
}
}
crates/clawhdf5/src/edit/earray.rs
+512
View File
@@ -0,0 +1,512 @@
//! Setting elements of an Extensible Array chunk index (layout v4, index
//! type 4), creating the index block, super blocks, data blocks and data
//! block pages the element needs, exactly as `H5EA__lookup_elmt` creates
//! them — including the header statistics libhdf5 keeps (blocks created,
//! their bytes, elements realised, one past the highest index set) and the
//! "block offset" each data block records.
use crate::edit::image::{Image, get_uint, put_uint, rechecksum, undef};
use crate::error::Error;
/// A chunk index element.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub(crate) struct Elem {
pub(crate) addr: u64,
pub(crate) size: u64,
pub(crate) mask: u32,
}
/// Encode an element: the address, and for a filtered array the stored
/// size (in `elem_size - os - 4` bytes) and filter mask. `None` is the
/// fill element (undefined address, zero size and mask).
pub(crate) fn encode_elem(
e: Option<Elem>,
filtered: bool,
elem_size: usize,
os: u8,
) -> Result<Vec<u8>, Error> {
let osz = os as usize;
let mut b = vec![0u8; if filtered { elem_size } else { osz }];
let addr = e.map_or(undef(os), |e| e.addr);
put_uint(&mut b, addr, os);
if filtered {
let width = elem_size - osz - 4;
if let Some(e) = e {
if width < 8 && e.size >> (8 * width) != 0 {
return Err(Error::Unsupported(format!(
"filtered chunk of {} bytes does not fit the index's {width}-byte size field",
e.size
)));
}
b[osz..osz + width].copy_from_slice(&e.size.to_le_bytes()[..width]);
b[osz + width..].copy_from_slice(&e.mask.to_le_bytes());
}
}
Ok(b)
}
/// The width libhdf5 gives the stored-size field of a filtered chunk index
/// element for chunks of `chunk_bytes` bytes (`H5D__earray_idx_create`,
/// `H5D__farray_idx_create`): one byte more than the nominal size needs —
/// except under layout message version 5 (HDF5 2.0's own format), which
/// always uses 8 bytes.
pub(crate) fn chunk_size_len(chunk_bytes: u64, layout_version: u8) -> usize {
if layout_version >= 5 {
return 8;
}
let log2 = if chunk_bytes <= 1 {
0
} else {
63 - chunk_bytes.leading_zeros()
};
(1 + ((log2 + 8) / 8) as usize).min(8)
}
/// Creation parameters, in the layout message's order.
#[derive(Debug, Clone, Copy)]
pub(crate) struct EaParams {
pub(crate) max_nelmts_bits: u8,
pub(crate) idx_blk_elmts: u8,
pub(crate) sup_blk_min_data_ptrs: u8,
pub(crate) data_blk_min_elmts: u8,
pub(crate) max_dblk_page_nelmts_bits: u8,
}
#[derive(Debug, Clone, Copy)]
struct Level {
ndblks: u64,
dblk_nelmts: u64,
/// First element of the level, counted after the index block's own.
start_idx: u64,
/// Number of data blocks in the levels before this one.
start_dblk: u64,
}
/// An open Extensible Array.
pub(crate) struct Ea {
hdr: u64,
filtered: bool,
elem_size: usize,
p: EaParams,
/// nsuper_blks, super_blk_size, ndata_blks, data_blk_size,
/// max_idx_set, nelmts.
stats: [u64; 6],
iblock: u64,
levels: Vec<Level>,
/// Levels whose data blocks the index block addresses directly.
direct_levels: usize,
ndblk_addrs: usize,
nsblk_addrs: usize,
dirty_hdr: bool,
/// Checksummed ranges changed by `set` (start -> checksum position),
/// recomputed once by `finish`.
dirty: std::collections::BTreeMap<u64, u64>,
}
fn bad(why: &str) -> Error {
Error::Format(clawhdf5_format::error::FormatError::ChunkedReadError(
format!("Extensible Array: {why}"),
))
}
impl Ea {
fn layout(p: EaParams) -> Result<(Vec<Level>, usize, usize, usize), Error> {
let dmin = u64::from(p.data_blk_min_elmts);
if dmin == 0 || !dmin.is_power_of_two() || p.max_nelmts_bits > 64 {
return Err(bad("bad creation parameters"));
}
let nsblks =
1 + (p.max_nelmts_bits as usize).saturating_sub(dmin.trailing_zeros() as usize);
let mut levels = Vec::with_capacity(nsblks);
let (mut start_idx, mut start_dblk) = (0u64, 0u64);
for u in 0..nsblks {
let ndblks = 1u64.checked_shl((u / 2) as u32).unwrap_or(u64::MAX);
let dblk_nelmts = dmin.checked_shl(u.div_ceil(2) as u32).unwrap_or(u64::MAX);
levels.push(Level {
ndblks,
dblk_nelmts,
start_idx,
start_dblk,
});
start_idx = start_idx.saturating_add(ndblks.saturating_mul(dblk_nelmts));
start_dblk = start_dblk.saturating_add(ndblks);
}
let ndblk_addrs = 2 * (p.sup_blk_min_data_ptrs as usize).saturating_sub(1);
let mut direct_levels = 0;
let mut n = 0u64;
while n < ndblk_addrs as u64 {
if direct_levels >= levels.len() {
return Err(bad("index block holds more data blocks than the array"));
}
n += levels[direct_levels].ndblks;
direct_levels += 1;
}
if n != ndblk_addrs as u64 {
return Err(bad("index block ends mid super block"));
}
Ok((levels, direct_levels, ndblk_addrs, nsblks - direct_levels))
}
fn arr_off_size(&self) -> usize {
(self.p.max_nelmts_bits as usize).div_ceil(8)
}
fn page_nelmts(&self) -> u64 {
1u64.checked_shl(u32::from(self.p.max_dblk_page_nelmts_bits))
.unwrap_or(u64::MAX)
}
fn slot_size(&self, os: u8) -> usize {
if self.filtered {
self.elem_size
} else {
os as usize
}
}
/// Open the array whose header is at `hdr`.
pub(crate) fn open(img: &Image<'_>, hdr: u64) -> Result<Self, Error> {
let os = img.os;
let ls = img.ls as usize;
let size = 12 + 6 * ls + os as usize + 4;
let d = img.read(hdr, size)?;
if &d[0..4] != b"EAHD" || d[4] != 0 {
return Err(bad("bad header"));
}
let filtered = match d[5] {
0 => false,
1 => true,
_ => return Err(bad("unknown client")),
};
let elem_size = d[6] as usize;
if filtered && elem_size < os as usize + 5 {
return Err(bad("element too small"));
}
let p = EaParams {
max_nelmts_bits: d[7],
idx_blk_elmts: d[8],
data_blk_min_elmts: d[9],
sup_blk_min_data_ptrs: d[10],
max_dblk_page_nelmts_bits: d[11],
};
let mut stats = [0u64; 6];
for (k, s) in stats.iter_mut().enumerate() {
*s = get_uint(&d[12 + k * ls..], img.ls);
}
let iblock = get_uint(&d[12 + 6 * ls..], os);
let stored = u32::from_le_bytes(d[size - 4..].try_into().unwrap_or([0; 4]));
if clawhdf5_format::checksum::jenkins_lookup3(&d[..size - 4]) != stored {
return Err(bad("header checksum mismatch"));
}
let (levels, direct_levels, ndblk_addrs, nsblk_addrs) = Self::layout(p)?;
Ok(Self {
hdr,
filtered,
elem_size,
p,
stats,
iblock,
levels,
direct_levels,
ndblk_addrs,
nsblk_addrs,
dirty_hdr: false,
dirty: Default::default(),
})
}
/// Create an empty array (header only; the index block comes with the
/// first element) and return it.
pub(crate) fn create(
img: &mut Image<'_>,
p: EaParams,
filtered: bool,
chunk_bytes: u64,
layout_version: u8,
) -> Result<Self, Error> {
let os = img.os;
let elem_size = if filtered {
os as usize + chunk_size_len(chunk_bytes, layout_version) + 4
} else {
os as usize
};
let size = 12 + 6 * img.ls as usize + os as usize + 4;
let hdr = img.alloc(size as u64)?;
let (levels, direct_levels, ndblk_addrs, nsblk_addrs) = Self::layout(p)?;
let mut ea = Self {
hdr,
filtered,
elem_size,
p,
stats: [0; 6],
iblock: undef(os),
levels,
direct_levels,
ndblk_addrs,
nsblk_addrs,
dirty_hdr: true,
dirty: Default::default(),
};
ea.write_header(img)?;
Ok(ea)
}
pub(crate) fn header_address(&self) -> u64 {
self.hdr
}
fn write_header(&mut self, img: &mut Image<'_>) -> Result<(), Error> {
let os = img.os;
let ls = img.ls as usize;
let size = 12 + 6 * ls + os as usize + 4;
let mut d = vec![0u8; size];
d[0..4].copy_from_slice(b"EAHD");
d[4] = 0;
d[5] = u8::from(self.filtered);
d[6] = self.elem_size as u8;
d[7] = self.p.max_nelmts_bits;
d[8] = self.p.idx_blk_elmts;
d[9] = self.p.data_blk_min_elmts;
d[10] = self.p.sup_blk_min_data_ptrs;
d[11] = self.p.max_dblk_page_nelmts_bits;
for (k, s) in self.stats.iter().enumerate() {
put_uint(&mut d[12 + k * ls..], *s, img.ls);
}
put_uint(&mut d[12 + 6 * ls..], self.iblock, os);
let sum = clawhdf5_format::checksum::jenkins_lookup3(&d[..size - 4]);
d[size - 4..].copy_from_slice(&sum.to_le_bytes());
img.write(self.hdr, &d)?;
self.dirty_hdr = false;
Ok(())
}
/// Recompute the checksums of the blocks `set` changed; store changed
/// header statistics.
pub(crate) fn finish(&mut self, img: &mut Image<'_>) -> Result<(), Error> {
for (start, end) in std::mem::take(&mut self.dirty) {
rechecksum(img, start, end)?;
}
if self.dirty_hdr {
self.write_header(img)?;
}
Ok(())
}
fn fill_elems(&self, n: u64, os: u8) -> Result<Vec<u8>, Error> {
let one = encode_elem(None, self.filtered, self.elem_size, os)?;
let n = usize::try_from(n).map_err(|_| bad("block too large"))?;
Ok(one.repeat(n))
}
fn iblock_prefix(&self, os: u8) -> u64 {
6 + u64::from(os)
}
fn iblock_len(&self, os: u8) -> u64 {
let osz = os as u64;
self.iblock_prefix(os)
+ u64::from(self.p.idx_blk_elmts) * self.slot_size(os) as u64
+ (self.ndblk_addrs + self.nsblk_addrs) as u64 * osz
}
fn create_iblock(&mut self, img: &mut Image<'_>) -> Result<(), Error> {
let os = img.os;
let len = self.iblock_len(os);
let addr = img.alloc(len + 4)?;
let mut d = Vec::with_capacity(len as usize + 4);
d.extend_from_slice(b"EAIB");
d.push(0);
d.push(u8::from(self.filtered));
let mut a = vec![0u8; os as usize];
put_uint(&mut a, self.hdr, os);
d.extend_from_slice(&a);
d.extend_from_slice(&self.fill_elems(u64::from(self.p.idx_blk_elmts), os)?);
let u = undef(os).to_le_bytes();
for _ in 0..self.ndblk_addrs + self.nsblk_addrs {
d.extend_from_slice(&u[..os as usize]);
}
let sum = clawhdf5_format::checksum::jenkins_lookup3(&d);
d.extend_from_slice(&sum.to_le_bytes());
img.write(addr, &d)?;
self.iblock = addr;
self.stats[5] += u64::from(self.p.idx_blk_elmts);
self.dirty_hdr = true;
Ok(())
}
fn block_prefix(&self, sig: &[u8; 4], off: u64, os: u8) -> Vec<u8> {
let mut d = Vec::new();
d.extend_from_slice(sig);
d.push(0);
d.push(u8::from(self.filtered));
let mut a = vec![0u8; os as usize];
put_uint(&mut a, self.hdr, os);
d.extend_from_slice(&a);
d.extend_from_slice(&off.to_le_bytes()[..self.arr_off_size()]);
d
}
fn dblk_prefix_len(&self, os: u8) -> u64 {
6 + u64::from(os) + self.arr_off_size() as u64
}
/// Create a data block of `nelmts` elements whose recorded block offset
/// is `off`; returns its address.
fn create_dblock(&mut self, img: &mut Image<'_>, nelmts: u64, off: u64) -> Result<u64, Error> {
let os = img.os;
let es = self.slot_size(os) as u64;
let page = self.page_nelmts();
let prefix = self.block_prefix(b"EADB", off, os);
let (size, body) = if nelmts > page {
// Paged: only the prefix (and its checksum) is written now; each
// page is written when an element in it is first set.
let npages = nelmts / page;
let size = prefix.len() as u64 + 4 + npages * (page * es + 4);
let mut d = prefix;
let sum = clawhdf5_format::checksum::jenkins_lookup3(&d);
d.extend_from_slice(&sum.to_le_bytes());
(size, d)
} else {
let mut d = prefix;
d.extend_from_slice(&self.fill_elems(nelmts, os)?);
let sum = clawhdf5_format::checksum::jenkins_lookup3(&d);
d.extend_from_slice(&sum.to_le_bytes());
(d.len() as u64, d)
};
let addr = img.alloc(size)?;
img.write(addr, &body)?;
self.stats[2] += 1;
self.stats[3] += size;
self.stats[5] += nelmts;
self.dirty_hdr = true;
Ok(addr)
}
/// Set element `idx` to `e`.
pub(crate) fn set(&mut self, img: &mut Image<'_>, idx: u64, e: Elem) -> Result<(), Error> {
let os = img.os;
let osz = u64::from(os);
let es = self.slot_size(os) as u64;
let enc = encode_elem(Some(e), self.filtered, self.elem_size, os)?;
if self.iblock == undef(os) {
self.create_iblock(img)?;
}
let ib = self.iblock;
let ib_len = self.iblock_len(os);
let idx_blk = u64::from(self.p.idx_blk_elmts);
if idx < idx_blk {
img.write(ib + self.iblock_prefix(os) + idx * es, &enc)?;
self.dirty.insert(ib, ib + ib_len);
} else {
let rel = idx - idx_blk;
let u = self
.levels
.iter()
.position(|l| {
rel < l
.start_idx
.saturating_add(l.ndblks.saturating_mul(l.dblk_nelmts))
})
.ok_or_else(|| bad("index beyond the array's maximum"))?;
let l = self.levels[u];
let dblks_at = ib + self.iblock_prefix(os) + idx_blk * es;
if u < self.direct_levels {
if l.dblk_nelmts > self.page_nelmts() {
return Err(Error::Unsupported(
"Extensible Array index block addressing a paged data block".into(),
));
}
let local = (rel - l.start_idx) / l.dblk_nelmts;
let dblk_idx = l.start_dblk + local;
let slot = dblks_at + dblk_idx * osz;
let mut addr = get_uint(&img.read(slot, os as usize)?, os);
if addr == undef(os) {
// libhdf5 records start_idx + (global data block index)
// * nelmts here (H5EA__lookup_elmt), not the block's
// own first element; kept for byte-for-byte parity.
let off = l.start_idx + dblk_idx * l.dblk_nelmts;
addr = self.create_dblock(img, l.dblk_nelmts, off)?;
let mut a = vec![0u8; os as usize];
put_uint(&mut a, addr, os);
img.write(slot, &a)?;
self.dirty.insert(ib, ib + ib_len);
}
let within = (rel - l.start_idx) % l.dblk_nelmts;
let at = addr + self.dblk_prefix_len(os) + within * es;
img.write(at, &enc)?;
self.dirty
.insert(addr, addr + self.dblk_prefix_len(os) + l.dblk_nelmts * es);
} else {
let s = (u - self.direct_levels) as u64;
let sslot = dblks_at + self.ndblk_addrs as u64 * osz + s * osz;
let page = self.page_nelmts();
let npages = if l.dblk_nelmts > page {
l.dblk_nelmts / page
} else {
0
};
let bitmap_len = npages.div_ceil(8) * l.ndblks;
let sb_prefix = self.dblk_prefix_len(os);
let sb_len = sb_prefix + bitmap_len + l.ndblks * osz;
let mut sb = get_uint(&img.read(sslot, os as usize)?, os);
if sb == undef(os) {
let mut d = self.block_prefix(b"EASB", l.start_idx, os);
d.resize(d.len() + bitmap_len as usize, 0);
let u8s = undef(os).to_le_bytes();
for _ in 0..l.ndblks {
d.extend_from_slice(&u8s[..os as usize]);
}
let sum = clawhdf5_format::checksum::jenkins_lookup3(&d);
d.extend_from_slice(&sum.to_le_bytes());
sb = img.alloc(d.len() as u64)?;
img.write(sb, &d)?;
self.stats[0] += 1;
self.stats[1] += d.len() as u64;
self.dirty_hdr = true;
let mut a = vec![0u8; os as usize];
put_uint(&mut a, sb, os);
img.write(sslot, &a)?;
self.dirty.insert(ib, ib + ib_len);
}
let local = (rel - l.start_idx) / l.dblk_nelmts;
let dslot = sb + sb_prefix + bitmap_len + local * osz;
let mut addr = get_uint(&img.read(dslot, os as usize)?, os);
if addr == undef(os) {
let off = l.start_idx + local * l.dblk_nelmts;
addr = self.create_dblock(img, l.dblk_nelmts, off)?;
let mut a = vec![0u8; os as usize];
put_uint(&mut a, addr, os);
img.write(dslot, &a)?;
self.dirty.insert(sb, sb + sb_len);
}
let within = (rel - l.start_idx) % l.dblk_nelmts;
let dprefix = self.dblk_prefix_len(os);
if npages == 0 {
img.write(addr + dprefix + within * es, &enc)?;
self.dirty.insert(addr, addr + dprefix + l.dblk_nelmts * es);
} else {
let pg = within / page;
let page_at = addr + dprefix + 4 + pg * (page * es + 4);
let bit = local * npages + pg;
let bpos = sb + sb_prefix + bit / 8;
let mut byte = img.read(bpos, 1)?[0];
let mask = 0x80u8 >> (bit % 8);
if byte & mask == 0 {
let fill = self.fill_elems(page, os)?;
img.write(page_at, &fill)?;
byte |= mask;
img.write(bpos, &[byte])?;
self.dirty.insert(sb, sb + sb_len);
}
img.write(page_at + (within % page) * es, &enc)?;
self.dirty.insert(page_at, page_at + page * es);
}
}
}
if idx + 1 > self.stats[4] {
self.stats[4] = idx + 1;
self.dirty_hdr = true;
}
Ok(())
}
}
crates/clawhdf5/src/edit/farray.rs
+185
View File
@@ -0,0 +1,185 @@
//! Setting elements of a Fixed Array chunk index (layout v4, index type 3),
//! creating the array (header and data block) when the dataset has none
//! yet, and a data block page when an element in it is first set.
use crate::edit::earray::{Elem, chunk_size_len, encode_elem};
use crate::edit::image::{Image, get_uint, put_uint, rechecksum, undef};
use crate::error::Error;
pub(crate) struct Fa {
filtered: bool,
elem_size: usize,
page_bits: u8,
nelmts: u64,
dblk: u64,
/// Checksummed ranges changed by `set`, recomputed by `finish`.
dirty: std::collections::BTreeMap<u64, u64>,
}
fn bad(why: &str) -> Error {
Error::Format(clawhdf5_format::error::FormatError::ChunkedReadError(
format!("Fixed Array: {why}"),
))
}
impl Fa {
fn slot(&self, os: u8) -> u64 {
if self.filtered {
self.elem_size as u64
} else {
u64::from(os)
}
}
fn page(&self) -> u64 {
1u64.checked_shl(u32::from(self.page_bits))
.unwrap_or(u64::MAX)
}
/// Open the array whose header is at `hdr`.
pub(crate) fn open(img: &Image<'_>, hdr: u64) -> Result<Self, Error> {
let os = img.os;
let size = 8 + img.ls as usize + os as usize + 4;
let d = img.read(hdr, size)?;
if &d[0..4] != b"FAHD" || d[4] != 0 {
return Err(bad("bad header"));
}
let filtered = match d[5] {
0 => false,
1 => true,
_ => return Err(bad("unknown client")),
};
let stored = u32::from_le_bytes(d[size - 4..].try_into().unwrap_or([0; 4]));
if clawhdf5_format::checksum::jenkins_lookup3(&d[..size - 4]) != stored {
return Err(bad("header checksum mismatch"));
}
let fa = Self {
filtered,
elem_size: d[6] as usize,
page_bits: d[7],
nelmts: get_uint(&d[8..], img.ls),
dblk: get_uint(&d[8 + img.ls as usize..], os),
dirty: Default::default(),
};
if fa.filtered && fa.elem_size < os as usize + 5 {
return Err(bad("element too small"));
}
if fa.page_bits >= 64 || fa.dblk == undef(os) {
return Err(bad("bad header fields"));
}
Ok(fa)
}
/// Create an array of `nelmts` fill elements; returns it and its
/// header address.
pub(crate) fn create(
img: &mut Image<'_>,
nelmts: u64,
page_bits: u8,
filtered: bool,
chunk_bytes: u64,
layout_version: u8,
) -> Result<(Self, u64), Error> {
let os = img.os;
let osz = os as usize;
let elem_size = if filtered {
osz + chunk_size_len(chunk_bytes, layout_version) + 4
} else {
osz
};
let mut fa = Self {
filtered,
elem_size,
page_bits,
nelmts,
dblk: 0,
dirty: Default::default(),
};
let hsize = 8 + img.ls as usize + osz + 4;
let hdr = img.alloc(hsize as u64)?;
// Data block.
let fill = encode_elem(None, filtered, elem_size, os)?;
let mut d = Vec::new();
d.extend_from_slice(b"FADB");
d.push(0);
d.push(u8::from(filtered));
let mut a = vec![0u8; osz];
put_uint(&mut a, hdr, os);
d.extend_from_slice(&a);
let page = fa.page();
let n = usize::try_from(nelmts).map_err(|_| bad("too many elements"))?;
let total = if nelmts > page {
let npages = nelmts.div_ceil(page);
d.resize(d.len() + npages.div_ceil(8) as usize, 0);
let sum = clawhdf5_format::checksum::jenkins_lookup3(&d);
d.extend_from_slice(&sum.to_le_bytes());
// Pages are written when first used; their space is reserved.
d.len() as u64 + nelmts * fa.slot(os) + npages * 4
} else {
d.extend_from_slice(&fill.repeat(n));
let sum = clawhdf5_format::checksum::jenkins_lookup3(&d);
d.extend_from_slice(&sum.to_le_bytes());
d.len() as u64
};
let dblk = img.alloc(total)?;
img.write(dblk, &d)?;
fa.dblk = dblk;
let mut h = vec![0u8; hsize];
h[0..4].copy_from_slice(b"FAHD");
h[5] = u8::from(filtered);
h[6] = elem_size as u8;
h[7] = page_bits;
put_uint(&mut h[8..], nelmts, img.ls);
put_uint(&mut h[8 + img.ls as usize..], dblk, os);
let sum = clawhdf5_format::checksum::jenkins_lookup3(&h[..hsize - 4]);
h[hsize - 4..].copy_from_slice(&sum.to_le_bytes());
img.write(hdr, &h)?;
Ok((fa, hdr))
}
/// Set element `idx` to `e`.
pub(crate) fn set(&mut self, img: &mut Image<'_>, idx: u64, e: Elem) -> Result<(), Error> {
let os = img.os;
if idx >= self.nelmts {
return Err(bad("index beyond the array"));
}
let enc = encode_elem(Some(e), self.filtered, self.elem_size, os)?;
let es = self.slot(os);
let prefix = 6 + u64::from(os);
let page = self.page();
if self.nelmts <= page {
img.write(self.dblk + prefix + idx * es, &enc)?;
self.dirty
.insert(self.dblk, self.dblk + prefix + self.nelmts * es);
return Ok(());
}
let npages = self.nelmts.div_ceil(page);
let bitmap_len = npages.div_ceil(8);
let pages_at = self.dblk + prefix + bitmap_len + 4;
let p = idx / page;
let count = page.min(self.nelmts - p * page);
let page_at = pages_at + p * (page * es + 4);
let bpos = self.dblk + prefix + p / 8;
let mut byte = img.read(bpos, 1)?[0];
let mask = 0x80u8 >> (p % 8);
if byte & mask == 0 {
let fill = encode_elem(None, self.filtered, self.elem_size, os)?;
img.write(page_at, &fill.repeat(count as usize))?;
byte |= mask;
img.write(bpos, &[byte])?;
self.dirty
.insert(self.dblk, self.dblk + prefix + bitmap_len);
}
img.write(page_at + (idx % page) * es, &enc)?;
self.dirty.insert(page_at, page_at + count * es);
Ok(())
}
/// Recompute the checksums of the blocks and pages `set` changed.
pub(crate) fn finish(&mut self, img: &mut Image<'_>) -> Result<(), Error> {
for (start, end) in std::mem::take(&mut self.dirty) {
rechecksum(img, start, end)?;
}
Ok(())
}
}
crates/clawhdf5/src/edit/image.rs
+345
View File
@@ -0,0 +1,345 @@
//! The file as one edit sees it: the bytes on disk plus the edit's pending
//! writes, and an allocator that hands out space at the end of the file.
//!
//! An edit never writes to the file while it is being planned. Every change
//! is recorded here first (reads see them), so an edit that fails half-way —
//! a filter that cannot encode, a chunk index this code does not handle —
//! leaves the file exactly as it was. [`Image::into_plan`] then detaches the
//! changes from the bytes they were planned over, and [`Plan::commit`]
//! writes them in an order that keeps the old metadata valid for as long as
//! possible (see there).
//!
//! **Invariant:** the base bytes an image reads are the reader's view of the
//! file — a memory map when the `mmap` feature is on. Nothing may write the
//! file while that view is alive: a write through another descriptor would
//! change memory behind a live `&[u8]`, which Rust's aliasing rules forbid.
//! So a [`Plan`] owns everything it writes and borrows nothing, and the
//! editor drops the reader (unmapping the file) before it commits.
use std::collections::BTreeMap;
use std::io::{Seek, SeekFrom, Write};
use crate::error::Error;
/// Pending writes over the file's bytes, addressed as HDF5 addresses
/// (relative to the superblock).
pub(crate) struct Image<'a> {
/// The file from the superblock to its recorded end of allocation.
base: &'a [u8],
/// Pending writes: start address -> bytes. Never overlapping.
patches: BTreeMap<u64, Vec<u8>>,
/// End of allocated space (grows with [`Self::alloc`]).
eoa: u64,
/// The end of allocated space when the edit started.
old_eoa: u64,
/// Width of addresses and lengths in the file.
pub(crate) os: u8,
pub(crate) ls: u8,
}
impl<'a> Image<'a> {
pub(crate) fn new(base: &'a [u8], os: u8, ls: u8) -> Self {
let eoa = base.len() as u64;
Self {
base,
patches: BTreeMap::new(),
eoa,
old_eoa: eoa,
os,
ls,
}
}
pub(crate) fn eoa(&self) -> u64 {
self.eoa
}
pub(crate) fn old_eoa(&self) -> u64 {
self.old_eoa
}
/// Whether the edit changes anything.
pub(crate) fn is_dirty(&self) -> bool {
!self.patches.is_empty() || self.eoa != self.old_eoa
}
/// Allocate `size` bytes at the end of the file. The space reads as
/// zeros until written. Nothing is ever freed: space an edit stops
/// using (a relocated chunk, say) is leaked, as there is no free-space
/// manager.
pub(crate) fn alloc(&mut self, size: u64) -> Result<u64, Error> {
let addr = self.eoa;
let end = addr
.checked_add(size)
.filter(|&e| self.os >= 8 || e < (1u64 << (8 * u32::from(self.os))) - 1)
.ok_or_else(|| Error::Unsupported("file would exceed its address size".into()))?;
self.eoa = end;
Ok(addr)
}
/// If `[addr, addr + old_len)` is the last allocated space, grow it to
/// `new_len` bytes (a structure at the end of the file can grow where
/// it is) and return true.
pub(crate) fn grow_tail(
&mut self,
addr: u64,
old_len: u64,
new_len: u64,
) -> Result<bool, Error> {
if addr.checked_add(old_len) != Some(self.eoa) || new_len < old_len {
return Ok(false);
}
let old_end = self.eoa;
self.eoa = addr;
if let Err(e) = self.alloc(new_len) {
self.eoa = old_end;
return Err(e);
}
Ok(true)
}
/// `len` bytes at `addr`, with the pending writes applied.
pub(crate) fn read(&self, addr: u64, len: usize) -> Result<Vec<u8>, Error> {
let end = addr
.checked_add(len as u64)
.filter(|&e| e <= self.eoa)
.ok_or_else(|| {
Error::Format(clawhdf5_format::error::FormatError::UnexpectedEof {
expected: addr.saturating_add(len as u64) as usize,
available: self.eoa as usize,
})
})?;
let mut out = vec![0u8; len];
let base_len = self.base.len() as u64;
if addr < base_len {
let b_end = end.min(base_len);
out[..(b_end - addr) as usize]
.copy_from_slice(&self.base[addr as usize..b_end as usize]);
}
// Patches overlapping [addr, end): the last one starting before
// `end`, walking back while they still reach `addr`.
for (&p_start, bytes) in self.patches.range(..end).rev() {
let p_end = p_start + bytes.len() as u64;
if p_end <= addr {
break;
}
let lo = p_start.max(addr);
let hi = p_end.min(end);
out[(lo - addr) as usize..(hi - addr) as usize]
.copy_from_slice(&bytes[(lo - p_start) as usize..(hi - p_start) as usize]);
}
Ok(out)
}
/// Record a write of `bytes` at `addr` (inside allocated space).
pub(crate) fn write(&mut self, addr: u64, bytes: &[u8]) -> Result<(), Error> {
if bytes.is_empty() {
return Ok(());
}
let end = addr
.checked_add(bytes.len() as u64)
.filter(|&e| e <= self.eoa)
.ok_or_else(|| Error::Unsupported("write past the end of allocated space".into()))?;
// Fast path: inside, or extending, the one patch that starts at or
// before `addr` and reaches it (sequential writes into a block, and
// chunks allocated back to back, stay linear).
if let Some((&p_start, p)) = self.patches.range_mut(..=addr).next_back()
&& p_start + p.len() as u64 >= addr
&& self
.patches
.range(addr + 1..end.max(addr + 1))
.next()
.is_none()
{
let p = self.patches.get_mut(&p_start).expect("found above");
let off = (addr - p_start) as usize;
if off + bytes.len() > p.len() {
p.resize(off + bytes.len(), 0);
}
p[off..off + bytes.len()].copy_from_slice(bytes);
return Ok(());
}
// Patches that overlap or touch [addr, end) merge into one.
let touching: Vec<u64> = self
.patches
.range(..=end)
.rev()
.take_while(|(s, b)| **s + b.len() as u64 >= addr)
.map(|(s, _)| *s)
.collect();
if touching.is_empty() {
self.patches.insert(addr, bytes.to_vec());
return Ok(());
}
let lo = touching.iter().copied().min().map_or(addr, |s| s.min(addr));
let hi = touching
.iter()
.map(|s| s + self.patches[s].len() as u64)
.max()
.map_or(end, |e| e.max(end));
let mut merged = self.read(lo, (hi - lo) as usize)?;
merged[(addr - lo) as usize..(end - lo) as usize].copy_from_slice(bytes);
for s in touching {
self.patches.remove(&s);
}
self.patches.insert(lo, merged);
Ok(())
}
/// The edit's writes, detached from the base bytes (see the module's
/// invariant: the reader that owns them can then be dropped before
/// anything is written).
pub(crate) fn into_plan(self) -> Plan {
Plan {
patches: self.patches,
eoa: self.eoa,
old_eoa: self.old_eoa,
}
}
}
/// The writes of a planned edit, owning all of their bytes.
pub(crate) struct Plan {
patches: BTreeMap<u64, Vec<u8>>,
eoa: u64,
old_eoa: u64,
}
impl Plan {
/// Write the edit to `file`, whose superblock is at `user_block`.
///
/// Order: first everything in newly allocated space (new chunks, new
/// index blocks, relocated structures), which nothing on disk refers to
/// yet, then a sync; then the changes to existing bytes — raw data
/// overwritten in place and the metadata that links the new space in
/// (superblock end of file, chunk index entries, object header
/// messages) — then a sync. A crash during the first phase leaves the
/// file as it was (plus unreferenced bytes past its end of file); a
/// crash during the second can leave it inconsistent, as with libhdf5
/// without SWMR: there is no journal.
pub(crate) fn commit(self, file: &mut std::fs::File, user_block: u64) -> Result<(), Error> {
let old_eoa = self.old_eoa;
let mut in_place: Vec<(u64, &[u8])> = Vec::new();
for (&addr, bytes) in &self.patches {
// A patch may run from existing bytes into new space (writes
// merge); its new part goes with the new space.
let split = old_eoa.saturating_sub(addr).min(bytes.len() as u64) as usize;
let (old, new) = bytes.split_at(split);
if !new.is_empty() {
write_at(file, user_block + addr + split as u64, new)?;
}
if !old.is_empty() {
in_place.push((addr, old));
}
}
if self.eoa > old_eoa {
let want = user_block + self.eoa;
if file.metadata()?.len() < want {
file.set_len(want)?;
}
}
file.sync_data()?;
for (addr, bytes) in in_place {
write_at(file, user_block + addr, bytes)?;
}
file.sync_all()?;
Ok(())
}
}
fn write_at(file: &mut std::fs::File, pos: u64, bytes: &[u8]) -> Result<(), Error> {
file.seek(SeekFrom::Start(pos))?;
file.write_all(bytes)?;
Ok(())
}
/// Little-endian encode of `v` in `width` bytes.
pub(crate) fn put_uint(buf: &mut [u8], v: u64, width: u8) {
let w = width as usize;
buf[..w].copy_from_slice(&v.to_le_bytes()[..w]);
}
/// Little-endian decode of `width` bytes.
pub(crate) fn get_uint(buf: &[u8], width: u8) -> u64 {
let mut b = [0u8; 8];
b[..width as usize].copy_from_slice(&buf[..width as usize]);
u64::from_le_bytes(b)
}
/// The undefined address for `os`-byte addresses.
pub(crate) fn undef(os: u8) -> u64 {
if os >= 8 {
u64::MAX
} else {
(1u64 << (8 * u32::from(os))) - 1
}
}
/// Recompute the Jenkins checksum over `[start, end)` and store it at `end`.
pub(crate) fn rechecksum(img: &mut Image<'_>, start: u64, end: u64) -> Result<(), Error> {
let bytes = img.read(start, (end - start) as usize)?;
let sum = clawhdf5_format::checksum::jenkins_lookup3(&bytes);
img.write(end, &sum.to_le_bytes())
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn reads_see_writes_and_merges() {
let base = vec![1u8; 32];
let mut img = Image::new(&base, 8, 8);
img.write(4, &[9, 9]).unwrap();
img.write(8, &[7]).unwrap();
img.write(5, &[3, 3, 3]).unwrap(); // extends the first up to the second
assert_eq!(img.read(3, 7).unwrap(), vec![1, 9, 3, 3, 3, 7, 1]);
img.write(2, &[4, 4, 4, 4, 4, 4, 4, 4]).unwrap(); // covers both: merged
assert_eq!(img.patches.len(), 1);
assert_eq!(img.read(1, 10).unwrap(), vec![1, 4, 4, 4, 4, 4, 4, 4, 4, 1]);
let a = img.alloc(10).unwrap();
assert_eq!(a, 32);
assert_eq!(img.read(30, 4).unwrap(), vec![1, 1, 0, 0]);
img.write(40, &[5]).unwrap();
assert_eq!(img.read(39, 3).unwrap(), vec![0, 5, 0]);
assert!(img.write(42, &[1]).is_err());
}
/// Random reads and writes against a flat copy of the bytes.
#[test]
fn matches_a_flat_model() {
let base: Vec<u8> = (0..200u32).map(|i| i as u8).collect();
let mut img = Image::new(&base, 8, 8);
img.alloc(100).unwrap();
let mut flat = base.clone();
flat.resize(300, 0);
let mut x = 12345u64;
let mut next = |n: u64| {
x ^= x << 13;
x ^= x >> 7;
x ^= x << 17;
x % n
};
for step in 0..5000 {
let at = next(300);
let len = 1 + next(20).min(299 - at);
if step % 3 == 0 {
assert_eq!(
img.read(at, len as usize).unwrap(),
flat[at as usize..(at + len) as usize]
);
} else {
let bytes: Vec<u8> = (0..len).map(|_| next(256) as u8).collect();
img.write(at, &bytes).unwrap();
flat[at as usize..(at + len) as usize].copy_from_slice(&bytes);
}
}
assert_eq!(img.read(0, 300).unwrap(), flat);
// Patches never overlap.
let mut end = 0;
for (s, b) in &img.patches {
assert!(*s >= end);
end = s + b.len() as u64;
}
}
}
crates/clawhdf5/src/edit/mod.rs
+1341
View File
File diff suppressed because it is too large Load Diff
crates/clawhdf5/src/edit/ohdr.rs
+509
View File
@@ -0,0 +1,509 @@
//! An object header as an edit sees it: every chunk and every message
//! (NIL and continuation messages included) with its position in the file,
//! so single messages can be changed in place, deleted (turned into NIL
//! messages) and added (into a NIL message big enough, or into a new
//! continuation chunk at the end of the file).
//!
//! Version-2 chunks carry a checksum, recomputed by [`Header::finish`] for
//! every chunk the edit touched; a version-1 header's message count is kept
//! up to date there too.
use std::collections::BTreeSet;
use crate::edit::image::{Image, get_uint, put_uint, rechecksum};
use crate::error::Error;
use clawhdf5_format::error::FormatError;
pub(crate) const MSG_NIL: u16 = 0x00;
pub(crate) const MSG_CONTINUATION: u16 = 0x10;
pub(crate) const MSG_ATTRIBUTE: u16 = 0x0C;
/// One message: where its header and body are, and what it is.
#[derive(Debug, Clone)]
pub(crate) struct Msg {
pub(crate) chunk: usize,
pub(crate) hdr_pos: u64,
pub(crate) data_pos: u64,
pub(crate) size: usize,
pub(crate) mtype: u16,
pub(crate) flags: u8,
pub(crate) corder: Option<u16>,
}
/// One chunk of the header.
#[derive(Debug, Clone)]
struct Chunk {
/// Where the checksummed bytes start (the `OHDR`/`OCHK` signature).
start: u64,
/// Where the checksum is (version 2 only).
checksum_at: Option<u64>,
/// Where the chunk's messages end.
end: u64,
/// Bytes at the end too few for a message header (version 2 only).
/// libhdf5 refuses a chunk with both a gap and a NIL message, so a
/// NIL message made in such a chunk must absorb the gap.
gap: u64,
}
#[derive(Debug)]
pub(crate) struct Header {
pub(crate) addr: u64,
pub(crate) version: u8,
/// Version-2 header flags (0 for version 1).
pub(crate) flags: u8,
chunks: Vec<Chunk>,
pub(crate) msgs: Vec<Msg>,
dirty: BTreeSet<usize>,
/// Messages added (a split NIL message, a new chunk's messages), for a
/// version-1 header's message count.
added: usize,
}
const MAX_CHUNKS: usize = 1024;
fn corrupt(why: &'static str) -> Error {
Error::Format(FormatError::InvalidObjectHeader(why))
}
impl Header {
/// Locate every chunk and message of the header at `addr`.
pub(crate) fn load(img: &Image<'_>, addr: u64) -> Result<Self, Error> {
let sig = img.read(addr, 4)?;
let mut h = Header {
addr,
version: 0,
flags: 0,
chunks: Vec::new(),
msgs: Vec::new(),
dirty: BTreeSet::new(),
added: 0,
};
let mut pending: Vec<(u64, u64)> = Vec::new();
if sig == b"OHDR" {
let pre = img.read(addr, 6)?;
if pre[4] != 2 {
return Err(corrupt("bad object header version"));
}
h.version = 2;
h.flags = pre[5];
let mut pos = addr + 6;
if h.flags & 0x20 != 0 {
pos += 16;
}
if h.flags & 0x10 != 0 {
pos += 4;
}
let w = 1u8 << (h.flags & 0x03);
let size = get_uint(&img.read(pos, w as usize)?, w);
pos += u64::from(w);
h.chunks.push(Chunk {
start: addr,
checksum_at: Some(pos + size),
end: pos + size,
gap: 0,
});
h.scan(img, 0, pos, pos + size, &mut pending)?;
} else {
let pre = img.read(addr, 16)?;
if pre[0] != 1 {
return Err(corrupt("bad object header version"));
}
h.version = 1;
let size = u64::from(u32::from_le_bytes([pre[8], pre[9], pre[10], pre[11]]));
h.chunks.push(Chunk {
start: addr,
checksum_at: None,
end: addr + 16 + size,
gap: 0,
});
h.scan(img, 0, addr + 16, addr + 16 + size, &mut pending)?;
}
while let Some((caddr, clen)) = pending.pop() {
if h.chunks.len() >= MAX_CHUNKS {
return Err(corrupt("too many object header chunks"));
}
let idx = h.chunks.len();
if h.version == 2 {
if clen < 8 || img.read(caddr, 4)? != b"OCHK" {
return Err(corrupt("bad continuation chunk"));
}
h.chunks.push(Chunk {
start: caddr,
checksum_at: Some(caddr + clen - 4),
end: caddr + clen - 4,
gap: 0,
});
h.scan(img, idx, caddr + 4, caddr + clen - 4, &mut pending)?;
} else {
h.chunks.push(Chunk {
start: caddr,
checksum_at: None,
end: caddr + clen,
gap: 0,
});
h.scan(img, idx, caddr, caddr + clen, &mut pending)?;
}
}
Ok(h)
}
/// Size of a message header in this object header.
pub(crate) fn hsize(&self) -> usize {
match (self.version, self.flags & 0x04 != 0) {
(1, _) => 8,
(_, true) => 6,
_ => 4,
}
}
fn scan(
&mut self,
img: &Image<'_>,
chunk: usize,
start: u64,
end: u64,
pending: &mut Vec<(u64, u64)>,
) -> Result<(), Error> {
let hs = self.hsize() as u64;
let bytes = img.read(start, (end - start) as usize)?;
let mut p = 0usize;
while (p as u64) + hs <= end - start {
let b = &bytes[p..];
let (mtype, size, flags, corder) = if self.version == 1 {
(
u16::from_le_bytes([b[0], b[1]]),
u16::from_le_bytes([b[2], b[3]]) as usize,
b[4],
None,
)
} else {
(
u16::from(b[0]),
u16::from_le_bytes([b[1], b[2]]) as usize,
b[3],
(hs == 6).then(|| u16::from_le_bytes([b[4], b[5]])),
)
};
let data_off = p + hs as usize;
if data_off + size > bytes.len() {
return Err(corrupt("message size exceeds buffer end"));
}
if mtype == MSG_CONTINUATION {
let d = &bytes[data_off..data_off + size];
let os = img.os as usize;
let ls = img.ls as usize;
if d.len() < os + ls {
return Err(corrupt("short continuation message"));
}
pending.push((get_uint(d, img.os), get_uint(&d[os..], img.ls)));
}
self.msgs.push(Msg {
chunk,
hdr_pos: start + p as u64,
data_pos: start + data_off as u64,
size,
mtype,
flags,
corder,
});
p = data_off + size;
}
self.chunks[chunk].gap = (end - start) - p as u64;
Ok(())
}
/// After message `i` became a NIL message: if its chunk ends in a gap,
/// grow the NIL message over it (it must be the chunk's last message).
fn absorb_gap(&mut self, img: &mut Image<'_>, i: usize) -> Result<(), Error> {
let m = self.msgs[i].clone();
let c = &self.chunks[m.chunk];
if c.gap == 0 {
return Ok(());
}
if m.data_pos + m.size as u64 + c.gap != c.end {
return Err(Error::Unsupported(
"object header chunk ends in a gap that a free message cannot absorb".into(),
));
}
let new_size = m.size + c.gap as usize;
if new_size > usize::from(u16::MAX) {
return Err(Error::Unsupported("object header message too large".into()));
}
self.write_msg_header(img, m.hdr_pos, MSG_NIL, new_size, 0, m.corder)?;
img.write(m.data_pos, &vec![0u8; new_size])?;
self.msgs[i].size = new_size;
self.chunks[m.chunk].gap = 0;
Ok(())
}
/// The first message of type `mtype`.
pub(crate) fn find(&self, mtype: u16) -> Option<usize> {
self.msgs.iter().position(|m| m.mtype == mtype)
}
pub(crate) fn data(&self, img: &Image<'_>, i: usize) -> Result<Vec<u8>, Error> {
let m = &self.msgs[i];
img.read(m.data_pos, m.size)
}
/// Overwrite bytes of message `i`'s body, from `offset`.
pub(crate) fn patch(
&mut self,
img: &mut Image<'_>,
i: usize,
offset: usize,
bytes: &[u8],
) -> Result<(), Error> {
let m = &self.msgs[i];
if offset + bytes.len() > m.size {
return Err(Error::Unsupported(
"change does not fit the header message".into(),
));
}
img.write(m.data_pos + offset as u64, bytes)?;
self.dirty.insert(m.chunk);
Ok(())
}
fn write_msg_header(
&mut self,
img: &mut Image<'_>,
hdr_pos: u64,
mtype: u16,
size: usize,
flags: u8,
corder: Option<u16>,
) -> Result<(), Error> {
let mut h = vec![0u8; self.hsize()];
if self.version == 1 {
h[0..2].copy_from_slice(&mtype.to_le_bytes());
h[2..4].copy_from_slice(&(size as u16).to_le_bytes());
h[4] = flags;
} else {
h[0] = mtype as u8;
h[1..3].copy_from_slice(&(size as u16).to_le_bytes());
h[3] = flags;
if h.len() == 6 {
h[4..6].copy_from_slice(&corder.unwrap_or(0).to_le_bytes());
}
}
img.write(hdr_pos, &h)
}
/// Turn message `i` into a NIL message (its space becomes free).
pub(crate) fn delete(&mut self, img: &mut Image<'_>, i: usize) -> Result<(), Error> {
let m = self.msgs[i].clone();
self.write_msg_header(img, m.hdr_pos, MSG_NIL, m.size, 0, m.corder)?;
img.write(m.data_pos, &vec![0u8; m.size])?;
self.msgs[i].mtype = MSG_NIL;
self.msgs[i].flags = 0;
self.dirty.insert(m.chunk);
self.absorb_gap(img, i)
}
/// Body size a message of `len` bytes occupies (version 1 pads to 8).
fn padded(&self, len: usize) -> usize {
if self.version == 1 {
len.next_multiple_of(8)
} else {
len
}
}
/// Whether a free slot of `slot` bytes can take a body of `need` bytes:
/// exactly, or with room left for a NIL message after it.
fn fits(&self, slot: usize, need: usize) -> bool {
slot == need || slot >= need + self.hsize()
}
/// The smallest NIL message that can take `need` body bytes.
fn best_nil(&self, need: usize) -> Option<usize> {
self.msgs
.iter()
.enumerate()
.filter(|(_, m)| m.mtype == MSG_NIL && self.fits(m.size, need))
.min_by_key(|(_, m)| m.size)
.map(|(i, _)| i)
}
/// Whether free space in the header can take a body of `len` bytes.
pub(crate) fn has_free(&self, len: usize) -> bool {
self.best_nil(self.padded(len)).is_some()
}
/// Put a message into slot `i` (a NIL message, or a message being
/// moved away), splitting off the rest as a NIL message.
fn place(
&mut self,
img: &mut Image<'_>,
i: usize,
mtype: u16,
flags: u8,
data: &[u8],
corder: Option<u16>,
) -> Result<(), Error> {
let slot = self.msgs[i].clone();
let need = self.padded(data.len());
debug_assert!(self.fits(slot.size, need));
let mut body = data.to_vec();
body.resize(need, 0);
self.write_msg_header(img, slot.hdr_pos, mtype, need, flags, corder)?;
img.write(slot.data_pos, &body)?;
self.msgs[i] = Msg {
size: need,
mtype,
flags,
corder,
..slot.clone()
};
if slot.size > need {
let hs = self.hsize();
let nil_hdr = slot.data_pos + need as u64;
let nil_size = slot.size - need - hs;
self.write_msg_header(img, nil_hdr, MSG_NIL, nil_size, 0, Some(0))?;
img.write(nil_hdr + hs as u64, &vec![0u8; nil_size])?;
self.msgs.push(Msg {
chunk: slot.chunk,
hdr_pos: nil_hdr,
data_pos: nil_hdr + hs as u64,
size: nil_size,
mtype: MSG_NIL,
flags: 0,
corder: (hs == 6).then_some(0),
});
self.added += 1;
let nil = self.msgs.len() - 1;
self.absorb_gap(img, nil)?;
}
self.dirty.insert(slot.chunk);
Ok(())
}
/// Add a message: into free space in the header when there is some,
/// else into a new continuation chunk at the end of the file (whose
/// continuation message takes a NIL slot, or the slot of another
/// message — an attribute if possible — that moves into the new chunk
/// with it).
pub(crate) fn insert(
&mut self,
img: &mut Image<'_>,
mtype: u16,
flags: u8,
data: &[u8],
corder: Option<u16>,
) -> Result<(), Error> {
if data.len() > usize::from(u16::MAX) {
return Err(Error::Unsupported(
"message larger than 64 KiB (would need dense storage)".into(),
));
}
let need = self.padded(data.len());
if let Some(i) = self.best_nil(need) {
return self.place(img, i, mtype, flags, data, corder);
}
let os = img.os as usize;
let ls = img.ls as usize;
let cont_need = self.padded(os + ls);
// Where the continuation message goes, and the message (if any)
// that moves out of that slot into the new chunk.
let (slot, moved) = match self.best_nil(cont_need) {
Some(i) => (i, None),
None => {
// Any message but a continuation can live in any chunk;
// prefer moving an attribute, then the smallest that fits.
let i = self
.msgs
.iter()
.enumerate()
.filter(|(_, m)| {
m.mtype != MSG_NIL
&& m.mtype != MSG_CONTINUATION
&& self.fits(m.size, cont_need)
})
.min_by_key(|(_, m)| (m.mtype != MSG_ATTRIBUTE, m.size))
.map(|(i, _)| i)
.ok_or_else(|| {
Error::Unsupported(
"no room in the object header for a continuation message".into(),
)
})?;
let m = self.msgs[i].clone();
let body = img.read(m.data_pos, m.size)?;
(i, Some((m, body)))
}
};
// The new chunk: [moved message] + new message + a NIL message
// holding spare room for later additions.
let hs = self.hsize();
let spare = 64usize;
let mut payload = hs + need;
if let Some((m, _)) = &moved {
payload += hs + m.size;
}
let msgs_len = payload + hs + spare;
let (prefix, suffix) = if self.version == 2 { (4, 4) } else { (0, 0) };
let chunk_len = prefix + msgs_len + suffix;
let caddr = img.alloc(chunk_len as u64)?;
if self.version == 2 {
img.write(caddr, b"OCHK")?;
}
let cidx = self.chunks.len();
self.chunks.push(Chunk {
start: caddr,
checksum_at: (self.version == 2).then_some(caddr + (prefix + msgs_len) as u64),
end: caddr + (prefix + msgs_len) as u64,
gap: 0,
});
let first = caddr + prefix as u64;
// Lay the chunk out as one NIL message, then place into it.
self.write_msg_header(img, first, MSG_NIL, msgs_len - hs, 0, Some(0))?;
self.msgs.push(Msg {
chunk: cidx,
hdr_pos: first,
data_pos: first + hs as u64,
size: msgs_len - hs,
mtype: MSG_NIL,
flags: 0,
corder: (hs == 6).then_some(0),
});
self.added += 1;
if let Some((m, body)) = &moved {
let nil = self.msgs.len() - 1;
self.place(img, nil, m.mtype, m.flags, body, m.corder)?;
}
let nil = self.msgs.len() - 1;
self.place(img, nil, mtype, flags, data, corder)?;
// Link it in.
let mut cont = vec![0u8; os + ls];
put_uint(&mut cont, caddr, img.os);
put_uint(&mut cont[os..], chunk_len as u64, img.ls);
if moved.is_some() {
self.msgs[slot].mtype = MSG_NIL; // its content now lives in the new chunk
}
self.place(img, slot, MSG_CONTINUATION, 0, &cont, Some(0))?;
self.dirty.insert(cidx);
Ok(())
}
/// Recompute the checksum of every changed version-2 chunk; store a
/// version-1 header's new message count.
pub(crate) fn finish(&mut self, img: &mut Image<'_>) -> Result<(), Error> {
for &c in &self.dirty {
if let Some(at) = self.chunks[c].checksum_at {
rechecksum(img, self.chunks[c].start, at)?;
}
}
if self.version == 1 && self.added > 0 {
let old = u16::from_le_bytes(img.read(self.addr + 2, 2)?.try_into().unwrap_or([0; 2]));
let new = usize::from(old) + self.added;
let new = u16::try_from(new)
.map_err(|_| Error::Unsupported("too many object header messages".into()))?;
img.write(self.addr + 2, &new.to_le_bytes())?;
}
self.dirty.clear();
self.added = 0;
Ok(())
}
}
crates/clawhdf5/src/edit/select.rs
+142
View File
@@ -0,0 +1,142 @@
//! A selection as runs of consecutive elements along the last dimension, in
//! the order the selection's elements are numbered (row-major over a
//! hyperslab, as h5py and libhdf5 number them; a point list in its order).
use clawhdf5_format::selection::Selection;
use crate::error::Error;
/// Call `f(coords, len, src)` for each run: `len` elements starting at
/// `coords` (consecutive in the last dimension), which are elements
/// `src..src + len` of the selection. Returns the number of elements.
/// The selection must already be validated against `dims`.
pub(crate) fn for_each_run(
sel: &Selection,
dims: &[u64],
mut f: impl FnMut(&[u64], u64, u64) -> Result<(), Error>,
) -> Result<u64, Error> {
let rank = dims.len();
let mut src = 0u64;
match sel {
Selection::None => {}
Selection::Points(pts) => {
for p in pts {
f(p, 1, src)?;
src += 1;
}
}
Selection::All => {
if rank == 0 {
f(&[], 1, 0)?;
return Ok(1);
}
if dims.contains(&0) {
return Ok(0);
}
let last = dims[rank - 1];
let mut coords = vec![0u64; rank];
loop {
f(&coords, last, src)?;
src += last;
if !advance(&mut coords[..rank - 1], &dims[..rank - 1]) {
break;
}
}
}
Selection::Hyperslab {
start,
stride,
count,
block,
} => {
if rank == 0 {
return Err(Error::InvalidArgument(
"hyperslab selection on a scalar dataset".into(),
));
}
if (0..rank).any(|d| count[d] == 0 || block[d] == 0) {
return Ok(0);
}
// Per-dimension extent of the selection: j in 0..count*block.
let ext: Vec<u64> = (0..rank).map(|d| count[d] * block[d]).collect();
let coord = |d: usize, j: u64| start[d] + (j / block[d]) * stride[d] + j % block[d];
let l = rank - 1;
// Along the last dimension, blocks merge when they touch.
let merged = stride[l] == block[l] || count[l] == 1;
let mut js = vec![0u64; rank - 1];
let mut coords = vec![0u64; rank];
loop {
for (d, &j) in js.iter().enumerate() {
coords[d] = coord(d, j);
}
if merged {
coords[l] = start[l];
f(&coords, ext[l], src)?;
src += ext[l];
} else {
for c in 0..count[l] {
coords[l] = start[l] + c * stride[l];
f(&coords, block[l], src)?;
src += block[l];
}
}
if !advance(&mut js, &ext[..l]) {
break;
}
}
}
}
Ok(src)
}
/// Odometer step over `0..lim[d]`; false when it wraps around.
fn advance(v: &mut [u64], lim: &[u64]) -> bool {
for d in (0..v.len()).rev() {
v[d] += 1;
if v[d] < lim[d] {
return true;
}
v[d] = 0;
}
false
}
#[cfg(test)]
mod tests {
use super::*;
fn collect(sel: &Selection, dims: &[u64]) -> Vec<(Vec<u64>, u64, u64)> {
let mut out = Vec::new();
for_each_run(sel, dims, |c, n, s| {
out.push((c.to_vec(), n, s));
Ok(())
})
.unwrap();
out
}
#[test]
fn runs() {
assert_eq!(
collect(&Selection::All, &[2, 3]),
vec![(vec![0, 0], 3, 0), (vec![1, 0], 3, 3)]
);
let h = Selection::Hyperslab {
start: vec![1, 0],
stride: vec![2, 3],
count: vec![2, 2],
block: vec![1, 2],
};
assert_eq!(
collect(&h, &[5, 6]),
vec![
(vec![1, 0], 2, 0),
(vec![1, 3], 2, 2),
(vec![3, 0], 2, 4),
(vec![3, 3], 2, 6)
]
);
assert_eq!(collect(&Selection::All, &[]), vec![(vec![], 1, 0)]);
assert_eq!(collect(&Selection::All, &[0, 4]), vec![]);
}
}
crates/clawhdf5/src/error.rs
+18
View File
@@ -6,7 +6,12 @@ use clawhdf5_format::error::FormatError;
use clawhdf5_format::message_type::MessageType;
/// Errors that can occur when using the high-level API.
///
/// Non-exhaustive: new failure modes add variants, so a `match` needs a
/// wildcard arm. Failures of a [`Storage`](clawhdf5_format::storage::Storage)
/// backend arrive as `Error::Format(FormatError::Storage(..))`.
#[derive(Debug)]
#[non_exhaustive]
pub enum Error {
/// I/O error from the filesystem.
Io(std::io::Error),
@@ -36,6 +41,16 @@ pub enum Error {
/// Actual alignment of the data pointer.
actual: usize,
},
/// The requested change is valid but not supported (by
/// [`FileEditor`](crate::FileEditor): a chunk index, filter or header
/// layout it cannot modify). Nothing was written.
Unsupported(String),
/// An argument does not fit the object (a selection outside the
/// dataset, a buffer of the wrong length, a shrinking resize, ...).
InvalidArgument(String),
/// The file is locked by another writer (another [`FileEditor`](crate::FileEditor),
/// or libhdf5 with file locking on).
Locked(String),
}
impl fmt::Display for Error {
@@ -58,6 +73,9 @@ impl fmt::Display for Error {
"zero-copy type mismatch: expected {expected}, got {actual}"
)
}
Error::Unsupported(msg) => write!(f, "unsupported: {msg}"),
Error::InvalidArgument(msg) => write!(f, "invalid argument: {msg}"),
Error::Locked(msg) => write!(f, "file is locked: {msg}"),
Error::ZeroCopyUnaligned { required, actual } => {
write!(
f,
crates/clawhdf5/src/lazy.rs
+44 -13
View File
@@ -28,7 +28,7 @@ use clawhdf5_format::superblock::Superblock;
use clawhdf5_io::HDF5Read;
use crate::error::Error;
use crate::types::{AttrValue, DType, classify_datatype, read_attrs};
use crate::types::{AttrValue, DType, classify_datatype, read_attr, read_attrs};
/// A lazy HDF5 file handle that parses metadata on demand.
///
@@ -304,12 +304,7 @@ impl<'f, R: HDF5Read> LazyGroup<'f, R> {
/// Get a dataset within this group by name.
pub fn dataset(&self, name: &str) -> Result<LazyDataset<'f, R>, Error> {
let entries = self.children()?;
let entry = entries
.iter()
.find(|e| e.name == name)
.ok_or_else(|| Error::Format(FormatError::PathNotFound(name.to_string())))?;
let hdr = self.file.get_or_parse_header(entry.object_header_address)?;
let hdr = self.file.get_or_parse_header(self.child_address(name)?)?;
if !has_message(&hdr, MessageType::DataLayout) {
return Err(Error::NotADataset(name.to_string()));
}
@@ -322,17 +317,38 @@ impl<'f, R: HDF5Read> LazyGroup<'f, R> {
/// Get a subgroup within this group by name.
pub fn group(&self, name: &str) -> Result<LazyGroup<'f, R>, Error> {
let entries = self.children()?;
let entry = entries
.iter()
.find(|e| e.name == name)
.ok_or_else(|| Error::Format(FormatError::PathNotFound(name.to_string())))?;
Ok(LazyGroup {
file: self.file,
address: entry.object_header_address,
address: self.child_address(name)?,
})
}
/// The attribute called `name`, or `None` if it has none by that name
/// (or it cannot be read) — the value [`attrs`](Self::attrs) has under
/// that name, found without reading the other attributes when they are
/// stored densely.
pub fn attr(&self, name: &str) -> Result<Option<AttrValue>, Error> {
let hdr = self.file.get_or_parse_header(self.address)?;
let data = self.file.hdf5_bytes();
read_attr(
data,
&hdr,
name,
self.file.offset_size(),
self.file.length_size(),
)
}
/// The object header address of the child called `name`: the entry of
/// the group's listing with that name, looked up through the group's
/// name index rather than by listing the group (see
/// [`group_v2::resolve_child`]).
fn child_address(&self, name: &str) -> Result<u64, Error> {
let data = self.file.hdf5_bytes();
group_v2::resolve_child(data, &self.file.superblock, self.address, name)
.map_err(Error::Format)
}
/// This group's links that can be opened: hard links, and soft links
/// resolved to their targets (see
/// [`group_v2::resolve_group_children`]); dangling, external and
@@ -555,6 +571,21 @@ impl<'f, R: HDF5Read> LazyDataset<'f, R> {
)
}
/// The attribute called `name`, or `None` if it has none by that name
/// (or it cannot be read) — the value [`attrs`](Self::attrs) has under
/// that name, found without reading the other attributes when they are
/// stored densely.
pub fn attr(&self, name: &str) -> Result<Option<AttrValue>, Error> {
let data = self.file.hdf5_bytes();
read_attr(
data,
&self.header,
name,
self.file.offset_size(),
self.file.length_size(),
)
}
/// A header message's payload, resolved through the shared-message
/// indirection when needed (e.g. a committed datatype). See
/// [`clawhdf5_format::shared_message::message_data`].
crates/clawhdf5/src/lib.rs
+18
View File
@@ -23,8 +23,25 @@
//! builder.set_attr("version", AttrValue::I64(1));
//! builder.write("output.h5").unwrap();
//! ```
//!
//! # Modifying a file in place
//!
//! ```no_run
//! use clawhdf5::{FileEditor, Selection};
//!
//! let mut ed = FileEditor::open("data.h5").unwrap();
//! ed.resize("series", &[1100]).unwrap(); // a chunked dataset, maxshape (None,)
//! let tail = Selection::Hyperslab {
//! start: vec![1000],
//! stride: vec![1],
//! count: vec![100],
//! block: vec![1],
//! };
//! ed.write_values("series", &tail, &[0.5f64; 100]).unwrap();
//! ```
mod cache_image;
mod edit;
pub mod error;
pub mod lazy;
#[cfg(feature = "mmap")]
@@ -34,6 +51,7 @@ pub mod types;
pub mod vlen;
pub mod writer;
pub use edit::FileEditor;
pub use error::Error;
pub use lazy::{LazyDataset, LazyFile, LazyGroup};
#[cfg(feature = "mmap")]
crates/clawhdf5/src/mmap_file.rs
+44 -13
View File
@@ -23,7 +23,7 @@ use clawhdf5_format::superblock::Superblock;
use clawhdf5_io::MmapReader;
use crate::error::Error;
use crate::types::{AttrValue, DType, classify_datatype, read_attrs};
use crate::types::{AttrValue, DType, classify_datatype, read_attr, read_attrs};
/// An HDF5 file opened via memory mapping.
///
@@ -242,12 +242,7 @@ impl<'f> MmapGroup<'f> {
/// Get a dataset within this group by name.
pub fn dataset(&self, name: &str) -> Result<MmapDataset<'f>, Error> {
let entries = self.children()?;
let entry = entries
.iter()
.find(|e| e.name == name)
.ok_or_else(|| Error::Format(FormatError::PathNotFound(name.to_string())))?;
let hdr = self.file.parse_header(entry.object_header_address)?;
let hdr = self.file.parse_header(self.child_address(name)?)?;
if !has_message(&hdr, MessageType::DataLayout) {
return Err(Error::NotADataset(name.to_string()));
}
@@ -260,17 +255,38 @@ impl<'f> MmapGroup<'f> {
/// Get a subgroup within this group by name.
pub fn group(&self, name: &str) -> Result<MmapGroup<'f>, Error> {
let entries = self.children()?;
let entry = entries
.iter()
.find(|e| e.name == name)
.ok_or_else(|| Error::Format(FormatError::PathNotFound(name.to_string())))?;
Ok(MmapGroup {
file: self.file,
address: entry.object_header_address,
address: self.child_address(name)?,
})
}
/// The attribute called `name`, or `None` if it has none by that name
/// (or it cannot be read) — the value [`attrs`](Self::attrs) has under
/// that name, found without reading the other attributes when they are
/// stored densely.
pub fn attr(&self, name: &str) -> Result<Option<AttrValue>, Error> {
let hdr = self.file.parse_header(self.address)?;
let data = self.file.hdf5_bytes();
read_attr(
data,
&hdr,
name,
self.file.offset_size(),
self.file.length_size(),
)
}
/// The object header address of the child called `name`: the entry of
/// the group's listing with that name, looked up through the group's
/// name index rather than by listing the group (see
/// [`group_v2::resolve_child`]).
fn child_address(&self, name: &str) -> Result<u64, Error> {
let data = self.file.meta()?;
group_v2::resolve_child(data, &self.file.superblock, self.address, name)
.map_err(Error::Format)
}
/// This group's links that can be opened: hard links, and soft links
/// resolved to their targets (see
/// [`group_v2::resolve_group_children`]); dangling, external and
@@ -506,6 +522,21 @@ impl<'f> MmapDataset<'f> {
)
}
/// The attribute called `name`, or `None` if it has none by that name
/// (or it cannot be read) — the value [`attrs`](Self::attrs) has under
/// that name, found without reading the other attributes when they are
/// stored densely.
pub fn attr(&self, name: &str) -> Result<Option<AttrValue>, Error> {
let data = self.file.hdf5_bytes();
read_attr(
data,
&self.header,
name,
self.file.offset_size(),
self.file.length_size(),
)
}
/// A header message's payload, resolved through the shared-message
/// indirection when needed (e.g. a committed datatype). See
/// [`clawhdf5_format::shared_message::message_data`].
crates/clawhdf5/src/reader.rs
+81 -20
View File
@@ -23,7 +23,7 @@ use clawhdf5_format::superblock::Superblock;
use crate::cache_image::{self, ImageView};
use crate::error::Error;
use crate::types::{AttrValue, DType, classify_datatype, read_attrs};
use crate::types::{AttrValue, DType, classify_datatype, read_attr, read_attrs};
// ---------------------------------------------------------------------------
// FileData — internal storage for either owned bytes or an mmap
@@ -230,9 +230,10 @@ impl File {
/// A `Dataset` handle for the object header at `address` (an address
/// from a group listing, or one kept from an earlier lookup), without
/// resolving a path. Resolving a path walks every group on it, which in
/// a large group costs a scan of its links; keep the address instead to
/// open the same dataset repeatedly.
/// resolving a path. Resolving a path looks each component up in its
/// group (through the name index of a dense group; a v1 group's entries
/// are scanned); keep the address instead to open the same dataset
/// repeatedly.
pub fn dataset_at(&self, address: u64) -> Result<Dataset<'_>, Error> {
let hdr = self.parse_header(address)?;
if !has_message(&hdr, MessageType::DataLayout) {
@@ -245,6 +246,17 @@ impl File {
.check_open()
}
/// A `Group` handle for the object header at `address` (from
/// [`Group::entries`], or kept from an earlier lookup), without
/// resolving a path. Like [`group`](Self::group), the object is not
/// checked to be a group; a non-group has no children.
pub fn group_at(&self, address: u64) -> Group<'_> {
Group {
file: self,
address,
}
}
/// Resolve a path and return a `Group` handle.
///
/// The path uses `/` separators (e.g., `"sensors"`).
@@ -483,12 +495,7 @@ impl<'f> Group<'f> {
/// Get a dataset within this group by name.
pub fn dataset(&self, name: &str) -> Result<Dataset<'f>, Error> {
let entries = self.children()?;
let entry = entries
.iter()
.find(|e| e.name == name)
.ok_or_else(|| Error::Format(FormatError::PathNotFound(name.to_string())))?;
let hdr = self.file.parse_header(entry.object_header_address)?;
let hdr = self.file.parse_header(self.child_address(name)?)?;
if !has_message(&hdr, MessageType::DataLayout) {
return Err(Error::NotADataset(name.to_string()));
}
@@ -501,17 +508,51 @@ impl<'f> Group<'f> {
/// Get a subgroup within this group by name.
pub fn group(&self, name: &str) -> Result<Group<'f>, Error> {
let entries = self.children()?;
let entry = entries
.iter()
.find(|e| e.name == name)
.ok_or_else(|| Error::Format(FormatError::PathNotFound(name.to_string())))?;
Ok(Group {
file: self.file,
address: entry.object_header_address,
address: self.child_address(name)?,
})
}
/// The attribute called `name`, or `None` if it has none by that name
/// (or it cannot be read) — the value [`attrs`](Self::attrs) has under
/// that name, found without reading the other attributes when they are
/// stored densely.
pub fn attr(&self, name: &str) -> Result<Option<AttrValue>, Error> {
let hdr = self.file.parse_header(self.address)?;
let data = self.file.data.as_bytes();
read_attr(
data,
&hdr,
name,
self.file.offset_size(),
self.file.length_size(),
)
}
/// The object header address of the child called `name`: the entry of
/// the group's listing with that name, looked up through the group's
/// name index rather than by listing the group (see
/// [`group_v2::resolve_child`]).
fn child_address(&self, name: &str) -> Result<u64, Error> {
let data = self.file.data.meta()?;
group_v2::resolve_child(data, &self.file.superblock, self.address, name)
.map_err(Error::Format)
}
/// This group's children that can be opened, as `(name, object header
/// address)` in listing order — the entries [`datasets`](Self::datasets)
/// and [`groups`](Self::groups) are drawn from. Open one with
/// [`File::dataset_at`] or [`File::group_at`] to skip looking its name
/// up again, or keep the addresses to revisit the objects.
pub fn entries(&self) -> Result<Vec<(String, u64)>, Error> {
Ok(self
.children()?
.into_iter()
.map(|e| (e.name, e.object_header_address))
.collect())
}
/// This group's links that can be opened: hard links, and soft links
/// resolved to their targets (see
/// [`group_v2::resolve_group_children`]); dangling, external and
@@ -1051,6 +1092,21 @@ impl<'f> Dataset<'f> {
)
}
/// The attribute called `name`, or `None` if it has none by that name
/// (or it cannot be read) — the value [`attrs`](Self::attrs) has under
/// that name, found without reading the other attributes when they are
/// stored densely.
pub fn attr(&self, name: &str) -> Result<Option<AttrValue>, Error> {
let data = self.file.data.as_bytes();
read_attr(
data,
&self.header,
name,
self.file.offset_size(),
self.file.length_size(),
)
}
/// Verify this dataset's content against its stored provenance hash
/// (`_provenance_sha256`, written automatically on save when a
/// [`Provenance`](clawhdf5_format::provenance::Provenance) is set — see
@@ -1104,13 +1160,18 @@ impl<'f> Dataset<'f> {
.ok_or(Error::MissingMessage(msg_type))
}
fn datatype(&self) -> Result<Datatype, Error> {
/// The dataset's object header as parsed.
pub(crate) fn header(&self) -> &ObjectHeader {
&self.header
}
pub(crate) fn datatype(&self) -> Result<Datatype, Error> {
let data = self.required_payload(MessageType::Datatype)?;
let (dt, _) = Datatype::parse_in_header(&data, self.header.version)?;
Ok(dt)
}
fn dataspace(&self) -> Result<Dataspace, Error> {
pub(crate) fn dataspace(&self) -> Result<Dataspace, Error> {
let data = self.required_payload(MessageType::Dataspace)?;
let mut ds = Dataspace::parse(&data, self.file.length_size())?;
// libhdf5 reports a virtual dataset with unlimited or printf-style
@@ -1130,7 +1191,7 @@ impl<'f> Dataset<'f> {
Ok(ds)
}
fn data_layout(&self) -> Result<DataLayout, Error> {
pub(crate) fn data_layout(&self) -> Result<DataLayout, Error> {
let msg = find_message(&self.header, MessageType::DataLayout)?;
Ok(DataLayout::parse(
&msg.data,
@@ -1143,7 +1204,7 @@ impl<'f> Dataset<'f> {
/// that is present but unparseable is an error: treating it as "no
/// filters" would hand the caller the still-compressed bytes as if they
/// were the data.
fn filter_pipeline(&self) -> Result<Option<FilterPipeline>, Error> {
pub(crate) fn filter_pipeline(&self) -> Result<Option<FilterPipeline>, Error> {
self.message_payload(MessageType::FilterPipeline)?
.map(|data| FilterPipeline::parse(&data).map_err(Error::Format))
.transpose()
crates/clawhdf5/src/types.rs
+29
View File
@@ -182,6 +182,35 @@ pub(crate) fn read_attrs(
))
}
/// The attribute called `name` on the object with header `header`, decoded
/// as [`read_attrs`] decodes it, or `None` (see
/// [`find_attribute_in_file`](clawhdf5_format::attribute::find_attribute_in_file)).
pub(crate) fn read_attr(
file_data: &[u8],
header: &clawhdf5_format::object_header::ObjectHeader,
name: &str,
offset_size: u8,
length_size: u8,
) -> Result<Option<AttrValue>, crate::Error> {
let Some(msg) = clawhdf5_format::attribute::find_attribute_in_file(
file_data,
header,
name,
offset_size,
length_size,
)?
else {
return Ok(None);
};
Ok(attrs_to_map(
std::slice::from_ref(&msg),
file_data,
offset_size,
length_size,
)
.remove(name))
}
pub(crate) fn attrs_to_map(
attrs: &[clawhdf5_format::attribute::AttributeMessage],
file_data: &[u8],
crates/clawhdf5/tests/dense_storage_interop.rs
+30
View File
@@ -211,6 +211,36 @@ fn dense_attribute_stored_as_a_huge_heap_object() {
assert!(matches!(&attrs["bigger"], AttrValue::I64Array(v) if *v == bigger));
}
/// Enough huge attributes that the huge-object B-tree has internal nodes:
/// each one is found by descending it by heap ID (libhdf5 orders the
/// records of indirectly addressed huge objects by ID), and every value
/// matches what was written.
#[test]
fn many_huge_attributes_are_found_through_their_index() {
skip_if_no_python!();
let (_dir, path) = h5py_file(
"d = f.create_dataset('d', data=[1.0])\n\
for i in range(300):\n\
\x20 d.attrs['h%03d' % i] = np.arange(600, dtype='i8') + i\n",
);
let f = File::open(&path).unwrap();
let d = f.dataset("d").unwrap();
let attrs = d.attrs().unwrap();
assert_eq!(attrs.len(), 300);
for i in 0..300i64 {
let name = format!("h{i:03}");
let want: Vec<i64> = (0..600).map(|v| v + i).collect();
assert!(
matches!(&attrs[&name], AttrValue::I64Array(v) if *v == want),
"{name}"
);
assert!(
matches!(d.attr(&name).unwrap(), Some(AttrValue::I64Array(v)) if v == want),
"{name}"
);
}
}
/// A group whose link heap has a deflate I/O filter (set on the group
/// creation property list), with 3 000 links and one link whose message is
/// larger than the heap's managed-object limit, so it is a huge object.
crates/clawhdf5/tests/edit_tests.rs
+138
View File
@@ -0,0 +1,138 @@
//! `FileEditor` on files clawhdf5 writes, read back with our own reader
//! (libhdf5 interop is in `clawhdf5-tools/tests/edit_interop.rs`, which can
//! also run `h5rs check`).
use clawhdf5::{AttrValue, Error, File, FileBuilder, FileEditor, Selection};
fn block(start: u64, count: u64) -> Selection {
Selection::Hyperslab {
start: vec![start],
stride: vec![1],
count: vec![count],
block: vec![1],
}
}
fn sample(dir: &std::path::Path) -> std::path::PathBuf {
let path = dir.join("f.h5");
let mut b = FileBuilder::new();
b.create_dataset("ext")
.with_i32_data(&[0, 1, 2, 3, 4])
.with_shape(&[5])
.with_maxshape(&[u64::MAX])
.with_chunks(&[4])
.with_deflate(6);
b.create_dataset("raw")
.with_f64_data(&[0.5; 8])
.with_shape(&[2, 4])
.with_maxshape(&[u64::MAX, 4])
.with_chunks(&[1, 4]);
b.create_dataset("flat").with_i64_data(&[1, 2, 3]);
b.set_attr("title", AttrValue::String("t".into()));
b.write(&path).unwrap();
path
}
#[test]
fn append_overwrite_and_attributes_round_trip() {
let dir = tempfile::tempdir().unwrap();
let path = sample(dir.path());
let len_before = std::fs::metadata(&path).unwrap().len();
let mut expect: Vec<i32> = (0..5).collect();
let mut raw = vec![0.5f64; 8];
{
let mut ed = FileEditor::open(&path).unwrap();
for k in 0..300u64 {
let n = expect.len() as u64;
let add = 1 + k % 5;
ed.resize("ext", &[n + add]).unwrap();
let vals: Vec<i32> = (0..add).map(|j| (n + j) as i32 * 2).collect();
ed.write_values("ext", &block(n, add), &vals).unwrap();
expect.extend(&vals);
}
// A filtered chunk rewritten with data that compresses worse moves.
let noisy: Vec<i32> = (0..4).map(|i| i * 7_919_993).collect();
ed.write_values("ext", &block(0, 4), &noisy).unwrap();
expect[..4].copy_from_slice(&noisy);
ed.resize("raw", &[5, 4]).unwrap();
raw.resize(20, 0.0);
let sel = Selection::Hyperslab {
start: vec![1, 1],
stride: vec![2, 2],
count: vec![2, 2],
block: vec![1, 1],
};
ed.write_values("raw", &sel, &[1.0f64, 2.0, 3.0, 4.0])
.unwrap();
for (i, (r, c)) in [(1, 1), (1, 3), (3, 1), (3, 3)].iter().enumerate() {
raw[r * 4 + c] = i as f64 + 1.0;
}
ed.write_values("flat", &Selection::Points(vec![vec![2]]), &[30i64])
.unwrap();
ed.set_attr("/", "title", &AttrValue::String("a longer title".into()))
.unwrap();
ed.set_attr("ext", "count", &AttrValue::I64(expect.len() as i64))
.unwrap();
}
let f = File::open(&path).unwrap();
assert_eq!(f.dataset("ext").unwrap().read_i32().unwrap(), expect);
assert_eq!(f.dataset("raw").unwrap().shape().unwrap(), vec![5, 4]);
assert_eq!(f.dataset("raw").unwrap().read_f64().unwrap(), raw);
assert_eq!(
f.dataset("flat").unwrap().read_i64().unwrap(),
vec![1, 2, 30]
);
let root = f.root().attrs().unwrap();
assert!(matches!(root.get("title"), Some(AttrValue::String(s)) if s == "a longer title"));
let ext = f.dataset("ext").unwrap().attrs().unwrap();
assert!(matches!(ext.get("count"), Some(AttrValue::I64(n)) if *n == expect.len() as i64));
assert!(std::fs::metadata(&path).unwrap().len() > len_before);
}
#[test]
fn errors_leave_the_file_untouched() {
let dir = tempfile::tempdir().unwrap();
let path = sample(dir.path());
let before = std::fs::read(&path).unwrap();
let mut ed = FileEditor::open(&path).unwrap();
assert!(matches!(FileEditor::open(&path), Err(Error::Locked(_))));
assert!(ed.write_all("missing", &[0; 4]).is_err());
// Wrong length, wrong type, outside the extent, beyond maxshape,
// shrinking, a rank change.
assert!(matches!(
ed.write_all("flat", &[0; 7]),
Err(Error::InvalidArgument(_))
));
assert!(matches!(
ed.write_values("flat", &Selection::All, &[1i32, 2, 3]),
Err(Error::InvalidArgument(_))
));
assert!(matches!(
ed.write_values("ext", &block(4, 2), &[1i32, 2]),
Err(Error::InvalidArgument(_))
));
assert!(matches!(
ed.resize("raw", &[3, 5]),
Err(Error::InvalidArgument(_))
));
assert!(matches!(ed.resize("ext", &[4]), Err(Error::Unsupported(_))));
assert!(matches!(
ed.resize("ext", &[4, 1]),
Err(Error::InvalidArgument(_))
));
assert!(matches!(
ed.resize("flat", &[4]),
Err(Error::InvalidArgument(_))
));
assert!(matches!(
ed.set_attr("/", "", &AttrValue::I64(1)),
Err(Error::InvalidArgument(_))
));
// No-ops write nothing.
ed.resize("ext", &[5]).unwrap();
ed.write_values("ext", &Selection::None, &[] as &[i32])
.unwrap();
drop(ed);
assert!(std::fs::read(&path).unwrap() == before);
}
crates/clawhdf5/tests/indexed_lookup_interop.rs
+612
View File
@@ -0,0 +1,612 @@
//! Looking one name up in a dense group (links in a fractal heap, indexed by
//! a v2 B-tree of name hashes) or in dense attribute storage reads the name
//! index, not every link: O(log n) index nodes and only the links whose
//! lookup3 hash equals the name's. Before, every lookup decoded all n links,
//! so opening each child of a 35 001-link group by name decoded ~1.2e9.
//!
//! The file is written by h5py (libhdf5 orders the index), with names whose
//! hashes collide, and every result is compared with what h5py reads.
//!
//! Skipped when python3 with h5py is unavailable, unless
//! `CLAWHDF5_REQUIRE_INTEROP=1`.
use std::collections::{BTreeMap, HashMap};
use std::process::Command;
use std::sync::OnceLock;
use std::time::{Duration, Instant};
use clawhdf5::{AttrValue, File, LazyFile, MmapFile};
use clawhdf5_format::checksum::jenkins_lookup3;
use clawhdf5_format::error::FormatError;
use clawhdf5_format::lookup_stats;
fn python() -> String {
std::env::var("CLAWHDF5_PYTHON").unwrap_or_else(|_| "python3".to_string())
}
fn interop_required() -> bool {
std::env::var("CLAWHDF5_REQUIRE_INTEROP").is_ok_and(|v| v == "1")
}
fn python_available() -> bool {
Command::new(python())
.args(["-c", "import h5py"])
.output()
.map(|o| o.status.success())
.unwrap_or(false)
}
macro_rules! skip_if_no_python {
() => {
if !python_available() {
assert!(
!interop_required(),
"CLAWHDF5_REQUIRE_INTEROP=1 but python3 with h5py is not available"
);
eprintln!("SKIP: python3 with h5py not available");
return;
}
};
}
fn run_python(script: &str) -> String {
let output = Command::new(python())
.args(["-c", script])
.output()
.expect("failed to run python");
assert!(
output.status.success(),
"Python script failed:\nSTDOUT: {}\nSTDERR: {}",
String::from_utf8_lossy(&output.stdout),
String::from_utf8_lossy(&output.stderr)
);
String::from_utf8_lossy(&output.stdout).trim().to_string()
}
/// Links in the big group, as libhdf5's `h5stat_newgrat.h5` has.
const LINKS: usize = 35_001;
/// Attributes on the dense-attribute dataset.
const ATTRS: usize = 3_000;
/// Pairs of distinct names with equal lookup3 hashes, found by search (the
/// hash is fixed, so the pairs are too).
fn colliding_pairs(count: usize) -> Vec<(String, String)> {
let mut seen: HashMap<u32, String> = HashMap::new();
let mut pairs = Vec::new();
for i in 0.. {
let name = format!("c{i}");
let h = jenkins_lookup3(name.as_bytes());
if let Some(first) = seen.insert(h, name.clone()) {
pairs.push((first, name));
if pairs.len() == count {
break;
}
}
}
pairs
}
/// What h5py reads: link values, attribute values, and which of the
/// missing names it finds as links and as attributes (none).
type H5pyView = (
BTreeMap<String, i64>,
BTreeMap<String, i64>,
Vec<String>,
Vec<String>,
);
struct Fixture {
_dir: tempfile::TempDir,
path: String,
/// Names in the big group, with the value of the scalar dataset each
/// links to, as h5py reads them.
links: BTreeMap<String, i64>,
/// Names that are not links but hash like one that is.
missing_links: Vec<String>,
/// Attributes of `/x`, as h5py reads them.
attrs: BTreeMap<String, i64>,
missing_attrs: Vec<String>,
}
fn fixture() -> &'static Fixture {
static FIXTURE: OnceLock<Fixture> = OnceLock::new();
FIXTURE.get_or_init(|| {
let pairs = colliding_pairs(6);
for (a, b) in &pairs {
assert_ne!(a, b);
assert_eq!(jenkins_lookup3(a.as_bytes()), jenkins_lookup3(b.as_bytes()));
}
// Pairs 0-2 both present (either can be the one libhdf5 orders
// first), pairs 3-5 only the first: its partner must not be found.
// "k69209"/"k155448" is the pair the writer once misordered.
let mut present: Vec<String> = vec!["k69209".into(), "k155448".into()];
let mut missing: Vec<String> = Vec::new();
for (i, (a, b)) in pairs.into_iter().enumerate() {
present.push(a);
if i < 3 {
present.push(b);
} else {
missing.push(b);
}
}
missing.extend(["", "nope", "n35001x", "N1"].map(String::from));
let mut links = present.clone();
let mut i = 0;
while links.len() < LINKS {
links.push(format!("n{i}"));
i += 1;
}
let mut attrs = present.clone();
attrs.extend((0..ATTRS - present.len()).map(|i| format!("a{i}")));
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("big.h5").display().to_string();
// The names go through a file: 35 001 of them overflow an argument.
let names = dir.path().join("names.json");
std::fs::write(
&names,
serde_json::to_string(&(&links, &attrs, &missing)).unwrap(),
)
.unwrap();
let names = names.display();
let out = run_python(&format!(
"import h5py, json, numpy as np\n\
links, attrs, missing = json.load(open(r'{names}'))\n\
with h5py.File(r'{path}', 'w', libver='latest') as f:\n\
\x20 g = f.create_group('g')\n\
\x20 for i, n in enumerate(links):\n\
\x20 g.create_dataset(n, data=np.int64(i))\n\
\x20 x = f.create_dataset('x', data=np.int64(0))\n\
\x20 for i, n in enumerate(attrs):\n\
\x20 x.attrs[n] = np.int64(1000 + i)\n\
with h5py.File(r'{path}', 'r') as f:\n\
\x20 g, a = f['g'], f['x'].attrs\n\
\x20 print(json.dumps([{{n: int(g[n][()]) for n in g}}, {{n: int(a[n]) for n in a}},\n\
\x20 [n for n in missing if n and n in g], [n for n in missing if n and n in a]]))",
));
let (links, attrs, found_links, found_attrs): H5pyView =
serde_json::from_str(&out).unwrap();
assert_eq!(links.len(), LINKS);
assert_eq!(attrs.len(), ATTRS);
assert!(found_links.is_empty() && found_attrs.is_empty());
Fixture {
_dir: dir,
path,
links,
missing_links: missing.clone(),
attrs,
missing_attrs: missing,
}
})
}
fn is_not_found(e: &clawhdf5::Error) -> bool {
matches!(e, clawhdf5::Error::Format(FormatError::PathNotFound(_)))
}
#[test]
fn one_link_lookup_reads_the_index_not_every_link() {
skip_if_no_python!();
let fx = fixture();
let f = File::open(&fx.path).unwrap();
let g = f.group("g").unwrap();
for (name, value) in &fx.links {
lookup_stats::reset();
let ds = g.dataset(name).unwrap();
// One link decoded per lookup, two where hashes collide — not 35 001.
let read = lookup_stats::heap_objects_read();
assert!(read <= 2, "looking up {name} read {read} heap objects");
assert_eq!(ds.read_i64().unwrap(), vec![*value], "{name}");
}
for name in &fx.missing_links {
lookup_stats::reset();
let err = g.dataset(name).unwrap_err();
assert!(is_not_found(&err), "{name:?}: {err:?}");
assert!(lookup_stats::heap_objects_read() <= 2, "{name:?}");
}
// A path resolves each component the same way.
for name in ["k155448", "n0", "n34000"] {
lookup_stats::reset();
let ds = f.dataset(&format!("/g/{name}")).unwrap();
assert!(lookup_stats::heap_objects_read() <= 2);
assert_eq!(ds.read_i64().unwrap(), vec![fx.links[name]]);
}
}
#[test]
fn one_attribute_lookup_reads_the_index_not_every_attribute() {
skip_if_no_python!();
let fx = fixture();
let f = File::open(&fx.path).unwrap();
let x = f.dataset("x").unwrap();
let all = x.attrs().unwrap();
assert_eq!(all.len(), ATTRS);
for (name, value) in &fx.attrs {
lookup_stats::reset();
let got = x.attr(name).unwrap();
assert!(lookup_stats::heap_objects_read() <= 2, "{name}");
assert!(
matches!(got, Some(AttrValue::I64(v)) if v == *value),
"{name}: {got:?}"
);
assert!(matches!(all.get(name), Some(AttrValue::I64(v)) if v == value));
}
for name in &fx.missing_attrs {
lookup_stats::reset();
assert!(x.attr(name).unwrap().is_none(), "{name:?}");
assert!(lookup_stats::heap_objects_read() <= 2, "{name:?}");
}
// Compact attributes (on the root group: none) and a group's attributes.
assert!(f.root().attr("k69209").unwrap().is_none());
}
/// Every child of the big group opened by name through each file type,
/// within `limit`: with a scan per lookup this is ~1.2e9 link decodes.
#[test]
fn opening_every_child_of_a_35001_link_group_by_name_is_quick() {
skip_if_no_python!();
let fx = fixture();
let limit = Duration::from_secs(120);
let started = Instant::now();
let check_time = |n: usize| {
assert!(
started.elapsed() < limit,
"{n} lookups took {:?}",
started.elapsed()
);
};
let f = File::open(&fx.path).unwrap();
let g = f.group("g").unwrap();
for (n, (name, value)) in fx.links.iter().enumerate() {
assert_eq!(g.dataset(name).unwrap().read_i64().unwrap(), vec![*value]);
check_time(n);
}
// The listing hands out entries: open each by address.
let entries = g.entries().unwrap();
assert_eq!(entries.len(), LINKS);
for (name, address) in &entries {
let ds = f.dataset_at(*address).unwrap();
assert_eq!(ds.read_i64().unwrap(), vec![fx.links[name]]);
}
assert!(f.group_at(g_address(&f)).dataset("n0").is_ok());
let m = MmapFile::open(&fx.path).unwrap();
let mg = m.group("g").unwrap();
for (n, (name, value)) in fx.links.iter().enumerate() {
assert_eq!(mg.dataset(name).unwrap().read_i64().unwrap(), vec![*value]);
check_time(n);
}
assert!(mg.group("nope").is_err_and(|e| is_not_found(&e)));
let l = LazyFile::open_mmap(&fx.path).unwrap();
let lg = l.group("g").unwrap();
for (n, (name, value)) in fx.links.iter().enumerate() {
assert_eq!(lg.dataset(name).unwrap().read_i64().unwrap(), vec![*value]);
check_time(n);
}
let lx = l.dataset("x").unwrap();
assert!(
matches!(lx.attr("k155448").unwrap(), Some(AttrValue::I64(v)) if v == fx.attrs["k155448"])
);
assert!(
lg.dataset(&fx.missing_links[0])
.is_err_and(|e| is_not_found(&e))
);
}
fn g_address(f: &File) -> u64 {
f.root()
.entries()
.unwrap()
.into_iter()
.find(|(n, _)| n == "g")
.unwrap()
.1
}
/// Every kind of link, looked up by name in a dense group (through the name
/// index) and in a compact one, opens what h5py opens and nothing it cannot:
/// hard links, soft links (absolute, relative, to a group), and not a
/// dangling soft link, an external link or a missing name.
#[test]
fn links_of_every_kind_resolve_by_name_as_in_h5py() {
skip_if_no_python!();
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("links.h5").display().to_string();
// For each group and name: "dataset <value>", "group", or "none" as
// h5py sees it.
let out = run_python(&format!(
"import h5py, json, numpy as np\n\
with h5py.File(r'{path}', 'w', libver='latest') as f:\n\
\x20 for gname, n in (('dense', 20), ('compact', 2)):\n\
\x20 g = f.create_group(gname)\n\
\x20 for i in range(n):\n\
\x20 g.create_dataset(f'd{{i}}', data=np.int64(100 + i))\n\
\x20 s = g.create_group('sub')\n\
\x20 s.create_dataset('x', data=np.int64(7))\n\
\x20 g['abs'] = h5py.SoftLink(f'/{{gname}}/d1')\n\
\x20 g['rel'] = h5py.SoftLink('sub/x')\n\
\x20 g['tosub'] = h5py.SoftLink('sub')\n\
\x20 g['dangling'] = h5py.SoftLink('/nowhere')\n\
\x20 g['ext'] = h5py.ExternalLink('other.h5', '/y')\n\
names = ['d0', 'd1', 'sub', 'abs', 'rel', 'tosub', 'dangling', 'ext', 'nope', '']\n\
seen = {{}}\n\
with h5py.File(r'{path}', 'r') as f:\n\
\x20 for gname in ('dense', 'compact'):\n\
\x20 g = f[gname]\n\
\x20 for n in names:\n\
\x20 try:\n\
\x20 o = g[n] if n else None\n\
\x20 except (KeyError, OSError):\n\
\x20 o = None\n\
\x20 if isinstance(o, h5py.Dataset):\n\
\x20 seen[f'{{gname}}/{{n}}'] = f'dataset {{int(o[()])}}'\n\
\x20 elif isinstance(o, h5py.Group):\n\
\x20 seen[f'{{gname}}/{{n}}'] = 'group'\n\
\x20 else:\n\
\x20 seen[f'{{gname}}/{{n}}'] = 'none'\n\
print(json.dumps(seen))",
));
let seen: BTreeMap<String, String> = serde_json::from_str(&out).unwrap();
assert_eq!(seen.len(), 20);
let f = File::open(&path).unwrap();
// The dense group's links are in a heap, the compact group's in its
// header.
for (gname, dense) in [("dense", true), ("compact", false)] {
let g = f.group(gname).unwrap();
lookup_stats::reset();
g.dataset("d0").unwrap();
assert_eq!(lookup_stats::heap_objects_read() > 0, dense, "{gname}");
}
let m = MmapFile::open(&path).unwrap();
let l = LazyFile::open_mmap(&path).unwrap();
for (key, want) in &seen {
let (gname, name) = key.split_once('/').unwrap();
let got = {
let g = f.group(gname).unwrap();
match (g.dataset(name), g.group(name)) {
(Ok(ds), _) => format!("dataset {}", ds.read_i64().unwrap()[0]),
(Err(clawhdf5::Error::NotADataset(_)), Ok(sub)) => {
// A group: it has the child `x` (checks the address).
assert!(sub.dataset("x").is_ok() || name == "sub" || name == "tosub");
"group".to_string()
}
(Err(e), Err(e2)) => {
assert!(
is_not_found(&e) && is_not_found(&e2),
"{key}: {e:?} / {e2:?}"
);
"none".to_string()
}
(Err(e), Ok(_)) => panic!("{key}: dataset {e:?} but group ok"),
}
};
assert_eq!(&got, want, "{key}");
// The other readers agree, and a path through the group resolves the
// same way.
let mg = m.group(gname).unwrap();
let lg = l.group(gname).unwrap();
match want.strip_prefix("dataset ") {
Some(v) => {
let v: i64 = v.parse().unwrap();
assert_eq!(mg.dataset(name).unwrap().read_i64().unwrap(), vec![v]);
assert_eq!(lg.dataset(name).unwrap().read_i64().unwrap(), vec![v]);
let ds = f.dataset(&format!("/{gname}/{name}")).unwrap();
assert_eq!(ds.read_i64().unwrap(), vec![v], "{key}");
}
None if want == "group" => {
assert!(mg.group(name).unwrap().dataset("x").is_ok(), "{key}");
assert!(lg.group(name).unwrap().dataset("x").is_ok(), "{key}");
let ds = f.dataset(&format!("/{gname}/{name}/x")).unwrap();
assert_eq!(ds.read_i64().unwrap(), vec![7]);
}
None => {
assert!(mg.dataset(name).is_err_and(|e| is_not_found(&e)), "{key}");
assert!(lg.group(name).is_err_and(|e| is_not_found(&e)), "{key}");
}
}
}
}
/// The link name index (v2 B-tree, record type 5) of the big group: its
/// depth and root node address, read from the one type-5 `BTHD` in the file.
fn name_index_root(bytes: &[u8]) -> (u16, usize) {
let headers: Vec<usize> = bytes
.windows(4)
.enumerate()
.filter(|(i, w)| *w == b"BTHD" && bytes.get(i + 5) == Some(&5))
.map(|(i, _)| i)
.collect();
assert_eq!(headers.len(), 1, "type-5 B-tree headers at {headers:?}");
let h = headers[0];
// signature, version, type, node size (4), record size (2), depth (2),
// split and merge percent, root address (8).
let depth = u16::from_le_bytes([bytes[h + 12], bytes[h + 13]]);
let root = u64::from_le_bytes(bytes[h + 16..h + 24].try_into().unwrap());
(depth, usize::try_from(root).unwrap())
}
/// One byte changed in a key of the name index's root (an internal node)
/// must be an error, not a name quietly routed to the wrong child and
/// reported missing: lookups prune children by those keys. libhdf5 checks
/// the internal node's checksum and refuses the group; so must we, for a
/// lookup and for a listing.
#[test]
fn a_corrupt_internal_index_node_is_an_error_not_a_missing_name() {
skip_if_no_python!();
let fx = fixture();
let mut bytes = std::fs::read(&fx.path).unwrap();
let (depth, root) = name_index_root(&bytes);
assert!(depth >= 2, "want a deep index, got depth {depth}");
assert_eq!(&bytes[root..root + 4], b"BTIN");
// Signature, version, type, then record 0: its name hash comes first.
bytes[root + 6] ^= 0x5a;
let dir = tempfile::tempdir().unwrap();
let bad = dir.path().join("bad.h5");
std::fs::write(&bad, &bytes).unwrap();
let bad = bad.display().to_string();
let is_checksum = |e: &clawhdf5::Error| {
matches!(
e,
clawhdf5::Error::Format(FormatError::ChecksumMismatch { .. })
)
};
let f = File::open(&bad).unwrap();
let g = f.group("g").unwrap();
// Every name, present or not, goes through the root.
for name in fx.links.keys().step_by(97).chain(&fx.missing_links) {
let err = g.dataset(name).map(|_| ()).unwrap_err();
assert!(is_checksum(&err), "dataset({name:?}): {err:?}");
}
let err = f.dataset("/g/n0").map(|_| ()).unwrap_err();
assert!(is_checksum(&err), "path: {err:?}");
let err = g.datasets().unwrap_err();
assert!(is_checksum(&err), "listing: {err:?}");
let err = g.entries().unwrap_err();
assert!(is_checksum(&err), "entries: {err:?}");
let m = MmapFile::open(&bad).unwrap();
let mg = m.group("g").unwrap();
assert!(mg.dataset("n0").is_err_and(|e| is_checksum(&e)));
assert!(mg.datasets().is_err_and(|e| is_checksum(&e)));
let l = LazyFile::open_mmap(&bad).unwrap();
let lg = l.group("g").unwrap();
assert!(lg.dataset("n0").is_err_and(|e| is_checksum(&e)));
assert!(lg.datasets().is_err_and(|e| is_checksum(&e)));
// libhdf5 refuses both too.
let out = run_python(&format!(
"import h5py\n\
r = []\n\
with h5py.File(r'{bad}', 'r') as f:\n\
\x20 g = f['g']\n\
\x20 for op in (lambda: g['n0'], lambda: list(g)):\n\
\x20 try:\n\
\x20 op()\n\
\x20 r.append('ok')\n\
\x20 except Exception as e:\n\
\x20 r.append('checksum' if 'checksum' in str(e) else repr(e))\n\
print(' '.join(r))",
));
assert_eq!(out, "checksum checksum");
}
/// Rename the one link called `from` to `to` (same length) in `bytes`, and
/// re-checksum the object header chunk holding it: two links of one name,
/// which libhdf5 cannot write.
fn rename_link_in_header(bytes: &mut [u8], from: &[u8], to: &[u8]) {
assert_eq!(from.len(), to.len());
let find = |hay: &[u8], needle: &[u8]| hay.windows(needle.len()).position(|w| w == needle);
let at = find(bytes, from).expect("link name");
assert!(find(&bytes[at + 1..], from).is_none(), "name not unique");
bytes[at..at + to.len()].copy_from_slice(to);
// The v2 object header (chunk 0) holding it.
let ohdr = bytes[..at]
.windows(4)
.rposition(|w| w == b"OHDR")
.expect("OHDR");
let flags = bytes[ohdr + 5];
let mut pos = ohdr + 6;
if flags & 0x20 != 0 {
pos += 16; // times
}
if flags & 0x10 != 0 {
pos += 4; // attribute phase change
}
let width = 1usize << (flags & 3);
let mut size = [0u8; 8];
size[..width].copy_from_slice(&bytes[pos..pos + width]);
let end = pos + width + usize::try_from(u64::from_le_bytes(size)).unwrap();
assert!(at < end, "name outside chunk 0");
let sum = jenkins_lookup3(&bytes[ohdr..end]);
bytes[end..end + 4].copy_from_slice(&sum.to_le_bytes());
}
/// Two soft links of one name (a damaged or hand-made group; libhdf5
/// cannot create one), one dangling: only the first counts, as in libhdf5,
/// which opens the first Link message of a name and fails if it dangles.
/// Lookup, path and listing agree — before, the listing skipped a dangling
/// first link and listed the name via the second, which lookup did not
/// follow, and path resolution followed the last.
#[test]
fn of_two_links_with_one_name_the_first_wins_everywhere() {
skip_if_no_python!();
let dir = tempfile::tempdir().unwrap();
for dangling_first in [true, false] {
let path = dir
.path()
.join(format!("dup_{dangling_first}.h5"))
.display()
.to_string();
let (first, second) = if dangling_first {
("/nowhere_xyz", "/d")
} else {
("/d", "/nowhere_xyz")
};
run_python(&format!(
"import h5py, numpy as np\n\
with h5py.File(r'{path}', 'w', libver='latest') as f:\n\
\x20 f.create_dataset('d', data=np.int64(42))\n\
\x20 s = f.create_group('s')\n\
\x20 s['dup_A'] = h5py.SoftLink('{first}')\n\
\x20 s['dup_B'] = h5py.SoftLink('{second}')",
));
let mut bytes = std::fs::read(&path).unwrap();
rename_link_in_header(&mut bytes, b"dup_B", b"dup_A");
std::fs::write(&path, &bytes).unwrap();
// What libhdf5 opens under that name: both names listed, first link
// followed.
let out = run_python(&format!(
"import h5py\n\
with h5py.File(r'{path}', 'r') as f:\n\
\x20 s = f['s']\n\
\x20 assert list(s) == ['dup_A', 'dup_A'], list(s)\n\
\x20 try:\n\
\x20 print(int(s['dup_A'][()]))\n\
\x20 except KeyError:\n\
\x20 print('none')",
));
let want = if dangling_first { "none" } else { "42" };
assert_eq!(out, want, "h5py, dangling first: {dangling_first}");
let want = (!dangling_first).then_some(42i64);
let f = File::open(&path).unwrap();
let s = f.group("s").unwrap();
let got = |r: Result<clawhdf5::Dataset<'_>, clawhdf5::Error>| match r {
Ok(ds) => Some(ds.read_i64().unwrap()[0]),
Err(e) => {
assert!(is_not_found(&e), "{e:?}");
None
}
};
assert_eq!(got(s.dataset("dup_A")), want, "lookup, {dangling_first}");
assert_eq!(got(f.dataset("/s/dup_A")), want, "path, {dangling_first}");
let listed = s.datasets().unwrap();
let listed_n = listed.iter().filter(|n| *n == "dup_A").count();
assert_eq!(listed_n, usize::from(want.is_some()), "{listed:?}");
let entries = s.entries().unwrap();
assert_eq!(entries.len(), listed_n, "{entries:?}");
let m = MmapFile::open(&path).unwrap();
let l = LazyFile::open_mmap(&path).unwrap();
let (mg, lg) = (m.group("s").unwrap(), l.group("s").unwrap());
match want {
Some(v) => {
assert_eq!(mg.dataset("dup_A").unwrap().read_i64().unwrap(), vec![v]);
assert_eq!(lg.dataset("dup_A").unwrap().read_i64().unwrap(), vec![v]);
}
None => {
assert!(mg.dataset("dup_A").is_err_and(|e| is_not_found(&e)));
assert!(lg.dataset("dup_A").is_err_and(|e| is_not_found(&e)));
}
}
assert_eq!(mg.datasets().unwrap(), listed);
assert_eq!(lg.datasets().unwrap(), listed);
}
}
crates/clawhdf5/tests/plugin_filters_interop.rs
+463 -7
View File
@@ -495,16 +495,15 @@ fn lzf_written_by_clawhdf5_reads_in_h5py() {
});
}
/// ZFP is not implemented, and Blosc2 is not in a build without the
/// `blosc2` feature: reading them must be a clear error naming the filter,
/// never data.
/// Blosc2 and ZFP are not in a build without their features: reading them
/// must be a clear error naming the filter, never data.
#[test]
fn unimplemented_filters_are_a_clear_error() {
fn filters_left_out_of_the_build_are_a_clear_error() {
if !have_python("h5py, hdf5plugin") {
return;
}
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("unimplemented.h5");
let path = dir.path().join("left_out.h5");
run_python(
r#"
import sys
@@ -517,7 +516,10 @@ with h5py.File(sys.argv[1], 'w') as f:
&[path.to_str().unwrap()],
);
let file = File::open(&path).unwrap();
let mut missing = vec![("zfp", 32013u16, "ZFP")];
let mut missing = Vec::new();
if !cfg!(feature = "zfp") {
missing.push(("zfp", 32013u16, "ZFP"));
}
if !cfg!(feature = "blosc2") {
missing.push(("blosc2", 32026, "Blosc2"));
}
@@ -526,7 +528,7 @@ with h5py.File(sys.argv[1], 'w') as f:
.dataset(name)
.unwrap()
.read_selection(&Selection::All)
.expect_err("an unimplemented filter must not read");
.expect_err("a filter left out of the build must not read");
let msg = err.to_string();
assert!(
msg.contains(&id.to_string()) && msg.contains(label),
@@ -586,3 +588,457 @@ with h5py.File(sys.argv[1], 'w') as f:
let want: Vec<i32> = (0..16).collect();
assert_eq!(file.dataset("lzf_ok").unwrap().read_i32().unwrap(), want);
}
/// A family of datasets for the filter-mask tests: element type, chunk
/// length along the last dimension, the h5py `create_dataset` keywords of
/// the same filters, and how chunk `k` is filled.
#[cfg(feature = "lzf")]
struct MaskFamily {
name: &'static str,
/// 1 (`u1`) or 4 (`<i4`).
elem: usize,
chunk: u64,
h5py_kw: &'static str,
build: fn(&mut clawhdf5_format::type_builders::DatasetBuilder),
fill: MaskFill,
}
#[cfg(feature = "lzf")]
#[derive(Clone, Copy)]
enum MaskFill {
/// `[x, 0, 0, 0, 0]`: LZF's output is exactly the chunk's size, so
/// libhdf5's LZF filter fails and h5py stores the chunk raw.
FiveBytes,
/// Even chunks random bytes, odd chunks one repeated value.
Alternating,
/// Every chunk one repeated value.
Compressible,
}
/// The index shapes of the mask tests: (label, shape, chunk dims, maxshape
/// with `u64::MAX` unlimited) for chunk length `c`. Single chunk, Fixed
/// Array, Extensible Array and version-2 B-tree, in that order — every
/// index the whole-file writer builds.
#[cfg(feature = "lzf")]
#[allow(clippy::type_complexity)]
fn mask_layouts(c: u64) -> Vec<(&'static str, Vec<u64>, Vec<u64>, Option<Vec<u64>>)> {
vec![
("single", vec![c], vec![c], None),
("fixed", vec![4 * c], vec![c], None),
("ea", vec![4 * c], vec![c], Some(vec![u64::MAX])),
(
"bt2",
vec![2, 2 * c],
vec![1, c],
Some(vec![u64::MAX, u64::MAX]),
),
]
}
/// Raw little-endian bytes of the dataset `fam` fills over `shape`.
#[cfg(feature = "lzf")]
fn mask_data(fam: &MaskFamily, shape: &[u64], seed: u64) -> Vec<u8> {
let c = fam.chunk as usize;
let cols = *shape.last().unwrap() as usize;
let n: usize = shape.iter().product::<u64>() as usize;
let chunks_per_row = cols.div_ceil(c);
let mut state = seed;
let mut noise = move || {
state = state.wrapping_add(0x9E37_79B9_7F4A_7C15);
let mut z = state;
z = (z ^ (z >> 30)).wrapping_mul(0xBF58_476D_1CE4_E5B9);
z = (z ^ (z >> 27)).wrapping_mul(0x94D0_49BB_1331_11EB);
z ^ (z >> 31)
};
let mut out = Vec::with_capacity(n * fam.elem);
for i in 0..n {
let (row, col) = (i / cols, i % cols);
let k = row * chunks_per_row + col / c;
let v: u64 = match fam.fill {
MaskFill::FiveBytes if col % c == 0 => 182 - k as u64,
MaskFill::FiveBytes => 0,
MaskFill::Alternating if k.is_multiple_of(2) => noise(),
MaskFill::Alternating | MaskFill::Compressible => 7,
};
out.extend_from_slice(&v.to_le_bytes()[..fam.elem]);
}
out
}
#[cfg(feature = "lzf")]
fn mask_families() -> Vec<MaskFamily> {
#[cfg_attr(not(feature = "blosc"), allow(unused_mut))]
let mut v = vec![
MaskFamily {
name: "lzf5",
elem: 1,
chunk: 5,
h5py_kw: "dict(compression='lzf')",
build: |d| {
d.with_lzf().without_shuffle();
},
fill: MaskFill::FiveBytes,
},
MaskFamily {
name: "lzf",
elem: 4,
chunk: 64,
h5py_kw: "dict(compression='lzf')",
build: |d| {
d.with_lzf().without_shuffle();
},
fill: MaskFill::Alternating,
},
MaskFamily {
name: "mix",
elem: 4,
chunk: 8,
h5py_kw: "dict(compression='lzf', shuffle=True, fletcher32=True)",
build: |d| {
d.with_lzf().with_shuffle().with_fletcher32();
},
fill: MaskFill::Alternating,
},
MaskFamily {
name: "lzfc",
elem: 4,
chunk: 64,
h5py_kw: "dict(compression='lzf')",
build: |d| {
d.with_lzf().without_shuffle();
},
fill: MaskFill::Compressible,
},
];
#[cfg(feature = "blosc")]
{
use clawhdf5_format::chunked_write::{BloscCodec, BloscShuffle};
v.push(MaskFamily {
name: "blosc",
elem: 4,
chunk: 64,
h5py_kw: "hdf5plugin.Blosc(cname='lz4', clevel=5, shuffle=hdf5plugin.Blosc.SHUFFLE)",
build: |d| {
d.with_blosc(BloscCodec::Lz4, 5, BloscShuffle::Byte);
},
fill: MaskFill::Alternating,
});
v.push(MaskFamily {
name: "blosc0",
elem: 4,
chunk: 64,
h5py_kw: "hdf5plugin.Blosc(cname='lz4', clevel=0, shuffle=hdf5plugin.Blosc.SHUFFLE)",
build: |d| {
d.with_blosc(BloscCodec::Lz4, 0, BloscShuffle::Byte);
},
fill: MaskFill::Compressible,
});
}
v
}
/// Builds h5py twins of our datasets and prints, per dataset, the filter
/// masks by chunk offset in our file and in the twin.
const MASK_TWIN: &str = r#"
import sys, numpy as np, h5py
try:
import hdf5plugin
except ImportError:
hdf5plugin = None
ours, twin, spec = sys.argv[1], sys.argv[2], eval(sys.argv[3])
def masks(ds):
return sorted((tuple(ds.id.get_chunk_info(k).chunk_offset), ds.id.get_chunk_info(k).filter_mask)
for k in range(ds.id.get_num_chunks()))
with h5py.File(ours, 'r') as o, h5py.File(twin, 'w', libver='v114') as t:
for name, dt, shape, chunks, maxshape, kw, raw in spec:
want = np.fromfile(raw, dtype=dt).reshape(shape)
assert np.array_equal(o[name][()], want), name
t.create_dataset(name, data=want, chunks=chunks, maxshape=maxshape, **eval(kw))
print(name, masks(o[name]), '|', masks(t[name]))
"#;
/// h5py (libhdf5) rewrites every chunk of our datasets — random chunks
/// become compressible and the other way round; the `[x,0,0,0,0]` chunks
/// change in place at the same size — then extends the resizable ones with
/// random data, and saves what each dataset must now hold. Prints the
/// datasets h5dump can decode (no chunk left LZF-encoded: h5dump has no
/// LZF filter).
const MASK_REWRITE: &str = r#"
import sys, numpy as np, h5py
try:
import hdf5plugin
except ImportError:
hdf5plugin = None
ours, spec = sys.argv[1], eval(sys.argv[2])
rng = np.random.default_rng(3)
def noise(shape, dt):
return rng.integers(0, 256, int(np.prod(shape)) * np.dtype(dt).itemsize,
dtype=np.uint8).view(dt).reshape(shape)
dumpable = []
with h5py.File(ours, 'r+') as f:
for name, dt, shape, chunks, maxshape, kw, raw in spec:
d = f[name]
want = d[()]
for s in d.iter_chunks():
blk = want[s]
if dt == 'u1':
blk.flat[-1] = 1
elif (blk == blk.flat[0]).all():
blk[...] = noise(blk.shape, dt)
else:
blk[...] = 5
d[...] = want
if maxshape is not None:
new = tuple(n + c for n, c in zip(shape, chunks))
grown = noise(new, dt)
grown[tuple(slice(0, n) for n in shape)] = want
d.resize(new)
d[...] = grown
want = grown
want.tofile(raw + '.want')
pl = d.id.get_create_plist()
ids = [pl.get_filter(i)[0] for i in range(pl.get_nfilters())]
if 32000 in ids:
bit = 1 << ids.index(32000)
if not all(d.id.get_chunk_info(k).filter_mask & bit for k in range(d.id.get_num_chunks())):
continue
dumpable.append(name)
with h5py.File(ours, 'r') as f:
for name, dt, shape, chunks, maxshape, kw, raw in spec:
want = np.fromfile(raw + '.want', dtype=dt).reshape(f[name].shape)
assert np.array_equal(f[name][()], want), name
print(' '.join(dumpable))
"#;
/// Optional filters that fail are skipped in files `FileBuilder` writes,
/// exactly as libhdf5 skips them: an LZF or Blosc output no smaller than the
/// chunk leaves the chunk stored unfiltered with the filter's mask bit set.
/// The writer used to store every chunk filtered with mask 0. For LZF, a
/// chunk whose LZF stream is exactly the chunk's size (`[x,0,0,0,0]`) was
/// then corrupted by the first libhdf5 rewrite of it: libhdf5 stores the
/// new data raw at the same size and, the size being unchanged, leaves the
/// stale mask in the index, so h5py could no longer read the dataset.
///
/// For every family × every chunk index the writer builds (single chunk,
/// Fixed Array, Extensible Array, version-2 B-tree): our masks equal those
/// of an h5py-written twin of the same data; then h5py r+ rewrites and
/// extends the datasets, and h5py, h5dump (where it has the filter) and our
/// reader read every value.
#[cfg(feature = "lzf")]
#[test]
fn skipped_optional_filters_are_masked_as_libhdf5_masks_them() {
let modules = if cfg!(feature = "blosc") {
"h5py, numpy, hdf5plugin"
} else {
"h5py, numpy"
};
if !have_python(modules) {
return;
}
let dir = tempfile::tempdir().unwrap();
let ours = dir.path().join("ours.h5");
let twin = dir.path().join("twin.h5");
let mut fb = clawhdf5::FileBuilder::new();
let mut spec = Vec::new();
let mut names = Vec::new();
for (fi, fam) in mask_families().iter().enumerate() {
for (label, shape, chunks, maxshape) in mask_layouts(fam.chunk) {
let name = format!("{}_{label}", fam.name);
let data = mask_data(fam, &shape, fi as u64 * 31 + shape.len() as u64);
let raw = dir.path().join(format!("{name}.raw"));
std::fs::write(&raw, &data).unwrap();
let ds = fb.create_dataset(&name);
if fam.elem == 1 {
ds.with_u8_data(&data);
} else {
let v: Vec<i32> = data
.as_chunks::<4>()
.0
.iter()
.map(|&b| i32::from_le_bytes(b))
.collect();
ds.with_i32_data(&v);
}
ds.with_shape(&shape).with_chunks(&chunks);
if let Some(ms) = &maxshape {
ds.with_maxshape(ms);
}
(fam.build)(ds);
let py_tuple = |v: &[u64]| {
let items: Vec<String> = v
.iter()
.map(|&d| {
if d == u64::MAX {
"None".into()
} else {
d.to_string()
}
})
.collect();
format!("({},)", items.join(","))
};
spec.push(format!(
"({name:?}, {:?}, {}, {}, {}, {:?}, {:?})",
if fam.elem == 1 { "u1" } else { "<i4" },
py_tuple(&shape),
py_tuple(&chunks),
maxshape.as_deref().map_or("None".into(), py_tuple),
fam.h5py_kw,
raw.to_str().unwrap(),
));
names.push((name, fam.elem));
}
}
fb.write(&ours).unwrap();
let spec = format!("[{}]", spec.join(", "));
// Our masks are h5py's, chunk by chunk.
let out = run_python(
MASK_TWIN,
&[ours.to_str().unwrap(), twin.to_str().unwrap(), &spec],
);
let mut skipped = 0;
for line in out.lines() {
let (name, rest) = line.split_once(' ').unwrap();
let (got, want) = rest.split_once(" | ").unwrap();
assert_eq!(got, want, "{name}: our filter masks (left) vs h5py's");
skipped += usize::from(got.contains("), 1)") || got.contains("), 2)"));
}
assert_eq!(out.lines().count(), names.len());
assert!(
skipped >= 12,
"too few datasets with skipped filters:\n{out}"
);
// libhdf5 rewrites and extends them; everyone reads the new values.
let dumpable = run_python(MASK_REWRITE, &[ours.to_str().unwrap(), &spec]);
let plugin_path = run_python("import hdf5plugin; print(hdf5plugin.PLUGIN_PATH)", &[]);
let file = File::open(&ours).unwrap();
for (name, elem) in &names {
let want = std::fs::read(dir.path().join(format!("{name}.raw.want"))).unwrap();
let got = file
.dataset(name)
.unwrap()
.read_selection(&Selection::All)
.unwrap();
assert!(got == want, "{name}: our reader after h5py r+");
if !dumpable.split(' ').any(|d| d == name) || Command::new("h5dump").output().is_err() {
continue;
}
let o = Command::new("h5dump")
.env("HDF5_PLUGIN_PATH", &plugin_path)
.args(["-d", name, "-y", "-w", "0", ours.to_str().unwrap()])
.output()
.unwrap();
assert!(
o.status.success(),
"h5dump -d {name}: {}",
String::from_utf8_lossy(&o.stderr)
);
let s = String::from_utf8_lossy(&o.stdout).into_owned();
let vals: Vec<i64> = s
.split_once("DATA {")
.and_then(|(_, r)| r.split_once('}'))
.map(|(d, _)| {
d.split(|c: char| c == ',' || c.is_whitespace())
.filter(|t| !t.is_empty())
.map(|t| t.parse::<i64>().unwrap())
.collect()
})
.unwrap_or_default();
let want_vals: Vec<i64> = want
.chunks_exact(*elem)
.map(|b| {
if *elem == 1 {
i64::from(b[0])
} else {
i64::from(i32::from_le_bytes(b.try_into().unwrap()))
}
})
.collect();
assert_eq!(vals, want_vals, "h5dump -d {name}");
}
assert!(
dumpable.split(' ').any(|d| d.starts_with("lzf5_")),
"{dumpable}"
);
}
/// Files whose chunks all compress are written exactly as before optional
/// filters could be skipped: every mask is 0 and nothing else changed. The
/// hashes are of the files the writer produced before that change.
#[cfg(feature = "lzf")]
#[test]
fn files_whose_chunks_all_compress_are_unchanged() {
use clawhdf5_format::checksum::jenkins_lookup3;
#[allow(clippy::type_complexity)]
#[cfg_attr(not(feature = "blosc"), allow(unused_mut))]
let mut cases: Vec<(
&str,
fn(&mut clawhdf5_format::type_builders::DatasetBuilder),
(usize, u32),
)> = vec![
(
"lzf_fixed",
|d| {
d.with_i32_data(&ramp_i32(4000))
.with_chunks(&[500])
.with_lzf();
},
(3965, 449169442),
),
(
"lzf_ea_noshuffle",
|d| {
d.with_i32_data(&ramp_i32(4000))
.with_chunks(&[700])
.with_maxshape(&[u64::MAX])
.with_lzf()
.without_shuffle();
},
(7213, 4277403206),
),
(
"mix_bt2",
|d| {
d.with_f64_data(&ramp_f64(40 * 60))
.with_shape(&[40, 60])
.with_chunks(&[16, 16])
.with_maxshape(&[u64::MAX, u64::MAX])
.with_lzf()
.with_fletcher32();
},
(11495, 3340532700),
),
(
"lzf_single",
|d| {
d.with_u8_data(&ramp_u8(3000))
.with_chunks(&[3000])
.with_lzf();
},
(546, 690805477),
),
];
#[cfg(feature = "blosc")]
cases.push((
"blosc_fixed",
|d| {
use clawhdf5_format::chunked_write::{BloscCodec, BloscShuffle};
d.with_i32_data(&ramp_i32(5000))
.with_chunks(&[1024])
.with_blosc(BloscCodec::Lz4, 5, BloscShuffle::Byte);
},
(2776, 4278611376),
));
for (name, build, want) in &cases {
let mut fb = clawhdf5::FileBuilder::new();
build(fb.create_dataset("d"));
let bytes = fb.finish().unwrap();
assert_eq!(
(bytes.len(), jenkins_lookup3(&bytes)),
*want,
"{name}: (length, lookup3 hash) of the file"
);
}
}
crates/clawhdf5/tests/zfp_interop.rs
+277
View File
@@ -0,0 +1,277 @@
//! ZFP (H5Z-ZFP, filter 32013) against libhdf5 + libzfp.
//!
//! h5py with hdf5plugin (H5Z-ZFP 1.1.1, zfp 1.0.1) writes datasets in every
//! ZFP mode (fixed rate, precision and accuracy, reversible, expert) for
//! each type ZFP supports (int32, int64, float, double), in 1 to 4
//! dimensions, with chunks that are partial at the dataset's edges, blocks
//! that are partial at the chunks' edges, and chunks with unit dimensions
//! (a lower-dimensional ZFP field). Next to each it stores what h5py reads
//! back, unfiltered, and clawhdf5 must read the ZFP dataset bit for bit
//! equal to that: the decoder is deterministic, so lossy modes have one
//! right answer.
//!
//! Skipped when python3 with h5py and hdf5plugin is unavailable, unless
//! `CLAWHDF5_REQUIRE_INTEROP=1`.
#![cfg(feature = "zfp")]
use std::collections::BTreeMap;
use std::process::Command;
use clawhdf5::File;
use clawhdf5_format::selection::Selection;
fn python() -> String {
std::env::var("CLAWHDF5_PYTHON").unwrap_or_else(|_| "python3".to_string())
}
fn have_python() -> bool {
let ok = Command::new(python())
.args(["-c", "import h5py, hdf5plugin"])
.output()
.map(|o| o.status.success())
.unwrap_or(false);
if ok {
return true;
}
assert!(
!std::env::var("CLAWHDF5_REQUIRE_INTEROP").is_ok_and(|v| v == "1"),
"CLAWHDF5_REQUIRE_INTEROP=1 but python3 with h5py and hdf5plugin is not available"
);
eprintln!("SKIP: python3 with h5py and hdf5plugin not available");
false
}
fn run_python(script: &str, args: &[&str]) -> String {
let output = Command::new(python())
.arg("-c")
.arg(script)
.args(args)
.output()
.expect("failed to run python");
assert!(
output.status.success(),
"Python script failed:\nSTDOUT: {}\nSTDERR: {}",
String::from_utf8_lossy(&output.stdout),
String::from_utf8_lossy(&output.stderr)
);
String::from_utf8_lossy(&output.stdout).trim().to_string()
}
/// Writes `f{i}` (ZFP) and `r{i}` (h5py's reading of `f{i}`, unfiltered)
/// for every mode x dtype x shape x data kind H5Z-ZFP accepts; each `f{i}`
/// has a `case` attribute. Prints the number of pairs.
const GENERATE: &str = r#"
import sys
import numpy as np, h5py, hdf5plugin
path = sys.argv[1]
MODES = [
('rate 2.5', dict(rate=2.5)),
('rate 8', dict(rate=8)),
('rate 16', dict(rate=16)),
('rate 31', dict(rate=31)),
('rate 64', dict(rate=64)),
('precision 6', dict(precision=6)),
('precision 20', dict(precision=20)),
('precision 64', dict(precision=64)),
('accuracy 0.5', dict(accuracy=0.5)),
('accuracy 1e-3', dict(accuracy=1e-3)),
('accuracy 1e-12', dict(accuracy=1e-12)),
('reversible', dict(reversible=True)),
('expert', dict(minbits=24, maxbits=600, maxprec=30, minexp=-20)),
# maxbits past ZFP_MAX_BITS: H5Z-ZFP stores it as fixed precision.
('expert maxbits 20000', dict(minbits=1, maxbits=20000, maxprec=40, minexp=-1074)),
# A budget of 10 bits a block: one bit left after a float's exponent.
# (Doubles are left out: 12 bits of exponent and flag overrun it, and
# libzfp's encoder then writes past its buffer.)
('expert maxbits 10', dict(minbits=5, maxbits=10, maxprec=64, minexp=-100)),
('expert minbits 900', dict(minbits=900, maxbits=4000, maxprec=64, minexp=-60)),
]
DTYPES = ['<f4', '<f8', '<i4', '<i8']
SHAPES = [
((37,), (16,)),
((13, 22), (5, 8)),
((9, 7, 6), (4, 5, 6)),
((5, 6, 7, 3), (3, 5, 6, 3)),
((4, 1, 30), (2, 1, 30)),
((3, 10, 1, 9), (2, 10, 1, 9)),
((64, 64), (64, 64)),
]
KINDS = ['smooth', 'noise', 'wide', 'zeros', 'special']
rng = np.random.default_rng(13)
def data(dt, shape, kind):
n = int(np.prod(shape))
d = np.dtype(dt)
idx = np.arange(n, dtype=np.float64)
if d.kind == 'f':
if kind == 'smooth':
v = 100 * np.sin(idx / 5.0) + idx / 3.0
elif kind == 'noise':
v = rng.normal(size=n) * 10.0 ** rng.uniform(-3, 3, n)
elif kind == 'wide':
lo, hi = (-150, 120) if d.itemsize == 4 else (-1075, 1000)
v = rng.choice([-1.0, 1.0], n) * np.exp2(rng.integers(lo, hi, n).astype(np.float64))
v[rng.random(n) < 0.1] = 0.0
elif kind == 'zeros':
v = np.zeros(n)
v[: n // 3] = 0.0
else:
v = rng.normal(size=n) * 1e3
v[rng.random(n) < 0.05] = np.inf
v[rng.random(n) < 0.05] = -np.inf
v[rng.random(n) < 0.05] = np.nan
v[rng.random(n) < 0.05] = -0.0
with np.errstate(over='ignore'):
return v.astype(dt).reshape(shape)
info = np.iinfo(d)
if kind == 'smooth':
v = ((idx * 7) % 1000 - 500).astype(np.int64)
elif kind == 'noise':
v = rng.integers(info.min, info.max, n, dtype=np.int64, endpoint=True)
elif kind == 'wide':
bits = rng.integers(0, d.itemsize * 8 - 2, n)
v = rng.integers(-(2 ** 20), 2 ** 20, n) << np.minimum(bits, d.itemsize * 8 - 22)
elif kind == 'zeros':
v = np.zeros(n, dtype=np.int64)
else:
v = rng.choice([info.min, info.max, 0, -1, 1], n)
return v.astype(dt).reshape(shape)
# Written first and read back after the file is closed: h5py reads a chunk
# still in libhdf5's chunk cache without decoding it.
i = 0
with h5py.File(path, 'w') as f:
for label, kw in MODES:
for dt in DTYPES:
if label == 'expert maxbits 10' and dt == '<f8':
continue
for shape, chunks in SHAPES:
for kind in KINDS:
v = data(dt, shape, kind)
name = f'f{i}'
try:
ds = f.create_dataset(name, data=v, chunks=chunks, **hdf5plugin.Zfp(**kw))
except Exception:
continue
ds.attrs['case'] = f'{label}|{dt}|{shape}|{chunks}|{kind}'
i += 1
with h5py.File(path, 'a') as f:
for k in range(i):
f.create_dataset(f'r{k}', data=f[f'f{k}'][()])
print(i)
"#;
#[test]
fn every_mode_reads_bit_exact() {
if !have_python() {
return;
}
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("zfp.h5");
let n: usize = run_python(GENERATE, &[path.to_str().unwrap()])
.parse()
.unwrap();
let file = File::open(&path).unwrap();
// mode -> dtypes read
let mut seen: BTreeMap<String, Vec<String>> = BTreeMap::new();
let mut failures = Vec::new();
for i in 0..n {
let ds = file.dataset(&format!("f{i}")).unwrap();
let case = match ds.attrs().unwrap().get("case") {
Some(clawhdf5::AttrValue::String(s)) => s.clone(),
other => panic!("f{i}: case attribute {other:?}"),
};
let want = file
.dataset(&format!("r{i}"))
.unwrap()
.read_selection(&Selection::All)
.unwrap();
match ds.read_selection(&Selection::All) {
Ok(got) if got == want => {
let mut parts = case.split('|');
let mode = parts.next().unwrap().to_string();
let dt = parts.next().unwrap().to_string();
let e = seen.entry(mode).or_default();
if !e.contains(&dt) {
e.push(dt);
}
}
Ok(got) => {
let first = got.iter().zip(&want).position(|(a, b)| a != b);
failures.push(format!("f{i} {case}: differs from byte {first:?}"));
}
Err(e) => failures.push(format!("f{i} {case}: {e}")),
}
}
assert!(
failures.is_empty(),
"{} of {n} datasets:\n{}",
failures.len(),
failures.join("\n")
);
// Every mode was exercised on every type H5Z-ZFP accepts it for.
for (mode, dts) in &seen {
assert!(dts.len() >= 2, "{mode}: only {dts:?}");
}
assert_eq!(seen.len(), 16, "{:?}", seen.keys());
eprintln!("{n} ZFP datasets bit-exact; modes and types: {seen:?}");
}
/// A header written on a big-endian machine: H5Z-ZFP finds the magic only
/// after byte-swapping the `cd_values`, and then byte-swaps the decoded
/// values, since the dataset's datatype is big-endian there. The file is
/// made by swapping the header words of a little-endian one in place, so
/// the datatype stays little-endian and libhdf5 reads the values swapped;
/// clawhdf5 must read the same bytes.
#[test]
fn big_endian_header_swaps_the_values() {
if !have_python() {
return;
}
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("zfp_be.h5");
let n: usize = run_python(
r#"
import sys, struct
import numpy as np, h5py, hdf5plugin
path = sys.argv[1]
cases = [('<f4', dict(rate=12)), ('<f8', dict(precision=30)), ('<i4', dict(reversible=True)),
('<i8', dict(accuracy=4))]
cds = []
with h5py.File(path, 'w') as f:
for k, (dt, kw) in enumerate(cases):
v = (np.arange(150) * 13 % 97 - 40).reshape(10, 15).astype(dt)
ds = f.create_dataset(f'f{k}', data=v, chunks=(4, 8), **hdf5plugin.Zfp(**kw))
cds.append(ds.id.get_create_plist().get_filter(0)[2])
raw = bytearray(open(path, 'rb').read())
for cd in cds:
le = struct.pack(f'<{len(cd)}I', *cd)
swapped = le[:4] + struct.pack(f'>{len(cd) - 1}I', *cd[1:])
at = raw.find(le)
assert at > 0 and raw.find(le, at + 1) < 0, 'cd_values not found once'
raw[at:at + len(le)] = swapped
open(path, 'wb').write(raw)
with h5py.File(path, 'a') as f:
for k in range(len(cases)):
f.create_dataset(f'r{k}', data=f[f'f{k}'][()])
f.create_dataset(f'o{k}', data=f[f'f{k}'][()].byteswap())
print(len(cases))
"#,
&[path.to_str().unwrap()],
)
.parse()
.unwrap();
let file = File::open(&path).unwrap();
for k in 0..n {
let read = |name: String| {
file.dataset(&name)
.unwrap()
.read_selection(&Selection::All)
.unwrap_or_else(|e| panic!("{name}: {e}"))
};
let got = read(format!("f{k}"));
assert!(got == read(format!("r{k}")), "f{k}: differs from h5py");
// Byte-swapped back, the values are the right ones.
assert!(got != read(format!("o{k}")), "f{k}: not swapped");
}
}
docs/design/range-reads.md
+26 -4
View File
@@ -1,7 +1,9 @@
# Design: range reads (reading HDF5 without holding the whole file)
Status: proposal, 2026-09-26. No library code has changed; this document is
the plan for Phase 3's largest architectural change. Every count below was
Status: proposal, 2026-09-26; the plan for Phase 3's largest architectural
change. Progress: M1, first part (the `Storage` trait and the metadata
parsers listed in `CHANGELOG.md` under "Range reads, milestone M1") is done;
group B-tree v2 lookups, dense groups and the facade are not converted yet. Every count below was
taken on `tank` on 2026-09-26 at commit `de2a53f`, with the commands given
next to it. No timing numbers appear here on purpose: the machine was shared
with other build jobs when this was written.
@@ -259,6 +261,11 @@ every `file_data[a..b]` becomes `file.read_at(a, b - a)?`.
(binary size matters for wasm); `&dyn Storage` costs one indirect call per
structure read, negligible next to parsing. Hot raw-data loops keep their
speed through `as_contiguous()`.
*M1 outcome:* `&dyn` was not negligible for small structures — with it,
`ObjectHeader::parse` was ~25% and a 400-group facade listing ~14% slower
than the slice code (provisional, shared machine). The cores are now
generic (`S: Storage + ?Sized`), so the `&[u8]` wrappers get a `[u8]`
instance and `dyn Storage` is one more instance, not one per backend.
### (b) A page-cache "virtual slice"
@@ -379,6 +386,16 @@ fast path within benchmark noise.
n children decodes its links O(n) times. Look names up through the index
(above) and let a listing hand out its entries, so the cache has less to
absorb.
- *Status 2026-09-26:* done on branch `perf/p3-indexed-lookups` — link and
attribute names through the name indexes (`group_v2::resolve_child`,
`attribute::find_attribute_in_file`; creation-order lookups by name do
not exist in the API, so the creation-order index is still only listed),
`addr::to_usize`/`saturating_usize` for all 119 truncating `u64 as usize`
casts clippy finds in `clawhdf5-format` under any CI-built feature set but
`szip` (the 133 above counted any `*addr*/*offset* as usize`, mostly
widening `u8`/`u32` casts; `scripts/check-32bit-casts.sh` lints those
feature sets for new ones), and `Group::entries`/`File::group_at`.
The facade, io and ann casts are not converted.
**M1 — metadata over the trait, in-memory impl identical to today (2–3 weeks).**
- Add `Storage` (above) to `clawhdf5-format`, `no_std`-compatible, with
@@ -390,7 +407,7 @@ fast path within benchmark noise.
B-tree v1/v2, fractal heap, fixed/extensible array, symbol table, group
v1/v2, shared messages, attributes, fill value, data layout — one commit
each. The old `&[u8]` signature stays as a thin wrapper over the new one
(`fn parse(data: &[u8], ..) { parse_in(data as &dyn Storage, ..) }`), so
(`fn parse(data: &[u8], ..) { parse_in(data, ..) }`, generic core), so
callers and the other crates don't move yet.
- Replace the 5 open-ended slices and 38 `len()` checks with bounded reads.
@@ -431,8 +448,13 @@ Total: roughly 6–10 engineer-weeks for M0–M4 (estimate, not measured).
- `impl Storage for [u8]` returns `Cow::Borrowed` — no copy, no allocation.
- Hot loops (raw-data copies, contiguous typed reads, `read_selection_native`)
branch once on `as_contiguous()` and then run today's code.
- `&dyn` dispatch is per structure, not per byte; parse code keeps working on
- The parser cores are generic over `S: Storage + ?Sized`, so the in-memory
instance has no dispatch at all; a remote backend behind `&dyn Storage`
pays one indirect call per structure read. Parse code keeps working on
the returned slice.
- Reads sized by untrusted fields cover what the parser uses (see
`CHANGELOG.md`, M1), so a hostile size costs no read of the rest of the
file.
- Gate: `crates/clawhdf5/benches/mmap_bench.rs`, the concurrent-read benches in
`clawhdf5-bench`, and the conformance run time, before and after each M1/M2
commit, on an otherwise idle machine. Anything outside noise blocks the
docs/known-issues.md
+65 -4
View File
@@ -7,6 +7,63 @@ deleting it.
---
## LZF/Blosc chunks written with a stale filter mask
**Status:** fixed 2026-09-26, before any release (the LZF and Blosc writers
were added the same day; v2.7.0 and earlier write neither).
`FileBuilder` stored every chunk of an LZF or Blosc dataset through the
filter with filter mask 0. libhdf5 counts LZF and Blosc output no smaller
than the chunk as a failure of the (optional) filter and stores the chunk
raw with the filter's mask bit set. When a chunk's LZF stream was exactly
the chunk's size, the first libhdf5 rewrite of it stored raw data at the
same size and left our mask 0 in the index, so h5py could no longer read
the dataset. `FileEditor` had the same bug, fixed earlier the same day.
Both now use `clawhdf5_format::filters::compress_chunk_masked`, and every
chunk index the writer builds records the real mask (see `CHANGELOG.md`).
Files written before the fix read correctly; rewrite them before letting
libhdf5 modify them.
## In-place modification (`FileEditor`) limits
**Status:** open (documented 2026-09-26). `clawhdf5::FileEditor` refuses,
with `Error::Unsupported` and without writing anything:
- new, moved or resized chunks in a **version-2 B-tree** chunk index (what
libhdf5 uses for two or more unlimited dimensions) — existing unfiltered
chunks, and filtered ones that re-encode to the same size and filter
mask, are
overwritten in place; `resize` works — and new chunks in an **implicit**
index (it has all of its chunks from the start);
- **shrinking** a dataset;
- variable-length and reference data;
- chunks through a filter this build cannot encode (scale-offset, N-Bit,
SZIP, or a plugin filter it lacks), even an optional one: libhdf5 skips
an optional filter only when its own build lacks it, which none does for
these;
- attributes of an object in **dense storage**, past its compact limit (8
by default) or with tracked **creation order**;
- partial edge chunks stored unfiltered (`H5Pset_chunk_opts`), external
raw data files, virtual datasets;
- files with a metadata cache image, paged or persistent free-space
management, a driver info block, or version-3 consistency flags set.
**Space is never reused.** There is no free-space manager: the old bytes of
a filtered chunk that grows and has to move, and of an attribute that is
replaced by a larger one, are leaked (`h5repack` reclaims them). A chunk
that is the last thing in the file grows in place instead, which covers the
usual append. Measured 2026-09-26 on tank with
`cargo test --release -p clawhdf5-tools --test edit_interop -- --ignored
--nocapture measure_append_waste` (file sizes are deterministic): 1000
appends of 100 `f8` values to a 1-D dataset with 1024-element chunks give
810 504 bytes unfiltered, as libhdf5's file, and 307 210 bytes with gzip
(libhdf5: 306 058; `h5repack`: 306 104); 2000 appends of 10 values with
4096-element gzip chunks give 119 684 bytes against libhdf5's 50 292
(`h5repack`: 49 930), because the chunk being appended to is followed by
new index blocks and moves each time it grows.
**No journal.** A crash while an edit patches existing structures can leave
the file inconsistent; see the `FileEditor` documentation.
## Selection reads that decode more than the selection
**Status:** open (documented 2026-09-26). `Dataset::read_selection` (and so
@@ -293,10 +350,14 @@ fill-value item that did is fixed).
697 ok, tank, `conformance/run.sh --no-fetch`). Blosc2 frames using
dictionaries, lazy chunks, variable-length blocks, user-defined codecs or
registered filters (e.g. bytedelta) are refused with an error.
**Still open:** ZFP (32013) fails with an `UnsupportedFilter` error that
names the filter, and can be plugged in with
`filter_registry::register_filter` (32023, Granular BitRound, too, since
2026-09-26 even with the `pcodec` feature). clawhdf5 cannot write Blosc2.
**Fixed 2026-09-26** for ZFP (32013, `zfp` feature, also in
`plugin-filters`), read only: every H5Z-ZFP mode and type, bit-exact
against h5py + hdf5plugin 7.1 (`crates/clawhdf5/tests/zfp_interop.rs`);
h5ex_d_zfp now reads (conformance 600 of 697 ok, tank,
`conformance/run.sh --no-fetch`). **Still open:** clawhdf5 cannot write
Blosc2 or ZFP. Other filters (32023, Granular BitRound, too, since
2026-09-26 even with the `pcodec` feature) can be plugged in with
`filter_registry::register_filter`.
- **Wrong data: a chunk whose filters decode to fewer bytes than the chunk
read with zeros for the missing bytes** (any filter; found reviewing the plugin
filters). **Fixed 2026-09-26:** it is an error naming the chunk. A corrupt
scripts/check-32bit-casts.sh
Executable
+68
View File
@@ -0,0 +1,68 @@
#!/usr/bin/env bash
# CI check: clawhdf5-format has no truncating `u64 as usize` cast on a 32-bit
# target. HDF5 addresses and lengths are 64-bit; on wasm32 (or any 32-bit
# target) such a cast silently wraps an address past 4 GiB onto another part
# of the file. File values go through `addr::to_usize` (a clean error) and
# in-memory counts through `addr::saturating_usize`.
#
# Lints with clippy's cast_possible_truncation and fails on any u64 -> usize
# finding (other truncations are not checked here), once per feature set
# below. Together the sets compile every feature-gated line of the crate that
# ci-test.sh builds: features only add code, except `not(feature = ...)`
# paths for std/checksum/fast-checksum/szip, which the no-default-features
# and default sets cover. szip is left out (it needs libaec), as in
# ci-test.sh.
#
# The sets are linted for wasm32 where they build there. zstd links a C
# library that does not build for wasm32, so the set with it is linted for
# the host: the lint reports u64 -> usize casts whatever the target's
# pointer width, and the crate has no pointer-width-dependent code.
#
# Usage:
# ./scripts/check-32bit-casts.sh
#
# Prerequisites:
# rustup target add wasm32-unknown-unknown
set -euo pipefail
WASM="wasm32-unknown-unknown"
ALL_BUT_ZSTD="parallel,lz4,pcodec,fast-checksum,blake3_hash,plugin-filters,lookup-stats"
# target|cargo feature arguments
SETS=(
"$WASM|--no-default-features"
"$WASM|--no-default-features --features std,checksum"
"$WASM|"
"$WASM|--features $ALL_BUT_ZSTD"
"host|--features $ALL_BUT_ZSTD,zstd"
)
status=0
for set in "${SETS[@]}"; do
target=${set%%|*}
args=${set#*|}
target_args=()
if [ "$target" != host ]; then
target_args=(--target "$target")
fi
echo "==> Checking for truncating u64 -> usize casts in clawhdf5-format ($target: ${args:-default features})"
# shellcheck disable=SC2086 # $args is a list of arguments
out=$(cargo clippy -p clawhdf5-format "${target_args[@]}" $args \
--message-format short \
-- -A clippy::all -W clippy::cast_possible_truncation 2>&1) || {
echo "$out"
echo "==> clippy failed" >&2
exit 1
}
found=$(grep -F 'casting `u64` to `usize`' <<<"$out" || true)
if [ -n "$found" ]; then
echo "$found"
status=1
fi
done
if [ "$status" -ne 0 ]; then
echo "==> use addr::to_usize (file values) or addr::saturating_usize (in-memory counts)" >&2
exit 1
fi
echo "==> no truncating u64 -> usize casts"
scripts/ci-test.sh
+7 -4
View File
@@ -67,7 +67,7 @@ run_step "cargo clippy --all-targets" cargo clippy \
# 3. Clippy over clawhdf5-format's optional features, which the default
# workspace build never compiles (szip is left out: it needs libaec).
# plugin-filters = bitshuffle, bzip2, blosc, blosc2 (and the default-on lzf).
# plugin-filters = bitshuffle, bzip2, blosc, blosc2, zfp (and the default-on lzf).
run_step "cargo clippy (format feature matrix)" cargo clippy \
-p clawhdf5-format \
--all-targets \
@@ -78,7 +78,7 @@ run_step "cargo clippy (format feature matrix)" cargo clippy \
# dependencies (bitshuffle and blosc share code).
plugin_filters_alone() {
local f
for f in bitshuffle bzip2 blosc blosc2; do
for f in bitshuffle bzip2 blosc blosc2 zfp; do
echo "--- $f"
cargo clippy -p clawhdf5-format --all-targets --features "$f" -- -D warnings || return 1
done
@@ -147,6 +147,8 @@ run_step "wasm32 clippy (clawhdf5-wasm)" cargo clippy \
--target wasm32-unknown-unknown \
--all-targets \
-- -D warnings
# A 64-bit file address must not wrap on a 32-bit target.
run_step "check-32bit-casts.sh" "$SCRIPT_DIR/check-32bit-casts.sh"
# The built wasm package, run under Node against h5py/netCDF4-written files,
# and the viewer page in headless Chromium when one is found.
@@ -208,9 +210,10 @@ if "$PYTHON" -c "import h5py" >/dev/null 2>&1 || [ "${CLAWHDF5_REQUIRE_INTEROP:-
# libhdf5's registered plugins) compile and run too.
run_step "h5py interop (format, ignored tests)" cargo test \
-p clawhdf5-format --features lz4,zstd --test writer_h5py_tests -- --include-ignored
# LZF, bitshuffle, bzip2 and Blosc both ways against h5py + hdf5plugin.
# LZF, bitshuffle, bzip2 and Blosc both ways against h5py + hdf5plugin;
# Blosc2 and ZFP (read-only) against what h5py reads.
run_step "h5py interop (plugin filters)" cargo test \
-p clawhdf5 --features plugin-filters --test plugin_filters_interop
-p clawhdf5 --features plugin-filters --test plugin_filters_interop --test zfp_interop
else
echo ""
echo "==> [h5py interop] SKIPPED: no h5py in $PYTHON"