diff --git a/CHANGELOG.md b/CHANGELOG.md index 6a4eeb6..50f6397 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,6 +19,37 @@ - `clawhdf5-agent`: `benches/bench.rs` and `benches/memory_bench.rs` no longer compiled against the current `strategy`/`consolidation` APIs. +### HDF5 Compatibility +- `clawhdf5-format`/`clawhdf5`: datasets and attributes that use a **committed + (named) datatype** now read correctly. They store a shared-message reference; + the facade parsed the reference bytes as the datatype (`Time { size: 0 }`, + unreadable data) and silently dropped such attributes. The shared-reference + parser itself was wrong for real files: version 2 has no reserved bytes, and + the version 3 types were inverted (1 = SOHM heap, 2 = committed). +- **Fill values are applied on read.** There was no Fill Value message parser: + the holes of a sparse chunked dataset read as zeros even when the fill value + was not zero (silently wrong data), and a dataset that was created but never + written failed with `NoDataAllocated` where h5py returns a filled array. + Messages v1–v3 and the old 0x0004 form are parsed; the fill value is written + into exactly the chunk-grid cells missing from the chunk index. +- **Soft links are followed** during path resolution, in old- and new-style + groups (absolute/relative targets, links to groups, links through links), + with a depth limit so a link cycle is an error rather than a hang. A dangling + link reports the target it could not find. +- Things the reader does not follow are now explicit errors instead of wrong + answers: an external link is `ExternalLinkUnsupported { filename, + object_path }` (was `PathNotFound`), and a dataset whose raw data lives in + external files (message 0x0007, now a known `MessageType`) is + `ExternalDataFilesUnsupported` (it would otherwise read as fill values). +- All of the above are covered by h5py interop tests under both default and + `libver='latest'` bounds, compared against h5py's own readback. + +### Security +- `clawhdf5`: virtual-dataset source file names are untrusted input but were + joined straight onto the opened file's directory, so a crafted file could + make the reader open any path the process can reach (absolute path, or `..` + components). Only plain relative paths inside that directory are accepted. + ### Durability & Integrity - `clawhdf5-agent`: a crash between writing a checkpoint and truncating the WAL no longer **duplicates every pending entry** on the next open. Each @@ -79,6 +110,11 @@ The `#[ignore]`d `writer_h5py_tests` suite is run explicitly. - h5py-generated-file tests now cover default libver bounds as well as `libver='latest'` (HDF5 2.0 raised the default low bound to 1.8). +- `clawhdf5-agent`: WAL property tests (round trip; after any corruption the + entries read back are an exact prefix of what was written — 1500 seeded + cases), a crash-recovery matrix (an on-disk image after every operation, the + checkpoint window, and the WAL torn at every byte length, each reopened and + checked against a model), and a WAL fuzz target. - Optional fuzz smoke run (`CLAWHDF5_FUZZ_SECONDS=N scripts/ci-test.sh`); new datatype corpus seeds for v1 compound and native complex messages. diff --git a/docs/known-issues.md b/docs/known-issues.md index d078acc..37fe945 100644 --- a/docs/known-issues.md +++ b/docs/known-issues.md @@ -107,3 +107,27 @@ only attributes convertible to `AttrValue`. An attribute with, e.g., a compound datatype is omitted from the map with no error or indication that it exists. Planned: surface these as an explicit `AttrValue` variant (raw bytes + datatype) or an error, as part of the "no silent skips" robustness work. + +## B-tree v2 chunk index (layout v4, index type 5) is not supported + +**Status:** open. + +**Summary:** a chunked dataset with **two or more unlimited dimensions** written +with `libver='latest'` indexes its chunks with a version-2 B-tree. Reading it +fails with `ChunkedReadError("unsupported chunked layout version=4, +index_type=Some(5)")`. Single-chunk, implicit, fixed-array and +extensible-array indexes (and the v3 B-tree v1) are supported. + +**Repro:** `f.create_dataset("d", shape=(5, 7), chunks=(2, 3), maxshape=(None, None))` +with `h5py.File(..., libver='latest')`. + +## External links and external raw data are not followed + +**Status:** open (by design for now); both are explicit errors. + +**Summary:** a path through an external link returns +`FormatError::ExternalLinkUnsupported { filename, object_path }`, and a dataset +created with `external=[...]` storage returns +`FormatError::ExternalDataFilesUnsupported`. Neither is resolved. If support is +added, file names must be confined to the opened file's directory, as the +virtual-dataset resolver now does.