osobhandClaude Fable 5.1
52cfcf20b2
HDF5 1.12 revised the reference datatype (class 7) in datatype message version 4: reference types 2-4 are the new H5T_STD_REF object / dataset-region / attribute references. Datatype::parse rejected them with InvalidReferenceType, so any dataset of that type was unreadable. h5py cannot write this type, which is why it had never been tested. A real file was produced by calling the libhdf5 bundled in the h5py wheel through ctypes (H5T_STD_REF_g, H5Rcreate_object, H5Dwrite); the 2 KB result is committed as tests/fixtures/std_ref_hdf5_2_0.h5 with its generator, gen_std_ref.py. - ReferenceType gains Object2, DatasetRegion2 and Attribute, accepted only from datatype version 4. - read_object_references decodes Object2 elements: type(1) flags(1) token_size(1) token, zero-padded to the element size; the token is the target's object header address. A null reference decodes to the undefined address; an external reference, a wrong type byte or a token that doesn't fit is an error. The fixture test follows both references and checks they resolve to the objects they were created from. Co-Authored-By: Claude Fable 5.1 <[email protected]>
149 lines
6.9 KiB
Markdown
149 lines
6.9 KiB
Markdown
# Known Issues
|
||
|
||
Bugs found during development or downstream use, tracked here because this
|
||
repository's issue tracker is disabled. One entry per bug; when an entry is
|
||
fixed, record the fix in `CHANGELOG.md` and update its status here rather than
|
||
deleting it.
|
||
|
||
---
|
||
|
||
## Compound datatype message version 5 is not parsed (HDF5 2.0)
|
||
|
||
**Status:** fixed on `main` in `a13ff51` (2026-06-03); **not in the v2.1.0
|
||
tag**, which was cut five commits earlier. Ships in the next release.
|
||
|
||
**Reported by:** M. Scot Breitenfeld (The HDF Group), 2026-09-08, against v2.1.0.
|
||
|
||
**Summary:** `clawhdf5-format` v2.1.0 rejects any dataset with a compound
|
||
(struct) datatype written by an HDF5 2.0 library in `libver='latest'` mode:
|
||
`InvalidDatatypeVersion { class: 6, version: 5 }`.
|
||
|
||
**Reproduction** (h5py 3.16.0 / HDF5 2.0.0):
|
||
|
||
```python
|
||
import h5py, numpy as np
|
||
dt = np.dtype([('x', 'f8'), ('y', 'f8'), ('id', 'i4')])
|
||
data = np.array([(1.0, 2.0, 10), (3.0, 4.0, 20)], dtype=dt)
|
||
f = h5py.File('compound.h5', 'w', libver='latest')
|
||
f.create_dataset('particles', data=data)
|
||
f.close()
|
||
```
|
||
|
||
Committed as `crates/clawhdf5-format/tests/writer_h5py_tests.rs::read_h5py_generated_compound`
|
||
(`#[ignore]`d; needs `python3` with h5py on `PATH`). Run with
|
||
`cargo test -p clawhdf5-format --test writer_h5py_tests -- --include-ignored`:
|
||
v2.1.0 gives 25 passed / 1 failed; `main` passes everything.
|
||
|
||
**Root cause:** the compound (class 6) branch of `Datatype::parse`
|
||
(`crates/clawhdf5-format/src/datatype.rs`) accepted only versions 1–4. Datatype
|
||
message versions 4 and 5 changed only the Reference and Complex classes, so a
|
||
v5-tagged compound uses the unchanged v3 member-list layout.
|
||
|
||
**Fix:** versions 3–5 are accepted for compound (class 6) and array (class 10)
|
||
datatypes, and data layout message version 5 is accepted too (needed for every
|
||
chunked dataset written by HDF5 2.0). Byte-level regression tests:
|
||
`test_compound_v5_from_hdf5_2_0`, `test_array_v5_from_hdf5_2_0`.
|
||
|
||
## Native complex datatype (class 11) is mis-parsed (HDF5 2.0)
|
||
|
||
**Status:** fixed 2026-09-18. Found while validating the report above.
|
||
|
||
**Summary:** HDF5 2.0 native complex types (`H5T_COMPLEX_IEEE_F64LE` etc.)
|
||
were parsed as if they carried a compound-style member list. The properties are
|
||
actually a single base floating-point datatype, so the parser produced a garbage
|
||
datatype, or `UnexpectedEof` when the complex type was a compound member. h5py's
|
||
default numpy-complex mapping is unaffected (it writes a `{r, i}` compound);
|
||
only files using the native type through the C API / h5py low-level API hit this.
|
||
|
||
**Fix:** class 11 parses its base type and is surfaced as the equivalent
|
||
`{r, i}` compound. Tests: `test_complex_v5_from_hdf5_2_0`,
|
||
`test_compound_with_complex_member_from_hdf5_2_0`,
|
||
`writer_h5py_tests.rs::read_h5py_generated_native_complex`.
|
||
|
||
## Revised reference datatype (class 7, version 4) is not parsed
|
||
|
||
**Status:** fixed 2026-09-19 for object references; region and attribute
|
||
references are recognised but not decoded.
|
||
|
||
**Summary:** HDF5 1.12+ `H5T_STD_REF` references use datatype message version 4
|
||
with reference types 2-4 (object / region / attribute), which `Datatype::parse`
|
||
rejected with `InvalidReferenceType`. h5py still writes the legacy references,
|
||
so no file had been available to test against.
|
||
|
||
**Fix:** a real file was produced by driving the libhdf5 bundled in the h5py
|
||
wheel through ctypes (`tests/fixtures/gen_std_ref.py` ->
|
||
`std_ref_hdf5_2_0.h5`). The three new types parse as
|
||
`ReferenceType::{Object2, DatasetRegion2, Attribute}`, and
|
||
`read_object_references` decodes `Object2` elements (type, flags, token size,
|
||
token = target object header address). External references (flag bit 0) and
|
||
the region/attribute payloads are errors rather than misreads.
|
||
|
||
## `clawhdf5-gpu` `gpu_tests` can hang under the default parallel test runner
|
||
|
||
**Status:** fixed 2026-09-19.
|
||
|
||
**Summary:** during `cargo test --workspace` the `gpu_tests` binary sat idle for
|
||
25+ minutes. Every test created its own `wgpu::Instance` + device (requesting
|
||
adapter-maximum limits) concurrently, and readback used an unbounded
|
||
`device.poll(Wait)`.
|
||
|
||
**Fix:** tests hold a process-wide lock while they own a device, and
|
||
`GpuAccelerator` readback waits time out after 30 s with `GpuError::BufferMap`.
|
||
|
||
## Compound datatype versions 1 and 2 are mis-parsed (default libver files)
|
||
|
||
**Status:** fixed 2026-09-19. Found by adding a default-libver axis to the h5py
|
||
interop tests.
|
||
|
||
**Summary:** any compound dataset written with default libver bounds (plain
|
||
`h5py.File(path, 'w')`, datatype message version 1) failed to read, typically
|
||
with `Overflow("compound member 'x': byte_offset(0) + field_size(4136977) ...")`.
|
||
Only `libver='latest'` files (version 3+) and files written by clawhdf5 itself
|
||
worked, which is why the existing tests never caught it.
|
||
|
||
**Root cause:** `Datatype::parse` skipped 24 bytes of legacy per-member array
|
||
fields for v1 where the format has 28 (dimensionality 1 + reserved 3 +
|
||
permutation 4 + reserved 4 + 4 dimension sizes 16), and treated v2 like v1 minus
|
||
name padding, whereas v2 keeps the 8-byte name padding and has no array fields.
|
||
|
||
## Attributes with unsupported datatypes are silently dropped
|
||
|
||
**Status:** fixed 2026-09-19.
|
||
|
||
**Summary:** `Dataset::attrs()` / `Group::attrs()` returned only attributes
|
||
convertible to `AttrValue` and omitted the rest without any indication — every
|
||
Python `bool` (an HDF5 enum), complex, compound and reference attributes. Unsigned
|
||
64-bit arrays were also cast to `I64Array`, turning values above `i64::MAX`
|
||
negative.
|
||
|
||
**Fix:** booleans decode as 0/1 integers, `AttrValue::U64Array` keeps unsigned
|
||
arrays unsigned, and `AttrValue::Raw { datatype, shape, data }` carries any other
|
||
attribute verbatim. Both new variants are writable. Still lossy: a
|
||
multi-dimensional numeric attribute is returned as a flat array (its shape is
|
||
not reported).
|
||
|
||
## B-tree v2 chunk index (layout v4, index type 5) is not supported
|
||
|
||
**Status:** fixed 2026-09-19.
|
||
|
||
**Summary:** a chunked dataset with **two or more unlimited dimensions** written
|
||
with `libver='latest'` indexes its chunks with a version-2 B-tree, and reading it
|
||
failed with `unsupported chunked layout version=4, index_type=Some(5)`.
|
||
|
||
**Fix:** record types 10 (unfiltered) and 11 (filtered) are decoded — address,
|
||
stored size, filter mask, scaled offsets — through the shared chunk-listing
|
||
function, so full reads, cached reads, partial reads and fill-value handling
|
||
all work. Covered by an h5py interop test (plain, gzip+shuffle, a 2500-chunk
|
||
tree with internal nodes, a sparse dataset with a fill value, a hyperslab).
|
||
|
||
## External links and external raw data are not followed
|
||
|
||
**Status:** open (by design for now); both are explicit errors.
|
||
|
||
**Summary:** a path through an external link returns
|
||
`FormatError::ExternalLinkUnsupported { filename, object_path }`, and a dataset
|
||
created with `external=[...]` storage returns
|
||
`FormatError::ExternalDataFilesUnsupported`. Neither is resolved. If support is
|
||
added, file names must be confined to the opened file's directory, as the
|
||
virtual-dataset resolver now does.
|