quantumclaw/clawhdf5
3
1
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases 1 Wiki Activity

fix(format): apply the base address of files with a user block

Browse Source 
A file may start with a user block (h5py userblock_size, h5jam), putting
the superblock at 512, 1024, ...; every address in the file is then
relative to the superblock. The signature search found it, but every
reader passed the whole file to the parsers, so addresses landed
userblock bytes early and the root group failed with
InvalidObjectHeaderVersion (twithub.h5, twithub513.h5,
h5clear_fsm_persist_user_*.h5).

Readers now view the file from the superblock on, taking the signature's
position as the base address as libhdf5 does: File (mmap, buffered,
from_bytes), MmapFile, LazyFile, AsyncHDF5File, the VOL and MPI VOL
readers, the HNSW loader and external VDS source files. File, MmapFile
and LazyFile gain user_block_size(). The new signature::split_user_block
returns the two parts, and Superblock::parse refuses a non-zero offset
(UserBlockNotStripped) so a format-level caller cannot silently apply
superblock-relative addresses to the whole file.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-25 22:07:50 -05:00
co-authored by Claude Opus 5.5
parent 0555794850
commit a6e90f3ee3
16 changed files with 540 additions and 79 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+13
View File
@@ -273,6 +273,19 @@
- CI keeps zlib-ng building and tested; the arm64 job no longer needs cmake.
### Correctness
- **Files with a user block** (`h5py.File(..., userblock_size=N)`, `h5jam`;
the superblock at 512, 1024, …) could not be read: every address in the
file is relative to the superblock, but it was applied from byte 0
(`InvalidObjectHeaderVersion` on the root group). `File` (mmap, buffered,
`from_bytes`), `MmapFile`, `LazyFile`, `AsyncHDF5File`, the VOL readers,
the HNSW loader and external VDS sources now view the file from the
superblock on, using the signature's position as the base address as
libhdf5 does; `user_block_size()` reports the user block (h5py's
`userblock_size`), and `as_bytes()` returns the bytes from the superblock
on. **Breaking (format crate):** `Superblock::parse` refuses a non-zero
signature offset with `FormatError::UserBlockNotStripped`, since the
addresses it returns would be applied to the wrong bytes; pass the slice
from `signature::split_user_block` (new) and parse at offset 0.
- `clawhdf5-format` reader: version-1 shared messages (HDF5 1.6-era files,
e.g. a dataset using a committed datatype in libhdf5's `tcompound.h5`)
read the heap-offset field of the embedded symbol-table entry as the