quantumclaw/clawhdf5
3
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

Chunks of 4 GiB or more: read in every index, write as libhdf5 2.x does #29

Merged
osobh merged 7 commits from feat/huge-chunks into main 2026-09-29 11:57:48 +00:00
Conversation 0 Commits 7 Files Changed 30
+5 -5
1 changed files with 5 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Showing only changes of commit 4b075656e7 - Show all commits
docs/known-issues.md
+5 -5
View File
@@ -542,7 +542,7 @@ given.
## NetCDF-4: variables' dimensions are guessed from sizes
**Status:** fixed 2026-09-28 (branch `fix/netcdf-dimension-list`). Affected
**Status:** fixed 2026-09-28 (#27). Affected
every release (v2.1.0 to v2.7.0: size matching dates from the crate's
first version). Wrong metadata only: stored values were always read right.
Users who read `_Netcdf4Coordinates` themselves can use
@@ -594,7 +594,7 @@ comparison over the conformance corpus's netCDF-readable files.
## HDF5 1.8 could not read the files we wrote
**Status:** fixed 2026-09-28 (branch `feat/libver-v18`), as an opt-in.
**Status:** fixed 2026-09-28 (#28), as an opt-in.
Affected every release (v2.1.0 to v2.7.0): the writer only ever wrote the
HDF5 1.10 format. Users who need HDF5 1.8 to read their files call
`FileBuilder::libver_bounds(LibVer::V18, LibVer::V18)` (format crate:
@@ -635,7 +635,7 @@ B-tree (see `BENCHMARKS.md`, "HDF5 1.8 format").
## Chunks of 4 GiB or more could not be read
**Status:** fixed 2026-09-28 (branch `feat/huge-chunks`); affected every
**Status:** fixed 2026-09-28 (#29); affected every
release (v2.1.0 to v2.7.0). Errors, and cost; no wrong values were
returned. Nothing for users to do but upgrade.
@@ -660,7 +660,7 @@ test time). Limits that remain:
## Chunks of 4 GiB or more were written unreadable
**Status:** fixed 2026-09-28 (branch `feat/huge-chunks`). The deflate
**Status:** fixed 2026-09-28 (#29). The deflate
truncation was before any release (the one-pass deflate dates from
2026-09-23); the rest affected every release (v2.1.0 to v2.7.0). Files
clawhdf5 wrote with a chunk of 4 GiB or more should be written again.
@@ -684,7 +684,7 @@ cap, and compressing one shuffled 4 GiB chunk now peaks at 4.3 GiB
## LZ4 chunks larger than 256 MiB were refused
**Status:** fixed 2026-09-28 (branch `feat/huge-chunks`); affected every
**Status:** fixed 2026-09-28 (#29); affected every
release (v2.1.0 to v2.7.0). An error, never wrong data. Nothing for users
to do but upgrade.