quantumclaw/clawhdf5
3
1
Fork 0
Code Issues Pull Requests 3 Actions Packages Projects Releases 1 Wiki Activity

Chunks of 4 GiB or more: read in every index, write as libhdf5 2.x does #29

Open
osobh wants to merge 10 commits from feat/huge-chunks into main
pull from: feat/huge-chunks
merge into: :main
Conversation 0 Commits 10 Files Changed 50
+5 -5
1 changed files with 5 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Showing only changes of commit 4b075656e7 - Show all commits
docs/known-issues.md
+5 -5
View File
@@ -542,7 +542,7 @@ given.
## NetCDF-4: variables' dimensions are guessed from sizes ## NetCDF-4: variables' dimensions are guessed from sizes
**Status:** fixed 2026-09-28 (branch `fix/netcdf-dimension-list`). Affected **Status:** fixed 2026-09-28 (#27). Affected
every release (v2.1.0 to v2.7.0: size matching dates from the crate's every release (v2.1.0 to v2.7.0: size matching dates from the crate's
first version). Wrong metadata only: stored values were always read right. first version). Wrong metadata only: stored values were always read right.
Users who read `_Netcdf4Coordinates` themselves can use Users who read `_Netcdf4Coordinates` themselves can use
@@ -594,7 +594,7 @@ comparison over the conformance corpus's netCDF-readable files.
## HDF5 1.8 could not read the files we wrote ## HDF5 1.8 could not read the files we wrote
**Status:** fixed 2026-09-28 (branch `feat/libver-v18`), as an opt-in. **Status:** fixed 2026-09-28 (#28), as an opt-in.
Affected every release (v2.1.0 to v2.7.0): the writer only ever wrote the Affected every release (v2.1.0 to v2.7.0): the writer only ever wrote the
HDF5 1.10 format. Users who need HDF5 1.8 to read their files call HDF5 1.10 format. Users who need HDF5 1.8 to read their files call
`FileBuilder::libver_bounds(LibVer::V18, LibVer::V18)` (format crate: `FileBuilder::libver_bounds(LibVer::V18, LibVer::V18)` (format crate:
@@ -635,7 +635,7 @@ B-tree (see `BENCHMARKS.md`, "HDF5 1.8 format").
## Chunks of 4 GiB or more could not be read ## Chunks of 4 GiB or more could not be read
**Status:** fixed 2026-09-28 (branch `feat/huge-chunks`); affected every **Status:** fixed 2026-09-28 (#29); affected every
release (v2.1.0 to v2.7.0). Errors, and cost; no wrong values were release (v2.1.0 to v2.7.0). Errors, and cost; no wrong values were
returned. Nothing for users to do but upgrade. returned. Nothing for users to do but upgrade.
@@ -660,7 +660,7 @@ test time). Limits that remain:
## Chunks of 4 GiB or more were written unreadable ## Chunks of 4 GiB or more were written unreadable
**Status:** fixed 2026-09-28 (branch `feat/huge-chunks`). The deflate **Status:** fixed 2026-09-28 (#29). The deflate
truncation was before any release (the one-pass deflate dates from truncation was before any release (the one-pass deflate dates from
2026-09-23); the rest affected every release (v2.1.0 to v2.7.0). Files 2026-09-23); the rest affected every release (v2.1.0 to v2.7.0). Files
clawhdf5 wrote with a chunk of 4 GiB or more should be written again. clawhdf5 wrote with a chunk of 4 GiB or more should be written again.
@@ -684,7 +684,7 @@ cap, and compressing one shuffled 4 GiB chunk now peaks at 4.3 GiB
## LZ4 chunks larger than 256 MiB were refused ## LZ4 chunks larger than 256 MiB were refused
**Status:** fixed 2026-09-28 (branch `feat/huge-chunks`); affected every **Status:** fixed 2026-09-28 (#29); affected every
release (v2.1.0 to v2.7.0). An error, never wrong data. Nothing for users release (v2.1.0 to v2.7.0). An error, never wrong data. Nothing for users
to do but upgrade. to do but upgrade.