quantumclaw/clawhdf5
3
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity
156 Commits 1 Branch 11 Tags
b36c6ec2af76ad172145a43d26d00ce1bb4e3e26
Commit Graph
8 Commits
Author SHA1 Message Date
osobhandClaude Fable 5.1 4aa3c5a1ca chore(release): v2.4.0 
Bump all workspace crates, the node package and pyproject to 2.4.0, finalize
the changelog and add upgrade notes for the search behaviour changes.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
 2026-09-19 13:32:47 -07:00
osobhandClaude Fable 5.1 0876796432 chore(release): v2.3.0 
Bump all workspace crates, the node package and pyproject to 2.3.0, finalize
the changelog and add upgrade notes for the behaviour changes.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
 2026-09-19 07:19:11 -07:00
osobhandClaude Fable 5.1 2053b69f07 chore(release): v2.2.0, point repository URLs at git.redclaw.dev 
Bump all workspace crates, the node package and pyproject to 2.2.0 and
finalize the changelog. The repository URL in every manifest pointed at a
GitHub location that does not resolve; use the real origin.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
 2026-09-18 20:58:29 -07:00
ClawHDF5 Coding Agent 5db1008eb7 security(format): wire provenance verify_dataset into the clawhdf5 read path 
verify_dataset existed and was tested, but was only ever called from
clawhdf5-format's own test files — no reader path in clawhdf5-io or the
clawhdf5 facade called it, so a corrupted dataset was silently readable
even though the write-side SHA-256 hash machinery (gated on the
provenance feature) had already written what it needed to detect that.

Add Dataset::verify_provenance() to the clawhdf5 facade, gated behind a
new `provenance` feature (on by default, forwarding to
clawhdf5-format/provenance which is already default-on). It surfaces a
typed VerifyResult (Ok/Mismatch/NoHash) via the existing Error type
rather than panicking. Deliberately NOT called automatically on
open()/dataset() — it decodes and hashes the entire dataset, which would
regress every read path (including the zero-copy/mmap ones) if run
unconditionally; callers opt in per dataset where the cost is
acceptable (e.g. a periodic integrity sweep).

Also re-export clawhdf5_format::provenance from the facade crate so
VerifyResult is reachable without depending on clawhdf5-format directly.

INT-08
 2026-08-17 00:54:44 +00:00
Omar Sobh 534331ffbe chore: Tier 4d — hoist tempfile/criterion/half/serde to workspace.dependencies
CI / test (push) Failing after 13s
Details 
Add [workspace.dependencies] to the root Cargo.toml for the four
duplicated-across-many-crates dependencies flagged by the earlier review:
tempfile (7 crates), criterion (6), half (4 — real version skew, clawhdf5-gpu
pinned 2.7 while others used bare 2), and serde (4). Update every consuming
crate to `dep = { workspace = true }`, preserving crate-local `optional =
true` where it already existed. half now resolves uniformly to 2.7.x
workspace-wide instead of two separate semver ranges.

Also fixed clawhdf5-filters/Cargo.toml's stale "rustyhdf5" description
while touching the file (same class of leftover rename as prior fixes).

Not touching rayon/byteorder/clap (no skew found, lower priority).
 2026-08-05 13:12:17 -07:00
Omar SobhandClaude Sonnet 4.6 5701e8045d feat: add Pcodec lossless numerical compression filter (arXiv:2502.06112) 
Implements Pcodec (filter ID 32023) via the `pco` 1.0.x crate as a new
optional compression codec. Pcodec achieves 30–94% better compression
ratio than Zstd for f32/f64 columnar data at 1–5 GiB/s decompression
speed, making it ideal for write-once/read-many embedding archives.

Write throughput at 512×512: 591 MiB/s (parity with Zstd-3 at 610 MiB/s).
For smaller chunks Zstd-3 remains faster due to Pcodec's fixed per-chunk
distributional analysis overhead.

- Add FILTER_PCODEC = 32023 constant to filter_pipeline.rs
- Add pcodec_compress/pcodec_decompress using pco::standalone API
- Wire into compress_chunk/decompress_chunk dispatch
- Add ChunkOptions.pcodec field and DatasetBuilder.with_pcodec() method
- Enable pcodec as highest-priority codec in build_pipeline()
- Add pco dep (optional, feature = "pcodec") to clawhdf5-format/clawhdf5
- Add write_2d_chunked_pcodec benchmark comparing pcodec vs zstd-3
- Document results in BENCHMARKS.md

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
 2026-07-01 01:16:08 +00:00
osobhandClaude Opus 4.8 a16e9ab8cb chore: bump workspace to 2.1.0 
Bumps [workspace.package] and all 17 crate package versions (and internal
path-dependency requirements) from 2.0.0 to 2.1.0 for a coordinated release.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
 2026-06-03 10:36:17 +00:00
redclawsystems 3f222f6956 Merge pull request 'docs(clawhdf5): document DType variants, fix unresolved doc links' (#17) from sdlc-docs/clawhdf5-types-20260514-165210 into main 2026-05-14 23:54:48 +00:00