quantumclaw/clawhdf5
3
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity
170 Commits 1 Branch 11 Tags
8ea455bbcbbae1a8b11d20506ee113d614ecbf64
Commit Graph
5 Commits
Author SHA1 Message Date
Omar Sobh 55959b4920 ci: wire up CI, fix no_std build, fix stale package names in scripts
CI / test (push) Failing after 15s
Details 
- Add .gitea/workflows/ci.yml running scripts/ci-test.sh (fmt, clippy,
  test, no_std check) on push/PR to main.
- Fix stale rustyhdf5-py/rustyhdf5-format package names in
  ci-test.sh/check-nostd.sh, which had been silently no-op'ing those
  checks (cargo warns but doesn't fail on an unknown --exclude/-p
  target).
- With those checks actually running, fix the real issues they surface:
  - clippy: useless_conversion in chunked_write.rs, byte_char_slices in
    global_heap.rs/object_header.rs.
  - cargo fmt: apply formatting across the workspace (whitespace only).
  - no_std (thumbv7em-none-eabihf) build errors in clawhdf5-format:
    core::sync::atomic::AtomicU64 doesn't exist on that target (no
    native 64-bit atomics) — switch profiling.rs's counters to
    portable-atomic, which falls back to a CAS-based emulation there
    and is a no-op wrapper elsewhere. Add missing alloc imports for
    Box (filters.rs), Vec (filters_szip.rs), and format! (dict_encoding.rs)
    on no_std paths. Replace f64::powi (std/libm-only) with a small
    local exponentiation-by-squaring helper in the scale-offset filter.
 2026-08-05 10:50:13 -07:00
osobhandClaude Opus 4.8 e0189cd5c4 harden: make the new readers panic-free on malformed input 
The readers added this cycle parse untrusted bytes, so malformed/hostile
input must produce errors — never a panic, OOM, or unbounded recursion.
Audited each new surface and fixed the concrete vectors, each covered by an
adversarial regression test:

- Paged Fixed Array: `1 << max_nelmts_bits` shift overflow (u8 up to 255);
  element-count bounded by file size; element/page offset multiplies checked.
- H5S selection decoder: ALL/NONE validate they have the 16 bytes they claim
  to consume; hyperslab rank capped at 32 (H5S_MAX_RANK); iter_linear
  coordinate/stride/product arithmetic uses checked ops.
- VDS mapping parser: drop pre-allocation from the untrusted `nused`;
  bounds-check all selection slicing.
- scale-offset / N-Bit filters: `1 << minbits` overflow at minbits==64; N-Bit
  `bit_offset + precision` overflow; N-Bit type-tree recursion depth capped to
  stop a crafted nested tree from overflowing the stack; element counts bounded
  by the chunk's expected decompressed size (threaded the previously-unused
  chunk_size into both decoders) so a bogus count can't over-allocate.
- VDS assembly: a virtual dataset whose source is itself virtual (a cycle) now
  errors instead of recursing into a stack overflow.

16 new adversarial tests; full format suite (482 lib) + agent + facade green;
clippy clean.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
 2026-06-04 00:28:33 +00:00
osobhandClaude Opus 4.8 908af40282 feat: assemble 1-D same-file Virtual Datasets (VDS) 
A virtual layout previously returned UnsupportedVersion. Implement reading
for the common 1-D, same-file case, reverse-engineered and validated against
HDF5 2.0.

- Rewrite parse_vds_mappings to the real global-heap block format
  (version(1) · nused(length_size) · entries · checksum(4)), where each
  entry is source-file(null) · source-dataset(null) · source-selection ·
  virtual-selection. Block version 1 encodes a same-file source as a single
  0x04 marker in place of the file name; version 0 stores an explicit file
  name. The selections are H5S-serialized and self-describing in length, so
  they are decoded to find entry boundaries. The previous parser used a
  guessed layout that did not match real files.

- Extend Selection with decode_serialized() (H5S_select_serialize: ALL,
  NONE, and version-3 regular hyperslabs) and iter_linear_1d().

- Add read_virtual_data: resolve the mapping block from the global heap,
  read each same-file source dataset, and scatter its selected elements into
  the virtual buffer; unmapped regions stay at the zero fill value.
  External-file sources and N-D selections return a clean unsupported error.

Tests: real-file integration test (vds_same_file.h5: partial source slice +
fill gap), selection decoder unit tests built from the fixture bytes, and
same-file/external mapping-parser unit tests.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
 2026-06-03 20:01:51 +00:00
osobhandClaude Opus 4.8 bc3a3a977a fix: read array datatypes and chunked layouts from HDF5 2.0 (version 5) 
Follow-up to the v5 compound fix, found by an interop sweep over diverse
h5py/HDF5 2.0 (libver=latest) datasets:

- Array datatype (class 10) version 5 was rejected. v3/v4/v5 share the same
  array encoding, so the parser now accepts 3-5.
- Data Layout message version 5 was rejected, which broke EVERY chunked/
  compressed dataset written by modern HDF5. v5 reuses the v4 message
  structure, so it now routes through parse_v4.

Validated end-to-end: a gzip-compressed, Fixed-Array-indexed v5 chunked dataset
now decodes to the correct values. Adds h5py-free regression tests using the
real v5 array-datatype and chunked-layout bytes, and updates the layout
invalid-version test to use v6.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
 2026-06-03 12:00:38 +00:00
redclawsystems 3f222f6956 Merge pull request 'docs(clawhdf5): document DType variants, fix unresolved doc links' (#17) from sdlc-docs/clawhdf5-types-20260514-165210 into main 2026-05-14 23:54:48 +00:00