Commit Graph
8 Commits
Author SHA1 Message Date
osobhandClaude Opus 5.5 c27a478e44 docs: fact-check the refreshed documentation against its sources
Numbers, API names, feature defaults and PR references checked against
CONFORMANCE.md, BENCHMARKS.md, CHANGELOG.md, the code and git history.

- int8 index figures (1.74x memory, 1.63x QPS) carry the dates git gives
  them (2026-09-19/20, machine not recorded, not re-run) instead of none;
  the Pi 5 1.18x carries 2026-09-21.
- BENCHMARKS headline: the libhdf5 chunked-write figure is the newest
  measurement (35x, 2026-09-23), not 45.3x (2026-08-03).
- Conformance counts follow the 2026-09-28 run (1 our-error, 2 ref-bug)
  in conformance/README.md, ROADMAP.md and CLAUDE.md, with a pointer to
  the bad_nbit_parms_walk.h5 flip.
- README: LZ4 is opt-in; the browser refuses reference/opaque/bitfield/
  time datasets too; zlib-rs byte-identity scoped to what was measured;
  macOS default links the system libz for inflate.
- Crate READMEs: system-zlib-decompress does something (macOS), SweepDetector
  lives in prefetch, checkpoint after more than 500 WAL entries, NetCDF-4
  unlimited-dimension size warning.
- agent-memory.md: string-dataset compression threshold, agents-md prints
  Markdown, float16 file sizes linked to their study.
- known-issues.md: contiguous selection reads, 1.21x vs h5py threads.
- docs/README.md, USE_CASES.md, ROADMAP.md, CLAUDE.md: range-read
  milestones M0-M5 and PRs #17-#19, missing README rows, CLI keygen/verify,
  dated figures, fast-math is not BLAS.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
2026-09-28 11:23:51 -05:00
osobhandClaude Opus 5.5 419cb52287 docs: ROADMAP rewritten from CHANGELOG, git log and known-issues
The old file was a tracker for the mid-2026 agent-memory tracks, last
updated 2026-08-05, with an OpenClaw track and "what's next" items that
have since shipped (CI, fuzz target, WAL checksums, HNSW parallel build).

Now: releases v2.0.0-v2.7.0 and every PR merged since (#3-#21, merge
dates from git log), range-read milestones M0-M5, a one-paragraph summary
of the agent-memory work, and what is genuinely next: crates.io and PyPI
publishing (plus the broken Node package), SWMR writing, MPI collective
I/O, paged-metadata single-request reads, Blosc2/ZFP encoders, and the
open items of docs/known-issues.md. OpenClaw and ZeroClaw are listed only
as withdrawn. No dates are given for future work.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
2026-09-28 11:13:40 -05:00
osobhandClaude Opus 5.5 b39a705e77 docs: LongMemEval with real embeddings re-run on 2026-09-27
Oracle and full longmemeval_s haystack, f32 and --float16, plus --sweep
(both corpora) and --rerank-sweep, on tank at 7a8fae0 (MiniLM on the RTX
5060 Ti). Not idle: two orphaned h5py test processes kept the 1-minute load
at 2.1-2.7 (up to 8.4 during runs), so no latency figure was updated.

The headline (hybrid 0.4/0.6 turn Hit@5 81.4%) and the float16 claim
reproduce. Oracle hybrid is 86.8%, not 85.2%: the old figure was at the
0.7/0.3 default of the time (today's 0.7/0.3 gives 85.2%). The ablation's
0.7/0.3 row, vector-only MRR and seven sweep rows move in the last digit or
by one or two questions, most likely from the index tie-break of 3ed0489.
RRF turn MRR 0.5967 -> 0.5969 is not explained. Recency counts vary by one
question between identical runs, so the float16 "flips" note is corrected.
ROADMAP 8.2 no longer repeats the retracted session-level and MemX claims.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
2026-09-27 19:25:36 -05:00
osobhandClaude Opus 5.5 0c65a27b00 docs: withdraw the OpenClaw integration claims
CI / test-arm64 (pull_request) Successful in 1m3s
CI / test (pull_request) Successful in 5m48s
The docs described a "drop-in" OpenClaw memory backend enabled with
`memory.backend = "clawhdf5"`. Checked against OpenClaw's source and
docs (v2026.2.26 through v2026.9.6): that config was never valid —
v2026.2-v2026.7 accepted only "builtin"/"qmd" and rejected unknown
keys, so a Gateway given it refuses to start, and v2026.8.1 (OpenClaw
2.0) removed the key. No plugin was ever built (no manifest, no
registration, no tools), nothing was tested against OpenClaw, the
linked github.com/redclawsystems/openclaw is a 404, and
@redclaw/clawhdf5 was never published.

Decision (2026-09-25): not pursuing an OpenClaw plugin for now; ZeroClaw
is the integration target.

- Remove openclaw-integration.md, openclaw-config.md and
  migration-guide.md; add docs/openclaw.md: the status, what a memory
  plugin needs against v2026.9.6 (plugins.slots.memory, manifest with
  kind "memory", registerMemoryCapability / MemorySearchManager,
  prebuilt native packages), and what this repo has as building blocks.
- README, QUICKSTART, USE_CASES, ROADMAP (Track 7 withdrawn), CLAUDE.md
  and the `openclaw` module docs describe ClawhdfBackend as what it is:
  a Markdown-oriented library backend, not an OpenClaw plugin. The
  QUICKSTART example is corrected (the old one called a three-argument
  create that does not exist) and states its limits.
- packages/clawhdf5-node: marked unpublished and broken, "private": true
  so it cannot be published by accident; its bugs (snake_case vs
  camelCase fields, wrong addon path, no way to store an embedding,
  wrong WAL name) are recorded in docs/known-issues.md.
- Two broken rustdoc links fixed along the way.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
2026-09-25 10:27:54 -05:00
Omar Sobh 429c29b76b docs: sync README/ROADMAP/CLAUDE/CHANGELOG with Tier 1-4 hardening work
CI / test (push) Failing after 3s
README.md:
- Fix badly stale LongMemEval numbers (badge said Hit@5 46%, table showed
  fabricated ~46%/~0.34/~72% figures that never matched BENCHMARKS.md's
  actual results of Hit@5 100% session / 84.4% turn-level, MRR 1.0/0.6597)
- Remove clawhdf5-types from the Crate Map — that crate was removed in an
  earlier cleanup pass but the README diagram was never updated; fix the
  crate count (16, not 17) and stale line-of-code figures (72,087/84K -> ~92K)
- Fix a dead #benchmarks badge anchor (no such heading exists) -> #performance
- Document the new clawhdf5-ann `parallel` feature (had no Feature Flags entry)
- Note WAL's CRC32 per-entry check, link the new tank LongMemEval/SIMD/
  vector-search reproduction section, update stale test-count comment
  (417+ -> 1,650+) and Phase 2 roadmap blurb (LongMemEval is now done)

ROADMAP.md:
- Check off "Academic benchmark cross-validation" (done via the tank
  LongMemEval re-run) and add a new "Recently closed out" section
  summarizing the Tier 3-4 hardening pass (Android JNI validation, pyo3
  bump, WAL CRC32, bounds-check audit + fuzz harness that found 3 real
  bugs, HNSW optional parallel feature, workspace.dependencies)
- Update stale test count (1,546 -> 1,650+) and last-updated date

CLAUDE.md: mention WAL's per-entry CRC32 check

CHANGELOG.md: add Security/Performance/Architecture/Documentation entries
under Unreleased summarizing all of Tiers 1-4 (this had not been touched
since 2026-06-04, predating the entire hardening pass)
2026-08-05 15:26:47 -07:00
Omar Sobh 62595d5ac0 chore: Tier 2 quick wins — version skew, docs, cleanup, overflow-safe bounds
CI / test (push) Failing after 14s
- Fix version skew: clawhdf5-py (pyproject.toml 1.93.0 -> 2.1.0) and
  packages/clawhdf5-node (package.json 2.0.0 -> 2.1.0) were both behind
  the actual crate version.
- Correct stale ROADMAP.md claims: the TypeScript bridge already has a
  complete napi-rs package (not "no package.json"); CI/CD is now wired
  up via .gitea/workflows/ci.yml.
- Fix CLAUDE.md: clawhdf5-gpu uses wgpu with hand-written WGSL compute
  shaders, not CubeCL.
- chunked_read.rs: drop 12 unnecessary chunk_dimensions[..rank].to_vec()
  allocations — all three callees already accept &[u32].
- btree_v1.rs: add an overflow-safe ensure_len(data, offset, needed)
  helper (checked_add) and use it at the two plain-arithmetic bounds
  guards, closing a usize-overflow edge case reachable from a crafted
  near-usize::MAX B-tree offset. Add a regression test.
- Clarify that the integrity hashes in clawhdf5-agent/provenance.rs
  (FNV-1a) and clawhdf5-format/provenance.rs (SHA-256) are unkeyed and
  only detect accidental corruption, not tampering — doc-only change.
- README.md: document that the mpi-io feature's read/write paths are
  root-read+broadcast / gather-to-rank-0, not true collective I/O.
2026-08-05 12:02:23 -07:00
Omar SobhandClaude Sonnet 5 6b1ea450f5 chore: cleanup pass — remove empty types stub, implement superblock v4, reconcile plan docs
- Remove clawhdf5-types (empty 1-line stub crate; type defs already live in
  clawhdf5-format). Update workspace Cargo.toml and CLAUDE.md accordingly.
- Implement HDF5 superblock v4 (page-buffer mode) read and write support in
  clawhdf5-format: Superblock::parse_v4, page_size field, v4 serialize
  branch, and FileWriter::with_page_size. This was the one task left
  unimplemented from docs/superpowers/plans/2026-06-29-format-write-extensions.md.
- Reconcile the three docs/superpowers/plans/*.md docs (filter codecs,
  format write extensions, MPI-IO VOL) against actual shipped code: they
  were pre-work plans for d6c4d4f (2026-06-30) committed to git late on
  2026-08-03 with all checkboxes still unchecked. Mark completed tasks done
  and add a status note so they read as historical records, not open work.
- Refresh ROADMAP.md's "What's Next" section against current repo state.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
2026-08-03 08:11:31 -07:00
redclawsystems 3f222f6956 Merge pull request 'docs(clawhdf5): document DType variants, fix unresolved doc links' (#17) from sdlc-docs/clawhdf5-types-20260514-165210 into main 2026-05-14 23:54:48 +00:00