Moved to docs/archive/ (kept for their history, not deleted), each with a
one-line header saying it is historical and what supersedes it:
- IMPROVEMENT_LOG.md: three automated-loop PRs from April-May 2026, on
the earlier quantumclaw PR numbering, which now collides with this
repo's #12-#15. Superseded by CHANGELOG.md and git log.
- IMPROVEMENT_SCAN.md: one scan's notes (2026-05-04) of changes merged
long ago. Superseded by CHANGELOG.md and git log.
- docs/superpowers/plans/*.md -> docs/archive/plans/: agent pre-work
plans for d6c4d4f (2026-06-30), already marked implemented. The MPI plan
promised collective MPI-IO, which is not what shipped (MpiVol is
root-read + broadcast); its header says so and points at ROADMAP.md,
where collective I/O is an open item.
Nothing links to the old paths (research/*.md mention IMPROVEMENT_LOG.md
and ROADMAP.md in prose only).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The old file was a tracker for the mid-2026 agent-memory tracks, last
updated 2026-08-05, with an OpenClaw track and "what's next" items that
have since shipped (CI, fuzz target, WAL checksums, HNSW parallel build).
Now: releases v2.0.0-v2.7.0 and every PR merged since (#3-#21, merge
dates from git log), range-read milestones M0-M5, a one-paragraph summary
of the agent-memory work, and what is genuinely next: crates.io and PyPI
publishing (plus the broken Node package), SWMR writing, MPI collective
I/O, paged-metadata single-request reads, Blosc2/ZFP encoders, and the
open items of docs/known-issues.md. OpenClaw and ZeroClaw are listed only
as withdrawn. No dates are given for future work.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Explains each class compare.py assigns, how ref_bugs.py confirms a
ref-bug in every run and how ref.py corrects h5py's big-endian VL values
(ref_fix), and records the latest result (602 of 697 ok, 3 ref-bug,
2026-09-27) with a link to CONFORMANCE.md. Mentions --no-fetch.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The size table predated openUrl (it said so). Re-measured on tank at
9b5803f with `bash examples/wasm-viewer/build.sh`, then `wc -c` and
`gzip -9 -n -c`: the wasm is 1,384,607 B (378,485 gzipped), was 627,501
(191,639); the glue 40,711 (8,181), was 21,826 (4,487); remote.js 9,326
(3,448). 476,062 B of the wasm is the function-name section. opt-level z
(CARGO_PROFILE_WASM_RELEASE_OPT_LEVEL=z) is now 1% smaller gzipped than
the profile's s; 3 is larger. h5wasm rows unchanged.
Also adds the measured cost of listing a large group and reading one
dataset by URL (passes / requests / bytes, from CHANGELOG, 2026-09-27).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Every crate under crates/ now has a README (android, bench, cli, napi and
wasm had none), each saying what the crate is, its main types and
functions (names checked against the code), its cargo features with
defaults and which ones build C (checked with `cargo tree`), and links to
the top-level docs.
Corrections to the old stubs:
- clawhdf5-derive: the derive is `H5Type`, not `HDF5Type`, and it needs
clawhdf5-format as a dependency.
- clawhdf5-filters: deflate backends only, and no library crate depends
on it; the filter pipeline and every other codec are in -format.
- clawhdf5-gpu: vector distance compute, not I/O; not used by
HDF5Memory::search.
- clawhdf5-io: MpiVol is root-read + broadcast, not collective MPI-IO.
- clawhdf5-ann: from_hdf5/search(q, k) did not exist; load_from_hdf5 and
search(q, k, ef).
- clawhdf5-accel: checksum::crc32_simd did not exist; the SSE4 and wasm
backends are reported but run the scalar kernels.
- clawhdf5-gpu: the old example called l2_distances, which does not
exist (l2_search).
- clawhdf5-agent: it described a "vector store" with "GPU acceleration";
it now covers HDF5Memory, search options, WAL, signing, the graph.
- crates.io/docs.rs badges removed and `cargo install <crate>` replaced:
nothing is published; depend on git.
- fuzz: the opt-in CLAWHDF5_FUZZ_SECONDS smoke run in ci-test.sh.
- tools: the FileEditor interop tests that live in this crate.
- remote, py: license, other front ends, limits, File.mode/flush/chunks.
The Rust examples of the facade, format, filters, accel, ann, derive and
agent READMEs were compiled and run as tests (netcdf4, gpu and remote
compiled only) in a scratch crate; the CLI example was run.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
One line per document: user guides, evidence (CONFORMANCE, BENCHMARKS,
the conformance README), design docs, crate and package READMEs, and the
historical notes (roadmap, improvement logs, June plans, research briefs).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
QUICKSTART used APIs that do not exist (file.dataset_names(),
File::attr, AttrValue::Str, memory.search(&q, 5), MemoryConfig::new with
a &str, consolidation without timestamps), `clawhdf5 = "2.0"` from
crates.io, and "3-45x faster than libhdf5". It now covers HDF5 in Rust
(write, read, strings, in-place append, remote, SWMR), Python (read, r+,
w, URLs), NetCDF-4, h5rs, agent memory and the CLI, every snippet
compiled and run (Python against a wheel built from the tree).
USE_CASES dropped claims with no source (the agent crate adds ~2MB,
IVF-PQ under 1.2 ms on modest hardware, an OpenClaw scenario, a .brain
layout and `clawhub publish` commands) and now covers the HDF5 cases
(no-C builds, threads, remote data, untrusted files, SWMR, in-place
edits), the agent cases with measured numbers, and when to use
something else.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Leads with what clawhdf5 is today for an HDF5 reader: the conformance
result (602 of 697, 0 mismatches, no panic/hang/crash; CONFORMANCE.md of
2026-09-28), the CVE corpus against h5dump and h5py, concurrent reads
against h5py threads and processes (BENCHMARKS.md, 2026-09-26, c5334b1)
and the libhdf5 comparison with its date and caveat; then a feature matrix
(supported / read only / not supported), install from git and maturin,
Rust and Python quick starts, remote files, the browser, SWMR, h5rs, a
short agent-memory section, the crate map and a documentation table.
Removed: the unverifiable "1850+ tests" badge and "~86K lines" footer,
the "What's new v2.2 -> v2.7" list (it is CHANGELOG.md), the agent
comparison table with other products, the Phase 1/2 roadmap checklist,
and the long agent sections (now docs/agent-memory.md). Fixed: crates
listed as C-free, SZIP / N-Bit / scale-offset as read-only, virtual
datasets as written too, Python 'r+' can create attributes (it cannot
create or delete objects). Every code snippet was compiled and run
against the workspace, the Python ones against a wheel built from it.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The agent-memory detail that lived only in README.md (architecture,
modules, performance and footprint tables, LongMemEval, feature flags and
settings, file schema, SQLite migration, research foundation) moves to its
own page, so the README can lead with the HDF5 library. Code examples are
updated to the current API (MemoryConfig::new takes a PathBuf,
HDF5Memory::search with SearchOptions, consolidation with timestamps) and
were compiled and run against the workspace; the CLI section was run
against the built `clawhdf5` binary. New: the CLI's search defaults to
0.7/0.3, not the library's 0.4/0.6; the /integrity group of signed stores.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Standing rules (no C by default, h5py must read what we write, one
float16 implementation, claims need evidence, OpenClaw/ZeroClaw
withdrawn, the known-issues rule) are gathered in one place; library and
agent-memory invariants are split; the CI section lists what
ci-test.sh and the conformance workflow run now instead of a dated
"two jobs, green" line. Adds the conformance, remote, wasm, Python and
benchmark workflows (idle load below 2, dated records), and warns that
scripts/run-benchmarks.sh is stale and overwrites BENCHMARKS.md. Crate
roles corrected (clawhdf5-io is I/O adapters; codecs live in
clawhdf5-format). Benchmark figures now live in BENCHMARKS.md only.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The committed CONFORMANCE.md counts bad_nbit_parms_walk.h5 as an
our-error because its six h5py reads agreed in that run; a rerun the
same day confirmed the over-read and counted it ref-bug. The ok count
(602 of 697) is the same either way.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
range-reads.md opens with a table of milestones M0-M5 and the PRs that
merged them (#17-#21), replacing a header left garbled by earlier
merges, and each milestone's status names its PR. swmr.md says the
reader is merged (PR #19) and the writer does not exist. openclaw.md
links the Node package's known-issues entry.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A "Current headline numbers" table gives each figure's newest dated
measurement with its machine, command and section. Sections a later run
replaced are marked superseded with a link to the newer one; stale
"still open" notes and cross-references now point at the fixes. No
measured value changed.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
An "Open issues" table at the top links each open entry; fixed entries
move to "Fixed (history)", newest first, keeping the date, PR, affected
releases and what users must do. Open entries re-checked against main
(9b5803f): the remaining audit gaps are gathered into one entry, the
range-read and remote limits no longer contradict themselves (Python and
the browser open URLs; SWMR reading is File::open_swmr), and the
nondeterministic damaged-chunk error, fixed in PR #19, has its own
history entry.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Regenerated on tank: ok 600 -> 602 (attr_datatypes.hdf5 and
tcomplex_be.h5, compared against h5py's big-endian VL values corrected),
mismatch 2 -> 0; cve-2025-2308.h5 and cve-2025-44904.h5 are ref-bug
(h5py's values varied across six heaps in this run);
bad_nbit_parms_walk.h5 read the same in all six this run, so it stays
our-error, as the classification rule requires. Baseline raised.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
CHANGELOG (Unreleased): the v1 B-tree lookup, `Storage::hint`, the walks
that go on past a missing node, and the counts before and after on an
h5py file like the reviewer's (3000 datasets, 198 MB, earliest and
latest libver, 1 MiB and 64 KiB blocks), the corpus read lazily at
512 B and 64 KiB blocks, and the Node/Chromium suite.
known-issues (browser limits, round trips): the new counts, why the
passes cannot go lower (the chain of addresses), why merging nearby
requests does not help such a file, and that a second listing refetches
at 1 MiB blocks when the file's metadata blocks exceed `cacheSize`.
range-reads.md M4 status and the viewer README follow.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Opening one dataset of a v1 (symbol table) group read every symbol
table node and every name of the group to find it: over openUrl, 74
requests and 193 MB to read one 64 KiB dataset of the reviewer's
3000-dataset h5py file (libver earliest) at 1 MiB blocks, 515 requests
and 34 MB at 64 KiB. Locally it made a lookup O(entries).
`group_v1::find_v1_entry` looks the name up as libhdf5's
`H5G__stab_lookup` does: `H5B_find`'s binary search at each B-tree node
with `H5G__node_cmp3` (left key < name <= right key, keys being names in
the local heap, compared bytewise like strcmp), then the one symbol
table node, after the heap's free list is checked as the listing does.
The heap's data segment (up to 1 MiB) is hinted, since the keys are read
one after another. Path resolution uses it for a v1 group; only when it
does not find a hard link of that name (a soft link, or a B-tree out of
name order, damaged or hand-made, where libhdf5 would report the name
missing) does it read every entry as before. A storage error is returned
as is (over the lazy reader, a miss: reading every entry would not get
further). The result differs from before only in a group holding two
entries of one name, where the B-tree's is now the one found, as in
libhdf5.
Measured with tests/lazy.rs listing_cost_of_a_given_file, open + read
one 64 KiB dataset of the reviewer-like file, passes/requests/bytes,
before -> after (open included):
earliest, 1 MiB: 7/74/193.6 MB -> 6/5/5.2 MB
earliest, 64 KiB: 9/515/34.1 MB -> 8/7/524 KB
latest (dense groups, already a name-index lookup): 1 MiB 8/7/6.7 MB
-> 7/7/6.7 MB, 64 KiB 9/8/581 KB -> 8/8/581 KB (the previous
commit's hints: the name index header with the heap header)
New tests, failing before: every child of v1_groups_400.h5 resolves
to its listed address reading under 1/8 of the listing's bytes, and
missing names are not found; a name moved out of B-tree order is still
found (by the fallback); reading one of 2000 datasets lazily at 512-byte
blocks takes at most 7 passes and 6 requests (earliest; 529 requests,
333 kB before) and 8 passes, 9 requests (latest).
Conformance 600 of 697 (baseline 600), no file's class or detail
changed against a run of main.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Listing a large group over openUrl still took 6-11 passes (network round
trips) for the reviewer's 3000-dataset h5py file: each pass only found
the structures the walk reached before its first miss.
- The v1 and v2 B-tree collectors descend into every child of a node
after one fails (they only read the siblings before, so a sibling's
subtree came a pass later), then return the first error: results and
errors unchanged. The v2 walk stops once its record budget is spent,
so a shared-subtree tree still cannot multiply the work.
- Hints (`Storage::hint`, a no-op for every backend but the lazy one):
a group B-tree node's and a symbol table node's body (read once their
header gives a length, a round trip later when the body is in the
next block), an object header's first chunk and its continuation
chunks, the symbol table nodes a B-tree leaf names, a dense group's
name index header and the heap's root block (both read right after
the heap header). A listing also hints every child's object header as
its entry is read, even after a failure, and every direct block of a
dense group's heap (reading the indirect blocks, at most 4096 entries
and 4 levels deep); a lookup does not.
- The fractal heap's indirect-block layout (entry sizes, where the first
n entries end) is one helper used by the object reads and the hints.
Measured with tests/lazy.rs listing_cost_of_a_given_file on an h5py file
like the reviewer's (3000 datasets of 64 KiB, 198 MB), list('/'),
passes/requests/bytes, before -> after:
earliest, 1 MiB: 6/73/192.5 MB -> 4/68/192.5 MB
earliest, 64 KiB: 8/531/35.2 MB -> 5/530/35.3 MB
latest, 1 MiB: 9/98/196.5 MB -> 5/86/196.5 MB
latest, 64 KiB: 11/452/29.6 MB -> 6/454/30.5 MB
listing_a_large_group_takes_a_few_passes (512-byte blocks), budgets
tightened to the new counts: FileBuilder 600 children 5 -> 4 passes,
h5py 2000 children earliest 8 -> 5, latest 11 -> 6.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A parser often learns where the next structures are (a node's children,
a structure's body once its prefix gives its length) before it reads
them one at a time. Over openUrl's restartable reader a structure only
reached after a miss costs a pass, and a round trip, of its own.
- clawhdf5-format: `Storage::hint(offset, len)`, "about to be read by
this operation". Default: nothing (every backend that reads when
asked); `&T`, `Box`, `Arc` and the facade's `FileData` forward it
(shifted past a user block, clamped to the file).
- clawhdf5-wasm `LazyStorage` records hinted blocks it lacks. A pass
that misses nothing ignores them (a hint never adds a round trip); a
pass that misses also asks for them, in file order, while the pass
stays within what is left of the operation's `maxFetch` budget (a
hint never makes a call fail). At most 1 MiB (or a block) per hint
and 65536 blocks per pass are recorded, whatever a file makes a
parser hint. `Operation::attempt` follows hints; the plain
`LazyStorage::attempt` does not. `run_blocking` and the browser's
driver use the former. `LazyStats::hinted_blocks` counts them.
No parser hints yet: results, passes and requests are unchanged.
Tests: hinted blocks come with a miss and never alone (cached ones
skipped, a plain attempt ignores them), and stay within the fetch
budget, a hint past the end or longer than the file harmless.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A ref-bug file's refused objects were still listed as our-error root
causes. The summary line now names every non-ok class and its count, and
an empty root-cause table says None.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
4313917 kept parse_v1_messages out of line (#[inline(never)]) so the
generic parser would not carry the loop; since ef428d7 the slice path is
compiled once in this crate, and the call itself was the remaining cost
of the chunk queue: A/B builds of object_header_parse_x401 with only this
attribute changed put #[inline(never)] and no attribute at 24.5-24.9 us
and #[inline] at 23.6-24.0 us, with 8f59b2e at 23.6-24.1 us. Lazily
creating the chunk list only when a continuation is found (tried too)
measured no faster and was not kept.
Same code otherwise: every chunk-queue check (65,536 chunks, cycles, file
size budget, one chunk buffer at a time, libhdf5 order, overlap allowed)
is unchanged.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The last 3 our-errors and 2 mismatches were documented as not ours but
still counted against us, on a heuristic (any big-endian VL mismatch) and
a fixed list.
- ref.py checks that the installed h5py returns big-endian VL elements
with the file's bytes under a little-endian dtype (writing and reading
a vlen('>f4') in memory) and, if so, relabels them with the file's byte
order before hashing, marking the object `ref_fix`. The values are now
compared: attr_datatypes.hdf5 /@vlen_uint64 and tcomplex_be.h5
/VariableLengthDatasetFloatComplex are identical to ours (h5dump 1.14.6
prints the same (1, 2), (3, 4, 5), (42)).
- ref_bugs.py re-reads each object h5py reads only through a libhdf5 bug
in six processes with different heaps (import order, MALLOC_PERTURB_).
Values the file determines are the same every time; these three change
(6, 6 and 3 distinct results), so they are over-read memory, not data
clawhdf5 could match. compare.py classifies a file `ref-bug` only when
every difference is such an object confirmed in the same run.
- report.py: the ref-bug class, the evidence table, the corrected
objects; test_ref.py covers both (run in the nightly job).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The first run of the day could not get an idle tank: two orphaned h5py
SWMR reader processes from earlier interop tests (since stopped) kept a
core each busy. Re-run with the load below 2 at every round:
ObjectHeader::parse is +4.2% (real: the ranges do not overlap; about 2.5
ns per header, not visible in the facade listing, which is -1.7%); full
deflate reads +1.7% (1 thread) to +36% (16 threads); the -5.6% single-
thread contiguous hyperslab result from the loaded run is noise (-1.7%,
overlapping ranges) and is withdrawn from known-issues.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Oracle and full longmemeval_s haystack, f32 and --float16, plus --sweep
(both corpora) and --rerank-sweep, on tank at 7a8fae0 (MiniLM on the RTX
5060 Ti). Not idle: two orphaned h5py test processes kept the 1-minute load
at 2.1-2.7 (up to 8.4 during runs), so no latency figure was updated.
The headline (hybrid 0.4/0.6 turn Hit@5 81.4%) and the float16 claim
reproduce. Oracle hybrid is 86.8%, not 85.2%: the old figure was at the
0.7/0.3 default of the time (today's 0.7/0.3 gives 85.2%). The ablation's
0.7/0.3 row, vector-only MRR and seven sweep rows move in the last digit or
by one or two questions, most likely from the index tie-break of 3ed0489.
RRF turn MRR 0.5967 -> 0.5969 is not explained. Recency counts vary by one
question between identical runs, so the float16 "flips" note is corrected.
ROADMAP 8.2 no longer repeats the retracted session-level and MemX claims.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
8f59b2e (main before PR #18) against 7a8fae0, separate binaries,
alternating rounds on tank (6 Criterion rounds of local_metadata_bench,
3 of concurrent_read --decode-threads 1). Still provisional: two orphaned
h5py test processes held the load at 2.1-2.6 and the load < 2 gate was
not met in 2 hours.
The facade listing regression is gone (-0.8%). ObjectHeader::parse x401
is +4.1% and one-thread contiguous hyperslab reads -5.6%; both listed as
open in known-issues. Full deflate reads are 7-23% faster.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
ChunkCache::chunks_for returned the chunk index's values in hash-map
order, which differs between File instances (a new HashMap per open).
Readers that stop at the first failing chunk therefore named different
chunks on different opens of the same damaged file: whole-dataset
selections here, and the indexed reader behind the storage harness's
intermittent cve-2025-2310.h5 failure. The cache now also keeps the
chunks in the order the index lists them (fetched or built under one
lock) and returns that order, as the uncached readers use.
Regression: several_damaged_chunks_report_the_same_chunk_every_time
(four chunks inflating to different short lengths; before the fix two
opens reported chunk [40, 0] and chunk [320, 0]).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The editor branch added File::from_std_file and the SWMR branch added the
swmr field to File; merged, the constructor did not set it.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A boolean array of the axis's length, or of the dataset's whole shape, is
a mask h5py would apply (NotImplementedError here); one of any other shape
(np.array(True), a wrong length) is a key h5py itself refuses with
TypeError, which test_errors_match_h5py requires us to match.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
CHANGELOG (M4 section), known-issues (wasm limits: maxFetch, the 1 GiB
decode limit, the 4 GiB file limit on wasm32, bodies cut off at their
length, listing passes, the cross-origin tests, and a pre-existing
nondeterministic error choice on cve-2025-2310.h5 that can fail the
native corpus comparison), the viewer README (options, how listing
costs, tests) and the M4 status in docs/design/range-reads.md.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
- opts.headers is read as fetch reads it (a Headers, [name, value] pairs
or a plain object); it was spread as an object, which silently dropped
a Headers instance (a common way to pass Authorization). A caller's
Range is not sent.
- When one range request of a batch fails, the others in flight are
aborted (one AbortController per batch, its signal passed to fetch)
and no new ones start; the first failure is the error. The workers
used to go on issuing requests nobody waited for.
- parallel must be an integer from 1 to 1024 (openUrl) or a positive
integer (fetchRanges): a non-number gave NaN workers, so none ran and
fetchRanges returned nothing.
Tests (test.mjs): headers as object, Headers and pairs reach the fetch;
bad parallel values are option errors; fetchRanges with a 500 on the
third of 20 ranges at parallel 3 starts 3 requests and aborts the 2 in
flight. Before: the Headers case sent no header, 17 requests started
after the failure with none aborted, and parallel "x" returned.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
serve.py always exposed Content-Range and ETag, and Node has no CORS, so
openUrl's documented cross-origin path (length from a HEAD request,
answers checked by body length alone, no validator) was never run.
- serve.py /noexpose/ serves ranges without Content-Range, ETag,
Last-Modified or Accept-Ranges (what a page sees of a server that
does not expose them); /unexposed/ sends them but exposes none, for a
real browser. HEAD requests are counted (0 bytes).
- test.mjs: every fixture check through /noexpose/ at 1 MiB and 512-byte
blocks (one HEAD each, requests and bytes as the server counted them),
concurrent reads with cacheSize 0, a short answer still caught, and a
server without a HEAD length a clear error.
- browser.sh: the page on 127.0.0.1 opens the file from localhost, once
with Content-Range exposed and once through /unexposed/, where the
server's log must show the HEAD.
Checked by breaking the HEAD length in remote.js: the new checks fail.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Planning from a mapping of a clone of the locked descriptor shared its
flock: a process forked by another thread meanwhile (any Command) kept the
lock alive for a moment after the editor was dropped, and a test that
reopened the file at once saw Error::Locked (once in a full run). On Linux
plan through /proc/self/fd (a new open file description of the same
file, which still follows a rename); elsewhere keep the clone.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Listing a group read every child's object header and stopped at the
first that was not fetched yet, and so did the traversals of the group's
index (v1 B-tree and symbol table nodes, the local heap's names, v2
B-tree nodes and fractal heap objects). Over openUrl's restartable
reader each block cost its own pass and round trip: 184 serial requests
to list 3000 datasets at 1 MiB blocks, 536 at 64 KiB.
- core::Reader::list reads every child's header before returning the
first error (the same error, in listing order, Group::groups/datasets
return), classifying them as those do.
- clawhdf5-format: after the first sibling that fails, the B-tree v1
and v2 collectors, the symbol table node loop and the dense-link loop
go on reading (not using) the remaining siblings, then return that
first error: results and errors are unchanged, only failing
traversals read more, and in memory that is free (storage::touch).
A v1 group's local heap segment (names) is read at once, up to 1 MiB.
- LazyStorage no longer fills a one-block hole that is already cached
(it was fetched again: 215 MB fetched from a 198 MB file).
Measured with tests/lazy.rs listing_cost_of_a_given_file on the
reviewer's file (h5py, 3000 datasets of 64 KiB, 198 MB), list('/'):
libver earliest, 1 MiB blocks: 185 passes/184 requests -> 6/73
libver earliest, 64 KiB: 537/536 -> 8/531 (6 in flight)
libver latest, 1 MiB: 189/188 -> 9/98
libver latest, 64 KiB: 453/452 -> 11/452
Bytes fetched are unchanged (the headers are spread through the file).
New test listing_a_large_group_takes_a_few_passes (512-byte blocks):
FileBuilder 600 children 102 -> 5 passes; h5py earliest/latest 2000
children 8 and 11 passes. Conformance 600 of 697 (baseline 600);
check-32bit-casts, check-nostd and h5rs-fuzz over the CVE corpus clean.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
h5py supports boolean masks for reads and writes; clawhdf5 supports
neither, so a mask is an unsupported operation (NotImplementedError, as
for every other edit the bindings cannot do), not an invalid key.
Tests: test_unsupported_edits_are_clear_errors (1-D, N-D and per-axis
mask writes, file unchanged) and test_boolean_masks_are_refused (reads).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A thread appending timestamps in a loop runs while a 1024x1024 gzip
dataset is rewritten through 'r+': the largest gap between its stamps
during the edit must be under half the edit's duration (an edit holding
the GIL stalls it for the whole edit; checked with a GIL-holding regex
standing in for the edit: one 0.20 s gap in a 0.21 s call). Edits already
ran detached; nothing tested it.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
FileEditor re-opened its path to plan each edit but wrote through the file
it held open, and the Python 'r+' handle re-opened the path after every
edit to read. When the path came to name another file between edits (a
rename or replacement, or a relative path after os.chdir), an edit was laid
out from the other file's metadata and written into the held one,
corrupting it, and later reads came from the other file (the review's
repro: h5py then reports "invalid dataset size, likely file corruption").
The editor now plans from a mapping of its own file (a clone of the held
descriptor, dropped before the edit writes) and canonicalises its path at
open. New FileEditor::reader() opens the held file anew for reading,
without sharing the editor's flock (a mapping of a cloned descriptor holds
the lock until unmapped): through /proc/self/fd on Linux, which follows a
renamed file; elsewhere by path, refused on Unix when the path no longer
names the held file. The Python handle reads through it and keeps no path;
a 'w' file is written at the absolute path it was opened with.
Tests: edit_tests.rs edits_go_to_the_file_held_not_the_path; test_edit.py
test_relative_path_and_chdir and test_path_replaced_between_edits.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
osobh
osobh