perf(py): datasets and groups keep their address; groups their links

Every ds[...] and g[k] resolved the path from the root again, two or
three times per open, and resolving a name in a large group scans its
links: visiting a group was O(n^2). 4000 scalar datasets in one group
took 39 s (v1 group) and 131 s (dense) to list, read and re-read; now
0.3 s each. A Dataset keeps its object address, a Group (and the file's
root) its address and, after the first lookup, its link table.

New facade API File::dataset_at(address), tested in integration_tests.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-26 09:01:54 -05:00
co-authored by Claude Opus 5.5
parent b43bd2e67f
commit f0ecae38b6
9 changed files with 349 additions and 163 deletions
+11
View File
@@ -80,6 +80,17 @@
(h5py 4.1 ms; release build on tank, best of 5).
`test_a_long_index_list_decodes_each_chunk_once` compares 1-D, 2-D and
contiguous cases with h5py under a 2 s bound (5.8 s before, debug build).
- **Groups and datasets remember where they are.** Every `ds[...]`, and
every `g[k]`, resolved its path from the root again (two or three times
per open), and in a large group each resolution scans the group's links,
so visiting a group was quadratic: 4000 scalar datasets in one group took
39 s (`libver='earliest'`) and 131 s (`'latest'`) to list, read and
re-read in `test_big_groups_are_not_quadratic`; now 0.3 s each (debug
build). A `Dataset` keeps its object's address, and a `Group` (and the
file's root) its address and, once listed, its link table. New facade
API: `File::dataset_at(address)` opens a dataset without resolving a
path. libhdf5's `h5stat_newgrat.h5` (35001 members in the root): listing
takes 0.03 s and 2000 opens 1 ms (h5py: 0.022 s).
- **CI builds and tests the Python package.** It was excluded from CI.
`scripts/ci-test.sh` now lints `clawhdf5-py`, builds the wheel with
maturin, unpacks it under `target/` and runs the pytest suite; skipped