perf(py): datasets and groups keep their address; groups their links
Every ds[...] and g[k] resolved the path from the root again, two or three times per open, and resolving a name in a large group scans its links: visiting a group was O(n^2). 4000 scalar datasets in one group took 39 s (v1 group) and 131 s (dense) to list, read and re-read; now 0.3 s each. A Dataset keeps its object address, a Group (and the file's root) its address and, after the first lookup, its link table. New facade API File::dataset_at(address), tested in integration_tests. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
@@ -80,6 +80,17 @@
|
||||
(h5py 4.1 ms; release build on tank, best of 5).
|
||||
`test_a_long_index_list_decodes_each_chunk_once` compares 1-D, 2-D and
|
||||
contiguous cases with h5py under a 2 s bound (5.8 s before, debug build).
|
||||
- **Groups and datasets remember where they are.** Every `ds[...]`, and
|
||||
every `g[k]`, resolved its path from the root again (two or three times
|
||||
per open), and in a large group each resolution scans the group's links,
|
||||
so visiting a group was quadratic: 4000 scalar datasets in one group took
|
||||
39 s (`libver='earliest'`) and 131 s (`'latest'`) to list, read and
|
||||
re-read in `test_big_groups_are_not_quadratic`; now 0.3 s each (debug
|
||||
build). A `Dataset` keeps its object's address, and a `Group` (and the
|
||||
file's root) its address and, once listed, its link table. New facade
|
||||
API: `File::dataset_at(address)` opens a dataset without resolving a
|
||||
path. libhdf5's `h5stat_newgrat.h5` (35001 members in the root): listing
|
||||
takes 0.03 s and 2000 opens 1 ms (h5py: 0.022 s).
|
||||
- **CI builds and tests the Python package.** It was excluded from CI.
|
||||
`scripts/ci-test.sh` now lints `clawhdf5-py`, builds the wheel with
|
||||
maturin, unpacks it under `target/` and runs the pytest suite; skipped
|
||||
|
||||
Reference in New Issue
Block a user