Omar SobhandClaude Opus 5
4a3f1cc2f9
phase_config::DECLARED_BUT_UNREAD lists four keys the recipes set and nothing reads. security_hardening.toml already carried a "what is real here and what is decoration" section and annotated its own dead keys inline; the other five did not, so `produces = ["md"]` and `loop = "until_no_more_int_items"` read like settings. The risk was never the dead keys themselves — it is a reader taking `loop = "until_no_more_int_items"` for a loop. Each now says it is inert and points at the registry. They stay because they state the intent. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01WZb5A2kfVfjpdwSochkuHz
67 lines
3.4 KiB
TOML
67 lines
3.4 KiB
TOML
key = "benchmark"
|
|
title = "Benchmark"
|
|
blurb = "Author + baseline benchmarks so subsequent refactors can be measured before/after."
|
|
requires_repo = true
|
|
|
|
default_team_template = "rust_sdlc"
|
|
|
|
# ── What this recipe actually does ───────────────────────────────────
|
|
#
|
|
# `harness` is REAL and was mislabelled: `benchmark_runner::harness_from_config`
|
|
# reads it, and `phase_runner`'s benchmark sweep runs a baseline through it
|
|
# automatically, recording a `benchmark_snapshots` row. It was listed in
|
|
# `phase_config.rs` as NOT IMPLEMENTED — now corrected there. `mode` genuinely
|
|
# is inert.
|
|
#
|
|
# The real gap was that the phase carried no `task` and no `done_when` — so a
|
|
# benchmark mission ran
|
|
# one unjudged phase with a generic directive and reported `completed` whether
|
|
# it wrote a benchmark or not. Since the entire point of this workflow is to
|
|
# produce numbers a later refactor is measured against, a run that silently
|
|
# produced none is worse than no run: the next mission compares against a
|
|
# baseline that does not exist.
|
|
#
|
|
# The instructions therefore live in `task`/`done_when`, which ARE read, rather
|
|
# than in `mode`/`harness`, which are not. `benchmark` is in PRODUCING_KINDS, so
|
|
# the empty-delivery rule applies once there is something to deliver.
|
|
|
|
[[phases]]
|
|
kind = "benchmark"
|
|
order_idx = 0
|
|
[phases.config]
|
|
# INERT. `produces` is listed in phase_config::DECLARED_BUT_UNREAD —
|
|
# artifact rendering is not driven by it. Kept because it states the
|
|
# intent, annotated because a reader should not take it for a setting.
|
|
produces = ["md"]
|
|
# NOTE: `mode` is INERT — no code selects behaviour from it.
|
|
mode = "author_and_baseline"
|
|
# READ by benchmark_runner. "auto" lets it detect the stack; naming a harness
|
|
# explicitly ("criterion" + `bench_name`, "cargo_bench", "vitest_bench",
|
|
# "pytest_bench", or "shell" + `cmd`) pins it. The automatic baseline this
|
|
# drives is separate from, and a check on, the numbers the agent reports.
|
|
harness = "auto"
|
|
task = """
|
|
Author benchmarks for this repository and record a baseline that a later \
|
|
refactor can be measured against.
|
|
|
|
Pick the harness that matches the stack — criterion for Rust, `vitest --bench` \
|
|
or mitata for TS/JS, pytest-benchmark for Python — and say in the report which \
|
|
you chose and why.
|
|
|
|
Benchmark what the project's own hot path is, not what is easy to measure. A \
|
|
microbenchmark of a function nobody calls produces a number that will never \
|
|
change and teaches a future refactor nothing. Read the code first and name the \
|
|
operation you are measuring and why it is the one that matters.
|
|
|
|
Record the baseline in BENCHMARKS.md: the machine, the command, the numbers, \
|
|
AND the run-to-run spread from at least three runs. The spread is not optional \
|
|
detail — without it, nobody can tell whether a later 5% "improvement" is real \
|
|
or noise, which makes the entire baseline unusable for its only purpose.
|
|
|
|
Commit the benchmark code itself, not just the results. The next mission has to \
|
|
be able to re-run exactly what you ran.
|
|
"""
|
|
done_when = "BENCHMARKS.md exists and records, for each benchmark authored, the operation measured, the exact command to re-run it, the baseline numbers from at least three runs including the observed spread, and the machine they were taken on — and the benchmark source is committed to the repository"
|
|
max_iterations = 2
|
|
commit_policy = "always"
|