Omar SobhandClaude Opus 5
18dc0b964b
Four defects, found by checking the audit's claims instead of trusting them. Two of the audit's own findings turned out to be wrong, and the registry that exists to record which config keys are read was itself inaccurate — so the corrections are part of the change. upsert_task raised 42P10 on every call, for every caller `mission_tasks_external_uniq` is a PARTIAL unique index (WHERE external_id IS NOT NULL). Postgres will not match a partial index to an ON CONFLICT target unless the statement repeats the predicate, so the upsert failed on its first row. Both callers — the task-card parser that turns INT markers into tasks, and the security scanner — map the error to a string their caller logs. Two features were broken and nothing was red. Regression test in cm-db with a negative control: reverting the WHERE reproduces 42P10 exactly. the security scan never ran `security_scan::run` was reachable only from an operator button, so security_hardening.toml — a workflow whose entire first phase is a scan — ran an agent that was never told to scan and never fired the scanner either. phase_runner now sweeps finished security_scan phases, mirroring the benchmark baseline sweep that was added for the identical defect. Guarded on a new completion marker rather than on findings: a clean scan writes no findings, so a findings-guard would rescan forever. The marker also answers the question an operator actually asks, which is not "how many findings" but "was this looked at, by what, and when". two recipes could not fail security_hardening.toml and benchmark.toml carried no `task` and no `done_when` on any phase. A phase without done_when never enters evaluating, is never judged, and reports completed whatever it did — so a security mission could scan nothing and go green, and a benchmark mission could record no baseline that the next refactor would then compare against. Both now state the work and the condition, with inert keys annotated inline rather than deleted, so the gap between what a recipe asks for and what a phase receives stays visible. the config registry was wrong in both directions `harness` was listed NOT IMPLEMENTED while benchmark_runner reads it and phase_runner runs a baseline through it. `tools` was listed NOT IMPLEMENTED while security_scan::run reads it. A registry that exists so an operator can trust what a recipe does is worse than useless when it is inaccurate. Both corrected, `bench_name` and `cmd` added, and `test_command` deleted — it had neither a reader nor a writer, so it described a situation that could not arise. Also: CLAWMATES_JUDGE_MODEL had two different defaults (opus-4-8 in routes/topology.rs vs opus-5 in cm_runtime::judge_model) and a doc comment naming a third; topology now calls the one function. GITEA_TOKEN's absence in mission_plan is stated rather than degrading to the same "could not be read" string a private repo produces. BRAINHUB_API_KEY needed no change — hub::push already rejects an unset key with a named error. That half of the finding was overstated. Co-Authored-By: Claude Opus 5 <[email protected]>
64 lines
3.2 KiB
TOML
64 lines
3.2 KiB
TOML
key = "benchmark"
|
|
title = "Benchmark"
|
|
blurb = "Author + baseline benchmarks so subsequent refactors can be measured before/after."
|
|
requires_repo = true
|
|
|
|
default_team_template = "rust_sdlc"
|
|
|
|
# ── What this recipe actually does ───────────────────────────────────
|
|
#
|
|
# `harness` is REAL and was mislabelled: `benchmark_runner::harness_from_config`
|
|
# reads it, and `phase_runner`'s benchmark sweep runs a baseline through it
|
|
# automatically, recording a `benchmark_snapshots` row. It was listed in
|
|
# `phase_config.rs` as NOT IMPLEMENTED — now corrected there. `mode` genuinely
|
|
# is inert.
|
|
#
|
|
# The real gap was that the phase carried no `task` and no `done_when` — so a
|
|
# benchmark mission ran
|
|
# one unjudged phase with a generic directive and reported `completed` whether
|
|
# it wrote a benchmark or not. Since the entire point of this workflow is to
|
|
# produce numbers a later refactor is measured against, a run that silently
|
|
# produced none is worse than no run: the next mission compares against a
|
|
# baseline that does not exist.
|
|
#
|
|
# The instructions therefore live in `task`/`done_when`, which ARE read, rather
|
|
# than in `mode`/`harness`, which are not. `benchmark` is in PRODUCING_KINDS, so
|
|
# the empty-delivery rule applies once there is something to deliver.
|
|
|
|
[[phases]]
|
|
kind = "benchmark"
|
|
order_idx = 0
|
|
[phases.config]
|
|
produces = ["md"]
|
|
# NOTE: `mode` is INERT — no code selects behaviour from it.
|
|
mode = "author_and_baseline"
|
|
# READ by benchmark_runner. "auto" lets it detect the stack; naming a harness
|
|
# explicitly ("criterion" + `bench_name`, "cargo_bench", "vitest_bench",
|
|
# "pytest_bench", or "shell" + `cmd`) pins it. The automatic baseline this
|
|
# drives is separate from, and a check on, the numbers the agent reports.
|
|
harness = "auto"
|
|
task = """
|
|
Author benchmarks for this repository and record a baseline that a later \
|
|
refactor can be measured against.
|
|
|
|
Pick the harness that matches the stack — criterion for Rust, `vitest --bench` \
|
|
or mitata for TS/JS, pytest-benchmark for Python — and say in the report which \
|
|
you chose and why.
|
|
|
|
Benchmark what the project's own hot path is, not what is easy to measure. A \
|
|
microbenchmark of a function nobody calls produces a number that will never \
|
|
change and teaches a future refactor nothing. Read the code first and name the \
|
|
operation you are measuring and why it is the one that matters.
|
|
|
|
Record the baseline in BENCHMARKS.md: the machine, the command, the numbers, \
|
|
AND the run-to-run spread from at least three runs. The spread is not optional \
|
|
detail — without it, nobody can tell whether a later 5% "improvement" is real \
|
|
or noise, which makes the entire baseline unusable for its only purpose.
|
|
|
|
Commit the benchmark code itself, not just the results. The next mission has to \
|
|
be able to re-run exactly what you ran.
|
|
"""
|
|
done_when = "BENCHMARKS.md exists and records, for each benchmark authored, the operation measured, the exact command to re-run it, the baseline numbers from at least three runs including the observed spread, and the machine they were taken on — and the benchmark source is committed to the repository"
|
|
max_iterations = 2
|
|
commit_policy = "always"
|