Original picker used min(WER), tie-broken by max(cosine). Fragile:
on emotion=surprised it picked "You can." (2 words, WER 0.93) over
"...Today I want to share something" (13 words, WER 1.00) because
WER weights all errors uniformly — terse-and-mostly-wrong beats
long-and-mostly-right.
New scoring:
score = WER + (1.0 if words(transcript) < 5 else 0)
sort_by(score, -cosine)
Verified on existing benches:
surprised: now picks seed=100 ("...Today I want to share something
with...", 13 words, score 1.0) over seed=7 ("You can.",
2 words, score 1.929 with +1 length penalty).
calm: still picks seed=100 (full transcript revealed: "It's a
good reflection. Not that that. I want to share
something with you that I've been thinking about." —
near-verbatim! the earlier 55-char display had been
truncating it).
disgust: all 3 candidates score ~1.93 (no seed has > 5 words,
all get the length penalty); picker honestly admits
none is good rather than picking a fake winner.
Worth noting: the calm seed=100 case is ANOTHER near-verbatim
single-shot result we missed in the previous bench because the
display truncation hid the full transcript content.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Productionizes yesterday's seed-variance finding. Wraps
emotional_speech.sh, rolls a list of seeds, scores each via
quality_eval (speaker_cosine + Moonshine WER), and copies the
lowest-WER candidate to --out. Defaults to 5 seeds; pass
--seeds 42,7 for cheaper runs.
Tie-breaking is `min(WER), then -max(cosine)` — text fidelity
takes precedence over speaker character because user-typed text
should be rendered verbatim, while voice character is only
secondary on top of context conditioning. Failed generations
(short clips that get the -1 cosine sentinel) sort to the bottom.
Smoke run on the canonical "Today I want to share..." prompt:
seed 7 → cos 0.845, WER 0.714 "Today, today I want to share..." ← picked
seed 100 → cos -1, WER 1.000 "It is." (premature EOT)
seed 42 → cos 0.916, WER 2.000 "The police are, if you're..." (drift)
Cost: N × single-shot cost. The recipe being unreliable per-seed
is the whole reason this wrapper exists — pay the multiplier in
exchange for a reliably-best output.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
