rustyverse/rustytorch
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity

rtx-csm: perf_history — recipe seed variance caveat

Browse Source 
3-seed × 3-prompt reproducibility bench on [email protected] reveals that the
recipe shifts speaker character reliably but produces high text-
fidelity variance:

  seed 42:  WER 2.00 "The police are, if you're, I can't recite..."
  seed 7:   WER 0.71 "Today, today I want to share..."  ← near-verbatim
  seed 100: 0.32 s premature EOT

Cross-prompt at seed 42 drifts uniformly across 3 prompts. Speaker
cosine is consistently elevated; text content is roll-the-dice.

Documenting this as the honest characterization rather than overclaim
the single-seed Sprint 2 results. Practical recipe: roll N seeds,
pick lowest-WER output.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-04-29 12:16:20 -07:00
co-authored by Claude Opus 4.7
parent 2008c9c4a6
commit 406c437d70
1 changed files with 15 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/models/rtx-csm/docs/perf_history.md
+15
View File
@@ -145,6 +145,21 @@ shipped on the FP backbone; quantized path unchanged.
outputs at every tested 0.15–0.3). The wrapper script outputs at every tested 0.15–0.3). The wrapper script
`emotional_speech.sh` picks these per-emotion defaults `emotional_speech.sh` picks these per-emotion defaults
automatically. automatically.
- *The recipe has **high seed variance** in text fidelity*. Same
prompt "Today I want to share something with you that I have been
thinking about." with `happy` @ scale 0.3:
- seed 42: WER 2.00, "The police are, if you're, I can't recite
this film." *(content drift)*
- seed 7: WER **0.71**, **"Today, today I want to share..."**
*(near-verbatim — one of three seeds nailed it)*
- seed 100: 0.32 s output, premature EOT
Cross-prompt at seed 42 also drifts uniformly. Speaker cosine is
consistently elevated (the steering DOES shift voice character),
but **text content fidelity is hit-or-miss**. Practical workflow:
roll N seeds, pick the lowest-WER output. The recipe is closer to
"biased random sampling toward emotional voice" than to
"deterministic emotional rendering." Reproducible voice cloning
(`--context-wav` alone) is much more deterministic than this.
- *RAVDESS corpus shape biases output length*: clips are ~3 s - *RAVDESS corpus shape biases output length*: clips are ~3 s
("Kids are talking by the door") so the steering biases the ("Kids are talking by the door") so the steering biases the
model toward terse outputs (one or two words). Longer-utterance model toward terse outputs (one or two words). Longer-utterance