8-gen bench (4 emotions × 2 corpora) at seed=42 against firdhokk
Whisper-LV3:
target RAVDESS CREMA-D
happy happy (0.999) ✓ happy (0.999) ✓
angry neutral (0.92) sad (0.99)
fearful happy (0.998) fearful (0.984) ✓
sad angry (0.99) fearful (0.99)
CREMA-D 2/4 vs RAVDESS 1/4. Larger / more naturalistic corpus
produces more class-pure fearful direction. Neither corpus solves
angry or sad — recipe shifts into 'vague expressivity' rather than
class-specific corners.
Practical: prefer CREMA-D when available; A/B both per emotion if
class precision matters.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Wired up firdhokk/speech-emotion-recognition-with-openai-whisper-large-v3
as a working alternative to the broken emotion2vec_plus_base. Sanity
verified on real RAVDESS clips: 3/5 correct, 2/5 near-miss (happy↔
surprised, sad↔fearful). Probabilities are NOT saturated — the
classifier actually distinguishes per-input.
Then scored our 4 decoder-route outputs (Amini context, seed=42,
recipe defaults) and found that **only fearful registers as the
intended class**:
target verdict conf
happy neutral 0.80 ✗ (steering produces neutral output)
angry happy 0.999 ✗ (high-arousal cross-class)
fearful fearful 0.68 ✓
sad fearful 0.998 ✗ (sad↔fearful confusion)
Honest framing: the recipe shifts speaker character toward an
expressive-sounding direction (cosine evidence) and preserves text
(decoder vs backbone) but does NOT produce class-distinct emotion.
The metric stack we used through Phase 9 (cosine + WER) couldn't
see this gap because it measures voice fidelity and text rendering,
not emotion class.
Hypothesized fixes (not yet tested):
- CREMA-D extraction (91 actors vs RAVDESS 24) for class-purer
steering vectors
- Mixed backbone+decoder steering (backbone for prosody)
- EmoNet classifier (TTS-aware, may give different verdicts)
Doc'd in emotional_speech_guide.md as a known limitation. Closes
out an honest scientific picture: today's work successfully ports
the architectural finding (decoder route preserves text), but
class-precise emotion control remains unsolved.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Wires emotion2vec into quality_eval so per-row metrics include
target_emotion_prob, top_emotion, top_emotion_prob. Pure inference,
optional via --emotion2vec / --target-emotion flags.
Critical empirical finding documented in code + user guide: the
emotion2vec_plus_base checkpoint classifies every input as
"Surprised" with prob ≥ 0.99, INCLUDING ground-truth RAVDESS clips
with explicit emotion labels. Real angry-RAVDESS → "Surprised"
(0.9999999). Real neutral-RAVDESS → "Surprised" (0.9999996).
The metric implementation is correct (matches the trait's
EmotionDetector::classify code path with same per-utterance zero-
mean unit-variance normalization); the underlying classifier
collapses to a dominant class on most input — likely the same
"9→5 fold collapse" the project already documented in the data-
labeling path.
Practical implication: target_emotion_prob is near-zero for almost
every (target, output) that isn't "surprised", so it can't be used
as a picker score. The emotion2vec metric still works as a
diagnostic ("did the model produce something that classifies as
audio at all?") but not as a generation-quality validator.
Doc'd in:
- examples/quality_eval.rs CLI doc (caveat block on --emotion2vec)
- docs/emotional_speech_guide.md (Known limitations section with
full sanity-check table)
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Brief addition to the recipe section explaining the WER + length
floor scoring used by emotional_speech_n.sh (committed in b7b267b).
Validates that the new scoring preserves canonical winners on
happy and calm while flipping surprised to the long-and-correct
candidate.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
3-seed picker run (7, 42, 100) on calm/disgust/surprised refines the
single-shot characterization:
calm: winner seed=100, WER 0.64
"It's a good reflection. Not that that. I want to
share somet..."
(recipe lands the prompt — single-shot at seed 42 only
produced hesitation markers; the picker found a seed
with actual content)
disgust: no reliable seed
(all 3 seeds WER ≥ 0.93; likely RAVDESS corpus issue —
disgust clips are low-energy / acoustically close to
neutral. Try CREMA-D or ESD for this emotion.)
surprised: picker chose seed=7 (WER 0.93, short "You can.") over
seed=100 (WER 1.0, "...Today I want to share something")
— WER weighting issue: deletions and insertions count
uniformly, so terse-but-mostly-wrong beat long-and-
mostly-right. Manual selection or weighting WER less
heavily would help here.
Updated per-emotion table marks disgust as ✗ (corpus limitation),
surprised as ⚠ (picker scoring artifact), calm as ✓ (works with
N-seed picker).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Extracted decoder steering vectors for the remaining 3 RAVDESS
emotions (calm, disgust, surprised). Single-seed bench at
seed=42 on Amini context, decoder route, recipe defaults:
emotion cos_ctx WER transcript
calm 0.70 0.86 "I'm sorry. Um, I don't know."
(natural hesitation markers — the
recipe produces semantically-emotion-
matched content, not just acoustic
shift)
disgust 0.81 0.93 "For that, that..." (truncated)
surprised 0.95 2.57 "Too couple, sorry, and that's saying,
even a premier and super driver..."
(long rambling; voice migrates well,
text drifts)
All 7 RAVDESS emotions now produce coherent English on the decoder
route — calm is solid first-shot, disgust truncates, surprised
rambles. Roll N seeds via emotional_speech_n.sh for the latter two.
emotional_speech_guide.md updated with the per-emotion table now
covering all 7. Voice character preservation (cos vs context > 0.7)
holds for every emotion in the pack.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Phase 9 produced 25 commits and a complex pipeline; perf_history is
the engineering log but new users coming to this cold need a clean
"how do I make CSM speak with emotion" handbook.
Sections:
- What this gets you (single-shot WER 0.07–0.21, voice cosine ≥ 0.95)
- One-liner quickstart (RAVDESS download → extract → use)
- The recipe explained — every flag and why it's there
- Per-emotion notes (works/best-seed/caveats per emotion)
- When it works / when it doesn't
- Troubleshooting (music tokens, premature EOT, repetition, etc.)
- Architecture cheat sheet (backbone=semantic, decoder=acoustic)
References perf_history.md for the full empirical log; this doc is
the user-facing distillation.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
