8-gen bench (4 emotions × 2 corpora) at seed=42 against firdhokk
Whisper-LV3:
target RAVDESS CREMA-D
happy happy (0.999) ✓ happy (0.999) ✓
angry neutral (0.92) sad (0.99)
fearful happy (0.998) fearful (0.984) ✓
sad angry (0.99) fearful (0.99)
CREMA-D 2/4 vs RAVDESS 1/4. Larger / more naturalistic corpus
produces more class-pure fearful direction. Neither corpus solves
angry or sad — recipe shifts into 'vague expressivity' rather than
class-specific corners.
Practical: prefer CREMA-D when available; A/B both per emotion if
class precision matters.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Three real bugs found while running a YouTube → train → eval pipeline
end-to-end on real corpora:
1. emotion2vec was producing near-constant logits regardless of input.
Per config.yaml `normalize: true` — data2vec2/emotion2vec expects
per-utterance zero-mean unit-variance normalization on the raw
waveform before the local_encoder. Added inside the EmotionDetector
trait impl so all callers get it.
Verified empirically: 4 different audio inputs (Carlini talk,
audience question, McConaughey speech) now produce different argmax
classes. Before fix: all 4 produced identical logits.
2. The 9→5 emotion fold was collapsing every real-world clip to
[excited]. happy / surprised / other all mapped to Excited covered
~95% of natural speech. Replaced with a direct 9-class identity
mapping; EmotionLabel gained Disgusted, Fearful, Happy, Surprised,
Unk variants. Now: 132 [surprised] + 12 [excited] across the
Carlini corpus instead of 144 [excited].
3. lora_train_emotional --peak-lr / --epochs flags. The canned 3-stage
recipe over-fits on small (~100 clip) corpora at extended rank 8;
users need to tune. (The recipe stays as defaults; flags are pure
overrides.)
Plus diagnostic: examples/emotion2vec_probe — feed real audio files
into emotion2vec and dump per-class logits. Used to find bug #1.
Lib suite still 131/131 (the test that locked the 9→5 fold updated
to lock the new identity mapping).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Closes the third gap from the audio-ML Rust ecosystem survey:
speech emotion recognition. Honest scope — this is a hand-tuned
placeholder, not a real classifier. The trait makes a future
emotion2vec_plus_base candle port a one-line swap.
src/ser.rs (~330 LOC):
- EmotionDetector trait
- ProsodyDetector impl: autocorrelation F0 (65-400 Hz, voiced via
autocorr peak ratio) + RMS + voiced-ratio aggregation
- 5 buckets compatible with Phase 12.2 emotion-hint format:
[neutral] [calm] [sad] [angry] [excited]
- 6 unit tests (autocorr accuracy on a pure tone, silence handling,
sad/excited/neutral edge cases, tag-format invariant)
audio_to_manifest gains --auto-emotion-tag: classifies each diarized
clip and writes the resolved label into the manifest row's
emotion_tag. Static --emotion-tag stays as a fallback.
End-to-end verified: 2-speaker concat → both clips classified
[neutral] (correct — synthetic CSM samples are prosodically flat).
Manifest round-trips through lora_train_emotional unchanged.
Lib suite 110/110 (6 new SER tests). Pure-DSP, zero ML deps, zero
runtime risk.
The data-prep pipeline is now end-to-end auto-labeled in-crate:
audio_to_manifest --auto-emotion-tag raw.wav → manifest.jsonl
→ lora_train_emotional → lora_eval → converse_server with --lora
Zero Python, zero ort, zero whisper.cpp.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
redclawsystems