8-gen bench (4 emotions × 2 corpora) at seed=42 against firdhokk
Whisper-LV3:
target RAVDESS CREMA-D
happy happy (0.999) ✓ happy (0.999) ✓
angry neutral (0.92) sad (0.99)
fearful happy (0.998) fearful (0.984) ✓
sad angry (0.99) fearful (0.99)
CREMA-D 2/4 vs RAVDESS 1/4. Larger / more naturalistic corpus
produces more class-pure fearful direction. Neither corpus solves
angry or sad — recipe shifts into 'vague expressivity' rather than
class-specific corners.
Practical: prefer CREMA-D when available; A/B both per emotion if
class precision matters.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Port complete. The Phase 13.3 prosody-rule placeholder is now
retire-able by setting one CLI flag — the real candle-ported
emotion2vec_plus_base classifier slots in behind the same
EmotionDetector trait the placeholder used.
impl EmotionDetector for Emotion2Vec — builds (1, 1, T) tensor on
stored device, runs forward, argmaxes the 9 logits, maps to the
5-bucket label via Classifier::tag_for_class. Empty input
short-circuits to Neutral.
Emotion2Vec struct gained a `device` field so the trait impl can
build tensors without an out-of-band handle. new() / load_from_pickle()
threaded through; existing tests + smoke binary updated.
audio_to_manifest --use-emotion2vec — pairs with --auto-emotion-tag
to swap ProsodyDetector for Emotion2Vec, boxed as
Box<dyn EmotionDetector> so the call site is unchanged.
converse_server --use-emotion2vec — same pattern; built once at boot
and stored in Shared as Box<dyn EmotionDetector + Send + Sync>.
~150 ms/turn forward cost vs <1 ms for prosody, but actually runs
SOTA SER. Removed redundant reactive_emotion: bool field — the
Option<Box<dyn>> already encodes the same state.
Verified end-to-end on Metal:
- audio_to_manifest --use-emotion2vec on 2-speaker concat → both
tagged [excited] (prosody had said [neutral] on same input)
- converse_server --quantized-gguf … --lora … --reactive-emotion
--use-emotion2vec boots, 1 bench turn 0 errors, /metrics shows
reactive_emotion_total{label="excited"} 1 — same tag
audio_to_manifest produced. Cross-consumer consistency.
Phase 13.8 complete (slices 1+2+3+4 shipped). The emotional-voice
stack now has a real, trained, candle-ported SER classifier with
no Python sidecar, no ort, no whisper.cpp.
Lib suite 120/120.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Closes the third gap from the audio-ML Rust ecosystem survey:
speech emotion recognition. Honest scope — this is a hand-tuned
placeholder, not a real classifier. The trait makes a future
emotion2vec_plus_base candle port a one-line swap.
src/ser.rs (~330 LOC):
- EmotionDetector trait
- ProsodyDetector impl: autocorrelation F0 (65-400 Hz, voiced via
autocorr peak ratio) + RMS + voiced-ratio aggregation
- 5 buckets compatible with Phase 12.2 emotion-hint format:
[neutral] [calm] [sad] [angry] [excited]
- 6 unit tests (autocorr accuracy on a pure tone, silence handling,
sad/excited/neutral edge cases, tag-format invariant)
audio_to_manifest gains --auto-emotion-tag: classifies each diarized
clip and writes the resolved label into the manifest row's
emotion_tag. Static --emotion-tag stays as a fallback.
End-to-end verified: 2-speaker concat → both clips classified
[neutral] (correct — synthetic CSM samples are prosodically flat).
Manifest round-trips through lora_train_emotional unchanged.
Lib suite 110/110 (6 new SER tests). Pure-DSP, zero ML deps, zero
runtime risk.
The data-prep pipeline is now end-to-end auto-labeled in-crate:
audio_to_manifest --auto-emotion-tag raw.wav → manifest.jsonl
→ lora_train_emotional → lora_eval → converse_server with --lora
Zero Python, zero ort, zero whisper.cpp.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
