rtx-csm: emotional_speech_guide — firdhokk classifier reveals emotion gap

Wired up firdhokk/speech-emotion-recognition-with-openai-whisper-large-v3
as a working alternative to the broken emotion2vec_plus_base. Sanity
verified on real RAVDESS clips: 3/5 correct, 2/5 near-miss (happy↔
surprised, sad↔fearful). Probabilities are NOT saturated — the
classifier actually distinguishes per-input.

Then scored our 4 decoder-route outputs (Amini context, seed=42,
recipe defaults) and found that **only fearful registers as the
intended class**:

  target    verdict       conf
  happy     neutral       0.80   ✗ (steering produces neutral output)
  angry     happy         0.999  ✗ (high-arousal cross-class)
  fearful   fearful       0.68   ✓
  sad       fearful       0.998  ✗ (sad↔fearful confusion)

Honest framing: the recipe shifts speaker character toward an
expressive-sounding direction (cosine evidence) and preserves text
(decoder vs backbone) but does NOT produce class-distinct emotion.
The metric stack we used through Phase 9 (cosine + WER) couldn't
see this gap because it measures voice fidelity and text rendering,
not emotion class.

Hypothesized fixes (not yet tested):
- CREMA-D extraction (91 actors vs RAVDESS 24) for class-purer
  steering vectors
- Mixed backbone+decoder steering (backbone for prosody)
- EmoNet classifier (TTS-aware, may give different verdicts)

Doc'd in emotional_speech_guide.md as a known limitation. Closes
out an honest scientific picture: today's work successfully ports
the architectural finding (decoder route preserves text), but
class-precise emotion control remains unsolved.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

crates/models/rtx-csm/docs/emotional_speech_guide.md

@@ -203,6 +203,71 @@ techniques compose; the recipe enables both.
 
 ## Known limitations
 
+### Steering produces voice-shift but NOT class-distinct emotion (firdhokk verdict, 2026-04-30)
+
+`firdhokk/speech-emotion-recognition-with-openai-whisper-large-v3`
+(Whisper-LV3 fine-tuned on RAVDESS+SAVEE+TESS+URDU; 7 categorical
+classes; sanity-checked NON-saturated on real RAVDESS clips)
+classifies our 4 decoder-route outputs (Amini context, seed=42,
+recipe defaults) as:
+
+  target    classifier-verdict   conf
+  happy     neutral              0.80   ✗
+  angry     happy                0.999  ✗
+  fearful   **fearful**          0.68   ✓
+  sad       fearful              0.998  ✗ (sad↔fearful is a
+                                          common classifier confusion)
+
+Sanity check on the same classifier against RAVDESS ground-truth
+clips confirms it works:
+
+  RAVDESS truth     classifier predicts   conf
+  neutral           neutral               0.975  ✓
+  happy             surprised             0.999  ✗ (near-miss)
+  sad               fearful               0.999  ✗ (near-miss)
+  angry             angry                 0.998  ✓
+  fearful           fearful               1.000  ✓
+
+So the classifier itself is competent (3/5 correct on real labeled
+clips, 2/5 near-miss in known confusion pairs), but our generated
+outputs **don't elicit class-distinct emotional predictions**:
+
+- `happy` target → classified as neutral (steering didn't shift voice
+  toward happy; output sounds emotionally neutral with elevated
+  speaker similarity)
+- `angry` target → classified as happy (high-energy steering produced
+  happy-sounding output, not angry — possibly high-arousal-positive
+  vs high-arousal-negative gets crossed)
+- `sad` target → classified as fearful (related but distinct emotion;
+  could be classifier's sad↔fearful confusion, or genuinely fearful
+  output)
+- `fearful` target → fearful (only direct hit)
+
+Practical implication: the recipe is more accurately characterized
+as **"shifts speaker character toward an emotionally-expressive
+direction"** than **"produces class-specific target emotion"**. The
+WER and cosine metrics that we have looked good throughout Phase 9
+because those measure voice fidelity and text rendering, not
+emotion class.
+
+Hypothesized causes (not yet tested):
+1. RAVDESS-derived steering vectors may encode actor-specific
+   prosodic patterns more than emotion-specific patterns. CREMA-D
+   has 91 actors vs 24, so the per-emotion mean should be more
+   class-pure.
+2. The decoder steering is at the acoustic codebook level and may
+   shift timbre/voice quality without shifting prosody (which is
+   where emotion lives).
+3. Our scale=1.0 might be too low for class-distinct effect on
+   the decoder route — but raising scale produced repetition before.
+
+Best next experiments:
+- Re-extract from CREMA-D and re-bench against firdhokk
+- Try mixing decoder and backbone steering (small backbone scale
+  for prosody, decoder for voice character)
+- Test the EmoNet (laion/Empathic-Insight-Voice-Small) classifier
+  which was trained explicitly on synthetic TTS audio
+
 ### emotion2vec classifier is saturated
 
 `quality_eval --emotion2vec <pickle> --target-emotion <name>` would