rustytorch/crates/models/rtx-csm/docs/emotional_speech_guide.md

Emotional speech with rtx-csm

A user-facing guide to the Phase 9 emotion-control pipeline. The empirical log lives in perf_history.md (Phase 9 section) — read that for the research arc and full benchmark tables. This doc is the practical "how do I make CSM speak with emotion" handbook.

What this gets you

End-to-end: a YouTube URL (or any yt-dlp source) + target text + target emotion, producing a 24 kHz mono WAV that:

• Speaks the target text (near-verbatim on a good seed; WER 0.07–0.21 single-shot)
• In the voice character of the source URL (cosine ≥ 0.95 vs context)
• With the prosody/timbre of the chosen emotion (cosine elevation ≥ 0.30 vs neutral)

No fine-tuning. No retraining. Pure inference-time intervention.

One-liner quickstart

# 1. One-time corpus setup (~10 min, requires ~210 MB download)
curl -L -O https://zenodo.org/records/1188976/files/Audio_Speech_Actors_01-24.zip
unzip Audio_Speech_Actors_01-24.zip -d /tmp/ravdess
scripts/build_ravdess_manifest.sh /tmp/ravdess > /tmp/ravdess/manifest.jsonl

# 2. One-time steering extraction (~10 min, 4 emotions × ~2 min each)
mkdir -p /tmp/ravdess_dec_steering
for emo in happy angry fearful sad; do
    target/release/examples/steering_extract \
        --in /tmp/ravdess/manifest.jsonl \
        --target $emo --baseline neutral --max-per-pool 50 \
        --target-module decoder \
        --out /tmp/ravdess_dec_steering/$emo.safetensors
done

# 3. Use it
scripts/emotional_clone.sh \
    --url "https://www.youtube.com/watch?v=..." \
    --text "What you want said." \
    --emotion happy \
    --steering-dir /tmp/ravdess_dec_steering \
    --out /tmp/cloned.wav

The first run pays the URL fetch + diarize + transcribe cost (~2–5 min for a 20-min lecture). Subsequent runs with the same --workdir reuse those artifacts and only spend time on the 5-seed generation (~7–10 min).

The recipe (and why each piece is there)

target = decoder
steering scale = 1.0
steering layers = [2, 3]
CFG schedule = linear:3.0:1.0:25
seeds rolled = 5

• target = decoder — CSM-1B has a Llama backbone (semantic codebook 0) and a small Llama100M depth decoder (acoustic codebooks 1..N-1). Steering the backbone shifts voice but destroys word content ("some needs of my prey"). Steering the decoder shifts voice while preserving word content ("I'm not that tall"). The decoder carries acoustic detail; the backbone carries semantics.

• scale = 1.0 — decoder vector magnitudes are ~10× smaller than backbone vectors (norm 0.85 deepest vs 14.9), so the apparent scale is ~10× higher. Backbone scales are 0.2–0.3 for the same magnitude of effect.

• layers = [2, 3] — the decoder has 4 layers; perturbing all 4 produces lexical repetition ("I'm not that tall ×3"). Restricting to the last 2 (analogue of the backbone's mid [8,10,12] recipe) eliminates the repetition while preserving the cosine boost. Same pattern at both scales of the model: don't perturb all layers at once.

• CFG schedule = linear:3.0:1.0:25 — Selective CFG (arXiv 2509.19668). Full CFG early (frame 0, scale 3.0) for speaker fidelity, ramping to 1.0 by frame 25 for text adherence. Step schedules (step:3:1.5:12) are prompt-fragile — they catastrophically fail dense prompts (0.08 s output observed). Linear is the robust default across prompt categories.

• 5 seeds — single-shot has high variance. The same recipe at one seed produces "All right, today I want to share..." (WER 0.21) and at another produces "Let me think, let him out." (WER 1.0). The N-seed picker (emotional_speech_n.sh) generates N candidates, scores each via WavLM-SV cosine + Moonshine WER, and copies the winner. Cost: 5× single-shot.

  Composite score: WER + length_floor, where length_floor = 1.0 if the transcript has fewer than 5 words. This sinks ultra-short outputs ("You can.", 2 words, WER 0.93) below long-and-mostly- right outputs ("...Today I want to share something", 13 words, WER 1.0). Original min(WER) tie-break preferred terse-and-wrong on emotion=surprised; the length floor inverts that ranking while preserving correct picks on emotions where the winner is already long.

Per-emotion notes

emotion	works?	best seed (Amini ctx)	notes
happy	✓	42	Cleanest. WER 0.071–0.21 across multiple prompts.
angry	✓	100	Voice migrates well; text drift on some prompts (URL spam observed once).
fearful	✓	7	Stable on Amini context.
sad	✓	7	Works on decoder route only. Failed on the backbone route at every scale we tested 0.15–0.3. The architectural switch was the unlock.
calm	✓	100	3-seed picker produced "It's a good reflection. Not that that. I want to share somet..." (WER 0.64) — recipe lands prompt content with hesitation/reflection prosody. Single-shot at seed 42 gave hesitation markers ("I'm sorry. Um, I don't know.").
disgust	✗	(none reliable)	All 3 picker seeds (7, 42, 100) produced WER ≥ 0.93. Recipe doesn't work for this emotion on RAVDESS — likely corpus issue (RAVDESS disgust clips are low-energy / acoustically close to neutral). Try with CREMA-D or ESD.
surprised	⚠	100 (manual pick)	Picker chose seed=7 ("You can.", WER 0.93). Seed 100 produced "...Today I want to share something" but WER scored higher because of prefix tokens — picker's WER metric weights deletions/insertions uniformly. Manual seed selection (or weighting WER less heavily) recommended for this emotion.

The per-emotion best seed is context-specific. Seed 42 wins for happy on both Amini and Carlini contexts, but seed 7 wins for sad/fearful on Amini and fails on Carlini. The N-seed picker absorbs this — don't try to memoize a single best seed.

When it works

The recipe lands well when:

• The prompt is grammatically complete (statement, question, imperative)
• The reference clip is 10–14 s (use pick_context.sh — duration sweet spot)
• The picker correctly selects the headline speaker (not a sub-speaker the diarizer found)
• You're rolling at least 3 seeds

Best results recorded:

• WER 0.071 — Carlini ctx (manual selection of spk1) × happy × seed 42 → "So today I want to share something with you that I have been thinking about."
• WER 0.125 — Amini × happy × seed 42 × imperative prompt → "That the weather has been absolutely beautiful this mor"
• WER 0.143 — Carlini ctx (auto-picked spk0) × happy × seed 42 (full automated pipeline) → "But Jason, today I want to share something with you that I h"

When it doesn't work

• Multi-speaker manifests — picker selects highest-scoring clip, not the dominant speaker. McConaughey's manifest has spk0 (McConaughey) and spk1 (Oscars announcer); picker picked spk1 and the recipe failed. Workaround: pre-filter the manifest to a single speaker, or pass an explicit --context-wav instead of using the auto-pick.
• Very short prompts ("Hello world.") — the model has too little text to anchor on; both backbone and decoder routes drift.
• Prompts the model "dislikes" — baseline produces music/song tokens (♪♪♪) on some prompts; the recipe stabilizes those but doesn't help prompts that are too out-of-distribution.
• Quantized backbone (--quantized-gguf) — steering is FP-only. Not supported on quantized.

Troubleshooting

"music tokens" / 🎵 in transcript The steering scale is too high for that emotion. Scale 0.3 works for happy on backbone but pushes angry/fearful past the speech manifold. Drop to 0.2 (backbone) or 0.5–0.7 (decoder).

Output is one word or premature EOT Try a different seed. The N-seed picker would have caught this; if you're calling emotional_speech.sh (single-shot) instead of emotional_speech_n.sh, switch.

Output is repetitive ("I'm not that tall ×3") You're steering all layers at once. Restrict to a subset:

• Backbone: --steering-layers 8,10,12
• Decoder: --decoder-steering-layers 2,3

These are the wrapper script's defaults; if you're calling generate directly, set them explicitly.

Output is gibberish ("some needs of my prey") You're using backbone steering. Switch to decoder steering by passing --target decoder to the wrapper, or --decoder-steering-vec instead of --steering-vec to generate directly.

Manifest contains only [surprised] labels emotion2vec on monotone speech (lecture audio) collapses everything to [surprised]. Use a real emotion-labeled corpus (RAVDESS, CREMA-D, ESD) for steering extraction. The auto-tagged labels from audio_to_manifest --auto-emotion-tag --use-emotion2vec are noisy and don't produce useful steering vectors.

Architecture cheat sheet

backbone (Llama1B, 16 layers × 2048 embed_dim)
  ↓ models codebook 0 (semantic, "what to say")
  ↓ steering here destroys word content

depth decoder (Llama100M, 4 layers × 1024 embed_dim)
  ↓ models codebooks 1..N-1 (acoustic, "how to say it")
  ↓ steering here preserves word content
  ↓ ← Phase 9 default route

Mimi (32 codebooks @ 12.5 Hz, 24 kHz audio)
  ↓ neural audio codec; not steered

CFG (Classifier-Free Guidance) operates on codebook 0 logits between the two backbones (conditional vs unconditional). Steering operates on hidden activations within either backbone or the decoder. The two techniques compose; the recipe enables both.

Known limitations

Steering produces voice-shift but NOT class-distinct emotion (firdhokk verdict, 2026-04-30)

firdhokk/speech-emotion-recognition-with-openai-whisper-large-v3 (Whisper-LV3 fine-tuned on RAVDESS+SAVEE+TESS+URDU; 7 categorical classes; sanity-checked NON-saturated on real RAVDESS clips) classifies our 4 decoder-route outputs (Amini context, seed=42, recipe defaults) as:

target classifier-verdict conf happy neutral 0.80 ✗ angry happy 0.999 ✗ fearful fearful 0.68 ✓ sad fearful 0.998 ✗ (sad↔fearful is a common classifier confusion)

Sanity check on the same classifier against RAVDESS ground-truth clips confirms it works:

RAVDESS truth classifier predicts conf neutral neutral 0.975 ✓ happy surprised 0.999 ✗ (near-miss) sad fearful 0.999 ✗ (near-miss) angry angry 0.998 ✓ fearful fearful 1.000 ✓

So the classifier itself is competent (3/5 correct on real labeled clips, 2/5 near-miss in known confusion pairs), but our generated outputs don't elicit class-distinct emotional predictions:

• happy target → classified as neutral (steering didn't shift voice toward happy; output sounds emotionally neutral with elevated speaker similarity)
• angry target → classified as happy (high-energy steering produced happy-sounding output, not angry — possibly high-arousal-positive vs high-arousal-negative gets crossed)
• sad target → classified as fearful (related but distinct emotion; could be classifier's sad↔fearful confusion, or genuinely fearful output)
• fearful target → fearful (only direct hit)

Practical implication: the recipe is more accurately characterized as "shifts speaker character toward an emotionally-expressive direction" than "produces class-specific target emotion". The WER and cosine metrics that we have looked good throughout Phase 9 because those measure voice fidelity and text rendering, not emotion class.

Hypothesized causes (not yet tested):

1. RAVDESS-derived steering vectors may encode actor-specific prosodic patterns more than emotion-specific patterns. CREMA-D has 91 actors vs 24, so the per-emotion mean should be more class-pure.
2. The decoder steering is at the acoustic codebook level and may shift timbre/voice quality without shifting prosody (which is where emotion lives).
3. Our scale=1.0 might be too low for class-distinct effect on the decoder route — but raising scale produced repetition before.

Best next experiments:

• Re-extract from CREMA-D and re-bench against firdhokk → DONE: CREMA-D wins on fearful (0.984 fearful) where RAVDESS got happy (0.998); both nail happy (0.999); both fail angry and sad. Top-1 match: RAVDESS 1/4, CREMA-D 2/4. Modest but real corpus advantage. Class-precise control remains partially open.
• Try mixing decoder and backbone steering (small backbone scale for prosody, decoder for voice character)
• Test the EmoNet (laion/Empathic-Insight-Voice-Small) classifier which was trained explicitly on synthetic TTS audio

CREMA-D vs RAVDESS bench (firdhokk verdict, 2026-04-30)

Same prompt, same seed (42), same recipe (decoder route, scale 1.0, layers [2,3], CFG linear:3.0:1.0:25, Amini context). Steering vectors derived from each corpus separately:

target RAVDESS verdict CREMA-D verdict happy happy (0.999) ✓ happy (0.999) ✓ angry neutral (0.92) sad (0.99) fearful happy (0.998) fearful (0.984) ✓ sad angry (0.99) fearful (0.99)

CREMA-D's fearful actually registers as fearful (0.984 confidence) — biggest single-corpus improvement we've measured. CREMA-D has 91 actors vs RAVDESS's 24, and the fearful-vs-neutral mean direction is apparently more class-pure with the larger pool.

Neither corpus produces angry or sad outputs that classify as angry/sad. The most likely confusion pairs (angry↔sad? both high-arousal-negative?) suggest the steering shifts the audio into a vague "expressive" subspace rather than a class-specific one.

Practical recommendation: use CREMA-D if you have it (scripts/build_crema_d_manifest.sh); fall back to RAVDESS for emotion classes CREMA-D lacks (calm, surprised). For best class precision on a specific emotion, A/B both corpora and pick the winner per class.

emotion2vec classifier is saturated

quality_eval --emotion2vec <pickle> --target-emotion <name> would ideally score whether the output actually sounds like the target emotion. Empirically (2026-04-30): the emotion2vec/emotion2vec_plus_base checkpoint as loaded through our pickle path classifies every input — including ground-truth RAVDESS clips with explicit emotion labels — as Surprised with probability ≥ 0.99.

Sanity check on real RAVDESS Actor_01 clips:

RAVDESS clip → emotion2vec prediction neutral → Surprised (0.9999996) happy → Surprised (1.000) sad → Surprised (0.9933) angry → Surprised (0.9999999) fearful → Surprised (1.000) disgust → Surprised (1.000) surprised → Surprised (0.9999988)

This is a classifier limitation, not a generation problem. The metric is wired correctly; the classifier itself collapses to a dominant class on most input. You cannot use this to verify emotion expression in generated audio. It's still useful for sanity-checking the emotion2vec pipeline (load, forward, normalize) and as a diagnostic of "does this audio at least classify as something."

The target_emotion_prob field in the metrics output is therefore near-zero for almost every (target, output) pair that isn't "surprised" — don't use it as a picker score component.

References

• Sesame's CSM blog — the architectural split this guide exploits
• arXiv 2508.03543 — EmoSteer-TTS (flow-matching specific; we adapted only the underlying ActAdd pattern)
• arXiv 2509.19668 — Selective CFG (the schedule we use)
• arXiv 2603.24430 — I2D evaluation (not used in the recipe; lives in i2d_loop.sh)
• perf_history.md Phase 9 — full empirical history with all benchmark tables