# Emotional speech with rtx-csm A user-facing guide to the Phase 9 emotion-control pipeline. The empirical log lives in `perf_history.md` (Phase 9 section) — read that for the research arc and full benchmark tables. This doc is the practical "how do I make CSM speak with emotion" handbook. ## What this gets you End-to-end: a YouTube URL (or any yt-dlp source) + target text + target emotion, producing a 24 kHz mono WAV that: - Speaks the target text (near-verbatim on a good seed; WER 0.07–0.21 single-shot) - In the voice character of the source URL (cosine ≥ 0.95 vs context) - With the prosody/timbre of the chosen emotion (cosine elevation ≥ 0.30 vs neutral) No fine-tuning. No retraining. Pure inference-time intervention. ## One-liner quickstart ```bash # 1. One-time corpus setup (~10 min, requires ~210 MB download) curl -L -O https://zenodo.org/records/1188976/files/Audio_Speech_Actors_01-24.zip unzip Audio_Speech_Actors_01-24.zip -d /tmp/ravdess scripts/build_ravdess_manifest.sh /tmp/ravdess > /tmp/ravdess/manifest.jsonl # 2. One-time steering extraction (~10 min, 4 emotions × ~2 min each) mkdir -p /tmp/ravdess_dec_steering for emo in happy angry fearful sad; do target/release/examples/steering_extract \ --in /tmp/ravdess/manifest.jsonl \ --target $emo --baseline neutral --max-per-pool 50 \ --target-module decoder \ --out /tmp/ravdess_dec_steering/$emo.safetensors done # 3. Use it scripts/emotional_clone.sh \ --url "https://www.youtube.com/watch?v=..." \ --text "What you want said." \ --emotion happy \ --steering-dir /tmp/ravdess_dec_steering \ --out /tmp/cloned.wav ``` The first run pays the URL fetch + diarize + transcribe cost (~2–5 min for a 20-min lecture). Subsequent runs with the same `--workdir` reuse those artifacts and only spend time on the 5-seed generation (~7–10 min). ## The recipe (and why each piece is there) ``` target = decoder steering scale = 1.0 steering layers = [2, 3] CFG schedule = linear:3.0:1.0:25 seeds rolled = 5 ``` - **`target = decoder`** — CSM-1B has a Llama backbone (semantic codebook 0) and a small Llama100M depth decoder (acoustic codebooks 1..N-1). Steering the **backbone** shifts voice but destroys word content ("some needs of my prey"). Steering the **decoder** shifts voice while preserving word content ("I'm not that tall"). The decoder carries acoustic detail; the backbone carries semantics. - **`scale = 1.0`** — decoder vector magnitudes are ~10× smaller than backbone vectors (norm 0.85 deepest vs 14.9), so the apparent scale is ~10× higher. Backbone scales are 0.2–0.3 for the same magnitude of effect. - **`layers = [2, 3]`** — the decoder has 4 layers; perturbing all 4 produces lexical repetition ("I'm not that tall ×3"). Restricting to the last 2 (analogue of the backbone's mid `[8,10,12]` recipe) eliminates the repetition while preserving the cosine boost. Same pattern at both scales of the model: don't perturb all layers at once. - **`CFG schedule = linear:3.0:1.0:25`** — Selective CFG (arXiv 2509.19668). Full CFG early (frame 0, scale 3.0) for speaker fidelity, ramping to 1.0 by frame 25 for text adherence. Step schedules (`step:3:1.5:12`) are **prompt-fragile** — they catastrophically fail dense prompts (0.08 s output observed). Linear is the robust default across prompt categories. - **`5 seeds`** — single-shot has high variance. The same recipe at one seed produces "All right, today I want to share..." (WER 0.21) and at another produces "Let me think, let him out." (WER 1.0). The N-seed picker (`emotional_speech_n.sh`) generates N candidates, scores each via WavLM-SV cosine + Moonshine WER, and copies the winner. Cost: 5× single-shot. Composite score: `WER + length_floor`, where `length_floor = 1.0` if the transcript has fewer than 5 words. This sinks ultra-short outputs ("You can.", 2 words, WER 0.93) below long-and-mostly- right outputs ("...Today I want to share something", 13 words, WER 1.0). Original min(WER) tie-break preferred terse-and-wrong on emotion=surprised; the length floor inverts that ranking while preserving correct picks on emotions where the winner is already long. ## Per-emotion notes | emotion | works? | best seed (Amini ctx) | notes | |---------|--------|----------------------|-------| | happy | ✓ | 42 | Cleanest. WER 0.071–0.21 across multiple prompts. | | angry | ✓ | 100 | Voice migrates well; text drift on some prompts (URL spam observed once). | | fearful | ✓ | 7 | Stable on Amini context. | | sad | ✓ | 7 | Works on decoder route only. **Failed on the backbone route** at every scale we tested 0.15–0.3. The architectural switch was the unlock. | | calm | ✓ | 100 | 3-seed picker produced "It's a good reflection. Not that that. I want to share somet..." (WER 0.64) — recipe lands prompt content with hesitation/reflection prosody. Single-shot at seed 42 gave hesitation markers ("I'm sorry. Um, I don't know."). | | disgust | ✗ | (none reliable) | All 3 picker seeds (7, 42, 100) produced WER ≥ 0.93. Recipe doesn't work for this emotion on RAVDESS — likely corpus issue (RAVDESS disgust clips are low-energy / acoustically close to neutral). Try with CREMA-D or ESD. | | surprised | ⚠ | 100 (manual pick) | Picker chose seed=7 ("You can.", WER 0.93). Seed 100 produced "...Today I want to share something" but WER scored higher because of prefix tokens — picker's WER metric weights deletions/insertions uniformly. Manual seed selection (or weighting WER less heavily) recommended for this emotion. | The per-emotion best seed is **context-specific**. Seed 42 wins for happy on both Amini and Carlini contexts, but seed 7 wins for sad/fearful on Amini and fails on Carlini. The N-seed picker absorbs this — don't try to memoize a single best seed. ## When it works The recipe lands well when: - The prompt is grammatically complete (statement, question, imperative) - The reference clip is 10–14 s (use `pick_context.sh` — duration sweet spot) - The picker correctly selects the headline speaker (not a sub-speaker the diarizer found) - You're rolling at least 3 seeds Best results recorded: - **WER 0.071** — Carlini ctx (manual selection of spk1) × happy × seed 42 → "So today I want to share something with you that I have been thinking about." - **WER 0.125** — Amini × happy × seed 42 × imperative prompt → "That the weather has been absolutely beautiful this mor" - **WER 0.143** — Carlini ctx (auto-picked spk0) × happy × seed 42 (full automated pipeline) → "But Jason, today I want to share something with you that I h" ## When it doesn't work - **Multi-speaker manifests** — picker selects highest-scoring clip, not the dominant speaker. McConaughey's manifest has spk0 (McConaughey) and spk1 (Oscars announcer); picker picked spk1 and the recipe failed. Workaround: pre-filter the manifest to a single speaker, or pass an explicit `--context-wav` instead of using the auto-pick. - **Very short prompts** ("Hello world.") — the model has too little text to anchor on; both backbone and decoder routes drift. - **Prompts the model "dislikes"** — baseline produces music/song tokens (♪♪♪) on some prompts; the recipe stabilizes those but doesn't help prompts that are too out-of-distribution. - **Quantized backbone (`--quantized-gguf`)** — steering is FP-only. Not supported on quantized. ## Troubleshooting **"music tokens" / 🎵 in transcript** The steering scale is too high for that emotion. Scale 0.3 works for happy on backbone but pushes angry/fearful past the speech manifold. Drop to 0.2 (backbone) or 0.5–0.7 (decoder). **Output is one word or premature EOT** Try a different seed. The N-seed picker would have caught this; if you're calling `emotional_speech.sh` (single-shot) instead of `emotional_speech_n.sh`, switch. **Output is repetitive ("I'm not that tall ×3")** You're steering all layers at once. Restrict to a subset: - Backbone: `--steering-layers 8,10,12` - Decoder: `--decoder-steering-layers 2,3` These are the wrapper script's defaults; if you're calling `generate` directly, set them explicitly. **Output is gibberish ("some needs of my prey")** You're using backbone steering. Switch to decoder steering by passing `--target decoder` to the wrapper, or `--decoder-steering-vec` instead of `--steering-vec` to `generate` directly. **Manifest contains only `[surprised]` labels** emotion2vec on monotone speech (lecture audio) collapses everything to `[surprised]`. Use a real emotion-labeled corpus (RAVDESS, CREMA-D, ESD) for steering extraction. The auto-tagged labels from `audio_to_manifest --auto-emotion-tag --use-emotion2vec` are noisy and don't produce useful steering vectors. ## Architecture cheat sheet ``` backbone (Llama1B, 16 layers × 2048 embed_dim) ↓ models codebook 0 (semantic, "what to say") ↓ steering here destroys word content depth decoder (Llama100M, 4 layers × 1024 embed_dim) ↓ models codebooks 1..N-1 (acoustic, "how to say it") ↓ steering here preserves word content ↓ ← Phase 9 default route Mimi (32 codebooks @ 12.5 Hz, 24 kHz audio) ↓ neural audio codec; not steered ``` CFG (Classifier-Free Guidance) operates on codebook 0 logits between the two backbones (conditional vs unconditional). Steering operates on hidden activations within either backbone or the decoder. The two techniques compose; the recipe enables both. ## Known limitations ### Steering produces voice-shift but NOT class-distinct emotion (firdhokk verdict, 2026-04-30) `firdhokk/speech-emotion-recognition-with-openai-whisper-large-v3` (Whisper-LV3 fine-tuned on RAVDESS+SAVEE+TESS+URDU; 7 categorical classes; sanity-checked NON-saturated on real RAVDESS clips) classifies our 4 decoder-route outputs (Amini context, seed=42, recipe defaults) as: target classifier-verdict conf happy neutral 0.80 ✗ angry happy 0.999 ✗ fearful **fearful** 0.68 ✓ sad fearful 0.998 ✗ (sad↔fearful is a common classifier confusion) Sanity check on the same classifier against RAVDESS ground-truth clips confirms it works: RAVDESS truth classifier predicts conf neutral neutral 0.975 ✓ happy surprised 0.999 ✗ (near-miss) sad fearful 0.999 ✗ (near-miss) angry angry 0.998 ✓ fearful fearful 1.000 ✓ So the classifier itself is competent (3/5 correct on real labeled clips, 2/5 near-miss in known confusion pairs), but our generated outputs **don't elicit class-distinct emotional predictions**: - `happy` target → classified as neutral (steering didn't shift voice toward happy; output sounds emotionally neutral with elevated speaker similarity) - `angry` target → classified as happy (high-energy steering produced happy-sounding output, not angry — possibly high-arousal-positive vs high-arousal-negative gets crossed) - `sad` target → classified as fearful (related but distinct emotion; could be classifier's sad↔fearful confusion, or genuinely fearful output) - `fearful` target → fearful (only direct hit) Practical implication: the recipe is more accurately characterized as **"shifts speaker character toward an emotionally-expressive direction"** than **"produces class-specific target emotion"**. The WER and cosine metrics that we have looked good throughout Phase 9 because those measure voice fidelity and text rendering, not emotion class. Hypothesized causes (not yet tested): 1. RAVDESS-derived steering vectors may encode actor-specific prosodic patterns more than emotion-specific patterns. CREMA-D has 91 actors vs 24, so the per-emotion mean should be more class-pure. 2. The decoder steering is at the acoustic codebook level and may shift timbre/voice quality without shifting prosody (which is where emotion lives). 3. Our scale=1.0 might be too low for class-distinct effect on the decoder route — but raising scale produced repetition before. Best next experiments: - ~~Re-extract from CREMA-D and re-bench against firdhokk~~ → **DONE**: CREMA-D wins on fearful (0.984 fearful) where RAVDESS got happy (0.998); both nail happy (0.999); both fail angry and sad. Top-1 match: RAVDESS 1/4, CREMA-D 2/4. Modest but real corpus advantage. Class-precise control remains partially open. - Try mixing decoder and backbone steering (small backbone scale for prosody, decoder for voice character) - Test the EmoNet (laion/Empathic-Insight-Voice-Small) classifier which was trained explicitly on synthetic TTS audio #### CREMA-D vs RAVDESS bench (firdhokk verdict, 2026-04-30) Same prompt, same seed (42), same recipe (decoder route, scale 1.0, layers [2,3], CFG linear:3.0:1.0:25, Amini context). Steering vectors derived from each corpus separately: target RAVDESS verdict CREMA-D verdict happy **happy (0.999)** ✓ **happy (0.999)** ✓ angry neutral (0.92) sad (0.99) fearful happy (0.998) **fearful (0.984)** ✓ sad angry (0.99) fearful (0.99) CREMA-D's fearful actually registers as fearful (0.984 confidence) — biggest single-corpus improvement we've measured. CREMA-D has 91 actors vs RAVDESS's 24, and the fearful-vs-neutral mean direction is apparently more class-pure with the larger pool. Neither corpus produces angry or sad outputs that classify as angry/sad. The most likely confusion pairs (angry↔sad? both high-arousal-negative?) suggest the steering shifts the audio into a vague "expressive" subspace rather than a class-specific one. Practical recommendation: use **CREMA-D** if you have it (`scripts/build_crema_d_manifest.sh`); fall back to RAVDESS for emotion classes CREMA-D lacks (calm, surprised). For best class precision on a specific emotion, A/B both corpora and pick the winner per class. ### emotion2vec classifier is saturated `quality_eval --emotion2vec --target-emotion ` would ideally score whether the output actually sounds like the target emotion. Empirically (2026-04-30): the `emotion2vec/emotion2vec_plus_base` checkpoint as loaded through our pickle path classifies **every** input — including ground-truth RAVDESS clips with explicit emotion labels — as `Surprised` with probability ≥ 0.99. Sanity check on real RAVDESS Actor_01 clips: RAVDESS clip → emotion2vec prediction neutral → Surprised (0.9999996) happy → Surprised (1.000) sad → Surprised (0.9933) angry → Surprised (0.9999999) fearful → Surprised (1.000) disgust → Surprised (1.000) surprised → Surprised (0.9999988) This is a classifier limitation, not a generation problem. The metric is wired correctly; the classifier itself collapses to a dominant class on most input. **You cannot use this to verify emotion expression in generated audio.** It's still useful for sanity-checking the emotion2vec pipeline (load, forward, normalize) and as a diagnostic of "does this audio at least classify as something." The `target_emotion_prob` field in the metrics output is therefore near-zero for almost every (target, output) pair that isn't "surprised" — don't use it as a picker score component. ## References - [Sesame's CSM blog](https://www.sesame.com/research/crossing_the_uncanny_valley_of_voice) — the architectural split this guide exploits - arXiv 2508.03543 — EmoSteer-TTS (flow-matching specific; we adapted only the underlying ActAdd pattern) - arXiv 2509.19668 — Selective CFG (the schedule we use) - arXiv 2603.24430 — I2D evaluation (not used in the recipe; lives in `i2d_loop.sh`) - `perf_history.md` Phase 9 — full empirical history with all benchmark tables