rustytorch/crates/models/rtx-csm/docs/personal_voice_training_guide.md

Personal voice training guide — beyond Maya, just for you

For the case where you want the most emotionally responsive voice ever, for personal use only (not distributed, not commercial), using all media formats you can legitimately access.

This is the iterative, hobby-project sibling of docs/maya_finetune_analysis.md. The legal calculus is different for personal use: training is in a gray zone but training-and-then- distributing-outputs is the actual liability vector. Keep outputs to yourself and the practical risk drops sharply.

TL;DR

• Realistic timeline: 60-120 hr of focused work for "clearly better than base CSM in your domain." 300+ hr for "genuinely beats Maya for me."
• Stack: yt-dlp → Demucs v4 → Silero V5 VAD → WhisperX (diarization+ASR+alignment) → DNSMOS gate → emotion2vec+ labels → resample to 24 kHz → Mimi tokenize.
• Best data sources by ROI: audiobooks > podcasts > anime/game VA

  YouTube interviews > movies. Skip Reddit clips, skip TTS-synthesized data (mode collapse).

• Biggest trap: spending 80% of time on data pipeline, 15% on training infra, 5% on actually listening to outputs critically. Listening is where the model gets good.

1. Data sources ranked by emotional richness × accessibility

Source	Hours	How to get	Audio	Transcripts	Emotion	Gotchas
Audiobooks (own Audible / LibriVox)	200-500 hr easy	OpenAudible + ffmpeg; LibriVox direct MP3	22-44 kHz clean	Aeneas / Montreal Forced Aligner against Project Gutenberg text	Narrator-acted: sadness, suspense, joy, anger	Single-speaker bias dominates
Podcasts (interview / drama / comedy)	100 hr / weekend	yt-dlp + RSS	44.1 kHz mostly clean	Many publish; else Whisper	Authentic conversation, laughter, hesitation	Ad reads, multiple speakers, room noise
Anime / game VA	50-200 hr	Wwise extractors (.bnk/.pck), vgmstream; anime via demuxed AC3/AAC	48 kHz studio	Subs (.ass/.srt) + scripts on Fandom	Hyper-expressive — the actual gold mine	Heavy stylization; subs aren't word-aligned
YouTube (interviews, dramatic reads, VA reels)	unlimited	yt-dlp -f bestaudio --extract-audio --audio-format wav --write-auto-sub	Varies 16-48 kHz	Auto-captions (noisy) or community subs	Highest variety	Music beds, compression, copyright
Movies / TV (own rips)	50-300 hr	MakeMKV → ffmpeg; center channel -ac 1 -af pan=mono|c0=FC	48 kHz from 5.1	PGS/SRT via Subtitle Edit OCR	Full dramatic spectrum	Music/SFX bleed even in center channel
Comedy specials / theater	20-50 hr	Same as movies	Good	Often subtitled	Laughter, timing, sarcasm	Audience noise
Your own recordings	5-20 hr realistic	USB mic + Audacity / Reaper	48 kHz pristine	Self-transcribe	Whatever you perform	You'll get bored by hour 3
TTS-synthesized	unlimited	ElevenLabs / PlayHT API	44.1 kHz	You wrote it	Limited to TTS expressiveness	DON'T DO THIS — mode collapse
Reddit voice clips	5-20 hr	Manual scraping	Junk to OK	None	Niche	Skip

Best ROI for first pass: audiobooks (single-speaker emotional performance) + podcasts (real conversational dynamics) + anime/game VA reels (extreme emotional range). 80% of the value with 20% of the effort.

2. End-to-end data preparation pipeline

In order:

1. Download: yt-dlp -x --audio-format wav --audio-quality 0 (use --cookies-from-browser for paywalled content).
2. Source separation: Demucs v4 htdemucs_ft (python -m demucs.separate -n htdemucs_ft --two-stems vocals). For dialogue with reverb: chain MDX23C-DeReverb before htdemucs.
3. VAD + segmentation: Silero V5 (we have a candle port — examples/silero_vad_smoke) → 3-15 s chunks.
4. Diarization + ASR + alignment, bundled: WhisperX (whisperx audio.wav --model large-v3 --diarize --hf_token ...). Wraps faster-whisper + pyannote 3.1 + wav2vec2 forced alignment in one command. Use this — don't roll your own.
5. Quality gate: DNSMOS P.835 (microsoft/DNSMOS HF model). Keep MOS-OVRL > 3.0. Drop chunks with SNR < 15 dB (pyloudnorm + noise floor estimate).
6. Single-speaker filter: keep diarized segments where one speaker covers > 95% of the chunk.
7. Resample to 24 kHz mono (CSM's native rate): ffmpeg -ar 24000 -ac 1.
8. Mimi tokenize offline once, cache to disk. Use our existing mimi.rs::Mimi::encode helper.

3. Emotion labeling — the hard part

Three-tier stack:

Primary (auto-tag every chunk):

• emotion2vec+ (Ma et al. 2024, IEEE/ACM TASLP). SOTA open-weight emotion encoder, ~75% on IEMOCAP. Outputs continuous valence / arousal / dominance + emotion classification.
• Or superb/wav2vec2-base-superb-er for a simpler discrete tag.

Secondary (LLM-as-judge for the 5-10% you actually train on):

• Feed Whisper transcript + emotion2vec output + 3 s audio clip to GPT-4o or Claude with a rubric: "rate valence -1 to 1, arousal 0 to 1, dominant emotion from GoEmotions 28-label set."
• Costs ~$50 per 100 h of audio. Don't go text-only — you'll miss prosody.

Validation:

• Hand-label 200 clips yourself, measure agreement with auto-labels (Cohen's κ). If κ < 0.5, your labels are noise — fix the classifier or change taxonomy.

Implicit alternative (what Sesame likely did):

• Skip explicit emotion labels entirely. Condition CSM training on the previous turn's audio tokens and let the model learn affect implicitly. Better for conversational naturalness, worse for controllability. Pragmatic answer: do BOTH — implicit conditioning + emotion2vec+ tags as control tokens prepended to text.

4. Training recipe (100-200 hr corpus, single A100 or H100)

• LoRA coverage: extend rtx-csm's Phase 3 q+v to q, k, v, o + MLP gate/up/down. q+v alone caps expressiveness; the FFN carries prosodic style. Rank 32-64, alpha 64-128.
• Epochs: 2-3 over full corpus. CSM overfits fast on hobby-scale corpora.
• LR: 1e-4 cosine, 500-step warmup, min 1e-5.
• Curriculum (this matters):
  • Stage 1: audiobooks only, 3 epochs, clean
  • Stage 2: + podcasts, 1 epoch, lower LR 3e-5
  • Stage 3: + VA / movies, 1 epoch, even lower LR
  • Prevents messy data from destabilizing acoustic priors learned on clean stage 1.
• Emotion conditioning: ONE LoRA, multiple control tokens. Per-emotion LoRAs sound nice but you can't switch them fast enough at inference. Prepend <emotion=joy:0.8,arousal=0.6> to text.
• External corpora mix-in: EmoV-DB, ESD, MEAD, RAVDESS at 5-10% sampling weight — NOT more, or you'll regress to acted-emotion stiffness.
• Eval: UTMOS + speaker-sim (WavLM-TDNN) + emotion-recognition accuracy on held-out clips.

5. Going BEYOND Maya

Maya is tuned for safe, generic warmth at scale. You can beat her on fit (your taste, your voice prefs, your conversational rhythms) without matching her engineering polish.

Concrete techniques:

• 50+ emotions: GoEmotions taxonomy (28 labels) + continuous valence/arousal. Bin V/A into a 5×5 grid for 25 pseudo-categories on top.
• Multi-persona: train persona embeddings (one 512-d vector per persona, like YourTTS / Voicebox speaker embedding). 5-10 personas trivially supported with the existing Generator.
• Multilingual: CSM-1B is English-only. Realistic add: French / Spanish via XTTS-v2 cross-distillation, but visibly worse than English. Don't promise this to yourself.
• Temporal coherence: condition on rolling window of last N turns' audio tokens. CSM already supports this; extend context to 2048.
• Reactive emotion (the "cry-back when user sounds sad" trick): run emotion2vec+ on user input audio at inference, feed result as control token. ~50 ms latency on Metal. Totally feasible with our existing pipeline.

Papers to read:

• CosyVoice 2 (Du, 2024) — emotion conditioning recipes
• Voicebox (Le, Meta, 2023) — flow matching alternative paradigm
• NaturalSpeech 3 (Ju, 2024) — disentangled prosody control
• emotion2vec+ (Ma, 2024) — the encoder you'll use
• Spirit-LM (Nguyen, 2024) — speech-text joint LM with affect

6. Realistic outcome + traps

After 2-3 focused weekends: noticeably warmer and more varied than base CSM, with maybe 5 reliable emotion modes. Still mispronounces names, occasionally glitches on long contexts, sounds almost like Maya — not better.

After 2-3 months committed: plausibly exceed Maya on emotional range for your specific use case, while losing on robustness / safety / latency engineering polish.

Biggest trap: spending 80% of time on data collection, 15% on training infra, 5% on actually listening critically and iterating. Listening is where the model gets good. Set a rule: every checkpoint, generate 20 prompts, listen to all, write notes.

Second trap: training on synthesized data (ElevenLabs out → CSM in). Mode collapse guaranteed.

Where motivation typically dies: hour 40, when WhisperX diarization fails on a podcast and you spend a Saturday debugging pyannote. Set a personal commitment number — 60 hr for "clearly better than base," 300 hr for "beats Maya for me" — and accept that crossing the line takes one boring weekend after another.

"Beat Maya" honestly means: Maya is tuned for safe generic warmth at scale. You can beat her on FIT (your taste, your voice prefs, your conversational rhythms) easily. You will NOT beat Sesame's acoustic engineering or latency. Aim for fit, not benchmarks.

7. Concrete next steps within rtx-csm

If you actually want to start this:

1. Extend LoRA coverage in our crate (~1-2 hr coding) — current add_lora_to_backbone covers q+v. Extend to k, o, MLP. This is a legitimately useful generic improvement; ship as Phase 12.x.
2. Wire WhisperX as a data-prep step — it's Python, not in our stack. Just shell out via a Makefile / scripts/. Don't try to port it.
3. Build a tiny emotion_label.rs that runs emotion2vec+ via the ort_conflict_probe-style path (pure-Rust ort), or as a Python sidecar. Outputs JSON sidecar files alongside each WAV.
4. Emotion control token plumbing in Generator::generate — accept an emotion_hint: Option<String> and prepend to text prompt. Tiny change, big impact.
5. Curriculum trainer in examples/lora_train_emotional.rs — takes a manifest of (wav, transcript, emotion_tag) tuples, stage-grouped, runs the 3-stage curriculum from §4. Mostly recipes around the existing trainer.

That's a plausible Phase 12 — without committing to the multi-week data-collection sprint. Each step is bounded and shippable. The data corpus is the user's project, not the crate's.

Cited tools / models

• yt-dlp https://github.com/yt-dlp/yt-dlp
• Demucs v4 https://github.com/facebookresearch/demucs
• WhisperX https://github.com/m-bain/whisperX
• pyannote 3.x https://github.com/pyannote/pyannote-audio
• Silero V5 https://github.com/snakers4/silero-vad (rtx-csm Phase 11)
• emotion2vec+ https://github.com/ddlBoJack/emotion2vec
• DNSMOS https://github.com/microsoft/DNSMOS
• GoEmotions taxonomy https://github.com/google-research/google-research/tree/master/goemotions
• EmoV-DB / ESD / MEAD / RAVDESS — see maya_finetune_analysis.md