# Personal voice training guide — beyond Maya, just for you For the case where you want **the most emotionally responsive voice ever**, for **personal use only** (not distributed, not commercial), using **all media formats you can legitimately access**. This is the iterative, hobby-project sibling of `docs/maya_finetune_analysis.md`. The legal calculus is different for personal use: training is in a gray zone but training-and-then- distributing-outputs is the actual liability vector. Keep outputs to yourself and the practical risk drops sharply. ## TL;DR - **Realistic timeline**: 60-120 hr of focused work for "clearly better than base CSM in your domain." 300+ hr for "genuinely beats Maya for me." - **Stack**: yt-dlp → Demucs v4 → Silero V5 VAD → WhisperX (diarization+ASR+alignment) → DNSMOS gate → emotion2vec+ labels → resample to 24 kHz → Mimi tokenize. - **Best data sources by ROI**: audiobooks > podcasts > anime/game VA > YouTube interviews > movies. **Skip Reddit clips, skip TTS-synthesized data** (mode collapse). - **Biggest trap**: spending 80% of time on data pipeline, 15% on training infra, 5% on actually listening to outputs critically. Listening is where the model gets good. ## 1. Data sources ranked by emotional richness × accessibility | Source | Hours | How to get | Audio | Transcripts | Emotion | Gotchas | |--------|-------|------------|-------|-------------|---------|---------| | **Audiobooks** (own Audible / LibriVox) | 200-500 hr easy | OpenAudible + ffmpeg; LibriVox direct MP3 | 22-44 kHz clean | Aeneas / Montreal Forced Aligner against Project Gutenberg text | Narrator-acted: sadness, suspense, joy, anger | Single-speaker bias dominates | | **Podcasts** (interview / drama / comedy) | 100 hr / weekend | yt-dlp + RSS | 44.1 kHz mostly clean | Many publish; else Whisper | Authentic conversation, laughter, hesitation | Ad reads, multiple speakers, room noise | | **Anime / game VA** | 50-200 hr | Wwise extractors (.bnk/.pck), vgmstream; anime via demuxed AC3/AAC | 48 kHz studio | Subs (.ass/.srt) + scripts on Fandom | Hyper-expressive — **the actual gold mine** | Heavy stylization; subs aren't word-aligned | | **YouTube** (interviews, dramatic reads, VA reels) | unlimited | `yt-dlp -f bestaudio --extract-audio --audio-format wav --write-auto-sub` | Varies 16-48 kHz | Auto-captions (noisy) or community subs | Highest variety | Music beds, compression, copyright | | **Movies / TV** (own rips) | 50-300 hr | MakeMKV → ffmpeg; center channel `-ac 1 -af pan=mono\|c0=FC` | 48 kHz from 5.1 | PGS/SRT via Subtitle Edit OCR | Full dramatic spectrum | Music/SFX bleed even in center channel | | **Comedy specials / theater** | 20-50 hr | Same as movies | Good | Often subtitled | Laughter, timing, sarcasm | Audience noise | | **Your own recordings** | 5-20 hr realistic | USB mic + Audacity / Reaper | 48 kHz pristine | Self-transcribe | Whatever you perform | You'll get bored by hour 3 | | **TTS-synthesized** | unlimited | ElevenLabs / PlayHT API | 44.1 kHz | You wrote it | Limited to TTS expressiveness | **DON'T DO THIS — mode collapse** | | **Reddit voice clips** | 5-20 hr | Manual scraping | Junk to OK | None | Niche | Skip | **Best ROI for first pass**: audiobooks (single-speaker emotional performance) + podcasts (real conversational dynamics) + anime/game VA reels (extreme emotional range). 80% of the value with 20% of the effort. ## 2. End-to-end data preparation pipeline In order: 1. **Download**: `yt-dlp -x --audio-format wav --audio-quality 0` (use `--cookies-from-browser` for paywalled content). 2. **Source separation**: **Demucs v4 htdemucs_ft** (`python -m demucs.separate -n htdemucs_ft --two-stems vocals`). For dialogue with reverb: chain **MDX23C-DeReverb** before htdemucs. 3. **VAD + segmentation**: **Silero V5** (we have a candle port — `examples/silero_vad_smoke`) → 3-15 s chunks. 4. **Diarization + ASR + alignment, bundled**: **WhisperX** (`whisperx audio.wav --model large-v3 --diarize --hf_token ...`). Wraps faster-whisper + pyannote 3.1 + wav2vec2 forced alignment in one command. **Use this — don't roll your own.** 5. **Quality gate**: DNSMOS P.835 (`microsoft/DNSMOS` HF model). Keep MOS-OVRL > 3.0. Drop chunks with SNR < 15 dB (`pyloudnorm` + noise floor estimate). 6. **Single-speaker filter**: keep diarized segments where one speaker covers > 95% of the chunk. 7. **Resample to 24 kHz mono** (CSM's native rate): `ffmpeg -ar 24000 -ac 1`. 8. **Mimi tokenize** offline once, cache to disk. Use our existing `mimi.rs::Mimi::encode` helper. ## 3. Emotion labeling — the hard part Three-tier stack: **Primary (auto-tag every chunk):** - **emotion2vec+** (Ma et al. 2024, IEEE/ACM TASLP). SOTA open-weight emotion encoder, ~75% on IEMOCAP. Outputs continuous valence / arousal / dominance + emotion classification. - Or `superb/wav2vec2-base-superb-er` for a simpler discrete tag. **Secondary (LLM-as-judge for the 5-10% you actually train on):** - Feed Whisper transcript + emotion2vec output + 3 s audio clip to GPT-4o or Claude with a rubric: "rate valence -1 to 1, arousal 0 to 1, dominant emotion from GoEmotions 28-label set." - Costs ~$50 per 100 h of audio. Don't go text-only — you'll miss prosody. **Validation**: - Hand-label 200 clips yourself, measure agreement with auto-labels (Cohen's κ). If κ < 0.5, your labels are noise — fix the classifier or change taxonomy. **Implicit alternative (what Sesame likely did)**: - Skip explicit emotion labels entirely. Condition CSM training on the previous turn's audio tokens and let the model learn affect implicitly. **Better for conversational naturalness, worse for controllability.** Pragmatic answer: do BOTH — implicit conditioning + emotion2vec+ tags as control tokens prepended to text. ## 4. Training recipe (100-200 hr corpus, single A100 or H100) - **LoRA coverage**: extend rtx-csm's Phase 3 q+v to **q, k, v, o + MLP gate/up/down**. q+v alone caps expressiveness; the FFN carries prosodic style. Rank 32-64, alpha 64-128. - **Epochs**: 2-3 over full corpus. CSM overfits fast on hobby-scale corpora. - **LR**: 1e-4 cosine, 500-step warmup, min 1e-5. - **Curriculum** (this matters): - Stage 1: audiobooks only, 3 epochs, clean - Stage 2: + podcasts, 1 epoch, lower LR 3e-5 - Stage 3: + VA / movies, 1 epoch, even lower LR - Prevents messy data from destabilizing acoustic priors learned on clean stage 1. - **Emotion conditioning**: **ONE LoRA, multiple control tokens**. Per-emotion LoRAs sound nice but you can't switch them fast enough at inference. Prepend `` to text. - **External corpora mix-in**: **EmoV-DB, ESD, MEAD, RAVDESS** at 5-10% sampling weight — NOT more, or you'll regress to acted-emotion stiffness. - **Eval**: UTMOS + speaker-sim (WavLM-TDNN) + emotion-recognition accuracy on held-out clips. ## 5. Going BEYOND Maya Maya is tuned for safe, generic warmth at scale. You can beat her on **fit** (your taste, your voice prefs, your conversational rhythms) without matching her engineering polish. Concrete techniques: - **50+ emotions**: GoEmotions taxonomy (28 labels) + continuous valence/arousal. Bin V/A into a 5×5 grid for 25 pseudo-categories on top. - **Multi-persona**: train **persona embeddings** (one 512-d vector per persona, like YourTTS / Voicebox speaker embedding). 5-10 personas trivially supported with the existing Generator. - **Multilingual**: CSM-1B is English-only. Realistic add: French / Spanish via XTTS-v2 cross-distillation, but visibly worse than English. Don't promise this to yourself. - **Temporal coherence**: condition on rolling window of last N turns' audio tokens. CSM already supports this; extend context to 2048. - **Reactive emotion** (the "cry-back when user sounds sad" trick): run emotion2vec+ on user input audio at inference, feed result as control token. ~50 ms latency on Metal. **Totally feasible** with our existing pipeline. Papers to read: - **CosyVoice 2** (Du, 2024) — emotion conditioning recipes - **Voicebox** (Le, Meta, 2023) — flow matching alternative paradigm - **NaturalSpeech 3** (Ju, 2024) — disentangled prosody control - **emotion2vec+** (Ma, 2024) — the encoder you'll use - **Spirit-LM** (Nguyen, 2024) — speech-text joint LM with affect ## 6. Realistic outcome + traps **After 2-3 focused weekends**: noticeably warmer and more varied than base CSM, with maybe 5 reliable emotion modes. Still mispronounces names, occasionally glitches on long contexts, sounds *almost* like Maya — not better. **After 2-3 months committed**: plausibly exceed Maya on emotional range for your specific use case, while losing on robustness / safety / latency engineering polish. **Biggest trap**: spending 80% of time on data collection, 15% on training infra, 5% on actually listening critically and iterating. Listening is where the model gets good. **Set a rule**: every checkpoint, generate 20 prompts, listen to all, write notes. **Second trap**: training on synthesized data (ElevenLabs out → CSM in). Mode collapse guaranteed. **Where motivation typically dies**: hour 40, when WhisperX diarization fails on a podcast and you spend a Saturday debugging pyannote. Set a personal commitment number — 60 hr for "clearly better than base," 300 hr for "beats Maya for me" — and accept that crossing the line takes one boring weekend after another. **"Beat Maya" honestly means**: Maya is tuned for safe generic warmth at scale. You can beat her on FIT (your taste, your voice prefs, your conversational rhythms) easily. You will NOT beat Sesame's acoustic engineering or latency. **Aim for fit, not benchmarks.** ## 7. Concrete next steps within rtx-csm If you actually want to start this: 1. **Extend LoRA coverage** in our crate (~1-2 hr coding) — current `add_lora_to_backbone` covers q+v. Extend to k, o, MLP. This is a legitimately useful generic improvement; ship as Phase 12.x. 2. **Wire WhisperX as a data-prep step** — it's Python, not in our stack. Just shell out via a Makefile / scripts/. Don't try to port it. 3. **Build a tiny `emotion_label.rs`** that runs emotion2vec+ via the `ort_conflict_probe`-style path (pure-Rust ort), or as a Python sidecar. Outputs JSON sidecar files alongside each WAV. 4. **Emotion control token plumbing** in `Generator::generate` — accept an `emotion_hint: Option` and prepend to text prompt. Tiny change, big impact. 5. **Curriculum trainer** in `examples/lora_train_emotional.rs` — takes a manifest of `(wav, transcript, emotion_tag)` tuples, stage-grouped, runs the 3-stage curriculum from §4. Mostly recipes around the existing trainer. That's a plausible Phase 12 — without committing to the multi-week data-collection sprint. Each step is bounded and shippable. The data corpus is the user's project, not the crate's. ## Cited tools / models - yt-dlp - Demucs v4 - WhisperX - pyannote 3.x - Silero V5 (rtx-csm Phase 11) - emotion2vec+ - DNSMOS - GoEmotions taxonomy - EmoV-DB / ESD / MEAD / RAVDESS — see `maya_finetune_analysis.md`