The hobby-project sibling of maya_finetune_analysis.md. For the case
where you want the most emotionally responsive voice ever, for personal
use only (not distributed, not commercial), using all media formats
legitimately accessible to one person.
Key practical guidance:
Best ROI sources: audiobooks (200-500 hr easy) > podcasts (100 hr per
weekend) > anime/game VA reels (the gold mine for extreme emotional
range) > YouTube > movies. Skip Reddit clips. SKIP TTS-synthesized
data (mode collapse).
Data prep pipeline (specific tool choices):
yt-dlp -> Demucs v4 htdemucs_ft -> Silero V5 VAD (we have the
candle port already) -> WhisperX (the right answer for
diarization+ASR+alignment, don't roll your own) -> DNSMOS quality
gate -> single-speaker filter -> resample 24 kHz -> Mimi tokenize
Emotion labeling: emotion2vec+ as primary auto-tagger, GPT-4o or
Claude as LLM-as-judge for the 5-10% you'll actually train on
(~$50/100hr), hand-label 200 clips for Cohen's kappa validation.
Plus implicit conditioning on previous-turn audio (what Sesame
likely did). Do BOTH.
Training recipe (100-200 hr corpus, single A100/H100):
- LoRA: extend from q+v to q,k,v,o + MLP gate/up/down. r=32-64.
- Curriculum: audiobooks (3 ep clean) -> podcasts (1 ep) -> VA/
movies (1 ep, lower LR). Prevents messy data destabilizing
acoustic priors.
- One LoRA, multiple emotion control tokens. Per-emotion LoRAs
can't switch fast enough at inference.
- 5-10% mix-in of EmoV-DB/ESD/MEAD/RAVDESS. Not more.
Reality check:
60-120 focused hr -> "clearly better than base CSM in your domain"
300+ hr -> "genuinely beats Maya for me"
Biggest trap: spending 80% of time on data, 15% on training infra,
5% on actually listening critically. Listening is where the model
gets good. Set a rule: every checkpoint, 20 prompts + notes.
Second trap: training on TTS-synthesized data. Mode collapse.
Where motivation dies: hour 40 when WhisperX diarization fails on
a podcast and you spend a Saturday debugging pyannote.
Going BEYOND Maya:
- GoEmotions 28-label taxonomy + V/A continuous (5x5 = 25 pseudo)
- Multi-persona via 512-d persona embeddings (YourTTS pattern)
- Reactive emotion: emotion2vec+ on user audio at inference,
feed as control token. ~50 ms latency. Feasible today.
Concrete Phase 12 candidates (bounded codable items, NOT the data
collection itself):
1. Extend rtx-csm LoRA coverage q+v -> k,o,MLP (~1-2 hr)
2. Wire WhisperX as scripts/ data-prep step (Python sidecar)
3. emotion2vec+ via ort sidecar, JSON labels
4. Emotion control token plumbing in Generator::generate
5. Curriculum trainer examples/lora_train_emotional.rs
Papers cited: CosyVoice 2, Voicebox, NaturalSpeech 3, emotion2vec+,
Spirit-LM. Tools: yt-dlp, Demucs v4, WhisperX, pyannote 3.x, Silero V5,
DNSMOS, GoEmotions taxonomy.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
