Extended docs/maya_finetune_analysis.md with the user's "train on
Hollywood scenes that denote emotions" angle. The instinct is sound;
the legal pitfalls are severe; legitimate alternatives exist.
Headline:
DON'T train on Hollywood movies. Copyright + right-of-publicity is
uninsurable for a shipped product (NYT v OpenAI / Andersen v
Stability / RIAA v Suno / Johansson v OpenAI "Sky" all 2024-2026).
Source separation works technically; the law doesn't.
Legitimate corpora that capture the same "actors performing emotion"
property (matrix added to the doc):
Commercial-clean (use these):
EmoV-DB 7 hr / 4 spk CC-BY 4.0 — explicit laughs/yawns
CREMA-D 5 hr / 91 spk ODC-By 1.0 — read but emotion-tagged
DailyTalk 20 hr / 2 spk CC-BY-SA — dyadic conversational
LAION Emo Speech ~5000 hr CC-BY 4.0 — but provenance risk
Hume Prosody proprietary paid commercial
Research-only (skip for shipped product):
Expresso (Meta) 47 hr / 4 spk CC-BY-NC — best quality
IEMOCAP 12 hr / 10 spk academic — best emotional range
MELD (Friends) 13 hr Warner Bros — audio is copyrighted
RAVDESS, ESD small/medium research
New "Path D" recipe added:
Stage 1 (~6 hr GPU on H100):
- EmoV-DB + CREMA-D combined (~12 hr, commercial-clean)
- LoRA r=8 α=16 on q+v +k+o + decoder cross-attn
- 3 epochs, lr 1e-4 cosine, bf16
Stage 2 (~weekend, 5-10 hr recording):
- One voice actor improvising LLM-prompted dialogue
- Stage-2 LoRA r=16 on the stage-1 checkpoint
Per the corpus research: gets ~70% of Maya's emotional
expressiveness, ~30% of her personality. The single-speaker stage 2
is the "uncanny news anchor doing feelings" -> "specific persona"
overlay. Crucially this is a WEEKEND with one actor, not the 40-hr
studio sprint Sesame did.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Honest research into what Sesame likely did to fine-tune the open
CSM-1B base into the deployed Maya/Miles persona. Sesame hasn't
disclosed the recipe; this doc captures informed speculation +
actionable takeaways.
Key findings:
1. Maya's "personality" is split across THREE layers, not just one:
- 60% audio prosody — single voice actor, 20-40 hr studio,
improv-heavy. Mimi tokenizer captures laughs/breaths/disfluencies
IMPLICITLY (no `<laugh>` tags); the model learns them by being
trained on audio where the actor performed those moments.
- 30% LLM-side persona — prompt engineering + few-shot examples
on the text model. NOT a voice-model property at all.
- 10% conversational dynamics — VAD + endpointing + barge-in +
streaming TTS. We're already at parity here.
2. Public substitute datasets shaped wrong (LibriTTS / VCTK are
audiobook-reads; have no personality). Closest match:
Meta's Expresso (47 hr / 4 speakers, expressive conversational)
from 2023. Worth investigating if we ever pursue real Maya-class
prosody.
3. Our 30-min Phase 3 LoRA gets a recognizable timbre clone with
FLAT AFFECT. Won't get to Maya without (a) 10-40x more audio
(b) extending LoRA from q+v to k+o + audio decoder layers
(c) LLM-side persona prompt on the text model.
Three concrete next chunks captured:
A. LLM-side persona prompt (~30 min, biggest ROI/minute)
B. Extended LoRA coverage (~1-2 hours)
C. Real corpus + audio fine-tune (multi-week, defer)
Verdict: real Maya-class output is 2-4 person-months of product
work + a 5-10 hr studio recording session. The IP gap is real and
not closeable with documentation alone. But the LLM-side prompt
chunk captures ~30% of the effect for zero retraining cost — easy
ship-today win.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
