rustytorch/crates/models/rtx-csm/docs/maya_finetune_analysis.md

Maya personality fine-tune — research + replication plan

Research (2026-04-27) into what Sesame likely did to fine-tune the base CSM-1B into Maya/Miles. Built on docs/sesame_gap_analysis.md's "gated on Sesame" verdict: Maya's recipe isn't open-sourced. Question we're answering: what would it take to get within shouting distance ourselves?

TL;DR

The 30-min LoRA pipeline rtx-csm ships today gets a recognizable voice clone with flat affect. Maya-class personality requires:

• ~10-40× more training audio (20-40 hr single-speaker studio, improv-heavy NOT audiobook)
• LLM-side prompt engineering (60% of perceived persona is text layer, not voice)
• Extending LoRA from q+v to also k+o and audio decoder layers
• NOT post-processing FX (Maya is dry-mic; quality is the model)

Realistic ceiling at 30 min audio = decent clone of a single voice's timbre. Personality (chuckles, "ums", thinking-pauses, witty asides) is downstream of training-data behavior, not a knob we can turn at inference.

What Sesame has actually disclosed

Substantiated:

• CSM-1B base trained on ~1M hr predominantly-English audio (crossing-the-uncanny-valley research blog)
• Open weights are the base model, not Maya
• Mimi RVQ tokenizer (12.5 Hz, 32 codebooks) handles laughs / breaths / disfluencies natively — they're audio codes, not special text tokens

Not disclosed:

• Maya fine-tune dataset size, speaker count, recording conditions
• Fine-tune method (full vs LoRA, hyperparameters, loss)
• Whether the Maya LLM prompt is published anywhere
• Whether CSM-3B / CSM-8B were used as the base before distillation

Verdict: marketing emphasizes "voice presence" + "emotional intelligence" but never describes the recipe. Founders' interviews (Iribe TechCrunch / Wired) similarly avoid technical specifics.

How the personality is split across the stack

The Maya effect is NOT a single-model property. It's three layers:

1. Audio prosody layer (~60% of perceived "Maya" — the special sauce)

The base CSM-1B is speaker-conditioned via reference embedding + prompt context. Maya is almost certainly:

• Single voice actor, studio-recorded, 20-100 hr (industry norm for this quality)
• Improv-heavy script with the actor reacting to LLM-generated prompts, NOT reading audiobook-style scripts
• Performed disfluencies (deliberate "ums", chuckles, thinking pauses) baked into the audio
• Fine-tune over the speaker-embedding-locked base — full-model or high-rank LoRA across many layers

The key insight: Mimi's tokenizer captures laughs / breaths implicitly. There's no <laugh> tag. The model learns "when this LLM text appears in a conversational moment, produce a chuckle here" through paired (text, audio) where the actor improvised exactly that.

2. Conversational dynamics layer (~10% — pacing, turn-taking, barge-in)

Pipeline-level engineering, not model:

• Aggressive VAD + endpointing (Silero V5 or similar)
• Streaming TTS chunked at ~160 ms (we have this — Phase 9.2)
• Smart barge-in with fade-out (we have this — Phase 6c.3c + 6c.3e)
• LLM token streaming so TTS starts on first sentence (we have this — Phase 6f.stream-tts)

We're already at parity here.

3. Text/persona layer (~30% — the witty content)

This is almost entirely prompt engineering on the LLM, not the voice model:

• System prompt establishing Maya's persona ("you are slightly flirty and witty, use disfluencies like 'um' and laugh occasionally")
• Few-shot examples of Maya-style turns
• Maybe a small DPO-style preference fine-tune on the LLM
• A short response style (1-2 sentence replies to keep conversation flowing)

We CAN replicate this layer fully via a prompt — no retraining needed.

Public datasets that could substitute

The user's intuition was spot-on: this is exactly what the emotional- speech research community has been building corpora for. The catch is licensing — most "best" corpora are academic-only. Concrete matrix below is the headline.

Skip Hollywood movies directly — DO NOT train on copyrighted film

The legal landscape (April 2026):

• NYT v. OpenAI survived motion to dismiss (2024)
• Andersen v. Stability class certified on training-data theory
• RIAA v. Suno/Udio still active and tilting against trainers
• Voice timbre adds right-of-publicity claims — Scarlett Johansson v. OpenAI "Sky" — strict-liability in CA/NY/TN even without copyright

Source separation (Demucs / MDX-Net) and diarization (pyannote 3.1) work technically — extracts clean dialog stems. The law doesn't. Public-domain film audio is < 50 hr total and acoustically poor (mono, hiss). Not worth it.

Legal commercial-clean corpora — the actual answer

Corpus	Size / Speakers	Emotions	Quality	License	Disfluency
EmoV-DB	7 hr / 4 spk	5 (neutral, amused, angry, sleepy, disgust)	44.1 kHz studio	CC-BY 4.0 — commercial OK	Yes — explicit laughs/yawns
CREMA-D	5 hr / 91 actors	6	16 kHz, varied	ODC-By 1.0 — commercial OK	Minimal (read 12 fixed sentences)
DailyTalk	20 hr / 2 spk dyadic	conversational	22 kHz	CC-BY-SA 4.0 — commercial OK	Some
LAION's Emotional Speech (2024)	~5000 hr synth+natural	broad	varied	CC-BY 4.0 — but provenance risk on web-scraped voices	Yes
Hume Prosody (2024)	proprietary	53 expressions	studio	Paid commercial license available	Yes

Best research-only corpora (use only if non-commercial OK)

Corpus	Size / Speakers	Emotions	License blocker
Expresso (Meta, 2023)	47 hr / 4 spk	26 styles incl. whisper/laughing/sad	CC-BY-NC 4.0
IEMOCAP	12 hr / 10 actors	9	Academic-only EULA
MELD (Friends TV)	~13 hr	7 emotions	Audio = Warner Bros copyright
RAVDESS	1.5 hr / 24 actors	8	CC-BY-NC-SA
ESD	29 hr / 20 spk en+zh	5	Research EULA

Comparable open-source persona work

• Hume EVI: emotion-labeled corpus + expression embedding head. Closer to conditioning than fine-tune. Different paradigm.
• ElevenLabs PVC: ~3 hr clean audio voice clones. Technique undisclosed; believed speaker-encoder + diffusion refinement. NOT personality, just timbre.
• Bark / Coqui: emotion via explicit [laughs] [sighs] tokens. Lower quality than CSM's implicit-via-audio approach.
• Moshi DPO: paper mentions DPO on dialogue quality; no released recipe. SpeechGPT-style preference tuning exists in papers but no production deploy.

Honest assessment vs our existing pipeline

Phase 3 LoRA shipped: rank-8 alpha-16 on backbone q+v projections with 30 min training audio.

Gap to Maya:

Component	rtx-csm Phase 3	Maya
Speaker timbre	✓ recognizable	✓
Basic prosody	partial (flat affect)	full
Spontaneous laughs	✗	✓ (in training data)
Witty asides / personality	✗	✓ (in training data + LLM prompt)
Long conversational arc	✗	✓ (hours of dialogue training)
Pacing / endpointing	✓ via Silero V5 (Phase 11)	✓
Barge-in	✓ Phase 6c.3c	✓
Stream TTS	✓ Phase 6f.stream-tts	✓

Two actionable next chunks

A. LLM-side persona (~30 min, biggest ROI per minute)

System prompt + few-shot examples that establish "Maya-class" personality on the text model. Drops into our existing --system "..." flag on converse_server. Tested by ear with the Z.AI / OpenAI loop.

This captures ~30% of the perceived persona for zero retraining cost and ships TODAY.

B. Extended LoRA layer coverage (~1-2 hours)

Phase 3's LoRA covers q_proj and v_proj of the backbone. For prosody adaptation, also wrap:

• k_proj and o_proj of attention
• The depth decoder's q/k/v/o (if the depth decoder governs intra-frame timbre, which it does)

Higher LoRA capacity → better adaptation at the same training-data budget. Still won't get to Maya, but ~25% improvement on the same 30 min audio.

C. (Long-term, multi-week) Real audio corpus

To approach Maya-class output:

• 5-10 hr single-speaker improv recording
• Studio quality (low reverb, ≥48 kHz)
• LLM-prompted scripts the speaker improvises around
• Train all attention + audio decoder layers (not just q+v)
• Also fine-tune on EmoV-DB / CREMA-D (commercial-clean) as pretraining data before the speaker-specific phase

This is real product work — likely 2-4 person-months total, not something to attempt without a concrete consumer.

D. Two-stage emotional corpus + small style session (the smart path)

Combining the user's "Hollywood emotion" instinct with the legal corpora:

Stage 1 — emotional baseline (~6 hr GPU on H100)

• Train on EmoV-DB + CREMA-D combined (~12 hr, commercial-clean, CC-BY)
• Extend LoRA: r=8 α=16 on backbone q+v and k+o, plus decoder cross-attn projections
• 3 epochs, lr 1e-4 cosine, batch 4, bf16
• Output: model that knows emotional prosody (laughs, sighs, anger inflection) but sounds like an "average emotional speaker"

Stage 2 — style transfer to single voice (~weekend, 5-10 hr session)

• Record ONE voice actor improvising LLM-prompted dialogue, 5-10 hr
• Studio room, 48 kHz, low reverb (NOT a real studio booth required — a treated home-office is fine)
• Stage-2 LoRA r=16 on the stage-1 checkpoint, 5 epochs
• Output: that actor's timbre + emotional prosody from stage 1

Realistic outcome (per the corpus research): ~70% of Maya's emotional expressiveness, ~30% of her personality. Multi-speaker stage-1 corpora produce "uncanny news anchor doing feelings" baseline; stage-2 single-speaker session locks in the personality.

The stage-2 session is a weekend with one actor, NOT a 40-hour studio sprint. The corpora do the heavy lifting on emotional range; stage 2 is just style-transfer overlay.

Why NOT Hollywood scenes directly

Tempting idea — train on TV/movies where actors already perform emotional dialogue at studio quality, no recording session needed. Don't do this for a shipped product.

• Copyright: NYT v OpenAI, Andersen v Stability, RIAA v Suno/Udio all tilting against training-on-copyrighted-material since 2024. Even if the audio works through fair-use, the trained model's outputs may be considered derivative works.
• Right-of-publicity: voice cloning has bonus IP exposure (CA, NY, TN have strict-liability voice-likeness statutes). Scarlett Johansson v. OpenAI "Sky" reset the bar.
• Practical: source separation cleans dialogue OK (~12 dB SDR via Demucs v4) but the legal exposure is uninsurable.
• Public-domain film: total < 50 hr with terrible audio (mono, hiss, pre-1929). Not enough volume even if usable.

The legal corpora above (EmoV-DB, CREMA-D, DailyTalk, LAION, optionally Hume Prosody) capture the same actor-performing-emotion property without the IP exposure. Stick to those.

Cited sources

• Sesame — Crossing the uncanny valley of conversational voice
• TechCrunch — Sesame raises $250M (Oct 2025)
• Meta Expresso dataset (2023)
• Hume EVI
• ElevenLabs PVC docs
• Moshi paper (Kyutai, arXiv 2410.00037)
• Bark (Suno)
• Coqui XTTS