# Maya personality fine-tune — research + replication plan

Research (2026-04-27) into what Sesame likely did to fine-tune the
base CSM-1B into Maya/Miles. Built on `docs/sesame_gap_analysis.md`'s
"gated on Sesame" verdict: Maya's recipe isn't open-sourced.
Question we're answering: **what would it take to get within shouting
distance ourselves?**

## TL;DR

**The 30-min LoRA pipeline rtx-csm ships today gets a recognizable
voice clone with flat affect. Maya-class personality requires:**

- ~10-40× more training audio (20-40 hr single-speaker studio,
  improv-heavy NOT audiobook)
- LLM-side prompt engineering (60% of perceived persona is text
  layer, not voice)
- Extending LoRA from `q+v` to also `k+o` and audio decoder layers
- NOT post-processing FX (Maya is dry-mic; quality is the model)

Realistic ceiling at 30 min audio = decent clone of a single voice's
*timbre*. Personality (chuckles, "ums", thinking-pauses, witty
asides) is downstream of training-data behavior, not a knob we can
turn at inference.

## What Sesame has actually disclosed

**Substantiated**:
- CSM-1B base trained on ~1M hr predominantly-English audio
  (`crossing-the-uncanny-valley` research blog)
- Open weights are the *base* model, not Maya
- Mimi RVQ tokenizer (12.5 Hz, 32 codebooks) handles laughs / breaths /
  disfluencies *natively* — they're audio codes, not special text
  tokens

**Not disclosed**:
- Maya fine-tune dataset size, speaker count, recording conditions
- Fine-tune method (full vs LoRA, hyperparameters, loss)
- Whether the Maya LLM prompt is published anywhere
- Whether CSM-3B / CSM-8B were used as the base before distillation

**Verdict**: marketing emphasizes "voice presence" + "emotional
intelligence" but never describes the recipe. Founders' interviews
(Iribe TechCrunch / Wired) similarly avoid technical specifics.

## How the personality is split across the stack

The Maya effect is NOT a single-model property. It's three layers:

### 1. Audio prosody layer (~60% of perceived "Maya" — the special sauce)

The base CSM-1B is speaker-conditioned via reference embedding +
prompt context. Maya is almost certainly:
- **Single voice actor, studio-recorded**, 20-100 hr (industry norm
  for this quality)
- **Improv-heavy script** with the actor reacting to LLM-generated
  prompts, NOT reading audiobook-style scripts
- Performed disfluencies (deliberate "ums", chuckles, thinking
  pauses) baked into the audio
- Fine-tune over the speaker-embedding-locked base — full-model or
  high-rank LoRA across many layers

The key insight: **Mimi's tokenizer captures laughs / breaths
implicitly**. There's no `<laugh>` tag. The model learns "when this
LLM text appears in a conversational moment, produce a chuckle here"
through paired (text, audio) where the actor improvised exactly that.

### 2. Conversational dynamics layer (~10% — pacing, turn-taking, barge-in)

Pipeline-level engineering, not model:
- Aggressive VAD + endpointing (Silero V5 or similar)
- Streaming TTS chunked at ~160 ms (we have this — Phase 9.2)
- Smart barge-in with fade-out (we have this — Phase 6c.3c + 6c.3e)
- LLM token streaming so TTS starts on first sentence (we have this —
  Phase 6f.stream-tts)

We're already at parity here.

### 3. Text/persona layer (~30% — the witty content)

This is **almost entirely prompt engineering on the LLM**, not the
voice model:
- System prompt establishing Maya's persona ("you are slightly flirty
  and witty, use disfluencies like 'um' and laugh occasionally")
- Few-shot examples of Maya-style turns
- Maybe a small DPO-style preference fine-tune on the LLM
- A short response style (1-2 sentence replies to keep conversation
  flowing)

We CAN replicate this layer fully via a prompt — no retraining needed.

## Public datasets that could substitute

The user's intuition was spot-on: this is exactly what the emotional-
speech research community has been building corpora for. The catch
is licensing — most "best" corpora are academic-only. Concrete
matrix below is the headline.

### Skip Hollywood movies directly — DO NOT train on copyrighted film

The legal landscape (April 2026):
- NYT v. OpenAI survived motion to dismiss (2024)
- Andersen v. Stability class certified on training-data theory
- RIAA v. Suno/Udio still active and tilting against trainers
- Voice timbre adds **right-of-publicity** claims — Scarlett Johansson
  v. OpenAI "Sky" — strict-liability in CA/NY/TN even without
  copyright

Source separation (Demucs / MDX-Net) and diarization (pyannote 3.1)
work technically — extracts clean dialog stems. The law doesn't.
Public-domain film audio is < 50 hr total and acoustically poor (mono,
hiss). Not worth it.

### Legal commercial-clean corpora — the actual answer

| Corpus | Size / Speakers | Emotions | Quality | License | Disfluency |
|--------|-----------------|----------|---------|---------|------------|
| **EmoV-DB** | 7 hr / 4 spk | 5 (neutral, amused, angry, sleepy, disgust) | 44.1 kHz studio | **CC-BY 4.0 — commercial OK** | **Yes — explicit laughs/yawns** |
| **CREMA-D** | 5 hr / 91 actors | 6 | 16 kHz, varied | **ODC-By 1.0 — commercial OK** | Minimal (read 12 fixed sentences) |
| **DailyTalk** | 20 hr / 2 spk dyadic | conversational | 22 kHz | **CC-BY-SA 4.0 — commercial OK** | Some |
| **LAION's Emotional Speech (2024)** | ~5000 hr synth+natural | broad | varied | **CC-BY 4.0** — but provenance risk on web-scraped voices | Yes |
| **Hume Prosody (2024)** | proprietary | 53 expressions | studio | **Paid commercial license available** | Yes |

### Best research-only corpora (use only if non-commercial OK)

| Corpus | Size / Speakers | Emotions | License blocker |
|--------|-----------------|----------|-----------------|
| **Expresso** (Meta, 2023) | **47 hr / 4 spk** | 26 styles incl. whisper/laughing/sad | CC-BY-NC 4.0 |
| **IEMOCAP** | 12 hr / 10 actors | 9 | Academic-only EULA |
| **MELD** (Friends TV) | ~13 hr | 7 emotions | Audio = Warner Bros copyright |
| **RAVDESS** | 1.5 hr / 24 actors | 8 | CC-BY-NC-SA |
| **ESD** | 29 hr / 20 spk en+zh | 5 | Research EULA |

## Comparable open-source persona work

- **Hume EVI**: emotion-labeled corpus + expression embedding head.
  Closer to conditioning than fine-tune. Different paradigm.
- **ElevenLabs PVC**: ~3 hr clean audio voice clones. Technique
  undisclosed; believed speaker-encoder + diffusion refinement.
  NOT personality, just timbre.
- **Bark / Coqui**: emotion via *explicit* `[laughs] [sighs]` tokens.
  Lower quality than CSM's implicit-via-audio approach.
- **Moshi DPO**: paper mentions DPO on dialogue quality; no released
  recipe. SpeechGPT-style preference tuning exists in papers but no
  production deploy.

## Honest assessment vs our existing pipeline

Phase 3 LoRA shipped: rank-8 alpha-16 on backbone q+v projections
with 30 min training audio.

Gap to Maya:

| Component | rtx-csm Phase 3 | Maya |
|-----------|-----------------|------|
| Speaker timbre | ✓ recognizable | ✓ |
| Basic prosody | partial (flat affect) | full |
| Spontaneous laughs | ✗ | ✓ (in training data) |
| Witty asides / personality | ✗ | ✓ (in training data + LLM prompt) |
| Long conversational arc | ✗ | ✓ (hours of dialogue training) |
| Pacing / endpointing | ✓ via Silero V5 (Phase 11) | ✓ |
| Barge-in | ✓ Phase 6c.3c | ✓ |
| Stream TTS | ✓ Phase 6f.stream-tts | ✓ |

## Two actionable next chunks

### A. LLM-side persona (~30 min, biggest ROI per minute)

System prompt + few-shot examples that establish "Maya-class"
personality on the text model. Drops into our existing
`--system "..."` flag on `converse_server`. Tested by ear with the
Z.AI / OpenAI loop.

This captures ~30% of the perceived persona for zero retraining cost
and ships TODAY.

### B. Extended LoRA layer coverage (~1-2 hours)

Phase 3's LoRA covers `q_proj` and `v_proj` of the backbone. For
prosody adaptation, also wrap:
- `k_proj` and `o_proj` of attention
- The depth decoder's q/k/v/o (if the depth decoder governs
  intra-frame timbre, which it does)

Higher LoRA capacity → better adaptation at the same training-data
budget. Still won't get to Maya, but ~25% improvement on the same
30 min audio.

### C. (Long-term, multi-week) Real audio corpus

To approach Maya-class output:
- 5-10 hr single-speaker improv recording
- Studio quality (low reverb, ≥48 kHz)
- LLM-prompted scripts the speaker improvises around
- Train all attention + audio decoder layers (not just q+v)
- Also fine-tune on EmoV-DB / CREMA-D (commercial-clean) as
  pretraining data before the speaker-specific phase

This is real product work — likely 2-4 person-months total, not
something to attempt without a concrete consumer.

### D. Two-stage emotional corpus + small style session (the smart path)

Combining the user's "Hollywood emotion" instinct with the legal
corpora:

**Stage 1 — emotional baseline (~6 hr GPU on H100)**
- Train on **EmoV-DB + CREMA-D combined** (~12 hr, commercial-clean,
  CC-BY)
- Extend LoRA: r=8 α=16 on backbone q+v *and* k+o, *plus* decoder
  cross-attn projections
- 3 epochs, lr 1e-4 cosine, batch 4, bf16
- Output: model that knows emotional prosody (laughs, sighs, anger
  inflection) but sounds like an "average emotional speaker"

**Stage 2 — style transfer to single voice (~weekend, 5-10 hr session)**
- Record ONE voice actor improvising LLM-prompted dialogue, 5-10 hr
- Studio room, 48 kHz, low reverb (NOT a real studio booth required —
  a treated home-office is fine)
- Stage-2 LoRA r=16 on the stage-1 checkpoint, 5 epochs
- Output: that actor's timbre + emotional prosody from stage 1

**Realistic outcome (per the corpus research)**: ~70% of Maya's
emotional expressiveness, ~30% of her personality. Multi-speaker
stage-1 corpora produce "uncanny news anchor doing feelings" baseline;
stage-2 single-speaker session locks in the personality.

The stage-2 session is **a weekend with one actor**, NOT a 40-hour
studio sprint. The corpora do the heavy lifting on emotional range;
stage 2 is just style-transfer overlay.

### Why NOT Hollywood scenes directly

Tempting idea — train on TV/movies where actors already perform
emotional dialogue at studio quality, no recording session needed.
**Don't do this for a shipped product.**

- **Copyright**: NYT v OpenAI, Andersen v Stability, RIAA v Suno/Udio
  all tilting against training-on-copyrighted-material since 2024.
  Even if the audio works through fair-use, the trained model's
  outputs may be considered derivative works.
- **Right-of-publicity**: voice cloning has bonus IP exposure (CA, NY,
  TN have strict-liability voice-likeness statutes). Scarlett
  Johansson v. OpenAI "Sky" reset the bar.
- **Practical**: source separation cleans dialogue OK (~12 dB SDR via
  Demucs v4) but the legal exposure is uninsurable.
- **Public-domain film**: total < 50 hr with terrible audio (mono,
  hiss, pre-1929). Not enough volume even if usable.

The legal corpora above (EmoV-DB, CREMA-D, DailyTalk, LAION,
optionally Hume Prosody) capture the same actor-performing-emotion
property without the IP exposure. Stick to those.

## Cited sources

- [Sesame — Crossing the uncanny valley of conversational voice](https://www.sesame.com/research/crossing_the_uncanny_valley_of_voice)
- [TechCrunch — Sesame raises $250M (Oct 2025)](https://techcrunch.com/2025/10/21/sesame-the-conversational-ai-startup-from-oculus-founders-raises-250m-and-launches-beta/)
- [Meta Expresso dataset (2023)](https://speechbot.github.io/expresso/)
- [Hume EVI](https://www.hume.ai/products/empathic-voice-interface-evi)
- [ElevenLabs PVC docs](https://elevenlabs.io/docs/product-guides/voices/voice-cloning/professional-voice-cloning)
- [Moshi paper (Kyutai, arXiv 2410.00037)](https://arxiv.org/abs/2410.00037)
- [Bark (Suno)](https://github.com/suno-ai/bark)
- [Coqui XTTS](https://github.com/coqui-ai/TTS)