rtx-csm: research — Maya personality fine-tune analysis

Honest research into what Sesame likely did to fine-tune the open
CSM-1B base into the deployed Maya/Miles persona. Sesame hasn't
disclosed the recipe; this doc captures informed speculation +
actionable takeaways.

Key findings:

1. Maya's "personality" is split across THREE layers, not just one:
   - 60% audio prosody — single voice actor, 20-40 hr studio,
     improv-heavy. Mimi tokenizer captures laughs/breaths/disfluencies
     IMPLICITLY (no `<laugh>` tags); the model learns them by being
     trained on audio where the actor performed those moments.
   - 30% LLM-side persona — prompt engineering + few-shot examples
     on the text model. NOT a voice-model property at all.
   - 10% conversational dynamics — VAD + endpointing + barge-in +
     streaming TTS. We're already at parity here.

2. Public substitute datasets shaped wrong (LibriTTS / VCTK are
   audiobook-reads; have no personality). Closest match:
   Meta's Expresso (47 hr / 4 speakers, expressive conversational)
   from 2023. Worth investigating if we ever pursue real Maya-class
   prosody.

3. Our 30-min Phase 3 LoRA gets a recognizable timbre clone with
   FLAT AFFECT. Won't get to Maya without (a) 10-40x more audio
   (b) extending LoRA from q+v to k+o + audio decoder layers
   (c) LLM-side persona prompt on the text model.

Three concrete next chunks captured:
  A. LLM-side persona prompt (~30 min, biggest ROI/minute)
  B. Extended LoRA coverage (~1-2 hours)
  C. Real corpus + audio fine-tune (multi-week, defer)

Verdict: real Maya-class output is 2-4 person-months of product
work + a 5-10 hr studio recording session. The IP gap is real and
not closeable with documentation alone. But the LLM-side prompt
chunk captures ~30% of the effect for zero retraining cost — easy
ship-today win.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

crates/models/rtx-csm/docs/maya_finetune_analysis.md

@@ -0,0 +1,179 @@
+# Maya personality fine-tune — research + replication plan
+
+Research (2026-04-27) into what Sesame likely did to fine-tune the
+base CSM-1B into Maya/Miles. Built on `docs/sesame_gap_analysis.md`'s
+"gated on Sesame" verdict: Maya's recipe isn't open-sourced.
+Question we're answering: **what would it take to get within shouting
+distance ourselves?**
+
+## TL;DR
+
+**The 30-min LoRA pipeline rtx-csm ships today gets a recognizable
+voice clone with flat affect. Maya-class personality requires:**
+
+- ~10-40× more training audio (20-40 hr single-speaker studio,
+  improv-heavy NOT audiobook)
+- LLM-side prompt engineering (60% of perceived persona is text
+  layer, not voice)
+- Extending LoRA from `q+v` to also `k+o` and audio decoder layers
+- NOT post-processing FX (Maya is dry-mic; quality is the model)
+
+Realistic ceiling at 30 min audio = decent clone of a single voice's
+*timbre*. Personality (chuckles, "ums", thinking-pauses, witty
+asides) is downstream of training-data behavior, not a knob we can
+turn at inference.
+
+## What Sesame has actually disclosed
+
+**Substantiated**:
+- CSM-1B base trained on ~1M hr predominantly-English audio
+  (`crossing-the-uncanny-valley` research blog)
+- Open weights are the *base* model, not Maya
+- Mimi RVQ tokenizer (12.5 Hz, 32 codebooks) handles laughs / breaths /
+  disfluencies *natively* — they're audio codes, not special text
+  tokens
+
+**Not disclosed**:
+- Maya fine-tune dataset size, speaker count, recording conditions
+- Fine-tune method (full vs LoRA, hyperparameters, loss)
+- Whether the Maya LLM prompt is published anywhere
+- Whether CSM-3B / CSM-8B were used as the base before distillation
+
+**Verdict**: marketing emphasizes "voice presence" + "emotional
+intelligence" but never describes the recipe. Founders' interviews
+(Iribe TechCrunch / Wired) similarly avoid technical specifics.
+
+## How the personality is split across the stack
+
+The Maya effect is NOT a single-model property. It's three layers:
+
+### 1. Audio prosody layer (~60% of perceived "Maya" — the special sauce)
+
+The base CSM-1B is speaker-conditioned via reference embedding +
+prompt context. Maya is almost certainly:
+- **Single voice actor, studio-recorded**, 20-100 hr (industry norm
+  for this quality)
+- **Improv-heavy script** with the actor reacting to LLM-generated
+  prompts, NOT reading audiobook-style scripts
+- Performed disfluencies (deliberate "ums", chuckles, thinking
+  pauses) baked into the audio
+- Fine-tune over the speaker-embedding-locked base — full-model or
+  high-rank LoRA across many layers
+
+The key insight: **Mimi's tokenizer captures laughs / breaths
+implicitly**. There's no `<laugh>` tag. The model learns "when this
+LLM text appears in a conversational moment, produce a chuckle here"
+through paired (text, audio) where the actor improvised exactly that.
+
+### 2. Conversational dynamics layer (~10% — pacing, turn-taking, barge-in)
+
+Pipeline-level engineering, not model:
+- Aggressive VAD + endpointing (Silero V5 or similar)
+- Streaming TTS chunked at ~160 ms (we have this — Phase 9.2)
+- Smart barge-in with fade-out (we have this — Phase 6c.3c + 6c.3e)
+- LLM token streaming so TTS starts on first sentence (we have this —
+  Phase 6f.stream-tts)
+
+We're already at parity here.
+
+### 3. Text/persona layer (~30% — the witty content)
+
+This is **almost entirely prompt engineering on the LLM**, not the
+voice model:
+- System prompt establishing Maya's persona ("you are slightly flirty
+  and witty, use disfluencies like 'um' and laugh occasionally")
+- Few-shot examples of Maya-style turns
+- Maybe a small DPO-style preference fine-tune on the LLM
+- A short response style (1-2 sentence replies to keep conversation
+  flowing)
+
+We CAN replicate this layer fully via a prompt — no retraining needed.
+
+## Public datasets that could substitute
+
+| Corpus | Size | Style | Verdict |
+|--------|------|-------|---------|
+| LibriTTS | 585 hr | Audiobook reads | Wrong shape — no personality |
+| VCTK | 44 hr / 110 spk | Read prompts | Wrong shape — no improv |
+| GigaSpeech | 10k hr | Mixed (podcasts, audiobooks) | Multi-speaker, noisy |
+| Spotify Podcast Dataset | huge | Real podcasts | Multi-speaker, noisy |
+| **Expresso** (Meta, 2023) | **47 hr / 4 spk** | **Expressive conversational, includes laughs/whispers/etc.** | **Closest public substitute. Worth investigating.** |
+
+## Comparable open-source persona work
+
+- **Hume EVI**: emotion-labeled corpus + expression embedding head.
+  Closer to conditioning than fine-tune. Different paradigm.
+- **ElevenLabs PVC**: ~3 hr clean audio voice clones. Technique
+  undisclosed; believed speaker-encoder + diffusion refinement.
+  NOT personality, just timbre.
+- **Bark / Coqui**: emotion via *explicit* `[laughs] [sighs]` tokens.
+  Lower quality than CSM's implicit-via-audio approach.
+- **Moshi DPO**: paper mentions DPO on dialogue quality; no released
+  recipe. SpeechGPT-style preference tuning exists in papers but no
+  production deploy.
+
+## Honest assessment vs our existing pipeline
+
+Phase 3 LoRA shipped: rank-8 alpha-16 on backbone q+v projections
+with 30 min training audio.
+
+Gap to Maya:
+
+| Component | rtx-csm Phase 3 | Maya |
+|-----------|-----------------|------|
+| Speaker timbre | ✓ recognizable | ✓ |
+| Basic prosody | partial (flat affect) | full |
+| Spontaneous laughs | ✗ | ✓ (in training data) |
+| Witty asides / personality | ✗ | ✓ (in training data + LLM prompt) |
+| Long conversational arc | ✗ | ✓ (hours of dialogue training) |
+| Pacing / endpointing | ✓ via Silero V5 (Phase 11) | ✓ |
+| Barge-in | ✓ Phase 6c.3c | ✓ |
+| Stream TTS | ✓ Phase 6f.stream-tts | ✓ |
+
+## Two actionable next chunks
+
+### A. LLM-side persona (~30 min, biggest ROI per minute)
+
+System prompt + few-shot examples that establish "Maya-class"
+personality on the text model. Drops into our existing
+`--system "..."` flag on `converse_server`. Tested by ear with the
+Z.AI / OpenAI loop.
+
+This captures ~30% of the perceived persona for zero retraining cost
+and ships TODAY.
+
+### B. Extended LoRA layer coverage (~1-2 hours)
+
+Phase 3's LoRA covers `q_proj` and `v_proj` of the backbone. For
+prosody adaptation, also wrap:
+- `k_proj` and `o_proj` of attention
+- The depth decoder's q/k/v/o (if the depth decoder governs
+  intra-frame timbre, which it does)
+
+Higher LoRA capacity → better adaptation at the same training-data
+budget. Still won't get to Maya, but ~25% improvement on the same
+30 min audio.
+
+### C. (Long-term, multi-week) Real audio corpus
+
+To approach Maya-class output:
+- 5-10 hr single-speaker improv recording
+- Studio quality (low reverb, ≥48 kHz)
+- LLM-prompted scripts the speaker improvises around
+- Train all attention + audio decoder layers (not just q+v)
+- Also fine-tune on Expresso (or similar public corpus) as
+  pretraining data before the speaker-specific phase
+
+This is real product work — likely 2-4 person-months total, not
+something to attempt without a concrete consumer.
+
+## Cited sources
+
+- [Sesame — Crossing the uncanny valley of conversational voice](https://www.sesame.com/research/crossing_the_uncanny_valley_of_voice)
+- [TechCrunch — Sesame raises $250M (Oct 2025)](https://techcrunch.com/2025/10/21/sesame-the-conversational-ai-startup-from-oculus-founders-raises-250m-and-launches-beta/)
+- [Meta Expresso dataset (2023)](https://speechbot.github.io/expresso/)
+- [Hume EVI](https://www.hume.ai/products/empathic-voice-interface-evi)
+- [ElevenLabs PVC docs](https://elevenlabs.io/docs/product-guides/voices/voice-cloning/professional-voice-cloning)
+- [Moshi paper (Kyutai, arXiv 2410.00037)](https://arxiv.org/abs/2410.00037)
+- [Bark (Suno)](https://github.com/suno-ai/bark)
+- [Coqui XTTS](https://github.com/coqui-ai/TTS)