# rtx-csm vs. Sesame Labs full voice stack — gap analysis

Research conducted 2026-04-27 to compare what we've shipped against
what Sesame Labs has actually disclosed publicly. The goal: an honest
punch-list of remaining gaps so we can prioritize what's worth porting
next vs. what's gated on Sesame releasing more.

## TL;DR

**rtx-csm has shipped a superset of Sesame's open release** (CSM-1B +
voice cloning + Q8 + production WebSocket server) and a rough peer of
Kyutai Unmute at the pipeline level.

The actual deployed Sesame product ("Maya") is, per their research
blog and absence of duplex paper, a **cascaded STT→LLM→TTS pipeline** —
same architecture pattern as rtx-csm. There is no public Sesame paper
describing a duplex production system. Their published "Crossing the
uncanny valley" research blog explicitly states future work is
"fully duplex models."

Three concrete gaps remain to "literal Maya parity":

1. **SilentCipher** is what Sesame actually ships as their watermarker
   (`SesameAILabs/silentcipher` on GitHub). rtx-csm has an AudioSeal
   scaffold instead — different system, same purpose. **~1-2 days to
   port** if literal parity matters.

2. **Maya's personality fine-tune dataset** — never released. Replicate
   via our existing LoRA pipeline + a custom corpus. This is product
   work, not infrastructure work.

3. **CSM-3B / CSM-8B variants** — Sesame trained them, never released.
   The Oct 2025 Series B + smart-glasses pivot suggests they're
   unlikely to ever release. Gated on Sesame, not us.

## What Sesame has actually disclosed

Substantiated:
- CSM-1B architecture (Llama-3.2 1B + 100M decoder + Mimi RVQ at 12.5 Hz)
- Apache-2.0 weights at `sesame/csm-1b` on HF
- ~1M hours predominantly-English training data
- 2048-token context (~2 min audio)
- Three new evaluation benchmarks (Homograph Disambiguation,
  Pronunciation Continuation Consistency, Contextual Expressivity)
- They watermark via SilentCipher (the `watermarking.py` in the open
  repo references it; `SesameAILabs/silentcipher` is the GitHub fork
  of Sony's SilentCipher implementation)
- They explicitly admit the open release "can only model the text and
  speech content in a conversation — not the structure" (i.e. no
  turn-taking, VAD, or duplex in the open weights)

Unsubstantiated marketing:
- "Sub-200 ms TTFA" — appears in third-party blogs, NOT in Sesame's
  own paper. Voice-agent benchmarks of cascaded stacks in 2025 show
  ~250-300 ms TTFT for production-tuned pipelines (OpenAI gpt-realtime,
  Kyutai Unmute) and 1-2 s for self-hosted. Treat the 200 ms figure as
  aspirational marketing.
- Maya's "interruption without stilted restart" — possible via barge-in
  + smart re-prompting. Not described in any technical doc. We have
  acoustic barge-in + fade-out shipped.

## Pipeline gap matrix

| Capability | Status | Detail |
|---|---|---|
| CSM-1B inference | **Closed** | Phase 1, candle 0.9 + Q8 GGUF |
| Voice cloning (LoRA) | **Closed** | Phase 3 — exceeds open release; Sesame doesn't ship a finetune pipeline |
| Mimi streaming codec | **Closed** | `mimi.rs::decode_step` chain |
| Long-form generation | **Closed** | `generate_long` with rolling context |
| Cascaded STT→LLM→TTS pipeline | **Closed** | WebSocket server, three STT backends (Phase 6/8) |
| Semantic VAD / EOT detection | **Closed** | Kyutai head-2 EOT (Phase 6c.3b) |
| Energy VAD silence skip | **Closed** | Phase 8.1.3b |
| Barge-in (acoustic) + fade-out | **Closed** | Phase 6c.3c + 6c.3e |
| Multi-speaker context conditioning | **Closed** | `Segment` / `SpeakerProfile` / `build_prompt` |
| Boot warm-up | **Closed** | Phase 8.2 — closes -89 % TTFA |
| Watermarking — AudioSeal | **Closed** | Phase 4, our scaffold |
| **Watermarking — SilentCipher (Sesame's actual)** | **Open** | ~1-2 days to port `SesameAILabs/silentcipher` |
| TTFA latency (real LLM) | **Partial** | 1.96 s end-to-end on Z.AI; ~280-380 ms TTS-side. The Z.AI provider TTFT is the dominant cost. Closed against any public open-stack benchmark; open against marketing claims |
| Full-duplex (listen-while-speaking) | **Open** | Sesame admitted not in CSM-1B. Only Moshi delivers this open-source. Substantial CSM extension needed |
| CSM-3B / CSM-8B port | **N/A** | Weights not released, likely never will be |
| Maya personality fine-tune | **N/A** | Dataset not released; replicate via our LoRA + custom corpus |
| On-device / wearable variant | **Open** | Q8 GGUF gets us part-way; full distillation (smaller decoder, fewer codebooks) not done |
| Memory across conversations | **N/A** | Maya marketing feature; belongs in LLM layer / clawsample, not the speech stack |
| Production STT backends | **Closed** | 3 backends (Kyutai 1B / Moonshine-tiny / Whisper-tiny) — superset of Sesame open release |

## What we have that Sesame doesn't ship

- **`tracing` instrumentation** with /metrics gauges (recv / stt_post /
  llm_to_first_audio / conv_total / total_turn) — Phase 6f.trace
- **Configurable LLM backend** via OpenAI-compatible HTTP — Sesame
  doesn't expose a swappable LLM in their open release
- **Auth, rate-limit, graceful shutdown** — production hardening Sesame
  doesn't ship
- **In-process Mimi (HF-named)** — rest of CSM stays on candle-0.9
  while we resist the moshi crate's incompatible Mimi naming
- **Comprehensive perf history doc** capturing every shipped + rejected
  optimization with reproducible benchmarks

## Closest open-source peers (anchor)

- **Kyutai Unmute**: closest analog to the deployed Maya — modular
  cascaded with Kyutai STT + any LLM + Kyutai TTS 1.6 B. Our pipeline
  is structurally equivalent.
- **Moshi**: the only open full-duplex system. If we ever need true
  listen-while-speaking, this is the reference.
- **Orpheus TTS**: TTS-only competitor to CSM (Llama-3 3 B + SNAC).
  Different codec; not directly portable.
- **OpenAI gpt-realtime**: closed; ~250-300 ms TTFT.
- **LiveKit Agents**: orchestration framework. Not a model. Could be
  the answer for multi-tenant / multi-user voice agents downstream.

## Recommended next chunks (prioritized)

1. **Swap AudioSeal scaffold for a SilentCipher port** (~1-2 days).
   This is the *actual* Sesame watermarker. Direct repo-to-port:
   - `git clone https://github.com/SesameAILabs/silentcipher`
   - Architecture is a small CNN — order of magnitude smaller than
     AudioSeal. Likely a 1-day port at this point.
   - Output: `--watermark-silentcipher` flag in converse_server,
     `examples/silentcipher_apply` CLI parallel to `audioseal_apply`.

2. **clawsample-csm integration** (~5-10 days, separate plan exists at
   `docs/clawsample_integration_plan.md`). Multi-tenant API surface,
   billing, R2 storage. Product work rather than ML work.

3. **Tier 2.1 VoXtream look-ahead** (~5-7 days, plan exists at
   `docs/voxtream_port_notes.md`). Closes the remaining ~270 ms TTS
   first-chunk gap. **Likely diminishing returns** post Phase 9.2 chunk
   tuning — TTS is already 280-380 ms; Z.AI TTFT (1300 ms) dominates.

4. **Tier 3 Frame-Stacked / VADUSA** (training-required, ~3-6 weeks).
   Reserved for when a real consumer hits the current ceiling.

5. **Distilled CSM for wearable target** (10-30 days). Speculative;
   probably wait until Sesame ships their distilled checkpoint or we
   have a real on-device deploy target.

## Honest verdict

**rtx-csm is feature-complete relative to Sesame's open release.** The
remaining gaps to "literal deployed Maya" are:

- **Closed by porting SilentCipher** (1-2 days, recommended next chunk)
- **Gated on Sesame releasing more** (CSM-3B, CSM-8B, training data,
  Maya fine-tune corpus) — probably never given the smart-glasses pivot
- **Architectural** (full-duplex via Moshi-style, distilled wearable
  variant) — multi-week investments worth pursuing only when product
  demand is concrete

Defensible product framing:

> rtx-csm is Sesame's open CSM-1B + voice cloning + production
> HTTP/WebSocket server + Kyutai-style cascaded duplex pipeline. The
> remaining gap to internal Maya is (a) SilentCipher watermarker
> (1-2 days), (b) Sesame's proprietary fine-tune dataset, (c) larger
> CSM variants Sesame chose not to release.

## Sources

- [Sesame — Crossing the uncanny valley of conversational voice](https://www.sesame.com/research/crossing_the_uncanny_valley_of_voice)
- [SesameAILabs/csm](https://github.com/SesameAILabs/csm)
- [SesameAILabs/silentcipher](https://github.com/SesameAILabs/silentcipher)
- [SilentCipher paper, arXiv 2406.03822](https://arxiv.org/abs/2406.03822)
- [TechCrunch — Sesame raises $250M (Oct 2025)](https://techcrunch.com/2025/10/21/sesame-the-conversational-ai-startup-from-oculus-founders-raises-250m-and-launches-beta/)
- [TechCrunch — Sesame open-sources CSM-1B (Mar 2025)](https://techcrunch.com/2025/03/13/sesame-the-startup-behind-the-viral-virtual-assistant-maya-releases-its-base-ai-model/)
- [Kyutai Unmute](https://kyutai.org/unmute) / [unmute repo](https://github.com/kyutai-labs/unmute)
- [Moshi paper, arXiv 2410.00037](https://arxiv.org/html/2410.00037v2)
- [Voice agent latency benchmarks 2025](https://dev.to/cloudx/cracking-the-1-second-voice-loop-what-we-learned-after-30-stack-benchmarks-427)
- [HF sesame/csm-1b realtime discussion (200ms claim)](https://huggingface.co/sesame/csm-1b/discussions/10)