8-gen bench (4 emotions × 2 corpora) at seed=42 against firdhokk
Whisper-LV3:
target RAVDESS CREMA-D
happy happy (0.999) ✓ happy (0.999) ✓
angry neutral (0.92) sad (0.99)
fearful happy (0.998) fearful (0.984) ✓
sad angry (0.99) fearful (0.99)
CREMA-D 2/4 vs RAVDESS 1/4. Larger / more naturalistic corpus
produces more class-pure fearful direction. Neither corpus solves
angry or sad — recipe shifts into 'vague expressivity' rather than
class-specific corners.
Practical: prefer CREMA-D when available; A/B both per emotion if
class precision matters.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Wired up firdhokk/speech-emotion-recognition-with-openai-whisper-large-v3
as a working alternative to the broken emotion2vec_plus_base. Sanity
verified on real RAVDESS clips: 3/5 correct, 2/5 near-miss (happy↔
surprised, sad↔fearful). Probabilities are NOT saturated — the
classifier actually distinguishes per-input.
Then scored our 4 decoder-route outputs (Amini context, seed=42,
recipe defaults) and found that **only fearful registers as the
intended class**:
target verdict conf
happy neutral 0.80 ✗ (steering produces neutral output)
angry happy 0.999 ✗ (high-arousal cross-class)
fearful fearful 0.68 ✓
sad fearful 0.998 ✗ (sad↔fearful confusion)
Honest framing: the recipe shifts speaker character toward an
expressive-sounding direction (cosine evidence) and preserves text
(decoder vs backbone) but does NOT produce class-distinct emotion.
The metric stack we used through Phase 9 (cosine + WER) couldn't
see this gap because it measures voice fidelity and text rendering,
not emotion class.
Hypothesized fixes (not yet tested):
- CREMA-D extraction (91 actors vs RAVDESS 24) for class-purer
steering vectors
- Mixed backbone+decoder steering (backbone for prosody)
- EmoNet classifier (TTS-aware, may give different verdicts)
Doc'd in emotional_speech_guide.md as a known limitation. Closes
out an honest scientific picture: today's work successfully ports
the architectural finding (decoder route preserves text), but
class-precise emotion control remains unsolved.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Wires emotion2vec into quality_eval so per-row metrics include
target_emotion_prob, top_emotion, top_emotion_prob. Pure inference,
optional via --emotion2vec / --target-emotion flags.
Critical empirical finding documented in code + user guide: the
emotion2vec_plus_base checkpoint classifies every input as
"Surprised" with prob ≥ 0.99, INCLUDING ground-truth RAVDESS clips
with explicit emotion labels. Real angry-RAVDESS → "Surprised"
(0.9999999). Real neutral-RAVDESS → "Surprised" (0.9999996).
The metric implementation is correct (matches the trait's
EmotionDetector::classify code path with same per-utterance zero-
mean unit-variance normalization); the underlying classifier
collapses to a dominant class on most input — likely the same
"9→5 fold collapse" the project already documented in the data-
labeling path.
Practical implication: target_emotion_prob is near-zero for almost
every (target, output) that isn't "surprised", so it can't be used
as a picker score. The emotion2vec metric still works as a
diagnostic ("did the model produce something that classifies as
audio at all?") but not as a generation-quality validator.
Doc'd in:
- examples/quality_eval.rs CLI doc (caveat block on --emotion2vec)
- docs/emotional_speech_guide.md (Known limitations section with
full sanity-check table)
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Brief addition to the recipe section explaining the WER + length
floor scoring used by emotional_speech_n.sh (committed in b7b267b).
Validates that the new scoring preserves canonical winners on
happy and calm while flipping surprised to the long-and-correct
candidate.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
3-seed picker run (7, 42, 100) on calm/disgust/surprised refines the
single-shot characterization:
calm: winner seed=100, WER 0.64
"It's a good reflection. Not that that. I want to
share somet..."
(recipe lands the prompt — single-shot at seed 42 only
produced hesitation markers; the picker found a seed
with actual content)
disgust: no reliable seed
(all 3 seeds WER ≥ 0.93; likely RAVDESS corpus issue —
disgust clips are low-energy / acoustically close to
neutral. Try CREMA-D or ESD for this emotion.)
surprised: picker chose seed=7 (WER 0.93, short "You can.") over
seed=100 (WER 1.0, "...Today I want to share something")
— WER weighting issue: deletions and insertions count
uniformly, so terse-but-mostly-wrong beat long-and-
mostly-right. Manual selection or weighting WER less
heavily would help here.
Updated per-emotion table marks disgust as ✗ (corpus limitation),
surprised as ⚠ (picker scoring artifact), calm as ✓ (works with
N-seed picker).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Extracted decoder steering vectors for the remaining 3 RAVDESS
emotions (calm, disgust, surprised). Single-seed bench at
seed=42 on Amini context, decoder route, recipe defaults:
emotion cos_ctx WER transcript
calm 0.70 0.86 "I'm sorry. Um, I don't know."
(natural hesitation markers — the
recipe produces semantically-emotion-
matched content, not just acoustic
shift)
disgust 0.81 0.93 "For that, that..." (truncated)
surprised 0.95 2.57 "Too couple, sorry, and that's saying,
even a premier and super driver..."
(long rambling; voice migrates well,
text drifts)
All 7 RAVDESS emotions now produce coherent English on the decoder
route — calm is solid first-shot, disgust truncates, surprised
rambles. Roll N seeds via emotional_speech_n.sh for the latter two.
emotional_speech_guide.md updated with the per-emotion table now
covering all 7. Voice character preservation (cos vs context > 0.7)
holds for every emotion in the pack.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Phase 9 produced 25 commits and a complex pipeline; perf_history is
the engineering log but new users coming to this cold need a clean
"how do I make CSM speak with emotion" handbook.
Sections:
- What this gets you (single-shot WER 0.07–0.21, voice cosine ≥ 0.95)
- One-liner quickstart (RAVDESS download → extract → use)
- The recipe explained — every flag and why it's there
- Per-emotion notes (works/best-seed/caveats per emotion)
- When it works / when it doesn't
- Troubleshooting (music tokens, premature EOT, repetition, etc.)
- Architecture cheat sheet (backbone=semantic, decoder=acoustic)
References perf_history.md for the full empirical log; this doc is
the user-facing distillation.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Cross-emotion × Carlini context at Amini-best seeds reveals the
"magic combo" doesn't fully transfer:
happy@42 Amini WER 0.21 → Carlini WER **0.071** (transfers!)
angry@100 Amini WER 0.93 → Carlini WER 1.21 (URL drift)
fearful@7 Amini WER 0.86 → Carlini WER 1.00 ("Screw it")
sad@7 Amini WER 0.93 → Carlini WER 1.00 (no transcript)
Only happy@42 cleanly generalizes across contexts. The previous
"context-robust" claim was too strong — the (emotion, seed, context)
interaction matters. Cosine vs context stays high for angry (0.95)
even when text drifts, so voice character preservation is the more
robust property than text fidelity.
Honest production interface: `emotional_speech_n.sh` rolling 5 seeds
per (context, prompt). The single-shot recipe lands well only when
all dimensions align, but the picker absorbs the variance.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
3-context bench at happy@seed=42 decoder recipe (Amini / MCC /
Carlini). Cosine measured vs the context wav (does recipe preserve
input voice character?), WER vs prompt:
context cos_ctx WER transcript
amini 0.972 0.21 "All right, today I want to share
something with you that I've been
thinking about."
mcc 0.58 0.93 "You" (sub-speaker mismatch)
carlini 0.958 0.071 ⭐ "So today I want to share something
with you that I have been thinking
about."
Carlini's WER 0.071 is the new single-shot best of Phase 9. Only
prefix "So" added to the verbatim prompt. Cos vs context > 0.95 on
the two working contexts means the recipe preserves speaker
character of the reference — does NOT impose RAVDESS speaker
identity on every output.
The recipe is context-robust on speaker identities the picker
selects correctly. McConaughey failed because we picked the
manifest's spk1 (likely the Oscars announcer), not McConaughey
himself. That's a context-selection issue, not a recipe issue.
Empirical capstone: single-shot near-verbatim emotional speech
with preserved voice character is achievable.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
3-prompt × 2-condition bench at happy@seed=42 decoder recipe vs
no-steering baseline. The recipe holds across prompts and produces
the lowest WER recorded across all Phase 9 experiments:
prompt base happy decoder
Today I want to… 0.52 / 0.93 "You" 0.82 / 0.21 ⭐
Have you ever… 0.72 / 1.39 drift 0.77 / 1.00 "With blames"
Weather has been… 0.73 / 1.88 ♪♪♪ 0.69 / 0.125 ⭐ "That the
weather has been
absolutely beautiful
this mor"
The imperative prompt's WER 0.125 is the lowest recorded.
Improvement vs baseline ranges 1.4× to 15× lower WER. The
no-steering baseline produced literal singing tokens (♪♪) on the
weather prompt, suggesting CSM's CFG-only path is fragile on
prompts the model "dislikes."
Empirical conclusion: decoder route + RAVDESS happy steering at
seed 42, scale 1.0, layers [2,3] is a reproducible recipe, not a
single-prompt anomaly. N-seed picker still the right consumer
interface, but this single configuration alone reaches
near-publishable quality on multiple prompts.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Records the breakthrough single-shot result: happy@seed=42,
decoder route, scale 1.0, layers [2,3] →
"All right, today I want to share something with you tha"
(WER 0.21, cos 0.82). Closest-to-perfect single-condition
result of the entire Phase 9 sprint.
Per-emotion seed winners diverge:
happy=42, angry=100, fearful=7, sad=7
No universal best seed exists; this validates emotional_speech_n.sh
as the production interface (rolls multiple, picks lowest WER).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Mirror of --steering-layers for the decoder: comma-separated layer
indices to actually steer (decoder has 4 layers; useful subsets are
[3], [2,3], [1,2]).
Sweep at seed 7, [email protected]:
[0,1,2,3] cos 0.86 WER 0.93 "I'm not that tall ×3" ← repetition
[3] cos 0.58 WER 0.86 "I'll be off and offense..."
[2,3] cos 0.80 WER 1.43 "My daughter, Penny Ryan, and I have…"
[0,1] cos 0.82 WER 1.57 "And I'll check on them..."
[1,2] cos 0.81 WER 1.43 "I'm going to call him an X-Man..."
The repetition is specific to all-layers-at-once steering. Any 2-layer
subset eliminates it while preserving most of the cosine boost. Same
pattern as the backbone's [8,10,12] finding: partial perturbation
lets the unsteered layers act as a stabilizing prior.
[2,3] (decoder last 2) is the new recommended recipe — best cosine
of the no-repetition subsets and the longest fluent transcript.
Documented in docs/perf_history.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
3-seed × 2-condition ([email protected] alone vs +CFG) bench plus a 4-step
scale sweep. Captures the honest tradeoff:
- Backbone steering destroys word content (semantic gibberish).
- Decoder steering preserves coherent English BUT produces
repetition or premature EOT.
Neither produces single-shot production-quality emotional speech;
emotional_speech_n.sh (N-seed picker, lowest-WER wins) remains the
right consumer interface — it doesn't care which failure mode
generated the bad samples, just discards them by metric.
Decoder vector magnitudes are ~10× smaller than backbone (norm 0.85
at deepest layer vs 14.9), so the apparent useful scale window is
~10× higher (0.5-1.0 instead of 0.2-0.3).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
3-seed × 3-prompt reproducibility bench on [email protected] reveals that the
recipe shifts speaker character reliably but produces high text-
fidelity variance:
seed 42: WER 2.00 "The police are, if you're, I can't recite..."
seed 7: WER 0.71 "Today, today I want to share..." ← near-verbatim
seed 100: 0.32 s premature EOT
Cross-prompt at seed 42 drifts uniformly across 3 prompts. Speaker
cosine is consistently elevated; text content is roll-the-dice.
Documenting this as the honest characterization rather than overclaim
the single-seed Sprint 2 results. Practical recipe: roll N seeds,
pick lowest-WER output.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Capstone wrapper for the Phase 9 recipe (Selective CFG + RAVDESS
steering + mid-layer subset). Picks the steering scale automatically
from an empirical per-emotion map:
- happy: 0.30
- angry: 0.20
- fearful: 0.20
- sad: 0.20 (note: sad is unreliable — see below)
These came from a follow-up sweep after the multi-emotion demo
revealed the recipe is emotion-sensitive: scale 0.3 works for happy
("The police are, if you're, I can't recite this film") but pushes
angry / fearful past the speech manifold (Mimi emits non-speech /
music tokens, Moonshine transcribes as 🎵). Dropping to 0.2 recovers
fluent speech for both:
- [email protected]: "The next disorder is completing kashim for more."
- [email protected]: "You just heard a little bit about this decision,
though."
- [email protected]: "The police are, if you're, I can't recite this
film. I"
Sad is the outlier — model resists "sad" steering at every scale
between 0.15 and 0.3. Likely a corpus issue (sad RAVDESS clips are
the lowest-energy subset). Documented as a known limitation rather
than worked around.
perf_history.md updated with the per-emotion sensitivity finding.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Sweep of 4 schedules × 4 prompts produced concrete data on schedule
sensitivity. step:3.0:1.5:12 (the Sprint 3 winner) catastrophically
fails dense prompts: lecture-style input → 0.08 s of audio (one
frame). step:2.0:1.0:8 produced the best single shot — near-verbatim
question rendering "Well, it had stem-wondered. Have you ever
wondered why we sometimes hear voices the way we do?" — but tanked
the lecture prompt (2.4 s "You").
linear:3.0:1.0:25 is the only schedule that's never the best AND
never the worst — graceful degradation across all four prompt
categories. Updates the recommended recipe in perf_history.md
(formerly step:3.0:1.5:12).
quality_eval: skip WavLM-SV scoring on clips shorter than 0.25 s
(emit -1 sentinel) — WavLM-SV's TDNN front-end requires a few
hundred samples and crashed mid-sweep on the 0.08 s clip. Now the
eval emits a row instead of bailing on the whole batch.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Full port of facebook/wav2vec2-base-960h (94.4 M params, MIT) closing
the WhisperX-class word-alignment gap from the audio-ML survey. Same
staged-scaffolding pattern that worked for emotion2vec — but landed
slices 1+2+3 in one session.
src/wav2vec2.rs ships:
- Wav2Vec2Config::base_960h
- FeatureExtractor — 7 Conv1d (1→512, total stride 320). Layer 0
uses GroupNorm with num_groups=num_channels=512 (HF's wav2vec2
feat_extract_norm: "group"). Critical: state-dict key is
layer_norm.* but the OP is GroupNorm — loading as LayerNorm
produces empty CTC output.
- FeatureProjection — LayerNorm(512) + Linear(512→768)
- ConvPosEmbedding — kernel 128 grouped Conv1d, materialized at
load time from upstream weight_g + weight_v (fairseq's weight_norm
on dim=2; eps-guarded division for numerical stability)
- Block — POST-norm transformer with separate Q/K/V (vs emotion2vec's
fused QKV), uses (B*H, T, D) Metal 3D-matmul workaround from
Phase 8.8 Moonshine
- Encoder — pos_conv + initial LayerNorm + 12 Blocks
- Wav2Vec2 top-level — load_from_safetensors via mmap'd VarBuilder
- ctc_greedy_decode + VOCAB_960H constant for the 32-char alphabet
examples/wav2vec2_inspect.rs (slice 1): dumps tensor layout + config
examples/wav2vec2_smoke.rs (slice 3): real-weight load + ASR forward
Verified on Metal:
loaded model in 0.28 s
forward in 9 ms for 10.42 s audio (~1150× realtime)
transcript: "HE HOPED THERE WOULD BE STEW FOR DINNER TURNIPS AND
CARROTS AND BRUISED POTATOES AND FAT MUTTON PIECES TO
BE LADLED OUT IN THICK PEPPERED FLOWER FAT AND SAUCE"
Numerical parity with upstream Python — the FLOWER-for-FLOUR typo is
the known wav2vec2-base-960h failure mode, matches HF reference exactly.
7 new unit tests; lib suite 127/127 (was 120).
Slice 4 remaining: Viterbi forced alignment given known transcript,
to emit (token, frame_start_ms, frame_end_ms) for word-boundary cuts.
The ASR path itself is now production-ready.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
The hobby-project sibling of maya_finetune_analysis.md. For the case
where you want the most emotionally responsive voice ever, for personal
use only (not distributed, not commercial), using all media formats
legitimately accessible to one person.
Key practical guidance:
Best ROI sources: audiobooks (200-500 hr easy) > podcasts (100 hr per
weekend) > anime/game VA reels (the gold mine for extreme emotional
range) > YouTube > movies. Skip Reddit clips. SKIP TTS-synthesized
data (mode collapse).
Data prep pipeline (specific tool choices):
yt-dlp -> Demucs v4 htdemucs_ft -> Silero V5 VAD (we have the
candle port already) -> WhisperX (the right answer for
diarization+ASR+alignment, don't roll your own) -> DNSMOS quality
gate -> single-speaker filter -> resample 24 kHz -> Mimi tokenize
Emotion labeling: emotion2vec+ as primary auto-tagger, GPT-4o or
Claude as LLM-as-judge for the 5-10% you'll actually train on
(~$50/100hr), hand-label 200 clips for Cohen's kappa validation.
Plus implicit conditioning on previous-turn audio (what Sesame
likely did). Do BOTH.
Training recipe (100-200 hr corpus, single A100/H100):
- LoRA: extend from q+v to q,k,v,o + MLP gate/up/down. r=32-64.
- Curriculum: audiobooks (3 ep clean) -> podcasts (1 ep) -> VA/
movies (1 ep, lower LR). Prevents messy data destabilizing
acoustic priors.
- One LoRA, multiple emotion control tokens. Per-emotion LoRAs
can't switch fast enough at inference.
- 5-10% mix-in of EmoV-DB/ESD/MEAD/RAVDESS. Not more.
Reality check:
60-120 focused hr -> "clearly better than base CSM in your domain"
300+ hr -> "genuinely beats Maya for me"
Biggest trap: spending 80% of time on data, 15% on training infra,
5% on actually listening critically. Listening is where the model
gets good. Set a rule: every checkpoint, 20 prompts + notes.
Second trap: training on TTS-synthesized data. Mode collapse.
Where motivation dies: hour 40 when WhisperX diarization fails on
a podcast and you spend a Saturday debugging pyannote.
Going BEYOND Maya:
- GoEmotions 28-label taxonomy + V/A continuous (5x5 = 25 pseudo)
- Multi-persona via 512-d persona embeddings (YourTTS pattern)
- Reactive emotion: emotion2vec+ on user audio at inference,
feed as control token. ~50 ms latency. Feasible today.
Concrete Phase 12 candidates (bounded codable items, NOT the data
collection itself):
1. Extend rtx-csm LoRA coverage q+v -> k,o,MLP (~1-2 hr)
2. Wire WhisperX as scripts/ data-prep step (Python sidecar)
3. emotion2vec+ via ort sidecar, JSON labels
4. Emotion control token plumbing in Generator::generate
5. Curriculum trainer examples/lora_train_emotional.rs
Papers cited: CosyVoice 2, Voicebox, NaturalSpeech 3, emotion2vec+,
Spirit-LM. Tools: yt-dlp, Demucs v4, WhisperX, pyannote 3.x, Silero V5,
DNSMOS, GoEmotions taxonomy.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Extended docs/maya_finetune_analysis.md with the user's "train on
Hollywood scenes that denote emotions" angle. The instinct is sound;
the legal pitfalls are severe; legitimate alternatives exist.
Headline:
DON'T train on Hollywood movies. Copyright + right-of-publicity is
uninsurable for a shipped product (NYT v OpenAI / Andersen v
Stability / RIAA v Suno / Johansson v OpenAI "Sky" all 2024-2026).
Source separation works technically; the law doesn't.
Legitimate corpora that capture the same "actors performing emotion"
property (matrix added to the doc):
Commercial-clean (use these):
EmoV-DB 7 hr / 4 spk CC-BY 4.0 — explicit laughs/yawns
CREMA-D 5 hr / 91 spk ODC-By 1.0 — read but emotion-tagged
DailyTalk 20 hr / 2 spk CC-BY-SA — dyadic conversational
LAION Emo Speech ~5000 hr CC-BY 4.0 — but provenance risk
Hume Prosody proprietary paid commercial
Research-only (skip for shipped product):
Expresso (Meta) 47 hr / 4 spk CC-BY-NC — best quality
IEMOCAP 12 hr / 10 spk academic — best emotional range
MELD (Friends) 13 hr Warner Bros — audio is copyrighted
RAVDESS, ESD small/medium research
New "Path D" recipe added:
Stage 1 (~6 hr GPU on H100):
- EmoV-DB + CREMA-D combined (~12 hr, commercial-clean)
- LoRA r=8 α=16 on q+v +k+o + decoder cross-attn
- 3 epochs, lr 1e-4 cosine, bf16
Stage 2 (~weekend, 5-10 hr recording):
- One voice actor improvising LLM-prompted dialogue
- Stage-2 LoRA r=16 on the stage-1 checkpoint
Per the corpus research: gets ~70% of Maya's emotional
expressiveness, ~30% of her personality. The single-speaker stage 2
is the "uncanny news anchor doing feelings" -> "specific persona"
overlay. Crucially this is a WEEKEND with one actor, not the 40-hr
studio sprint Sesame did.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Honest research into what Sesame likely did to fine-tune the open
CSM-1B base into the deployed Maya/Miles persona. Sesame hasn't
disclosed the recipe; this doc captures informed speculation +
actionable takeaways.
Key findings:
1. Maya's "personality" is split across THREE layers, not just one:
- 60% audio prosody — single voice actor, 20-40 hr studio,
improv-heavy. Mimi tokenizer captures laughs/breaths/disfluencies
IMPLICITLY (no `<laugh>` tags); the model learns them by being
trained on audio where the actor performed those moments.
- 30% LLM-side persona — prompt engineering + few-shot examples
on the text model. NOT a voice-model property at all.
- 10% conversational dynamics — VAD + endpointing + barge-in +
streaming TTS. We're already at parity here.
2. Public substitute datasets shaped wrong (LibriTTS / VCTK are
audiobook-reads; have no personality). Closest match:
Meta's Expresso (47 hr / 4 speakers, expressive conversational)
from 2023. Worth investigating if we ever pursue real Maya-class
prosody.
3. Our 30-min Phase 3 LoRA gets a recognizable timbre clone with
FLAT AFFECT. Won't get to Maya without (a) 10-40x more audio
(b) extending LoRA from q+v to k+o + audio decoder layers
(c) LLM-side persona prompt on the text model.
Three concrete next chunks captured:
A. LLM-side persona prompt (~30 min, biggest ROI/minute)
B. Extended LoRA coverage (~1-2 hours)
C. Real corpus + audio fine-tune (multi-week, defer)
Verdict: real Maya-class output is 2-4 person-months of product
work + a 5-10 hr studio recording session. The IP gap is real and
not closeable with documentation alone. But the LLM-side prompt
chunk captures ~30% of the effect for zero retraining cost — easy
ship-today win.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Phase 8.1.3 deferred Silero V5 VAD because the only Rust crate path
(`voice_activity_detector` via `ort`) collides with `sentencepiece-sys`
on protobuf 3.14 vs 3.21 and panics at process startup. This commit
closes that gap with a NATIVE candle port.
Direct port from `Snakers4/silero-vad/src/silero_vad/tinygrad_model.py`
(71 LOC reference). Architecture:
stft_conv Conv1d(1, 258, k=256, s=128) no bias
conv1 Conv1d(129, 128, k=3, p=1)
conv2 Conv1d(128, 64, k=3, s=2, p=1)
conv3 Conv1d(64, 64, k=3, s=2, p=1)
conv4 Conv1d(64, 128, k=3, p=1)
lstm_cell LSTMCell(128, 128)
final_conv Conv1d(128, 1, k=1)
Forward: reflect-pad input by 64, STFT-as-conv1d, sqrt(real² + imag²),
4-layer Conv1d feature stack with ReLU, single LSTM step (state across
chunks), 1x1 conv + sigmoid -> speech probability.
Files added:
src/silero_vad.rs ~310 LOC (incl. LSTM cell + downloader)
docs/silero_vad_port_notes.md architecture + port plan
examples/silero_vad_smoke.rs real-audio discrimination test
Plus a new `ureq` direct dep (transport already pulled in via hf-hub).
Weights ship via download-on-first-run from the upstream GitHub raw
URL into `~/.cache/rtx-csm/silero_vad_16k.safetensors` (1.24 MB). No
repo bloat; no .gitignore wrestling.
End-to-end smoke (synthetic 50/50 silence/speech WAV at 16 kHz):
load (cold): download + parse, < 100 ms after first run
VAD sweep: 170 ms over 9.99 s of audio = 0.017x realtime (59x faster)
unit test: passes (load weights + run one step)
Probability output (per 32 ms chunk):
0-1.5 s: p ~ 0.01-0.07 silence
1.5-5 s: p ~ 1.000 speech (clean ramp at speech onset)
5-10 s: p ~ 0.001 silence
Speech-chunk fraction 33% on the 50/50 layout — matches expected.
Production angle: dramatically better silence/speech discrimination
than the Phase 8.1.3b energy VAD (which only catches obvious silence).
Silero V5 catches whisper-quiet speech, breath/lip noise, music vs
speech distinction. Drop-in candidate for `--vad-gate` in a future
iteration.
The ort/protobuf conflict that blocked this for two months is now
permanently resolved by NOT using ort.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Captures the Phase 10 work in the consolidated perf doc. New section
"Watermarker backend matrix" lists AudioSeal (Meta) and SilentCipher
(Sesame's actual) side by side with measured RTF, capacity, conflicts,
and a per-use-case recommendation table.
Headline: SilentCipher is Sesame's literal production watermarker, now
shipping in pure candle with bit-perfect round-trip on real LibriSpeech
audio (15/15 codes, confidence 1.0000) and ~10× smaller than AudioSeal
(~3M params vs ~30M). The "blow them out of the water" item from the
Sesame gap analysis is closed.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Foundation for porting Sesame's actual production watermarker (NOT
AudioSeal — the gap analysis identified this as the literal Sesame
parity item). Same iterative-shipping pattern as Phase 8.4 for
Moonshine.
`docs/silentcipher_port_notes.md`:
- Full architecture from SesameAILabs/silentcipher/src/.../model.py
(verified against 95 LOC of source)
- Three small networks of gated 2D convs on STFT:
enc_c 3 layers 1 -> 32 channels
dec_c 4 layers 96 -> 1 channels
dec_m 10 layers 1 -> 128 -> message_dim, plus Linear
- Each Layer = Conv2d * sigmoid(Conv2d) + BatchNorm2d
- Pipeline (encode + decode) walked through step by step
- 10 ordered porting tasks with hour estimates totaling ~1-2 days
- Risks flagged: STFT helper needed, BatchNorm running stats loading,
phase passthrough, message-length differences vs AudioSeal
`examples/silentcipher_inspect`:
- Downloads sony/silentcipher 16 kHz checkpoint from HuggingFace
- Dumps hparams.yaml + tensor shapes per .ckpt file
- Verified output:
N_FFT 2048 HOP 1024 SR 16000
message_dim 4 message_len 16 message_band 512
enc_c 0.17 MB 40 k params
dec_c 2.01 MB 500 k params
dec_m_0 9.54 MB 2.38 M params
Total ~2.92 M params
That's ~10x smaller than AudioSeal's gen+det combined. Port
estimated 1-2 days.
`.ckpt` files are pickle (PyTorch state_dict) — direct loadable via
candle_core::pickle::read_all, same path as audioseal_convert.rs.
No safetensors conversion needed.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Researched what Sesame has actually disclosed publicly (vs. marketed)
and compared systematically against rtx-csm's shipped surface.
Key findings:
1. **rtx-csm has shipped a SUPERSET of Sesame's open release.** CSM-1B
inference + voice cloning + Q8 + production WebSocket server +
three-backend STT pipeline + per-phase observability — Sesame ships
inference code only.
2. **Sesame's deployed Maya is a cascaded STT->LLM->TTS pipeline**,
same architecture pattern as rtx-csm. Their research blog explicitly
states future work is "fully duplex models" — Maya today isn't
duplex either. We're structurally equivalent at the pipeline level
to Kyutai Unmute, Sesame's closest peer.
3. **Marketing latency claims are unverified.** "Sub-200 ms TTFA"
appears in third-party blogs, not in any Sesame paper. Production
benchmarks of similar cascaded stacks show 250-300 ms TTFT
(gpt-realtime, Unmute) — our 280-380 ms TTS-side is competitive.
Z.AI provider TTFT (~1.3 s) is the dominant cost in our 1.96 s
end-to-end.
4. **Critical correction**: Sesame ships SilentCipher (their fork of
Sony's), NOT AudioSeal (which is Meta's). Our Phase 4 AudioSeal
work is functionally equivalent but isn't the *literal* Sesame
watermarker. SilentCipher port is ~1-2 days.
5. **What's gated on Sesame**: CSM-3B / CSM-8B variants (trained but
never released), Maya personality fine-tune dataset, distilled
wearable variant. The Oct 2025 Series B + smart-glasses pivot
suggests they're unlikely to release any of these.
Punch-list of remaining gaps captured in the doc with status (Closed/
Partial/Open/N/A) per capability.
Recommended next chunks (prioritized):
1. SilentCipher port (~1-2 days) — literal Sesame watermarker parity
2. clawsample-csm integration (~5-10 days) — separate plan exists
3. Tier 2.1 VoXtream look-ahead (~5-7 days) — diminishing returns
after Phase 9.2's chunk_frames tuning
4. Tier 3 Frame-Stacked / VADUSA (training-required, ~3-6 weeks)
5. Distilled CSM (speculative, wait for product target)
Defensible framing: "rtx-csm is Sesame's open release + voice cloning
+ production HTTP/WS server + Kyutai-style cascaded duplex. Remaining
gap to internal Maya is (a) SilentCipher watermarker, (b) Sesame's
proprietary fine-tune dataset, (c) larger CSM variants Sesame chose
not to release."
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Phase 9.1 is the foundation step for the Tier 2.1 (VoXtream-style
look-ahead) optimization from the Phase 8 plan. Same pattern as Phase
8.4: don't attempt the full port in one session; ship the architecture
analysis + ecosystem audit + ordered porting plan so a future session
starts cold from concrete data.
Phonemizer ecosystem audited (results in the doc):
Crate Approach Latency Conflicts
espeak-ng 0.1.1 pure Rust eSpeak NG 1-5 ms/word none <- PICK
voirs-g2p 0.1.0-rc.1 neural, candle 0.9.2 10-50 ms/w libc dep
grapheme_to_phoneme 0.1.0 seq2seq RNN ARPAbet 2-8 ms/word none (abandoned 2020)
phonetisaurus-g2p 0.1.1 FST sub-ms none (no pre-trained FST)
espeak-ng is the right pick: zero C linkage default, no conflicts with
candle 0.9 / sentencepiece-sys / hound / symphonia / ebur128, mature.
Per-word latency fits VoXtream's 102 ms first-packet target with room
to spare (5-10 word lookahead = 5-50 ms total phonemizer cost).
Doc captures:
- Why this is worth doing (current ~600 ms TTS first chunk vs 102 ms
claim from arXiv 2509.15969)
- Three-piece integration architecture (phonemizer service, look-ahead
window in Converse, optional Generator-side phoneme hint)
- Three open questions that MUST be resolved before implementing
(mechanism, CSM training compatibility, empirical win on this HW)
- Six ordered porting tasks with hour estimates totaling 5-7 days
- Alternative quick win: drop streaming chunk_frames from 4 to 2 or 1
first; if that closes most of the gap, full VoXtream port may not
be worth the complexity
Recommended order of attack written for the next session: try the
chunk_frames tuning before implementing the phonemizer. The cheapest
move is a 1-line config change.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Updated docs/perf_history.md to reflect the full Phase 8 work:
- New TL;DR has TWO production configs (English-only with --moonshine,
multilingual with default Kyutai). The English path is the new
headline recommendation.
- Captured live Z.AI bench numbers from the new production stack
(Q8 + stream + warmup + Moonshine + thinking-disabled): total_turn
8765 ms vs Phase 6f.q8 era 17145 ms — half the wall-clock latency
end-to-end.
- Phase 8 commit table extended to 8.4 through 8.10.
- Added an STT backend matrix (Kyutai 1B / Whisper-rs / Moonshine)
with measured RTF, build flags, and per-deploy recommendation rows.
Bench command used (single turn, real Z.AI glm-4.5 thinking-disabled,
10.43 s LibriSpeech /tmp/asr_test.flac):
target/release/examples/converse_server \\
--bind 127.0.0.1:18099 \\
--quantized-gguf /tmp/csm_q8.gguf \\
--stream-tts --moonshine \\
--llm-base https://api.z.ai/api/coding/paas/v4 \\
--llm-model glm-4.5 \\
--llm-extra-body '{"thinking":{"type":"disabled"}}'
Server-side timing:
recv_phase 0 ms (Moonshine batch — no parallel STT)
stt_post 332 ms (Moonshine transcribe at EOT)
llm_to_first_audio 1627 ms (Z.AI TTFT ~1.3 s + first TTS chunk)
conv_total 8432 ms
total_turn 8765 ms
Client TTFA: 1959 ms (vs Phase 6f.q8 era ~2 s — comparable; the dominant
remaining latency is the Z.AI provider TTFT, not anything we control).
Z.AI returned a coherent reply: "He eagerly anticipated a hearty stew
with turnips, carrots, potatoes, and savory mutton pieces for dinner."
matching the LibriSpeech ground-truth meaning.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
First step on the Tier 2.2 (Moonshine v2 candle port) item from the
Phase 8 plan. Full port is honestly multi-session work (~12-15 hours
of focused implementation across encoder, decoder, generation loop,
tokenizer, weight mapping, smoke test). This commit ships the
foundation so future sessions start from concrete data instead of
arxiv reading.
Two ships:
1. examples/moonshine_inspect — downloads UsefulSensors/moonshine-tiny
from HF, parses safetensors header, dumps all 160 tensors grouped
by prefix with shapes + dtypes. Verified output: 27.1 M params,
108.4 MB safetensors (F32), encoder + decoder layers laid out as
expected.
2. docs/moonshine_port_notes.md — captures every architectural fact
established by the inspector + HF config.json:
- Hyperparameter table (hidden=288, 6+6 layers, vocab=32768,
partial_rotary=0.9, etc.)
- Tensor layout per layer (encoder, decoder)
- Architecture summary (raw waveform input, 3-layer Conv1d stem,
SwiGLU decoder MLP via fused fc1, tied LM head)
- Ordered porting tasks with hour estimates totaling ~12-15 h
- Risks / unknowns (conv strides not in config, tied output head
question, quality-vs-Kyutai concern)
- Recommended order of attack for the next session
The full port itself is deferred. Ship the foundation now so the
remaining work has a clean handoff.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Consolidated record of every shipped optimization, every rejected path
with the data behind the rejection, the production-recommended config,
the architectural lessons captured, and the deferred multi-session
work with realistic sizing.
Headline:
- client first_audio_ms p50 = 529 ms (mock LLM, Q8 + stream + extended warmup + VAD gate)
- real Z.AI loop: ~2 s TTFA p50
- boot cost: ~2.7 s (one-time)
Rejected paths captured (so future sessions don't redo the work):
Q4_K_M (2.85x slower than Q8), whisper-rs linkage (2-3x CSM regression),
Silero V5 via ort (protobuf 3.14 vs 3.21 conflict), Mimi codec Q8
(no candle conv-quant path), KV cache reuse (variance is content-
dependent not state-dependent), rayon for single-connection
(overhead exceeds gain on <100µs tasks), codec swaps (require
backbone retrain), custom distillation (no published checkpoint),
Kyutai 4x flush (hardware-bound on M-series).
Architectural lessons:
1. In-process linkage of external ML runtimes is a recurring trap;
default to sidecar-process pattern.
2. Bench thermals dominate single-machine A/B; 90s cooldown often
necessary.
3. First-frame compilation is the dominant cold-start cost — long
warm-ups are essential.
4. Conv-phase variance is content-dependent, not state-dependent.
5. tokio::join! polls cooperatively — spawn separate tasks for real
concurrency between sync compute and async pump.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Detailed plan for exposing rtx-csm as a managed service on the
clawsample platform: crate layout (mirrors clawsample-demucs +
clawsample-gen), HTTP routes (/v1/tts, /v1/tts/async, /v1/voice_profile,
WS /v1/converse), DB schema, R2 paths, webhook dispatch, ordered TDD
task breakdown, open decisions, acceptance criteria, reference commits.
Estimated effort: 5-10 days. Trigger to start: a real consumer for the
public TTS API, OR standalone converse_server hits a hard ceiling.
Until then, integration is product/platform work, not ML work.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
