Cross-emotion × Carlini context at Amini-best seeds reveals the
"magic combo" doesn't fully transfer:
happy@42 Amini WER 0.21 → Carlini WER **0.071** (transfers!)
angry@100 Amini WER 0.93 → Carlini WER 1.21 (URL drift)
fearful@7 Amini WER 0.86 → Carlini WER 1.00 ("Screw it")
sad@7 Amini WER 0.93 → Carlini WER 1.00 (no transcript)
Only happy@42 cleanly generalizes across contexts. The previous
"context-robust" claim was too strong — the (emotion, seed, context)
interaction matters. Cosine vs context stays high for angry (0.95)
even when text drifts, so voice character preservation is the more
robust property than text fidelity.
Honest production interface: `emotional_speech_n.sh` rolling 5 seeds
per (context, prompt). The single-shot recipe lands well only when
all dimensions align, but the picker absorbs the variance.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
3-context bench at happy@seed=42 decoder recipe (Amini / MCC /
Carlini). Cosine measured vs the context wav (does recipe preserve
input voice character?), WER vs prompt:
context cos_ctx WER transcript
amini 0.972 0.21 "All right, today I want to share
something with you that I've been
thinking about."
mcc 0.58 0.93 "You" (sub-speaker mismatch)
carlini 0.958 0.071 ⭐ "So today I want to share something
with you that I have been thinking
about."
Carlini's WER 0.071 is the new single-shot best of Phase 9. Only
prefix "So" added to the verbatim prompt. Cos vs context > 0.95 on
the two working contexts means the recipe preserves speaker
character of the reference — does NOT impose RAVDESS speaker
identity on every output.
The recipe is context-robust on speaker identities the picker
selects correctly. McConaughey failed because we picked the
manifest's spk1 (likely the Oscars announcer), not McConaughey
himself. That's a context-selection issue, not a recipe issue.
Empirical capstone: single-shot near-verbatim emotional speech
with preserved voice character is achievable.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
3-prompt × 2-condition bench at happy@seed=42 decoder recipe vs
no-steering baseline. The recipe holds across prompts and produces
the lowest WER recorded across all Phase 9 experiments:
prompt base happy decoder
Today I want to… 0.52 / 0.93 "You" 0.82 / 0.21 ⭐
Have you ever… 0.72 / 1.39 drift 0.77 / 1.00 "With blames"
Weather has been… 0.73 / 1.88 ♪♪♪ 0.69 / 0.125 ⭐ "That the
weather has been
absolutely beautiful
this mor"
The imperative prompt's WER 0.125 is the lowest recorded.
Improvement vs baseline ranges 1.4× to 15× lower WER. The
no-steering baseline produced literal singing tokens (♪♪) on the
weather prompt, suggesting CSM's CFG-only path is fragile on
prompts the model "dislikes."
Empirical conclusion: decoder route + RAVDESS happy steering at
seed 42, scale 1.0, layers [2,3] is a reproducible recipe, not a
single-prompt anomaly. N-seed picker still the right consumer
interface, but this single configuration alone reaches
near-publishable quality on multiple prompts.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Records the breakthrough single-shot result: happy@seed=42,
decoder route, scale 1.0, layers [2,3] →
"All right, today I want to share something with you tha"
(WER 0.21, cos 0.82). Closest-to-perfect single-condition
result of the entire Phase 9 sprint.
Per-emotion seed winners diverge:
happy=42, angry=100, fearful=7, sad=7
No universal best seed exists; this validates emotional_speech_n.sh
as the production interface (rolls multiple, picks lowest WER).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Mirror of --steering-layers for the decoder: comma-separated layer
indices to actually steer (decoder has 4 layers; useful subsets are
[3], [2,3], [1,2]).
Sweep at seed 7, [email protected]:
[0,1,2,3] cos 0.86 WER 0.93 "I'm not that tall ×3" ← repetition
[3] cos 0.58 WER 0.86 "I'll be off and offense..."
[2,3] cos 0.80 WER 1.43 "My daughter, Penny Ryan, and I have…"
[0,1] cos 0.82 WER 1.57 "And I'll check on them..."
[1,2] cos 0.81 WER 1.43 "I'm going to call him an X-Man..."
The repetition is specific to all-layers-at-once steering. Any 2-layer
subset eliminates it while preserving most of the cosine boost. Same
pattern as the backbone's [8,10,12] finding: partial perturbation
lets the unsteered layers act as a stabilizing prior.
[2,3] (decoder last 2) is the new recommended recipe — best cosine
of the no-repetition subsets and the longest fluent transcript.
Documented in docs/perf_history.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
3-seed × 2-condition ([email protected] alone vs +CFG) bench plus a 4-step
scale sweep. Captures the honest tradeoff:
- Backbone steering destroys word content (semantic gibberish).
- Decoder steering preserves coherent English BUT produces
repetition or premature EOT.
Neither produces single-shot production-quality emotional speech;
emotional_speech_n.sh (N-seed picker, lowest-WER wins) remains the
right consumer interface — it doesn't care which failure mode
generated the bad samples, just discards them by metric.
Decoder vector magnitudes are ~10× smaller than backbone (norm 0.85
at deepest layer vs 14.9), so the apparent useful scale window is
~10× higher (0.5-1.0 instead of 0.2-0.3).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
3-seed × 3-prompt reproducibility bench on [email protected] reveals that the
recipe shifts speaker character reliably but produces high text-
fidelity variance:
seed 42: WER 2.00 "The police are, if you're, I can't recite..."
seed 7: WER 0.71 "Today, today I want to share..." ← near-verbatim
seed 100: 0.32 s premature EOT
Cross-prompt at seed 42 drifts uniformly across 3 prompts. Speaker
cosine is consistently elevated; text content is roll-the-dice.
Documenting this as the honest characterization rather than overclaim
the single-seed Sprint 2 results. Practical recipe: roll N seeds,
pick lowest-WER output.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Capstone wrapper for the Phase 9 recipe (Selective CFG + RAVDESS
steering + mid-layer subset). Picks the steering scale automatically
from an empirical per-emotion map:
- happy: 0.30
- angry: 0.20
- fearful: 0.20
- sad: 0.20 (note: sad is unreliable — see below)
These came from a follow-up sweep after the multi-emotion demo
revealed the recipe is emotion-sensitive: scale 0.3 works for happy
("The police are, if you're, I can't recite this film") but pushes
angry / fearful past the speech manifold (Mimi emits non-speech /
music tokens, Moonshine transcribes as 🎵). Dropping to 0.2 recovers
fluent speech for both:
- [email protected]: "The next disorder is completing kashim for more."
- [email protected]: "You just heard a little bit about this decision,
though."
- [email protected]: "The police are, if you're, I can't recite this
film. I"
Sad is the outlier — model resists "sad" steering at every scale
between 0.15 and 0.3. Likely a corpus issue (sad RAVDESS clips are
the lowest-energy subset). Documented as a known limitation rather
than worked around.
perf_history.md updated with the per-emotion sensitivity finding.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Sweep of 4 schedules × 4 prompts produced concrete data on schedule
sensitivity. step:3.0:1.5:12 (the Sprint 3 winner) catastrophically
fails dense prompts: lecture-style input → 0.08 s of audio (one
frame). step:2.0:1.0:8 produced the best single shot — near-verbatim
question rendering "Well, it had stem-wondered. Have you ever
wondered why we sometimes hear voices the way we do?" — but tanked
the lecture prompt (2.4 s "You").
linear:3.0:1.0:25 is the only schedule that's never the best AND
never the worst — graceful degradation across all four prompt
categories. Updates the recommended recipe in perf_history.md
(formerly step:3.0:1.5:12).
quality_eval: skip WavLM-SV scoring on clips shorter than 0.25 s
(emit -1 sentinel) — WavLM-SV's TDNN front-end requires a few
hundred samples and crashed mid-sweep on the 0.08 s clip. Now the
eval emits a row instead of bailing on the whole batch.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Captures the Phase 10 work in the consolidated perf doc. New section
"Watermarker backend matrix" lists AudioSeal (Meta) and SilentCipher
(Sesame's actual) side by side with measured RTF, capacity, conflicts,
and a per-use-case recommendation table.
Headline: SilentCipher is Sesame's literal production watermarker, now
shipping in pure candle with bit-perfect round-trip on real LibriSpeech
audio (15/15 codes, confidence 1.0000) and ~10× smaller than AudioSeal
(~3M params vs ~30M). The "blow them out of the water" item from the
Sesame gap analysis is closed.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Updated docs/perf_history.md to reflect the full Phase 8 work:
- New TL;DR has TWO production configs (English-only with --moonshine,
multilingual with default Kyutai). The English path is the new
headline recommendation.
- Captured live Z.AI bench numbers from the new production stack
(Q8 + stream + warmup + Moonshine + thinking-disabled): total_turn
8765 ms vs Phase 6f.q8 era 17145 ms — half the wall-clock latency
end-to-end.
- Phase 8 commit table extended to 8.4 through 8.10.
- Added an STT backend matrix (Kyutai 1B / Whisper-rs / Moonshine)
with measured RTF, build flags, and per-deploy recommendation rows.
Bench command used (single turn, real Z.AI glm-4.5 thinking-disabled,
10.43 s LibriSpeech /tmp/asr_test.flac):
target/release/examples/converse_server \\
--bind 127.0.0.1:18099 \\
--quantized-gguf /tmp/csm_q8.gguf \\
--stream-tts --moonshine \\
--llm-base https://api.z.ai/api/coding/paas/v4 \\
--llm-model glm-4.5 \\
--llm-extra-body '{"thinking":{"type":"disabled"}}'
Server-side timing:
recv_phase 0 ms (Moonshine batch — no parallel STT)
stt_post 332 ms (Moonshine transcribe at EOT)
llm_to_first_audio 1627 ms (Z.AI TTFT ~1.3 s + first TTS chunk)
conv_total 8432 ms
total_turn 8765 ms
Client TTFA: 1959 ms (vs Phase 6f.q8 era ~2 s — comparable; the dominant
remaining latency is the Z.AI provider TTFT, not anything we control).
Z.AI returned a coherent reply: "He eagerly anticipated a hearty stew
with turnips, carrots, potatoes, and savory mutton pieces for dinner."
matching the LibriSpeech ground-truth meaning.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Consolidated record of every shipped optimization, every rejected path
with the data behind the rejection, the production-recommended config,
the architectural lessons captured, and the deferred multi-session
work with realistic sizing.
Headline:
- client first_audio_ms p50 = 529 ms (mock LLM, Q8 + stream + extended warmup + VAD gate)
- real Z.AI loop: ~2 s TTFA p50
- boot cost: ~2.7 s (one-time)
Rejected paths captured (so future sessions don't redo the work):
Q4_K_M (2.85x slower than Q8), whisper-rs linkage (2-3x CSM regression),
Silero V5 via ort (protobuf 3.14 vs 3.21 conflict), Mimi codec Q8
(no candle conv-quant path), KV cache reuse (variance is content-
dependent not state-dependent), rayon for single-connection
(overhead exceeds gain on <100µs tasks), codec swaps (require
backbone retrain), custom distillation (no published checkpoint),
Kyutai 4x flush (hardware-bound on M-series).
Architectural lessons:
1. In-process linkage of external ML runtimes is a recurring trap;
default to sidecar-process pattern.
2. Bench thermals dominate single-machine A/B; 90s cooldown often
necessary.
3. First-frame compilation is the dominant cold-start cost — long
warm-ups are essential.
4. Conv-phase variance is content-dependent, not state-dependent.
5. tokio::join! polls cooperatively — spawn separate tasks for real
concurrency between sync compute and async pump.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
