3-seed × 3-prompt reproducibility bench on [email protected] reveals that the
recipe shifts speaker character reliably but produces high text-
fidelity variance:
seed 42: WER 2.00 "The police are, if you're, I can't recite..."
seed 7: WER 0.71 "Today, today I want to share..." ← near-verbatim
seed 100: 0.32 s premature EOT
Cross-prompt at seed 42 drifts uniformly across 3 prompts. Speaker
cosine is consistently elevated; text content is roll-the-dice.
Documenting this as the honest characterization rather than overclaim
the single-seed Sprint 2 results. Practical recipe: roll N seeds,
pick lowest-WER output.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Capstone wrapper for the Phase 9 recipe (Selective CFG + RAVDESS
steering + mid-layer subset). Picks the steering scale automatically
from an empirical per-emotion map:
- happy: 0.30
- angry: 0.20
- fearful: 0.20
- sad: 0.20 (note: sad is unreliable — see below)
These came from a follow-up sweep after the multi-emotion demo
revealed the recipe is emotion-sensitive: scale 0.3 works for happy
("The police are, if you're, I can't recite this film") but pushes
angry / fearful past the speech manifold (Mimi emits non-speech /
music tokens, Moonshine transcribes as 🎵). Dropping to 0.2 recovers
fluent speech for both:
- [email protected]: "The next disorder is completing kashim for more."
- [email protected]: "You just heard a little bit about this decision,
though."
- [email protected]: "The police are, if you're, I can't recite this
film. I"
Sad is the outlier — model resists "sad" steering at every scale
between 0.15 and 0.3. Likely a corpus issue (sad RAVDESS clips are
the lowest-energy subset). Documented as a known limitation rather
than worked around.
perf_history.md updated with the per-emotion sensitivity finding.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Sweep of 4 schedules × 4 prompts produced concrete data on schedule
sensitivity. step:3.0:1.5:12 (the Sprint 3 winner) catastrophically
fails dense prompts: lecture-style input → 0.08 s of audio (one
frame). step:2.0:1.0:8 produced the best single shot — near-verbatim
question rendering "Well, it had stem-wondered. Have you ever
wondered why we sometimes hear voices the way we do?" — but tanked
the lecture prompt (2.4 s "You").
linear:3.0:1.0:25 is the only schedule that's never the best AND
never the worst — graceful degradation across all four prompt
categories. Updates the recommended recipe in perf_history.md
(formerly step:3.0:1.5:12).
quality_eval: skip WavLM-SV scoring on clips shorter than 0.25 s
(emit -1 sentinel) — WavLM-SV's TDNN front-end requires a few
hundred samples and crashed mid-sweep on the 0.08 s clip. Now the
eval emits a row instead of bailing on the whole batch.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Captures the Phase 10 work in the consolidated perf doc. New section
"Watermarker backend matrix" lists AudioSeal (Meta) and SilentCipher
(Sesame's actual) side by side with measured RTF, capacity, conflicts,
and a per-use-case recommendation table.
Headline: SilentCipher is Sesame's literal production watermarker, now
shipping in pure candle with bit-perfect round-trip on real LibriSpeech
audio (15/15 codes, confidence 1.0000) and ~10× smaller than AudioSeal
(~3M params vs ~30M). The "blow them out of the water" item from the
Sesame gap analysis is closed.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Updated docs/perf_history.md to reflect the full Phase 8 work:
- New TL;DR has TWO production configs (English-only with --moonshine,
multilingual with default Kyutai). The English path is the new
headline recommendation.
- Captured live Z.AI bench numbers from the new production stack
(Q8 + stream + warmup + Moonshine + thinking-disabled): total_turn
8765 ms vs Phase 6f.q8 era 17145 ms — half the wall-clock latency
end-to-end.
- Phase 8 commit table extended to 8.4 through 8.10.
- Added an STT backend matrix (Kyutai 1B / Whisper-rs / Moonshine)
with measured RTF, build flags, and per-deploy recommendation rows.
Bench command used (single turn, real Z.AI glm-4.5 thinking-disabled,
10.43 s LibriSpeech /tmp/asr_test.flac):
target/release/examples/converse_server \\
--bind 127.0.0.1:18099 \\
--quantized-gguf /tmp/csm_q8.gguf \\
--stream-tts --moonshine \\
--llm-base https://api.z.ai/api/coding/paas/v4 \\
--llm-model glm-4.5 \\
--llm-extra-body '{"thinking":{"type":"disabled"}}'
Server-side timing:
recv_phase 0 ms (Moonshine batch — no parallel STT)
stt_post 332 ms (Moonshine transcribe at EOT)
llm_to_first_audio 1627 ms (Z.AI TTFT ~1.3 s + first TTS chunk)
conv_total 8432 ms
total_turn 8765 ms
Client TTFA: 1959 ms (vs Phase 6f.q8 era ~2 s — comparable; the dominant
remaining latency is the Z.AI provider TTFT, not anything we control).
Z.AI returned a coherent reply: "He eagerly anticipated a hearty stew
with turnips, carrots, potatoes, and savory mutton pieces for dinner."
matching the LibriSpeech ground-truth meaning.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Consolidated record of every shipped optimization, every rejected path
with the data behind the rejection, the production-recommended config,
the architectural lessons captured, and the deferred multi-session
work with realistic sizing.
Headline:
- client first_audio_ms p50 = 529 ms (mock LLM, Q8 + stream + extended warmup + VAD gate)
- real Z.AI loop: ~2 s TTFA p50
- boot cost: ~2.7 s (one-time)
Rejected paths captured (so future sessions don't redo the work):
Q4_K_M (2.85x slower than Q8), whisper-rs linkage (2-3x CSM regression),
Silero V5 via ort (protobuf 3.14 vs 3.21 conflict), Mimi codec Q8
(no candle conv-quant path), KV cache reuse (variance is content-
dependent not state-dependent), rayon for single-connection
(overhead exceeds gain on <100µs tasks), codec swaps (require
backbone retrain), custom distillation (no published checkpoint),
Kyutai 4x flush (hardware-bound on M-series).
Architectural lessons:
1. In-process linkage of external ML runtimes is a recurring trap;
default to sidecar-process pattern.
2. Bench thermals dominate single-machine A/B; 90s cooldown often
necessary.
3. First-frame compilation is the dominant cold-start cost — long
warm-ups are essential.
4. Conv-phase variance is content-dependent, not state-dependent.
5. tokio::join! polls cooperatively — spawn separate tasks for real
concurrency between sync compute and async pump.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
