8-gen bench (4 emotions × 2 corpora) at seed=42 against firdhokk
Whisper-LV3:
target RAVDESS CREMA-D
happy happy (0.999) ✓ happy (0.999) ✓
angry neutral (0.92) sad (0.99)
fearful happy (0.998) fearful (0.984) ✓
sad angry (0.99) fearful (0.99)
CREMA-D 2/4 vs RAVDESS 1/4. Larger / more naturalistic corpus
produces more class-pure fearful direction. Neither corpus solves
angry or sad — recipe shifts into 'vague expressivity' rather than
class-specific corners.
Practical: prefer CREMA-D when available; A/B both per emotion if
class precision matters.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Restructured the converse_server's user-audio receive loop to feed STT
incrementally as PCM frames arrive, instead of accumulating all audio
then transcribing in one block at end-of-turn.
Both VAD and non-VAD paths share the same incremental-ingest logic:
- Reset STT once per turn
- On every binary frame: append to user_audio_24k AND step_pcm with
the new slice
- collect_transcript_events() helper pairs Word/EndWord, detoks via
sentencepiece, accumulates transcript_words
- On EOT: stt.finish() drains the asr_delay buffer, transcript_words
are joined into user_text — essentially instant
- Carry-over barge-in audio gets fed first (preserving the start of
the next user turn's speech)
Empirical numbers (3 turns, 10.43s LibriSpeech, mock LLM, --realtime):
Before (sync STT): transcript_ms p50=5219ms p95=5444ms
After (parallel STT): transcript_ms p50=45ms p95=48ms [-99%]
Total turn time went UP (24.9s vs 17.7s p50) only because the realtime
client now actually takes 10s to send 10s of speech (previously it
dumped instantly — unrealistic for voice).
For real voice traffic the user-perceived latency improvement is:
Before: ~9s of silence after user stops speaking
After: ~4.5s of silence (transcript ready in 45ms + 4.4s LLM+TTS)
Bench harness gains --realtime flag that paces frames at audio
playback rate — required to measure the parallel-STT win since the
default dump-everything-at-once mode can't show overlap.
Phase 6 status: feature-complete and now performance-optimized.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
End-to-end conversation latency bench. Drives N sequential turns
through a single WebSocket and reports per-phase stats:
audio_send_ms (client streaming PCM in until EOT)
transcript_ms (server EOT → "transcript" event)
first_audio_ms (server "transcript" → first audio chunk)
turn_total_ms (full audio_send → "done" event)
Pulls /metrics at end for the server-side averages.
First numbers on Metal (M-series, 10.43s LibriSpeech FLAC, 3 turns,
mock LLM with 50ms/token sleep):
audio_send p50=1ms p95=1ms
transcript p50=5.2s p95=5.4s (STT, 1.9x realtime)
first_audio p50=3.8s p95=4.4s (LLM stream + first sentence TTS)
turn_total p50=17.7s p95=18.5s
server stt avg 5.3s
server tts avg 2.7s/utterance
server e2e avg 9.3s
These are the empirical baselines for the Rust Unmute MVP. Optimization
opportunities: parallel STT during receive (already wired for VAD path),
smaller STT model, quantized CSM-1B (already shipped via Q8 GGUF), and
the obvious one — replace mock LLM with a real fast endpoint.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
