Phase 8.1.1 quality fix from the perf plan. Tokens emitted at LM step
`t` correspond to audio frame `t - ASR_DELAY_FRAMES` (6 frames /
0.48 s), so when a caller stops feeding audio without trailing
silence the last few words trail off — they're still inside the
delay pipeline.
finish() now steps ASR_DELAY_FRAMES additional silent frames after
handling any partial sub-frame buffer, giving the LM the chance to
emit those buffered tokens. Cost: 7 extra step_pcm calls per turn.
Verified end-to-end via stt_demo on a mid-utterance trim of the
LibriSpeech reference clip:
pre-flush: 11 words ("...turnips and carrots and bruised")
post-flush: 13 words ("...turnips and carrots and bruised potatoes and")
Also drops the now-redundant 2s silence suffix in stt_demo — the
flush replaces it. Affects converse_server's real-time end-of-turn
path where suffix padding wasn't possible.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Original plan: gate STT work via Silero V5 VAD using
voice_activity_detector / ort to skip silent chunks. Phase 8.1.2's
ort_conflict_probe initially passed (no Metal regression). But on
first server boot, sentencepiece-sys's protobuf 3.14 collides with
ort's bundled protobuf 3.21 at process startup:
[libprotobuf FATAL ...] This program was compiled against version
3.14.0 of the Protocol Buffer runtime library, which is not
compatible with the installed version (3.21.12) ... in
sentencepiece-sys-0.13.1
Updated the probe binary to also load Kyutai STT (which links
sentencepiece) — now correctly catches the conflict at probe time.
Same risk class as the whisper-rs/ggml issue from Phase 7.6: external
ML runtimes don't always coexist in the same Rust process.
Pivoted to a pure-Rust energy-based VAD: RMS amplitude threshold per
incoming chunk. ~30 LOC, no external runtime, no protobuf risk.
Catches obvious silence (room tone, pauses) — misses quiet speech
(whispering). Acceptable for typical mic-distance voice loops.
Wired into examples/converse_server as `--vad-gate` /
`--vad-gate-threshold 0.01`. The receive loop checks RMS before each
binary chunk's STT call; silence chunks skip step_pcm entirely.
Orthogonal to existing `--vad` (Kyutai semantic-VAD-for-EOT).
Bench (mock LLM, Q8+stream, 10s LibriSpeech in, ~90% speech):
baseline: recv_phase = 4834 ms
--vad-gate: recv_phase = 4509 ms (-7%, matches the ~10% silence
in this audio)
Real-world voice-agent audio is 30-50% silence; expect proportional
savings (~30-50% recv_phase reduction). Larger silence content =
larger VAD win.
Cargo.toml: keeps the optional `vad` feature + voice_activity_detector
dep wired (path remains for a future Silero-via-candle port). The
ort_conflict_probe example continues to require `--features vad` so
the conflict-detection path stays exercised.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Wraps the per-frame Kyutai STT calls in handle_connection's receive
loop with tokio::task::spawn_blocking. The async runtime worker is
freed during the ~80ms/frame compute so other tasks/connections can
run on the same worker pool.
Single-connection latency is unchanged (STT is hardware-bound at ~1×
realtime). The win is multi-tenant concurrency: with N WebSocket
connections, no longer blocks all N on one connection's STT work.
Two call sites updated:
- carry-over feed (turn-start barge-in audio)
- main receive loop (per-binary-chunk step_pcm)
Pattern: clone Arc<Shared>, move into spawn_blocking, use
tokio::sync::Mutex::blocking_lock() inside the closure (legal there).
The blocking_lock requires the Mutex to be reachable via Send + 'static
captures — that's why we clone shared rather than borrow.
Cannot show the multi-tenant win in the existing single-connection
bench harness; bench thermals also confound direct A/B. Functional
correctness verified via running bench (transcripts produced, no
errors).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Phase 6d.rate-limit:
Per-connection sliding-window token bucket (60s window) on the
WebSocket. Two limits: --rate-audio-secs-per-min (default 600,
60 of which exhausts the bucket on a 1-minute monologue), and
--rate-turns-per-min (default 60). Excess fires an error event +
closes the connection.
Verified: with --rate-turns-per-min 1, two SEPARATE WS connections
each get a fresh per-connection bucket (by design — defends against
a single misbehaving client; for per-IP, the bucket would need to
live globally on Shared).
Phase 6c.3b: semantic VAD via extra_heads
- Added Stt::load_default_with_vad() which downloads
`kyutai/stt-1b-en_fr-candle` (the VAD-enabled variant: 4 extra
heads × 6-dim categorical, trained for end-of-turn detection).
- config_stt_1b_en_fr_vad() builds the LM config with the
ExtraHeadsConfig that loads `extra_heads.X.weight` from the
checkpoint. The standard 1B en/fr config has extra_heads = None.
- Stt::end_of_turn_probability(&AsrEvent::Step) extracts head index
2's probability (per the Kyutai reference Python script). 2 unit
tests.
- converse_server gains --vad / --vad-threshold / --vad-consecutive
flags. When --vad is set: STT runs incrementally during the WS
receive loop, watches Step events for end-of-turn probability,
and auto-fires EOT when the threshold is exceeded for K
consecutive frames. Sends {"event":"vad_eot"} to the client when
triggered, then proceeds to LLM + TTS without needing a manual
"EOT" text frame.
87 lib tests pass.
Phase 6 status:
6a STT: working
6b LLM client: working
6c.1 text->LLM->TTS: working
6c.2 WebSocket duplex MVP: working
6c.3a streaming TTS chunks: working
6c.3b semantic VAD: shipped (this commit)
6d.{auth,shutdown,metrics,rate-limit}: shipped (this commit)
6c.3c barge-in (interrupt assistant mid-speak): deferred
The Rust Unmute conversational stack is now feature-complete for the
strategic Phase 6 deliverable.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Python-reference diff revealed the all-pad debugging session was on the
wrong audio source. /tmp/csm_24k.wav (CSM-generated speech) is not
intelligible enough for Kyutai STT — even the Python reference emits
nothing. On a real LibriSpeech-style speech sample the pipeline works
correctly.
Verified end-to-end on Metal (10s FLAC, "He hoped there would be stew
for dinner..."):
Rust port: 23 words, transcript matches Python reference
Python ref: 25 words (last 2 cut off in our run due to asr_delay
off-by-one — cosmetic, fixable by setting delay=7)
Changes:
- Add sentencepiece = "0.13" dep for token detok
- Stt::decode_word_text(tokens) returns the detokenized word text
(filters padding token id 3, calls SentencePieceProcessor::decode_piece_ids)
- examples/stt_demo: pair Word/EndWord events into timed segments,
detokenize each, print transcript + concatenated text
- Update module docs to reflect WORKING status
Phase 6 progress:
6a STT: WORKING (this commit)
6b LLM client: shipped
6c.1 text->LLM->TTS: shipped
6c.2 full duplex: ready to build now that 6a works
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Refinements on the Kyutai STT integration after debugging session:
- dtype: BF16 on accelerators (matches checkpoint storage), F32 on CPU.
Previously F16 on Metal which can overflow in the LM's RmsNorm.
- examples/stt_demo: pad input with 0.5s silence suffix per the HF
stt_config.audio_delay_seconds, matching the Python reference loop.
- src/stt.rs: tightened module docs with debugging notes for the
remaining all-pad-output issue. Removed RTX_STT_DEBUG callback path
(was useful for one-off debugging; can be re-added with cleaner shape).
Status: weights load cleanly, LM forward advances every frame, but
predictions are all-pad on real speech. Bisection plan documented in
the module rustdoc — next session should diff against the official
delayed-streams-modeling Python reference at frame-by-frame granularity.
77 lib tests + 2 stt tests all pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Integrates the moshi crate (0.6.4, candle 0.9.1) for streaming STT.
Module + demo + custom config for kyutai/stt-1b-en_fr. Model loads
cleanly, LM forward pass advances (model_step_idx increments correctly),
but word events don't yet emit on a 10s CSM speech sample.
What works:
- moshi 0.6.4 added as dependency (candle 0.9.1, version-compatible)
- src/stt.rs wraps moshi::asr::State + moshi::lm + moshi::mimi
- Stt::load_default downloads kyutai/stt-1b-en_fr (~3 GB) from HF
- Custom config_stt_1b_en_fr() matching the released checkpoint:
d_model=2048, num_layers=16, dim_feedforward=8192 (moshi's SwiGLU
hidden = 11/4 * d_model = 5632 — verified vs safetensors), text vocab
8001/8000, audio vocab 2049, 32 codebooks, no depformer
- AsrEvent enum + From<moshi::asr::AsrMsg> conversion
- examples/stt_demo.rs streams a WAV through the pipeline
- 2 unit tests for AsrEvent conversion
What needs more work:
- Word emission: 0 words detected on 10s of clean CSM speech, even
though LM forward advances every frame. Likely culprits:
a) asr_delay_in_tokens 6 vs HF stt_config.audio_delay_seconds=0.5
(6.25 frames). Off-by-one possible.
b) Sentencepiece detok not yet wired (tokens emitted but text=None).
c) Subtle weight-key remap differences between moshi's expected
layout and the released checkpoint that don't trip a shape check.
d) renormalize/audio preprocessing mismatch.
Next step (Phase 6a polish): compare against the official
delayed-streams-modeling/scripts/stt_from_file_pytorch.py reference to
identify the missing piece. The integration framework is sound; only
the final LM-output-to-text-event step needs work.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
