Two additions for downstream voice-channel prosody / quality wiring:
1. Converse::with_pre_sentence_hook
New PreSentenceHook (Box<dyn FnMut(&mut Generator, &str) -> Result<()>>)
that fires before each sentence's synth call. Receives mutable
access to the underlying Generator + the sentence text — lets
callers apply per-sentence steering (e.g. emotion shifts mid-reply)
without touching crate internals. Wired into both `synthesize` and
`synthesize_streaming` paths.
2. PostProcess streaming split
- New StreamingHpfState — stateful biquad whose IIR taps carry
across chunk boundaries so streaming HPF doesn't click at chunk
joins. Identical filter coefficients to the one-shot path.
- PostProcess::apply_chunk_safe(samples, hpf_state) — HPF + declick
per chunk, no LUFS (needs full utterance).
- PostProcess::apply_lufs(samples, sample_rate) -> Result<f32> —
full-utterance loudness gain, returns the linear gain applied
so streaming pipelines can compensate retroactively if needed.
- compute_lufs_gain helper extracted from loudness_normalize.
Used by zeroclaw-channel-voice for the Maya-gap-closure pack.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Without a voice anchor, CSM-1B picks a different speaker each turn
and drifts mid-sentence on longer outputs (high-pitch squeaks,
female/male swap mid-utterance). The fix is the standard CSM
speaker-prompt pattern: pass a Segment with reference audio + its
transcript as context to every generate() call.
Previously Converse::synthesize and synthesize_streaming hardcoded
`&[]` for the context arg. Add a `context: Vec<Segment>` field on
Converse plus a builder method:
let conv = Converse::new(&llm, &mut gen)
.with_context(vec![Segment::new(0, transcript, audio)]);
Both synth paths now pass `&self.context` instead of `&[]`. Empty
context (default) keeps prior behavior.
Verified end-to-end with zeroclaw-channel-voice + macOS `say`-
generated reference: same input now produces deterministic-length
output across turns (2.64s vs. previously varying 6/19/38s) and the
voice matches the seed throughout.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
8-gen bench (4 emotions × 2 corpora) at seed=42 against firdhokk
Whisper-LV3:
target RAVDESS CREMA-D
happy happy (0.999) ✓ happy (0.999) ✓
angry neutral (0.92) sad (0.99)
fearful happy (0.998) fearful (0.984) ✓
sad angry (0.99) fearful (0.99)
CREMA-D 2/4 vs RAVDESS 1/4. Larger / more naturalistic corpus
produces more class-pure fearful direction. Neither corpus solves
angry or sad — recipe shifts into 'vague expressivity' rather than
class-specific corners.
Practical: prefer CREMA-D when available; A/B both per emotion if
class precision matters.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
GenerateOptions gains emotion_hint: Option<String>. After text
normalization, the hint (if Some) is prepended as `<tag> <text>` so the
Llama BPE tokenizer encodes it as ordinary tokens. Plumbed through
generate, generate_streaming, and generate_with_profile (via delegation),
plus a `--emotion-hint` flag on examples/generate and a Shared field
+ CLI flag on examples/converse_server (per-turn ConverseOptions).
GenerateOptions lost Copy because Option<String> isn't Copy; updated
the four callers that depended on it (bench, longform, converse synth
+ stream) to .clone() the opts at the call site. Cheap — the struct
is small and clones are per-turn, not per-frame.
On the un-adapted base this is a no-op cosmetic prefix. The point is to
unlock Phase 12.1-fine-tuned adapters: train with `[whisper] X` paired
with whispered audio, and the adapter learns the tag→prosody mapping at
inference time.
Verified end-to-end: --emotion-hint "[whisper]" --max-audio-ms 3000
produced a valid 24kHz mono WAV through tokenizer → backbone → Mimi
with no panics. Lib suite 96/96 (added 4 apply_emotion_hint unit tests).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Adds Converse::run_streaming using Generator::generate_streaming. Chunks
are emitted via a per-chunk callback as they're decoded; per-sentence
callback fires once for /metrics + transcript bookkeeping. Skips
post-process and watermark — both require full-sentence context.
Wires --stream-tts and --stream-chunk-frames into examples/converse_server.
Combining --stream-tts with --watermark-* is rejected at boot.
Critical architectural fix: tokio::join!(conv_fut, pump_fut) polls
cooperatively in ONE task, so the sync TTS gen blocks the runtime worker
and prevents pump_fut from draining the chunk channel. Switched to
tokio::spawn(conv_fut) + pump_fut.await on the original task — now the
runtime can schedule pump_fut on a different worker while conv_fut is
mid-synthesize. Without the spawn, chunks pile up and arrive in burst at
sentence boundaries (defeating the streaming purpose).
A/B (warmed FP, mock LLM, 10.43s LibriSpeech in):
Phase Non-stream Stream+spawn Δ
llm_to_first_audio 4892ms 1942ms -2950ms
total_turn 18408ms 19628ms +1220ms (variance)
Client-side TTFA 9345ms 6789ms -2556ms
~3s TTFA reduction is the production win. total_turn is roughly unchanged
(chunk delivery overhead exists but is small). For voice-loop deployments
where time-to-first-audio dominates UX, this is the right default.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Adds info-level logs inside Converse::run with a stable "conv-phase:"
prefix. Three event types per turn:
conv-phase: ttft={ms} LLM first token (TTFT)
conv-phase: sentence[i] llm_buffer={ms} tts_gen={ms} chars={n}
conv-phase: sentence[i] (trailing) llm_buffer={ms} tts_gen={ms} chars={n}
llm_buffer = wall time accumulating tokens since the previous sentence
boundary (or run start, for the first sentence). tts_gen reports the
synthesize() cost (Generator.generate + post + watermark).
Together with the handle_connection turn-timing log shipped earlier,
this gives complete attribution of where voice-loop wall time goes.
Mock LLM measurement (10.43s LibriSpeech in, 4-sentence canned reply):
ttft=50ms
sentence[0] llm_buffer=103ms tts_gen=5679ms chars=12
sentence[1] llm_buffer=1449ms tts_gen=646ms chars=156
sentence[2] llm_buffer=154ms tts_gen=3661ms chars=20
sentence[3] llm_buffer=259ms tts_gen=4894ms chars=22
Surprise: first-sentence TTS gen (5.7s) is the dominant cost — Metal
warm-up + KV cache init on the first generate() call. Subsequent
sentences are 3-5x cheaper. For real LLMs (e.g., Z.AI thinking-disabled),
ttft becomes the dominant cost; this instrumentation distinguishes the
two cleanly.
No API change — pure observability.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Python-reference diff revealed the all-pad debugging session was on the
wrong audio source. /tmp/csm_24k.wav (CSM-generated speech) is not
intelligible enough for Kyutai STT — even the Python reference emits
nothing. On a real LibriSpeech-style speech sample the pipeline works
correctly.
Verified end-to-end on Metal (10s FLAC, "He hoped there would be stew
for dinner..."):
Rust port: 23 words, transcript matches Python reference
Python ref: 25 words (last 2 cut off in our run due to asr_delay
off-by-one — cosmetic, fixable by setting delay=7)
Changes:
- Add sentencepiece = "0.13" dep for token detok
- Stt::decode_word_text(tokens) returns the detokenized word text
(filters padding token id 3, calls SentencePieceProcessor::decode_piece_ids)
- examples/stt_demo: pair Word/EndWord events into timed segments,
detokenize each, print transcript + concatenated text
- Update module docs to reflect WORKING status
Phase 6 progress:
6a STT: WORKING (this commit)
6b LLM client: shipped
6c.1 text->LLM->TTS: shipped
6c.2 full duplex: ready to build now that 6a works
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Composable orchestrator stitching the LLM-output side of the
conversational stack:
prompt + history -> LlmClient.generate_stream -> sentence buffer
-> per-sentence Generator.generate -> post-process -> watermark
-> Vec<Utterance> stream of (text, audio, latency)
CSM is sentence-level (best prosody on full sentences), so the buffer
flushes when terminal punctuation appears anywhere in the buffer
(Punctuation policy: . ! ? \n) or at any clause boundary
(Eager policy: + , ; :).
src/converse.rs:
- Converse<L: LlmClient> orchestrator
- Utterance { text, audio, tts_latency_ms } per sentence
- FlushPolicy { Punctuation, Eager }
- find_first_boundary scans the whole buffer (not just the last char)
so "Sentence one. Word two" emits "Sentence one." immediately rather
than waiting for the next terminal mark
- 3 unit tests for boundary detection + policy modes
examples/converse.rs:
- --mock mode: hardcoded 20-token "sleepy turtle" stream, no API key
- live mode: any OpenAI-compatible endpoint via OpenAiCompatibleClient
- writes the concatenated audio to a single WAV
Verified mock end-to-end on Metal: 2 utterances emitted as expected
(sentence 1 hits max_audio_ms cap at 6s; sentence 2 EOTs naturally at
4.88s), total 10.88s of audio in 31.5s wall-clock.
Phase 6c.1 ships the half-duplex (text-in -> voice-out) pipeline. Full
duplex (audio-in -> voice-out) is 6c.2, blocked on 6a's STT word
emission landing.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
redclawsystems