rustytorch/crates/models/rtx-csm/docs/voxtream_port_notes.md

VoXtream-style look-ahead port notes (Tier 2.1)

Working notes for the planned look-ahead-window optimization on the TTS side. Captures architecture, ecosystem facts established by Phase 9.1 spike, and an ordered porting plan so a future session can pick up cold.

Why this exists

Production stack post-Phase 8.10:

• recv_phase (Moonshine): 0 ms
• stt_post: 332 ms
• LLM TTFT (Z.AI): ~1300 ms
• TTS first chunk: ~600 ms ← target of this work
• total client TTFA: ~1959 ms

VoXtream (arXiv 2509.15969) claims 102 ms first packet by adding a phoneme-level look-ahead window before TTS gen starts. The look-ahead runs ahead of the audio stream so it doesn't delay onset.

Ports cleanly to CSM (Llama backbone + depth decoder) — does not require retraining the model. Just a new pre-conditioning pipeline.

Phonemizer ecosystem (Phase 9.1 finding)

Crate	Approach	Latency	Conflicts	Verdict
espeak-ng 0.1.1	pure Rust port of eSpeak NG	1-5 ms/word	none	PICK THIS
voirs-g2p 0.1.0-rc.1	neural, candle 0.9.2	10-50 ms/word	libc dep	only if neural needed
grapheme_to_phoneme 0.1.0	seq2seq RNN, ARPAbet	2-8 ms/word	none	abandoned 2020; viable fallback
phonetisaurus-g2p 0.1.1	FST	sub-ms	none	needs pre-trained FST

espeak-ng is the right pick:

• Zero C linkage (the c-oracle feature is opt-in for testing only)
• 100+ languages, IPA output
• Mature, used in real TTS pipelines
• No dep conflicts with candle 0.9, sentencepiece-sys, hound, symphonia, ebur128

For our latency budget: 5-10 word look-ahead × 1-5 ms = 5-50 ms total phonemizer cost. Well within the 50 ms allowance for a 102 ms first- packet target.

Architecture (planned)

Three integration points — all in src/converse.rs and src/generator.rs:

1. Phonemizer service

A small wrapper around espeak-ng exposed as rtx_csm::phonemizer::Phonemizer (new module). Methods:

pub struct Phonemizer { /* espeak_ng::Voice */ }
impl Phonemizer {
    pub fn english() -> Result<Self>;
    /// Phonemize a sentence-fragment into IPA phonemes.
    pub fn phonemize(&self, text: &str) -> Result<Vec<Phoneme>>;
}

Use IPA for portability; an ARPAbet-tagged fork is doable later if CSM is found to phonologize better with ARPAbet.

2. Look-ahead window in Converse

Modify Converse::run_streaming so that as the LLM emits tokens, we:

1. Phonemize each new word (call espeak-ng on the latest token-completed word — usually triggered on whitespace boundary).
2. Maintain a sliding "look-ahead phoneme buffer" of the next N words (N=5-10).
3. Pass the look-ahead phonemes to the TTS as a hint before the first audio frame is generated.

The hint is a free-form input; CSM's prompt format already accepts text with formatting. We can experiment with formats:

• Inline phonemes in the text: [hə hoʊpt] he hoped...
• A separate "phoneme prompt" segment before the assistant turn
• A Segment::PhonemeHint(...) variant added to the prompt builder

3. Generator look-ahead-aware forward

Optionally modify Generator::generate_streaming to accept a "phoneme prefix" that conditions the early frames on look-ahead phonemes, helping the model start with prosody appropriate to the upcoming word.

This is the most speculative piece — the VoXtream paper's exact mechanism (does the look-ahead enter the model as text? Embeddings? Cross-attention?) needs careful reading. The arxiv abstract says "phoneme transformer + monotonic alignment + dynamic look-ahead." We may need to read the full paper before implementing this part.

Open questions (resolve before implementing)

1. Phoneme integration mechanism. VoXtream's paper says the look-ahead doesn't delay onset, which implies phonemes are NOT just prepended to the text input. They're presumably consumed by an auxiliary path. Read full paper for the exact mechanism.

2. CSM pre-training compatibility. CSM-1B was trained on text + audio tokens. It has never seen IPA or ARPAbet during training. The look-ahead might need to be encoded as natural-language hints or phonetic-alphabet transliteration (ARPAbet → spelled-out form).

3. Empirical win on this hardware. VoXtream's 102 ms claim is on their training-aware model. CSM-1B was NOT trained for this. The port might give us a smaller win — say 200-400 ms TTS first chunk vs current 600 ms. Need a concrete bench to commit.

Porting tasks (~5-7 days, multi-session)

In order of dependency:

1. src/phonemizer.rs skeleton (~0.5 day)

   • Add espeak-ng = "0.1" to Cargo.toml as optional dep behind phonemizer feature
   • Wrap espeak_ng::Voice with our Phonemizer API
   • Standalone smoke test: phonemize "He hoped there would be stew" → verify IPA output

2. docs/voxtream_paper_summary.md (~0.5 day)

   • Read arxiv 2509.15969 carefully
   • Document the exact mechanism: phoneme-transformer, monotonic alignment, dynamic look-ahead
   • Resolve open question (1) above

3. Phoneme hint integration in Generator (~2 days)

   • Decide format (inline / segment / new prompt slot)
   • Modify prompt::build_prompt to accept phoneme hints
   • Test with a known phrase: does CSM produce more natural prosody with phoneme hints?

4. Look-ahead window in Converse::run_streaming (~1 day)

   • Maintain a sliding phoneme buffer
   • Trigger phonemize on word boundary in the LLM stream
   • Pass updated buffer to Generator on each TTS call

5. A/B bench (~1 day)

   • Same converse_server_bench setup, with vs without look-ahead
   • Measure: sentence[0] tts_gen, llm_to_first_audio, client TTFA
   • If win is < 100 ms on M-series, deprioritize and document

6. Wire into converse_server (~1 day, if perf justified)

   • New --lookahead-words N flag on converse_server
   • Default 0 (off); recommend 5 for production English
   • Update docs/perf_history.md

Alternative if VoXtream mechanism is too speculative

If the paper's exact mechanism turns out to require model retraining (e.g., a phoneme cross-attention head), we have a simpler win available: streaming-Mimi reduction in chunk_frames from 4 to 2. Empirically measured this in Phase 6c.3a; current default 4 frames = 320 ms latency. Drop to 2 = 160 ms but adds decode overhead (Mimi has per-call setup cost).

That's a 1-line change, no phonemizer needed. Worth measuring before committing to the full VoXtream port.

Recommended order of attack for the next session

1. Drop chunk_frames=4 to chunk_frames=2 and chunk_frames=1 — measure first-chunk latency. If win ≥ 200 ms, ship it as Phase 9.0 and skip the phonemizer port entirely.
2. If chunk_frames tuning isn't enough, then implement the espeak-ng wrapper (task 1 above) — bounded 0.5-day spike.
3. Read the VoXtream paper carefully (task 2). DO NOT IMPLEMENT until the mechanism is clear from the paper, not the abstract.
4. Iterate per the task list above.

Cited sources

• Paper: arXiv 2509.15969 — VoXtream: Full-Stream TTS with Extremely Low Latency
• Phonemizer crates verified on crates.io (Phase 9.1 spike, agent ID <not_returned_this_session>)
• Production baseline numbers: docs/perf_history.md Phase 8.11 row