rtx-csm: Phase 9.1 spike — VoXtream port notes (Tier 2.1 prep)
Phase 9.1 is the foundation step for the Tier 2.1 (VoXtream-style
look-ahead) optimization from the Phase 8 plan. Same pattern as Phase
8.4: don't attempt the full port in one session; ship the architecture
analysis + ecosystem audit + ordered porting plan so a future session
starts cold from concrete data.
Phonemizer ecosystem audited (results in the doc):
Crate Approach Latency Conflicts
espeak-ng 0.1.1 pure Rust eSpeak NG 1-5 ms/word none <- PICK
voirs-g2p 0.1.0-rc.1 neural, candle 0.9.2 10-50 ms/w libc dep
grapheme_to_phoneme 0.1.0 seq2seq RNN ARPAbet 2-8 ms/word none (abandoned 2020)
phonetisaurus-g2p 0.1.1 FST sub-ms none (no pre-trained FST)
espeak-ng is the right pick: zero C linkage default, no conflicts with
candle 0.9 / sentencepiece-sys / hound / symphonia / ebur128, mature.
Per-word latency fits VoXtream's 102 ms first-packet target with room
to spare (5-10 word lookahead = 5-50 ms total phonemizer cost).
Doc captures:
- Why this is worth doing (current ~600 ms TTS first chunk vs 102 ms
claim from arXiv 2509.15969)
- Three-piece integration architecture (phonemizer service, look-ahead
window in Converse, optional Generator-side phoneme hint)
- Three open questions that MUST be resolved before implementing
(mechanism, CSM training compatibility, empirical win on this HW)
- Six ordered porting tasks with hour estimates totaling 5-7 days
- Alternative quick win: drop streaming chunk_frames from 4 to 2 or 1
first; if that closes most of the gap, full VoXtream port may not
be worth the complexity
Recommended order of attack written for the next session: try the
chunk_frames tuning before implementing the phonemizer. The cheapest
move is a 1-line config change.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
This commit is contained in:
osobh
@@ -0,0 +1,174 @@
|
|||||||
|
# VoXtream-style look-ahead port notes (Tier 2.1)
|
||||||
|
|
||||||
|
Working notes for the planned look-ahead-window optimization on the TTS
|
||||||
|
side. Captures architecture, ecosystem facts established by Phase 9.1
|
||||||
|
spike, and an ordered porting plan so a future session can pick up cold.
|
||||||
|
|
||||||
|
## Why this exists
|
||||||
|
|
||||||
|
Production stack post-Phase 8.10:
|
||||||
|
- recv_phase (Moonshine): 0 ms
|
||||||
|
- stt_post: 332 ms
|
||||||
|
- LLM TTFT (Z.AI): ~1300 ms
|
||||||
|
- **TTS first chunk: ~600 ms** ← target of this work
|
||||||
|
- total client TTFA: ~1959 ms
|
||||||
|
|
||||||
|
VoXtream (arXiv 2509.15969) claims **102 ms first packet** by adding a
|
||||||
|
phoneme-level look-ahead window before TTS gen starts. The look-ahead
|
||||||
|
runs ahead of the audio stream so it doesn't delay onset.
|
||||||
|
|
||||||
|
Ports cleanly to CSM (Llama backbone + depth decoder) — does not require
|
||||||
|
retraining the model. Just a new pre-conditioning pipeline.
|
||||||
|
|
||||||
|
## Phonemizer ecosystem (Phase 9.1 finding)
|
||||||
|
|
||||||
|
| Crate | Approach | Latency | Conflicts | Verdict |
|
||||||
|
|-------|----------|---------|-----------|---------|
|
||||||
|
| **espeak-ng 0.1.1** | pure Rust port of eSpeak NG | 1-5 ms/word | none | **PICK THIS** |
|
||||||
|
| voirs-g2p 0.1.0-rc.1 | neural, candle 0.9.2 | 10-50 ms/word | libc dep | only if neural needed |
|
||||||
|
| grapheme_to_phoneme 0.1.0 | seq2seq RNN, ARPAbet | 2-8 ms/word | none | abandoned 2020; viable fallback |
|
||||||
|
| phonetisaurus-g2p 0.1.1 | FST | sub-ms | none | needs pre-trained FST |
|
||||||
|
|
||||||
|
`espeak-ng` is the right pick:
|
||||||
|
- Zero C linkage (the `c-oracle` feature is opt-in for testing only)
|
||||||
|
- 100+ languages, IPA output
|
||||||
|
- Mature, used in real TTS pipelines
|
||||||
|
- No dep conflicts with candle 0.9, sentencepiece-sys, hound, symphonia, ebur128
|
||||||
|
|
||||||
|
For our latency budget: 5-10 word look-ahead × 1-5 ms = **5-50 ms total
|
||||||
|
phonemizer cost.** Well within the 50 ms allowance for a 102 ms first-
|
||||||
|
packet target.
|
||||||
|
|
||||||
|
## Architecture (planned)
|
||||||
|
|
||||||
|
Three integration points — all in `src/converse.rs` and `src/generator.rs`:
|
||||||
|
|
||||||
|
### 1. Phonemizer service
|
||||||
|
|
||||||
|
A small wrapper around `espeak-ng` exposed as
|
||||||
|
`rtx_csm::phonemizer::Phonemizer` (new module). Methods:
|
||||||
|
|
||||||
|
```rust
|
||||||
|
pub struct Phonemizer { /* espeak_ng::Voice */ }
|
||||||
|
impl Phonemizer {
|
||||||
|
pub fn english() -> Result<Self>;
|
||||||
|
/// Phonemize a sentence-fragment into IPA phonemes.
|
||||||
|
pub fn phonemize(&self, text: &str) -> Result<Vec<Phoneme>>;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Use IPA for portability; an ARPAbet-tagged fork is doable later if CSM
|
||||||
|
is found to phonologize better with ARPAbet.
|
||||||
|
|
||||||
|
### 2. Look-ahead window in Converse
|
||||||
|
|
||||||
|
Modify `Converse::run_streaming` so that as the LLM emits tokens, we:
|
||||||
|
|
||||||
|
1. Phonemize each new word (call espeak-ng on the latest token-completed
|
||||||
|
word — usually triggered on whitespace boundary).
|
||||||
|
2. Maintain a sliding "look-ahead phoneme buffer" of the next N words
|
||||||
|
(N=5-10).
|
||||||
|
3. Pass the look-ahead phonemes to the TTS as a hint *before* the first
|
||||||
|
audio frame is generated.
|
||||||
|
|
||||||
|
The hint is a free-form input; CSM's prompt format already accepts text
|
||||||
|
with formatting. We can experiment with formats:
|
||||||
|
- Inline phonemes in the text: `[hə hoʊpt] he hoped...`
|
||||||
|
- A separate "phoneme prompt" segment before the assistant turn
|
||||||
|
- A `Segment::PhonemeHint(...)` variant added to the prompt builder
|
||||||
|
|
||||||
|
### 3. Generator look-ahead-aware forward
|
||||||
|
|
||||||
|
Optionally modify `Generator::generate_streaming` to accept a "phoneme
|
||||||
|
prefix" that conditions the early frames on look-ahead phonemes,
|
||||||
|
helping the model start with prosody appropriate to the upcoming word.
|
||||||
|
|
||||||
|
This is the most speculative piece — the VoXtream paper's exact
|
||||||
|
mechanism (does the look-ahead enter the model as text? Embeddings?
|
||||||
|
Cross-attention?) needs careful reading. The arxiv abstract says
|
||||||
|
"phoneme transformer + monotonic alignment + dynamic look-ahead." We
|
||||||
|
may need to read the full paper before implementing this part.
|
||||||
|
|
||||||
|
## Open questions (resolve before implementing)
|
||||||
|
|
||||||
|
1. **Phoneme integration mechanism.** VoXtream's paper says the
|
||||||
|
look-ahead doesn't delay onset, which implies phonemes are NOT just
|
||||||
|
prepended to the text input. They're presumably consumed by an
|
||||||
|
auxiliary path. Read full paper for the exact mechanism.
|
||||||
|
|
||||||
|
2. **CSM pre-training compatibility.** CSM-1B was trained on text +
|
||||||
|
audio tokens. It has never seen IPA or ARPAbet during training. The
|
||||||
|
look-ahead might need to be encoded as natural-language hints or
|
||||||
|
phonetic-alphabet transliteration (ARPAbet → spelled-out form).
|
||||||
|
|
||||||
|
3. **Empirical win on this hardware.** VoXtream's 102 ms claim is on
|
||||||
|
their training-aware model. CSM-1B was NOT trained for this. The
|
||||||
|
port might give us a smaller win — say 200-400 ms TTS first chunk
|
||||||
|
vs current 600 ms. Need a concrete bench to commit.
|
||||||
|
|
||||||
|
## Porting tasks (~5-7 days, multi-session)
|
||||||
|
|
||||||
|
In order of dependency:
|
||||||
|
|
||||||
|
1. **`src/phonemizer.rs` skeleton (~0.5 day)**
|
||||||
|
- Add `espeak-ng = "0.1"` to Cargo.toml as optional dep behind
|
||||||
|
`phonemizer` feature
|
||||||
|
- Wrap `espeak_ng::Voice` with our Phonemizer API
|
||||||
|
- Standalone smoke test: phonemize "He hoped there would be stew" →
|
||||||
|
verify IPA output
|
||||||
|
|
||||||
|
2. **`docs/voxtream_paper_summary.md` (~0.5 day)**
|
||||||
|
- Read arxiv 2509.15969 carefully
|
||||||
|
- Document the exact mechanism: phoneme-transformer, monotonic
|
||||||
|
alignment, dynamic look-ahead
|
||||||
|
- Resolve open question (1) above
|
||||||
|
|
||||||
|
3. **Phoneme hint integration in Generator (~2 days)**
|
||||||
|
- Decide format (inline / segment / new prompt slot)
|
||||||
|
- Modify `prompt::build_prompt` to accept phoneme hints
|
||||||
|
- Test with a known phrase: does CSM produce more natural prosody
|
||||||
|
with phoneme hints?
|
||||||
|
|
||||||
|
4. **Look-ahead window in Converse::run_streaming (~1 day)**
|
||||||
|
- Maintain a sliding phoneme buffer
|
||||||
|
- Trigger phonemize on word boundary in the LLM stream
|
||||||
|
- Pass updated buffer to Generator on each TTS call
|
||||||
|
|
||||||
|
5. **A/B bench (~1 day)**
|
||||||
|
- Same `converse_server_bench` setup, with vs without look-ahead
|
||||||
|
- Measure: sentence[0] tts_gen, llm_to_first_audio, client TTFA
|
||||||
|
- If win is < 100 ms on M-series, deprioritize and document
|
||||||
|
|
||||||
|
6. **Wire into converse_server (~1 day, if perf justified)**
|
||||||
|
- New `--lookahead-words N` flag on converse_server
|
||||||
|
- Default 0 (off); recommend 5 for production English
|
||||||
|
- Update `docs/perf_history.md`
|
||||||
|
|
||||||
|
## Alternative if VoXtream mechanism is too speculative
|
||||||
|
|
||||||
|
If the paper's exact mechanism turns out to require model retraining
|
||||||
|
(e.g., a phoneme cross-attention head), we have a simpler win
|
||||||
|
available: **streaming-Mimi reduction in chunk_frames** from 4 to 2.
|
||||||
|
Empirically measured this in Phase 6c.3a; current default 4 frames =
|
||||||
|
320 ms latency. Drop to 2 = 160 ms but adds decode overhead (Mimi has
|
||||||
|
per-call setup cost).
|
||||||
|
|
||||||
|
That's a 1-line change, no phonemizer needed. Worth measuring before
|
||||||
|
committing to the full VoXtream port.
|
||||||
|
|
||||||
|
## Recommended order of attack for the next session
|
||||||
|
|
||||||
|
1. Drop `chunk_frames=4` to `chunk_frames=2` and `chunk_frames=1` —
|
||||||
|
measure first-chunk latency. If win ≥ 200 ms, ship it as Phase 9.0
|
||||||
|
and skip the phonemizer port entirely.
|
||||||
|
2. If chunk_frames tuning isn't enough, then implement the espeak-ng
|
||||||
|
wrapper (task 1 above) — bounded 0.5-day spike.
|
||||||
|
3. Read the VoXtream paper carefully (task 2). DO NOT IMPLEMENT until
|
||||||
|
the mechanism is clear from the paper, not the abstract.
|
||||||
|
4. Iterate per the task list above.
|
||||||
|
|
||||||
|
## Cited sources
|
||||||
|
|
||||||
|
- Paper: arXiv 2509.15969 — VoXtream: Full-Stream TTS with Extremely Low Latency
|
||||||
|
- Phonemizer crates verified on crates.io (Phase 9.1 spike, agent ID `<not_returned_this_session>`)
|
||||||
|
- Production baseline numbers: `docs/perf_history.md` Phase 8.11 row
|
||||||
Reference in New Issue
Block a user