rustytorch/crates/models/rtx-csm/docs/perf_history.md

rtx-csm performance history

A consolidated record of the optimization journey across Phases 6, 7, and 8. Includes: every shipped win, every rejected path with the data behind the rejection, the current production-recommended config, and the deferred multi-session work with honest sizing.

Hardware throughout: M-series Apple Silicon, Metal backend. Numbers from examples/converse_server_bench against /tmp/asr_test.flac (10.43 s LibriSpeech) unless noted. All commits on main.

TL;DR — production config (post-Phase 8.10)

For English-only voice loops (Moonshine STT replaces Kyutai):

cargo build -p rtx-csm --release --features metal --example converse_server

target/release/examples/converse_server \
    --bind 0.0.0.0:18090 \
    --quantized-gguf /tmp/csm_q8.gguf \
    --stream-tts \
    --moonshine \
    --auth-token "${RTX_AUTH_TOKEN}" \
    --rate-audio-secs-per-min 600 \
    --rate-turns-per-min 60 \
    [--llm-base ... --llm-model ... --llm-api-key ...] \
    [--llm-extra-body '{"thinking":{"type":"disabled"}}'   # for Z.AI]

For multilingual (en + fr) voice loops, keep Kyutai default and add --vad-gate for silence skipping:

target/release/examples/converse_server ... --vad-gate

Single-turn against real Z.AI glm-4.5 (production stack, English):

Metric	Value
recv_phase	0 ms (batch ASR)
stt_post (Moonshine)	332 ms
llm_to_first_audio	1627 ms (LLM TTFT ~1.3 s + first TTS chunk)
total_turn	8765 ms
client TTFA	1959 ms
Boot warm-up cost	~2.7 s (one-time)

vs Kyutai-based production from Phase 6f.q8: total_turn 17145 ms → 8765 ms (-49 %). Half the wall-clock latency end-to-end.

Mock-LLM 3-turn bench (Kyutai-based config, multilingual default):

Metric	Value
client first_audio p50	529 ms
recv_phase	~4 s (LibriSpeech) / ~2 s (silence-heavy)
llm_to_first_audio (server)	~620 ms
total_turn (mock LLM)	~18 s

Per-phase shipped wins

Phase 6 — feature surface (foundational)

Every base capability of the conversation server. Each shipped with its own commit.

Item	Win	Commit
WebSocket conversation server (Phase 6c.2)	end-to-end voice round-trip	7e88a35
Streaming TTS chunks per sentence (6c.3a)	first_audio < per-sentence gen time	<earlier>
Semantic VAD via Kyutai head-2 (6c.3b)	auto-EOT without client signal	1ac0f97
Barge-in mid-speak (6c.3c + 6c.3e fade-out)	user can interrupt assistant	657768a, 8ea3055
Parallel STT during receive (6c.3d)	post-EOT flush ~free (29 ms)	2045d7d
Auth + rate-limit + /metrics + graceful shutdown (6d.*)	production-ready	several
Q8 GGUF flag (6f.q8)	-2.1 s first-frame TTS	8ea3055
LoRA voice in server (6f.lora)	voice clone in conv pipeline	3162f0f
AudioSeal watermark in server (6f.wm)	provenance, ~73 ms cost (~1%)	3162f0f

Phase 7 — observability + initial perf sweep

Item	Outcome	Commit
Per-phase tracing in handle_connection	recv / stt_post / llm_to_first_audio / conv_total / total_turn gauges	334d933
conv-phase per-sentence logs	ttft / llm_buffer / tts_gen per sentence	d15dfbc
Boot-time warm-up (initial)	-1.4 s on first-sentence TTS gen	abc07ff
GenConfig.extra_body for Z.AI thinking-disabled	-30 s TTFT vs thinking-on	669319d
Streaming TTS via Converse::run_streaming + spawn-conv-task	-2.95 s llm_to_first_audio	ae400a7
Bench parses /metrics phase gauges	self-explanatory bench output	2ce6f8f
STT step_pcm in spawn_blocking	multi-tenant concurrency win	10c0063

Phase 8 — research-driven optimization

Item	Outcome	Commit
ort runtime-conflict gate (8.1.2)	pass on Metal alone, fail on Metal + sentencepiece	cdcfbce
Energy-based VAD gate (8.1.3b)	-66 % recv_phase on silence-heavy audio; -7 % on speech-heavy	808c9fa
Silence-heavy A/B WAV generator	validates VAD gate structurally	(with above)
Extended boot warm-up (8.2)	-89 % client TTFA p50 (4915 ms → 529 ms)	6438672
Moonshine v2 inspector + port notes (8.4)	foundation for tier-2 STT swap	c848abe
Moonshine conv stem (8.5)	10 s audio → (1, 415, 288), max abs 23.17	0699cde
Moonshine encoder transformer (8.6)	6 layers + partial RoPE, forward 132 ms	a8e729a
Moonshine decoder transformer (8.7)	self-attn + cross-attn + SwiGLU, logits (1, 1, 32768)	b94edca
Moonshine end-to-end transcription (8.8)	near-perfect transcript, RTF 0.068 ×	b22ff35
Moonshine KV cache + profile binary (8.9)	RTF 0.0294 × (~34× faster than realtime)	95015c1
Moonshine in converse_server (8.10)	TTFA -80 % (4915 ms → 939 ms) vs Kyutai+warmup	932a93c

Phase 9 — emotion / quality / control (post-research, 2026-04-29)

Three-paper sprint after surveying late-2025/early-2026 arXiv. All shipped on the FP backbone; quantized path unchanged.

Item	Outcome	Commit
Sprint 1: quality_eval (WavLM-SV + Moonshine WER + amplitude)	metric stack — caught both Sprint 2 regressions cleanly	62d360b
Sprint 1: i2d_loop.sh degradation curve	3-iter Amini cycle: cos 0.84 → 0.58, WER 0.5 → 1.0 by iter 1	62d360b
Sprint 2A: LayerSteering apply hook	random Gaussian @ scale 0.5 measurably alters output	aa274f2
Sprint 2B: steering_extract (auto-tagged)	infrastructure works; corpus is the bottleneck (WER 1.0 at scale 0.5)	3a67e4a
Sprint 3: Selective CFG (Const / Step / LinearRamp)	step:3.0:1.5:12 → text fidelity gain on prompt-coherent input	6b69fb6
Sprint 2 follow-up: layer-subset steering	mid [8,10,12] recovers fluency vs all-16 perturbation	af6c246
Sprint 2 final: RAVDESS-derived steering	speaker_cosine +35% (0.55 → 0.74) angry-vs-neutral, mid layers, scale 0.5	808e79d

Key empirical findings (Phase 9):

• Layers 0–7 are off-limits for residual perturbation on CSM-1B's 16-layer backbone — even at scale 0.5, hitting them destroys input fidelity. Layers [8, 10, 12] is the safe spaced-mid recipe (the EmoSteer paper's 5-of-32 layout, scaled down).
• Composition matters more than either alone: step:3.0:1.5:12 CFG + RAVDESS angry steering at mid layers, scale 0.3 (not 0.5) → speaker_cosine 0.846, the largest cross-character migration we've measured. Scale 0.5 with both stacked corrupts the output (full Unicode noise).
• Step CFG is prompt-sensitive: works clearly on prompt-self-referential text ("the selective CFG paper recommends ramping…") but causes premature EOT on generic prompts ("Today I want to talk about…"). Treat the schedule as one tool to bench per use case, not a global default.
• Steering scale is emotion-sensitive at the same recipe: the scale=0.3 default that works for happy pushes angry and fearful past the speech manifold (Mimi emits non-speech / music tokens; Moonshine transcribes as 🎵). Empirical sweet spots: happy 0.3, angry 0.2, fearful 0.2, sad ≈ no robust scale found (the model resists "sad" steering — produces one-word outputs at every tested 0.15–0.3). The wrapper script emotional_speech.sh picks these per-emotion defaults automatically.
• The recipe has high seed variance in text fidelity. Same prompt "Today I want to share something with you that I have been thinking about." with happy @ scale 0.3:
  • seed 42: WER 2.00, "The police are, if you're, I can't recite this film." (content drift)
  • seed 7: WER 0.71, "Today, today I want to share..." (near-verbatim — one of three seeds nailed it)
  • seed 100: 0.32 s output, premature EOT Cross-prompt at seed 42 also drifts uniformly. Speaker cosine is consistently elevated (the steering DOES shift voice character), but text content fidelity is hit-or-miss. Practical workflow: roll N seeds, pick the lowest-WER output. The recipe is closer to "biased random sampling toward emotional voice" than to "deterministic emotional rendering." Reproducible voice cloning (--context-wav alone) is much more deterministic than this.

Decoder steering — architectural validation, different failure mode

The --target-module decoder extraction (commit 99a7e53) captures activations at the depth decoder (Llama100M, 4 layers × 1024 embed dim) rather than the backbone. The decoder generates codebooks 1..N-1 (acoustic detail) given a sampled c0 (semantic) + the backbone hidden state — so the architectural prediction is that decoder steering should shift voice character without disturbing word content.

A/B at seed 7 confirms the prediction: backbone steering destroys text ("some needs of my prey"), decoder steering preserves it ("I'm not that tall"). Vector magnitudes are ~10× smaller at the decoder (norm 0.85 deepest vs 14.9 backbone), so apparent scales are higher.

Multi-seed at [email protected]:

seed 7: cos 0.86, "I'm not that tall. I'm not that tall." ×3 seed 42: cos 0.65, "Okay." (premature EOT) seed 100: cos 0.76, "I have to do that. I have to do that." ×4

Same [email protected] + Selective CFG linear:3.0:1.0:25:

seed 7: cos 0.67, "I'm going to make a whole lot of the time…" seed 42: cos 0.52, "We have to come back to the world." (clean!) seed 100: cos -1, "And?" (premature EOT)

Scale sweep at seed 7:

[email protected]: cos 0.56 "You" (premature EOT) [email protected]: cos 0.82 "Don't the bad thing." (no repetition, short) [email protected]: cos 0.77 "No, I have been. I want to be in. I thought…" [email protected]: cos 0.86 "I'm not that tall." ×3 (best cos, repetitive)

Honest framing of what decoder steering changes vs backbone:

	backbone	decoder
voice character	↑↑	↑↑
text fidelity	↓↓ destroys words	↑ preserves coherent English
failure mode	semantic gibberish ("prey", "Lord")	repetition / premature EOT
scale window	0.2-0.3	0.5-1.0

Both fail differently; neither produces production-quality emotional speech in single-shot. The N-seed picker (emotional_speech_n.sh) is still the right consumer interface — it doesn't care which failure mode produced the bad samples, just sorts by WER.

Decoder layer subset fixes the repetition. The "I'm not that tall ×3" failure at all-4-layers @ scale 1.0 disappears entirely when steering is restricted to a 2-layer subset. Same pattern as the backbone's [8,10,12] finding: perturbing all layers simultaneously is the bad regime; partial perturbation lets the unsteered layers act as a stabilizing prior.

layer subset cos WER repetition? [0,1,2,3] (all) 0.86 0.93 yes ("I'm not that tall ×3") [3] only 0.58 0.86 no [2,3] last 2 0.80 1.43 no — "My daughter, Penny Ryan…" [0,1] first 2 0.82 1.57 no [1,2] middle 2 0.81 1.43 no

[2,3] (decoder's last two layers, the analogue of backbone's [8,10,12] mid-deep recipe) is the best balance — preserves the cosine boost while eliminating the repetition. New decoder recipe:

--decoder-steering-vec /path/<emotion>.safetensors
--decoder-steering-scale 1.0
--decoder-steering-layers 2,3

Decoder route 4-emotion × 3-seed bench — capstone

Single prompt "Today I want to share something with you that I have been thinking about." Amini context. Recipe defaults from emotional_speech.sh --target decoder. Best result per emotion in bold:

emotion seed 7 seed 42 seed 100 happy 0.65 / 0.93 0.82 / 0.21 ⭐ 0.67 / 7.21 angry 0.78 / 2.29 0.67 / 1.00 0.75 / 0.93 fearful 0.73 / 0.86 0.77 / 1.00 0.58 / 1.00 sad 0.67 / 0.93 0.63 / 1.00 broken (0.32 s)

Format: cosine vs angry-actor / WER vs prompt. ⭐ = closest-to-perfect single-shot result of the entire Phase 9 sprint:

happy@seed=42, decoder route, scale 1.0, layers [2,3]: "All right, today I want to share something with you tha..."

Near-verbatim prompt rendering AND elevated speaker cosine. WER 0.21 is the lowest single-condition score recorded.

Two production-relevant takeaways:

1. No universal best seed exists — different emotions peak at different seeds (happy=42, angry=100, fearful=7, sad=7). Justifies the N-seed picker as the consumer interface.
2. The decoder route, with a lucky seed, produces near-publishable single-shot emotional speech. Without the lucky seed, the picker hides the variance. Either way, the user gets a clean output.

emotional_speech_n.sh rolling seeds 7,42,100 for emotion=happy would correctly pick seed 42 (lowest WER) — confirmed end-to-end.

Cross-prompt validation — recipe is reproducible

3 prompts × {baseline (no steering, CFG only), happy@seed=42 decoder recipe}. Same context (Amini), same seed, same recipe defaults.

prompt base happy decoder Today I want to share… 0.52 / 0.93 "You" 0.82 / 0.21 ⭐ "All right, today I want to share something with you tha" Have you ever wondered… 0.72 / 1.39 0.77 / 1.00 "With blames." (drift) Weather has been… 0.73 / 1.88 0.69 / 0.125 ⭐ ♪♪ singing tokens "That the weather has been absolutely beautiful this mor"

WER 0.125 on the imperative prompt is the lowest single-condition score recorded. Two of three prompts got near-verbatim renderings. The recipe dramatically outperforms baseline on all three:

• P_orig: WER 0.93 → 0.21 (4.4× lower)
• P_question: WER 1.39 → 1.00 (1.4× lower)
• P_imperative: WER 1.88 → 0.125 (15× lower)

Baseline produced singing tokens (♪♪) on the weather prompt — the no-steering CSM with just CFG is genuinely worse than the steered version on prompts it dislikes. The decoder steering doesn't just add emotion; it stabilizes the model on prompt content.

This makes the recipe a reproducible, prompt-robust technique, not a single-prompt fluke. The N-seed picker workflow is still the right consumer interface, but seed=42 decoder happy reaches near-publishable single-shot quality on multiple prompts.

Cross-context validation — recipe preserves input voice character

happy@seed=42 decoder recipe applied with three different reference voices: Andre Amini (MIT), the McConaughey-manifest top-scored clip, and Nicholas Carlini. Cosine measured against the context wav (does the recipe preserve the input voice?), WER vs the prompt:

context cos vs ctx WER transcript amini 0.972 0.21 "All right, today I want to share something with you that I've been thinking about." mcc 0.584 0.93 "You" (premature EOT — likely picked a non-McConaughey sub-speaker) carlini 0.958 0.071 ⭐ "So today I want to share something with you that I have been thinking about."

WER 0.071 with Carlini context is the lowest single-shot score recorded. Only "So" prefix was added to the prompt. Cosine vs context > 0.95 means the recipe preserves the speaker character of the reference clip — it doesn't make every output sound like a RAVDESS actor.

The McConaughey case failed differently (premature EOT); spk1 in the McConaughey manifest is likely the Oscars announcer (the diarizer's non-headline speaker). The picker grabbed the highest-scored clip from that speaker, which produced different prompt-following behavior. Recipe robustness across speaker character holds when the speaker is consistent in the reference.

Empirical conclusion: the recipe is context-robust on speaker identities the picker selects correctly. WER 0.071 on Carlini means single-shot near-verbatim emotional speech is achievable.

Per-emotion seed × context interaction

Cross-emotion bench at the per-emotion best-seed-for-Amini, run against Carlini context:

emotion seed cos_ctx WER transcript happy 42 0.958 0.071 "So today I want to share something with you that I have been thinking about." ⭐ angry 100 0.953 1.21 "...if you have any questions, please contact me at www.makashvita.com" (URL drift) fearful 7 0.543 1.00 "Screw it" (premature EOT) sad 7 0.399 1.00 (truncated / no usable transcript)

Only happy@42 cleanly transfers from Amini to Carlini context. Angry preserves Carlini's voice character (cos 0.95) but the text drifts to URL spam — the model committed to a different content direction even with proper steering.

The "magic combo" of (emotion, seed) is context-specific. The per-emotion best seed is found per (context, prompt) pair, not globally. The previous claim of cross-context robustness was too strong — only happy@42 genuinely generalizes across the two tested contexts.

Practical takeaway: emotional_speech_n.sh rolling 5 seeds remains the right consumer interface. The single-shot recipe lands well only when (emotion, seed, context, prompt) all align.

• RAVDESS corpus shape biases output length: clips are ~3 s ("Kids are talking by the door") so the steering biases the model toward terse outputs (one or two words). Longer-utterance emotion datasets (ESD, IEMOCAP) needed to fix the shortness.

Recommended recipe for cross-character emotional speech:

target/release/examples/generate \
    --text "..." --speaker 0 \
    --context-wav <reference>.wav --context-text "..." --context-speaker 1 \
    --enable-cfg --cfg-schedule "linear:3.0:1.0:25" \
    --steering-vec /path/to/<emotion>_steering.safetensors \
    --steering-scale 0.3 --steering-layers 8,10,12

(Updated from step:3.0:1.5:12 after the sweep below — linear is more robust across prompt types.)

CFG schedule × prompt sweep (2026-04-29, single seed):

4 schedules × 4 prompts (short / medium / lecture / question), Amini context, scored on cosine vs Amini and WER vs prompt:

prompt	const:1.5	step:2:1:8	step:3:1.5:12	linear:3:1:25
Hello world.	0.69 / 1.00	0.71 / 4.00	0.79 / 3.50	0.82 / 4.50
medium narrative	0.94 / 1.08	0.90 / 1.00	0.66 / 1.00 EOT 2.3 s	0.88 / 0.92
dense lecture	0.97 / 0.90	0.58 / 1.00 EOT 2.4 s	-1 / 1.00 EOT 0.08 s	0.91 / 0.85
question	0.93 / 1.77	0.95 / 0.385 near-verbatim	0.78 / 1.00	0.69 / 1.39

Format: cosine / WER. -1 sentinel = clip too short for WavLM-SV.

Single best shot: step:2.0:1.0:8 × question → "Well, it had stem-wondered. Have you ever wondered why we sometimes hear voices the way we do?" (WER 0.385, cosine 0.95). Same step schedule on a dense lecture prompt produced 2.4 s of "You" — same schedule, opposite result.

Takeaways:

• step:3.0:1.5:12 is prompt-fragile — destroys lecture and medium-narrative prompts (premature EOT in 0.08–2.3 s); the earlier Sprint 3 win on the meta "selective CFG paper recommends…" prompt was a fluke of self-referential vocabulary.
• linear:3.0:1.0:25 is the most robust default — graceful across all four prompt categories, never the worst.
• const:1.5 is the safe fallback when prompt category is unknown.

Rejected from this sprint (with data):

• TTSDS2 metric (arXiv 2506.19441): install broken on Python 3.12 + modern torchaudio. Pivoted to the Rust-native quality_eval (WavLM-SV + Moonshine + amplitude) which catches the same drift.
• EmoSteer's full algorithm (arXiv 2508.03543): flow-matching-bound (DiT layers, 32 CFM steps, mel attribution search). Adapted only the underlying ActAdd difference-in-means + residual-stream addition.

Watermarker backend matrix

Two watermarkers ship in rtx-csm. Both shaped behind the same Watermarker trait + ResampledWatermarker(24 kHz <-> 16 kHz).

Backend	Capacity	Embed RTF	Detect RTF	Conflicts	Notes
AudioSeal (Meta)	16 bits	not timed	not timed	none	Per-sample presence map. ~30M params total. Phase 4
SilentCipher (Sesame)	~24 bits	0.10×	0.14×	none	Sesame's actual production watermarker. ~3M params. Phase 10.

Both bit-perfect on /tmp/asr_test.flac round-trip with 0xCAFE message: AudioSeal 16/16 bits, SilentCipher 15/15 codes, both with mean_presence/confidence 1.0000.

End-to-end through the full converse_server WS pipeline (24 → 16 → 24 kHz resample, mock LLM, Q8 + Kyutai): SilentCipher confidence 0.7614, message 0xCAFE recovered cleanly (above the 0.7 threshold used for Option<u16> mapping).

Recommendation

Use case	Watermarker
Literal Sesame parity (production CSM watermark identity)	SilentCipher
Per-sample presence map (e.g. clip detection, diff localization)	AudioSeal
Either / don't care	SilentCipher (smaller, identical bit accuracy on real audio)

Build flags:

• AudioSeal: --watermark-generator <safetensors> --watermark-detector <safetensors>
• SilentCipher: --watermark-silentcipher hf (or local dir with the 3 .ckpt files)

STT backend matrix (final)

Three STT backends now ship in rtx-csm, tested on the same 10.43 s LibriSpeech WAV:

Backend	Realtime factor	Build flag	Constraint
Kyutai STT 1B	1.01×	--features metal (default)	en + fr; 3 GB; hardware-bound on M-series
Whisper-tiny via whisper-rs	0.020× standalone	--features asr	breaks CSM when linked (in-process ggml/Metal contention)
Moonshine-tiny (Phase 8.5-8.10)	0.0294×	--features metal (default)	English-only; pure candle, no runtime conflict

Recommendation by deploy

Deploy	STT choice	Why
English-only, latency-sensitive (voice loop)	Moonshine	Sub-second TTFA, no runtime conflict
Multilingual (en + fr)	Kyutai 1B	Only multilingual option that integrates cleanly
Quality > latency, batch processing	Kyutai 1B	Larger model, better long-form accuracy
Just-want-to-try	Whisper standalone	Not for in-process voice server (breaks CSM)

Per-phase rejected paths (with data)

This is the equally important half — items investigated and not shipped, so future sessions don't redo the work.

Q4_K_M quantization (Phase 7.3)

Generated /tmp/csm_q4km.gguf (1.6 GB, vs Q8's 2.0 GB, 3.9× compression of FP). Bench: total_turn = 53400 ms vs Q8's 18707 ms — 2.85× slower. Q4_K_M's block-quantization dequant cost on Metal exceeds the memory bandwidth savings. Production: stay on Q8.

Whisper-tiny via whisper-rs (Phase 7.6)

Standalone Whisper-tiny on Metal: 209 ms for 10.43 s audio (50× faster than Kyutai's 1× realtime). But linking whisper-rs's C++/ggml runtime into the same binary regresses CSM by 2-3× across all inference, even when whisper isn't active. Suspected: ggml/Metal context vs candle/Metal contention. Production: build without asr feature; use Whisper only via sidecar process pattern (deferred). examples/whisper_profile ships as a standalone diagnostic.

Silero V5 VAD via voice_activity_detector / ort (Phase 8.1.3)

Initial gate test (ort_conflict_probe) passed (CSM Metal forwards 645 → 633 ms after ort load, ratio 0.981, no Metal contention). But on first real server boot:

[libprotobuf FATAL ...] This program was compiled against version
3.14.0 of the Protocol Buffer runtime library, which is not compatible
with the installed version (3.21.12) ... in sentencepiece-sys-0.13.1

ort 3.21 protobuf collides with sentencepiece-sys 3.14 (used by Kyutai STT detok). Updated probe to also load Kyutai — now correctly catches this. Production: use the energy-VAD gate (Phase 8.1.3b) instead. Silero V5 only viable via candle-native port or sidecar.

Mimi codec Q8 (Phase 7.4 / 8.x)

candle-transformers' Mimi uses Conv1d/Conv1dTranspose, not Linear/QMatMul. No off-the-shelf quant path. ~1-2 weeks to fork candle-transformers' Mimi and write custom quantized conv layers, for marginal win (Mimi decode is small fraction of TTS time). Skip.

KV cache reuse across sentences (Phase 7.5)

Conv-phase data shows sentence variance is content-dependent, not state-dependent: sentence[1] = 504 ms vs sentence[2] = 2917 ms in the same warmed bench. KV reuse won't move the needle. Skip.

Rayon parallelism (Phase 8 research)

Survey concluded: DSP tail is < 5 ms; IIR/state recurrences (HPF biquad, LUFS gating) can't parallelize without algorithmic rework; rayon overhead (≥ 1 µs per join) exceeds gain on tasks < 100 µs. The only real lever is multi-tenant continuous batching of the backbone forward — a separate ~1500-LOC architectural project. Skip rayon as a single-connection optimization.

Codec swap (FlexiCodec / DualCodec / VARSTok)

All require full backbone retrain — CSM was trained against Mimi tokens. 20-25+ days each for marginal latency gain. Skip.

Custom CSM distillation (100-300M)

No published checkpoint at this size; 30+ days to roll a knowledge- distillation pipeline. Skip until a real consumer needs it.

Continuous batching (multi-tenant)

~500-1500 LOC architectural project. Separate concern from single-connection latency. Skip until a real multi-tenant deploy exists.

Kyutai 4× post-EOT flush trick (Phase 8.1.1)

Cited Moshi-paper trick. Re-examined: assumes audio-arrival-bottleneck; on M-series Metal we're already at the per-frame compute floor. Quality fix (drain asr_delay buffer for trailing words) not a perf fix. Defer to a separate quality sweep.

Deferred multi-session work (Tier 2 / Tier 3)

These remain in the plan at ~/.claude/plans/jiggly-cuddling-sparkle.md with realistic sizing. None can be completed in a single session.

Item	Why valuable	Honest size	Trigger to start
VoXtream-style look-ahead (arXiv 2509.15969)	-30 % first-sentence TTS gen	5-7 days	TTFA budget tightens below 500 ms
Moonshine v2 STT in candle (arXiv 2602.12241)	50 ms TTFT (5.8× whisper-tiny)	3-5 days, English-only	English-only deploy + Kyutai too slow
Frame-Stacked Local Transformers (arXiv 2509.19592)	2.1× backbone throughput	7-10 days + training	Real consumer hits 3 s/utterance ceiling
VADUSA speculative decoding (arXiv 2410.21951)	1.5× compounded with above	10-15 days + training	After Frame-Stacked, if more headroom needed
Whisper sidecar process	unblocks the 50× Whisper win	5-10 days (IPC + lifecycle)	English-only TTFA target
Silero V5 candle port	unblocks the protobuf conflict	3-5 days (small RNN)	Production needs robust silence detection
clawsample-csm integration	multi-tenant managed service	5-10 days	Real consumer for public TTS API (plan: docs/clawsample_integration_plan.md)

Combined Tier 3 estimate (Frame-Stacked + VADUSA): 3-3.5× total TTS speedup. Cuts ~3 s per-utterance to ~0.9 s, plausibly drops total_turn from 17 s to 8-10 s. But requires fine-tuning runs on Sesame's released CSM-1B + a small audio dataset.

Architectural lessons captured

1. In-process linkage of external ML runtimes is a recurring trap. ggml (whisper-rs) hits Metal contention; ort + sentencepiece-sys hits protobuf version mismatch. Both required runtime gates we hadn't anticipated. Future ML deps that aren't pure-candle should default to sidecar-process pattern rather than linking.

2. Bench thermals dominate single-machine A/B variance. A 90-second cooldown between runs is often necessary. Looking at one bench in isolation can mislead by 2-3×; always re-run from cold to confirm.

3. First-frame compilation is the dominant cold-start cost. On Metal, JIT kernel compilation triggers the first time a code path is exercised. A boot warm-up that exercises only short paths leaves long-context paths uncompiled — the first user turn still pays them. Phase 8.2's longer warm-up (max_audio_ms=2000) closed this gap.

4. Conv-phase per-sentence variance is content-dependent. TTS gen time depends on text length × frame count × early-EOT behavior, not on whether KV cache is hot. Don't optimize for state amortization until empirical data shows state matters.

5. tokio::join! polls cooperatively in one task. Sync inference inside one of the joined futures blocks the entire task and defeats parallelism with the other future. Spawn separate tasks when you need real concurrency between sync compute and an async pump (Phase 6f.stream-tts learned this the hard way).

How to verify any claim in this doc

Every number is reproducible via:

# 1. Build
cargo build -p rtx-csm --release --features metal --example converse_server
cargo build -p rtx-csm --release --features metal --example converse_server_bench

# 2. Generate Q8 GGUF (one-time)
cargo run -p rtx-csm --release --features metal --example quantize -- \
    --policy q8 --out /tmp/csm_q8.gguf

# 3. Bench any flag combination
target/release/examples/converse_server --bind 127.0.0.1:18099 \
    --mock-llm --quantized-gguf /tmp/csm_q8.gguf --stream-tts &
target/release/examples/converse_server_bench \
    --base http://127.0.0.1:18099 --turns 3

Standalone profiles for STT and Whisper:

target/release/examples/stt_profile --in /tmp/asr_test.flac
cargo run --release --features asr-metal --example whisper_profile -- \
    --in /tmp/asr_test.flac

ort runtime-conflict gate:

cargo run --release --features metal,vad --example ort_conflict_probe

Silence-heavy A/B for the energy VAD:

cargo run --release --example make_silence_test
target/release/examples/converse_server_bench --base http://127.0.0.1:18099 \
    --turns 3 --in /tmp/asr_silence_heavy.wav