# rtx-csm performance history

A consolidated record of the optimization journey across Phases 6, 7, and
8. Includes: every shipped win, every rejected path with the data behind
the rejection, the current production-recommended config, and the
deferred multi-session work with honest sizing.

Hardware throughout: M-series Apple Silicon, Metal backend. Numbers from
`examples/converse_server_bench` against `/tmp/asr_test.flac` (10.43 s
LibriSpeech) unless noted. All commits on `main`.

## TL;DR — production config (post-Phase 8.10)

For **English-only** voice loops (Moonshine STT replaces Kyutai):

```bash
cargo build -p rtx-csm --release --features metal --example converse_server

target/release/examples/converse_server \
    --bind 0.0.0.0:18090 \
    --quantized-gguf /tmp/csm_q8.gguf \
    --stream-tts \
    --moonshine \
    --auth-token "${RTX_AUTH_TOKEN}" \
    --rate-audio-secs-per-min 600 \
    --rate-turns-per-min 60 \
    [--llm-base ... --llm-model ... --llm-api-key ...] \
    [--llm-extra-body '{"thinking":{"type":"disabled"}}'   # for Z.AI]
```

For **multilingual** (en + fr) voice loops, keep Kyutai default and add
`--vad-gate` for silence skipping:

```bash
target/release/examples/converse_server ... --vad-gate
```

### Single-turn against real Z.AI glm-4.5 (production stack, English):

| Metric                | Value     |
|-----------------------|-----------|
| recv_phase            | 0 ms (batch ASR) |
| stt_post (Moonshine)  | 332 ms    |
| llm_to_first_audio    | 1627 ms (LLM TTFT ~1.3 s + first TTS chunk) |
| **total_turn**        | **8765 ms** |
| **client TTFA**       | **1959 ms** |
| Boot warm-up cost     | ~2.7 s (one-time) |

vs Kyutai-based production from Phase 6f.q8: total_turn 17145 ms → 8765 ms
**(-49 %)**. Half the wall-clock latency end-to-end.

### Mock-LLM 3-turn bench (Kyutai-based config, multilingual default):

| Metric                         | Value          |
|--------------------------------|----------------|
| **client first_audio p50**     | **529 ms**     |
| recv_phase                     | ~4 s (LibriSpeech) / ~2 s (silence-heavy) |
| llm_to_first_audio (server)    | ~620 ms        |
| total_turn (mock LLM)          | ~18 s          |

## Per-phase shipped wins

### Phase 6 — feature surface (foundational)

Every base capability of the conversation server. Each shipped with its
own commit.

| Item | Win | Commit |
|------|-----|--------|
| WebSocket conversation server (Phase 6c.2) | end-to-end voice round-trip | `7e88a35` |
| Streaming TTS chunks per sentence (6c.3a) | first_audio < per-sentence gen time | `<earlier>` |
| Semantic VAD via Kyutai head-2 (6c.3b) | auto-EOT without client signal | `1ac0f97` |
| Barge-in mid-speak (6c.3c + 6c.3e fade-out) | user can interrupt assistant | `657768a`, `8ea3055` |
| Parallel STT during receive (6c.3d) | post-EOT flush ~free (29 ms) | `2045d7d` |
| Auth + rate-limit + /metrics + graceful shutdown (6d.*) | production-ready | several |
| Q8 GGUF flag (6f.q8) | -2.1 s first-frame TTS | `8ea3055` |
| LoRA voice in server (6f.lora) | voice clone in conv pipeline | `3162f0f` |
| AudioSeal watermark in server (6f.wm) | provenance, ~73 ms cost (~1%) | `3162f0f` |

### Phase 7 — observability + initial perf sweep

| Item | Outcome | Commit |
|------|---------|--------|
| Per-phase tracing in handle_connection | recv / stt_post / llm_to_first_audio / conv_total / total_turn gauges | `334d933` |
| conv-phase per-sentence logs | ttft / llm_buffer / tts_gen per sentence | `d15dfbc` |
| Boot-time warm-up (initial) | -1.4 s on first-sentence TTS gen | `abc07ff` |
| `GenConfig.extra_body` for Z.AI thinking-disabled | -30 s TTFT vs thinking-on | `669319d` |
| Streaming TTS via `Converse::run_streaming` + spawn-conv-task | -2.95 s `llm_to_first_audio` | `ae400a7` |
| Bench parses /metrics phase gauges | self-explanatory bench output | `2ce6f8f` |
| STT step_pcm in spawn_blocking | multi-tenant concurrency win | `10c0063` |

### Phase 8 — research-driven optimization

| Item | Outcome | Commit |
|------|---------|--------|
| ort runtime-conflict gate (8.1.2) | pass on Metal alone, fail on Metal + sentencepiece | `cdcfbce` |
| Energy-based VAD gate (8.1.3b) | -66 % recv_phase on silence-heavy audio; -7 % on speech-heavy | `808c9fa` |
| Silence-heavy A/B WAV generator | validates VAD gate structurally | (with above) |
| Extended boot warm-up (8.2) | **-89 % client TTFA p50** (4915 ms → 529 ms) | `6438672` |
| Moonshine v2 inspector + port notes (8.4) | foundation for tier-2 STT swap | `c848abe` |
| Moonshine conv stem (8.5) | 10 s audio → (1, 415, 288), max abs 23.17 | `0699cde` |
| Moonshine encoder transformer (8.6) | 6 layers + partial RoPE, forward 132 ms | `a8e729a` |
| Moonshine decoder transformer (8.7) | self-attn + cross-attn + SwiGLU, logits (1, 1, 32768) | `b94edca` |
| Moonshine end-to-end transcription (8.8) | near-perfect transcript, RTF 0.068 × | `b22ff35` |
| Moonshine KV cache + profile binary (8.9) | RTF 0.0294 × (~34× faster than realtime) | `95015c1` |
| **Moonshine in converse_server (8.10)** | **TTFA -80 % (4915 ms → 939 ms)** vs Kyutai+warmup | `932a93c` |

## Watermarker backend matrix

Two watermarkers ship in rtx-csm. Both shaped behind the same
`Watermarker` trait + `ResampledWatermarker(24 kHz <-> 16 kHz)`.

| Backend | Capacity | Embed RTF | Detect RTF | Conflicts | Notes |
|---------|----------|-----------|------------|-----------|-------|
| AudioSeal (Meta) | 16 bits | not timed | not timed | none | Per-sample presence map. ~30M params total. Phase 4 |
| **SilentCipher (Sesame)** | ~24 bits | 0.10× | 0.14× | none | **Sesame's actual production watermarker.** ~3M params. Phase 10. |

Both bit-perfect on `/tmp/asr_test.flac` round-trip with `0xCAFE`
message: AudioSeal 16/16 bits, SilentCipher 15/15 codes, both with
mean_presence/confidence 1.0000.

End-to-end through the full converse_server WS pipeline (24 → 16 →
24 kHz resample, mock LLM, Q8 + Kyutai): SilentCipher confidence
0.7614, message 0xCAFE recovered cleanly (above the 0.7 threshold
used for `Option<u16>` mapping).

### Recommendation

| Use case | Watermarker |
|----------|-------------|
| Literal Sesame parity (production CSM watermark identity) | **SilentCipher** |
| Per-sample presence map (e.g. clip detection, diff localization) | AudioSeal |
| Either / don't care | SilentCipher (smaller, identical bit accuracy on real audio) |

Build flags:
- AudioSeal: `--watermark-generator <safetensors> --watermark-detector <safetensors>`
- SilentCipher: `--watermark-silentcipher hf` (or local dir with the 3 .ckpt files)

## STT backend matrix (final)

Three STT backends now ship in rtx-csm, tested on the same 10.43 s
LibriSpeech WAV:

| Backend | Realtime factor | Build flag | Constraint |
|---------|-----------------|------------|------------|
| Kyutai STT 1B | 1.01× | `--features metal` (default) | en + fr; 3 GB; hardware-bound on M-series |
| Whisper-tiny via whisper-rs | 0.020× standalone | `--features asr` | **breaks CSM** when linked (in-process ggml/Metal contention) |
| **Moonshine-tiny (Phase 8.5-8.10)** | **0.0294×** | `--features metal` (default) | English-only; pure candle, **no runtime conflict** |

### Recommendation by deploy

| Deploy | STT choice | Why |
|--------|------------|-----|
| English-only, latency-sensitive (voice loop) | **Moonshine** | Sub-second TTFA, no runtime conflict |
| Multilingual (en + fr) | Kyutai 1B | Only multilingual option that integrates cleanly |
| Quality > latency, batch processing | Kyutai 1B | Larger model, better long-form accuracy |
| Just-want-to-try | Whisper standalone | Not for in-process voice server (breaks CSM) |

## Per-phase rejected paths (with data)

This is the equally important half — items investigated and not shipped,
so future sessions don't redo the work.

### Q4_K_M quantization (Phase 7.3)
Generated `/tmp/csm_q4km.gguf` (1.6 GB, vs Q8's 2.0 GB, 3.9× compression
of FP). Bench: `total_turn = 53400 ms` vs Q8's `18707 ms` — **2.85×
slower**. Q4_K_M's block-quantization dequant cost on Metal exceeds the
memory bandwidth savings. **Production: stay on Q8.**

### Whisper-tiny via whisper-rs (Phase 7.6)
Standalone Whisper-tiny on Metal: 209 ms for 10.43 s audio (50× faster
than Kyutai's 1× realtime). But linking whisper-rs's C++/ggml runtime
into the same binary regresses CSM by **2-3× across all inference**, even
when whisper isn't active. Suspected: ggml/Metal context vs candle/Metal
contention. **Production: build without `asr` feature; use Whisper only
via sidecar process pattern (deferred).** `examples/whisper_profile`
ships as a standalone diagnostic.

### Silero V5 VAD via `voice_activity_detector` / `ort` (Phase 8.1.3)
Initial gate test (`ort_conflict_probe`) passed (CSM Metal forwards 645
→ 633 ms after ort load, ratio 0.981, no Metal contention). But on first
real server boot:
```
[libprotobuf FATAL ...] This program was compiled against version
3.14.0 of the Protocol Buffer runtime library, which is not compatible
with the installed version (3.21.12) ... in sentencepiece-sys-0.13.1
```
`ort` 3.21 protobuf collides with `sentencepiece-sys` 3.14 (used by
Kyutai STT detok). Updated probe to also load Kyutai — now correctly
catches this. **Production: use the energy-VAD gate (Phase 8.1.3b)
instead.** Silero V5 only viable via candle-native port or sidecar.

### Mimi codec Q8 (Phase 7.4 / 8.x)
candle-transformers' Mimi uses `Conv1d`/`Conv1dTranspose`, not
`Linear`/`QMatMul`. No off-the-shelf quant path. ~1-2 weeks to fork
candle-transformers' Mimi and write custom quantized conv layers, for
marginal win (Mimi decode is small fraction of TTS time). **Skip.**

### KV cache reuse across sentences (Phase 7.5)
Conv-phase data shows sentence variance is content-dependent, not
state-dependent: `sentence[1] = 504 ms` vs `sentence[2] = 2917 ms` in
the same warmed bench. KV reuse won't move the needle. **Skip.**

### Rayon parallelism (Phase 8 research)
Survey concluded: DSP tail is < 5 ms; IIR/state recurrences (HPF biquad,
LUFS gating) can't parallelize without algorithmic rework; rayon
overhead (≥ 1 µs per join) exceeds gain on tasks < 100 µs. The only
real lever is multi-tenant continuous batching of the backbone forward
— a separate ~1500-LOC architectural project. **Skip rayon as a
single-connection optimization.**

### Codec swap (FlexiCodec / DualCodec / VARSTok)
All require full backbone retrain — CSM was trained against Mimi tokens.
20-25+ days each for marginal latency gain. **Skip.**

### Custom CSM distillation (100-300M)
No published checkpoint at this size; 30+ days to roll a knowledge-
distillation pipeline. **Skip until a real consumer needs it.**

### Continuous batching (multi-tenant)
~500-1500 LOC architectural project. Separate concern from
single-connection latency. **Skip until a real multi-tenant deploy
exists.**

### Kyutai 4× post-EOT flush trick (Phase 8.1.1)
Cited Moshi-paper trick. Re-examined: assumes audio-arrival-bottleneck;
on M-series Metal we're already at the per-frame compute floor. Quality
fix (drain asr_delay buffer for trailing words) not a perf fix.
**Defer to a separate quality sweep.**

## Deferred multi-session work (Tier 2 / Tier 3)

These remain in the plan at `~/.claude/plans/jiggly-cuddling-sparkle.md`
with realistic sizing. None can be completed in a single session.

| Item | Why valuable | Honest size | Trigger to start |
|------|--------------|-------------|------------------|
| **VoXtream-style look-ahead** (arXiv 2509.15969) | -30 % first-sentence TTS gen | 5-7 days | TTFA budget tightens below 500 ms |
| **Moonshine v2 STT in candle** (arXiv 2602.12241) | 50 ms TTFT (5.8× whisper-tiny) | 3-5 days, English-only | English-only deploy + Kyutai too slow |
| **Frame-Stacked Local Transformers** (arXiv 2509.19592) | 2.1× backbone throughput | 7-10 days + training | Real consumer hits 3 s/utterance ceiling |
| **VADUSA speculative decoding** (arXiv 2410.21951) | 1.5× compounded with above | 10-15 days + training | After Frame-Stacked, if more headroom needed |
| **Whisper sidecar process** | unblocks the 50× Whisper win | 5-10 days (IPC + lifecycle) | English-only TTFA target |
| **Silero V5 candle port** | unblocks the protobuf conflict | 3-5 days (small RNN) | Production needs robust silence detection |
| **clawsample-csm integration** | multi-tenant managed service | 5-10 days | Real consumer for public TTS API (plan: `docs/clawsample_integration_plan.md`) |

**Combined Tier 3 estimate (Frame-Stacked + VADUSA): 3-3.5× total TTS
speedup.** Cuts ~3 s per-utterance to ~0.9 s, plausibly drops total_turn
from 17 s to 8-10 s. But requires fine-tuning runs on Sesame's released
CSM-1B + a small audio dataset.

## Architectural lessons captured

1. **In-process linkage of external ML runtimes is a recurring trap.**
   ggml (whisper-rs) hits Metal contention; ort + sentencepiece-sys hits
   protobuf version mismatch. Both required runtime gates we hadn't
   anticipated. Future ML deps that aren't pure-candle should default
   to **sidecar-process pattern** rather than linking.

2. **Bench thermals dominate single-machine A/B variance.** A 90-second
   cooldown between runs is often necessary. Looking at one bench in
   isolation can mislead by 2-3×; always re-run from cold to confirm.

3. **First-frame compilation is the dominant cold-start cost.** On
   Metal, JIT kernel compilation triggers the first time a code path is
   exercised. A boot warm-up that exercises only short paths leaves
   long-context paths uncompiled — the *first user turn* still pays
   them. Phase 8.2's longer warm-up (`max_audio_ms=2000`) closed this gap.

4. **Conv-phase per-sentence variance is content-dependent.** TTS gen
   time depends on text length × frame count × early-EOT behavior, not
   on whether KV cache is hot. Don't optimize for state amortization
   until empirical data shows state matters.

5. **`tokio::join!` polls cooperatively in one task.** Sync inference
   inside one of the joined futures blocks the entire task and
   defeats parallelism with the other future. Spawn separate tasks
   when you need real concurrency between sync compute and an async
   pump (Phase 6f.stream-tts learned this the hard way).

## How to verify any claim in this doc

Every number is reproducible via:
```bash
# 1. Build
cargo build -p rtx-csm --release --features metal --example converse_server
cargo build -p rtx-csm --release --features metal --example converse_server_bench

# 2. Generate Q8 GGUF (one-time)
cargo run -p rtx-csm --release --features metal --example quantize -- \
    --policy q8 --out /tmp/csm_q8.gguf

# 3. Bench any flag combination
target/release/examples/converse_server --bind 127.0.0.1:18099 \
    --mock-llm --quantized-gguf /tmp/csm_q8.gguf --stream-tts &
target/release/examples/converse_server_bench \
    --base http://127.0.0.1:18099 --turns 3
```

Standalone profiles for STT and Whisper:
```bash
target/release/examples/stt_profile --in /tmp/asr_test.flac
cargo run --release --features asr-metal --example whisper_profile -- \
    --in /tmp/asr_test.flac
```

ort runtime-conflict gate:
```bash
cargo run --release --features metal,vad --example ort_conflict_probe
```

Silence-heavy A/B for the energy VAD:
```bash
cargo run --release --example make_silence_test
target/release/examples/converse_server_bench --base http://127.0.0.1:18099 \
    --turns 3 --in /tmp/asr_silence_heavy.wav
```