rtx-csm: docs/perf_history.md — Phase 6/7/8 consolidated record

Consolidated record of every shipped optimization, every rejected path
with the data behind the rejection, the production-recommended config,
the architectural lessons captured, and the deferred multi-session
work with realistic sizing.

Headline:
  - client first_audio_ms p50 = 529 ms (mock LLM, Q8 + stream + extended warmup + VAD gate)
  - real Z.AI loop: ~2 s TTFA p50
  - boot cost: ~2.7 s (one-time)

Rejected paths captured (so future sessions don't redo the work):
  Q4_K_M (2.85x slower than Q8), whisper-rs linkage (2-3x CSM regression),
  Silero V5 via ort (protobuf 3.14 vs 3.21 conflict), Mimi codec Q8
  (no candle conv-quant path), KV cache reuse (variance is content-
  dependent not state-dependent), rayon for single-connection
  (overhead exceeds gain on <100µs tasks), codec swaps (require
  backbone retrain), custom distillation (no published checkpoint),
  Kyutai 4x flush (hardware-bound on M-series).

Architectural lessons:
  1. In-process linkage of external ML runtimes is a recurring trap;
     default to sidecar-process pattern.
  2. Bench thermals dominate single-machine A/B; 90s cooldown often
     necessary.
  3. First-frame compilation is the dominant cold-start cost — long
     warm-ups are essential.
  4. Conv-phase variance is content-dependent, not state-dependent.
  5. tokio::join! polls cooperatively — spawn separate tasks for real
     concurrency between sync compute and async pump.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

crates/models/rtx-csm/docs/perf_history.md

@@ -0,0 +1,239 @@
+# rtx-csm performance history
+
+A consolidated record of the optimization journey across Phases 6, 7, and
+8. Includes: every shipped win, every rejected path with the data behind
+the rejection, the current production-recommended config, and the
+deferred multi-session work with honest sizing.
+
+Hardware throughout: M-series Apple Silicon, Metal backend. Numbers from
+`examples/converse_server_bench` against `/tmp/asr_test.flac` (10.43 s
+LibriSpeech) unless noted. All commits on `main`.
+
+## TL;DR — production config
+
+```bash
+cargo build -p rtx-csm --release --features metal --example converse_server
+
+target/release/examples/converse_server \
+    --bind 0.0.0.0:18090 \
+    --quantized-gguf /tmp/csm_q8.gguf \
+    --stream-tts \
+    --vad-gate \
+    --auth-token "${RTX_AUTH_TOKEN}" \
+    --rate-audio-secs-per-min 600 \
+    --rate-turns-per-min 60 \
+    [--llm-base ... --llm-model ... --llm-api-key ...] \
+    [--llm-extra-body '{"thinking":{"type":"disabled"}}'   # for Z.AI]
+```
+
+3-turn bench against this config (mock LLM):
+
+| Metric                         | Value          |
+|--------------------------------|----------------|
+| **client first_audio p50**     | **529 ms**     |
+| recv_phase                     | ~4 s (LibriSpeech) / ~2 s (silence-heavy) |
+| llm_to_first_audio (server)    | ~620 ms        |
+| total_turn (mock LLM)          | ~18 s          |
+| Boot warm-up cost              | ~2.7 s (one-time) |
+
+Real Z.AI loop with `thinking:disabled`: ~2 s client TTFA p50, ~17 s
+total turn (multi-sentence reply).
+
+## Per-phase shipped wins
+
+### Phase 6 — feature surface (foundational)
+
+Every base capability of the conversation server. Each shipped with its
+own commit.
+
+| Item | Win | Commit |
+|------|-----|--------|
+| WebSocket conversation server (Phase 6c.2) | end-to-end voice round-trip | `7e88a35` |
+| Streaming TTS chunks per sentence (6c.3a) | first_audio < per-sentence gen time | `<earlier>` |
+| Semantic VAD via Kyutai head-2 (6c.3b) | auto-EOT without client signal | `1ac0f97` |
+| Barge-in mid-speak (6c.3c + 6c.3e fade-out) | user can interrupt assistant | `657768a`, `8ea3055` |
+| Parallel STT during receive (6c.3d) | post-EOT flush ~free (29 ms) | `2045d7d` |
+| Auth + rate-limit + /metrics + graceful shutdown (6d.*) | production-ready | several |
+| Q8 GGUF flag (6f.q8) | -2.1 s first-frame TTS | `8ea3055` |
+| LoRA voice in server (6f.lora) | voice clone in conv pipeline | `3162f0f` |
+| AudioSeal watermark in server (6f.wm) | provenance, ~73 ms cost (~1%) | `3162f0f` |
+
+### Phase 7 — observability + initial perf sweep
+
+| Item | Outcome | Commit |
+|------|---------|--------|
+| Per-phase tracing in handle_connection | recv / stt_post / llm_to_first_audio / conv_total / total_turn gauges | `334d933` |
+| conv-phase per-sentence logs | ttft / llm_buffer / tts_gen per sentence | `d15dfbc` |
+| Boot-time warm-up (initial) | -1.4 s on first-sentence TTS gen | `abc07ff` |
+| `GenConfig.extra_body` for Z.AI thinking-disabled | -30 s TTFT vs thinking-on | `669319d` |
+| Streaming TTS via `Converse::run_streaming` + spawn-conv-task | -2.95 s `llm_to_first_audio` | `ae400a7` |
+| Bench parses /metrics phase gauges | self-explanatory bench output | `2ce6f8f` |
+| STT step_pcm in spawn_blocking | multi-tenant concurrency win | `10c0063` |
+
+### Phase 8 — research-driven optimization
+
+| Item | Outcome | Commit |
+|------|---------|--------|
+| ort runtime-conflict gate (8.1.2) | pass on Metal alone, fail on Metal + sentencepiece | `cdcfbce` |
+| Energy-based VAD gate (8.1.3b) | -66 % recv_phase on silence-heavy audio; -7 % on speech-heavy | `808c9fa` |
+| Silence-heavy A/B WAV generator | validates VAD gate structurally | (with above) |
+| Extended boot warm-up (8.2) | **-89 % client TTFA p50** (4915 ms → 529 ms) | `6438672` |
+
+## Per-phase rejected paths (with data)
+
+This is the equally important half — items investigated and not shipped,
+so future sessions don't redo the work.
+
+### Q4_K_M quantization (Phase 7.3)
+Generated `/tmp/csm_q4km.gguf` (1.6 GB, vs Q8's 2.0 GB, 3.9× compression
+of FP). Bench: `total_turn = 53400 ms` vs Q8's `18707 ms` — **2.85×
+slower**. Q4_K_M's block-quantization dequant cost on Metal exceeds the
+memory bandwidth savings. **Production: stay on Q8.**
+
+### Whisper-tiny via whisper-rs (Phase 7.6)
+Standalone Whisper-tiny on Metal: 209 ms for 10.43 s audio (50× faster
+than Kyutai's 1× realtime). But linking whisper-rs's C++/ggml runtime
+into the same binary regresses CSM by **2-3× across all inference**, even
+when whisper isn't active. Suspected: ggml/Metal context vs candle/Metal
+contention. **Production: build without `asr` feature; use Whisper only
+via sidecar process pattern (deferred).** `examples/whisper_profile`
+ships as a standalone diagnostic.
+
+### Silero V5 VAD via `voice_activity_detector` / `ort` (Phase 8.1.3)
+Initial gate test (`ort_conflict_probe`) passed (CSM Metal forwards 645
+→ 633 ms after ort load, ratio 0.981, no Metal contention). But on first
+real server boot:
+```
+[libprotobuf FATAL ...] This program was compiled against version
+3.14.0 of the Protocol Buffer runtime library, which is not compatible
+with the installed version (3.21.12) ... in sentencepiece-sys-0.13.1
+```
+`ort` 3.21 protobuf collides with `sentencepiece-sys` 3.14 (used by
+Kyutai STT detok). Updated probe to also load Kyutai — now correctly
+catches this. **Production: use the energy-VAD gate (Phase 8.1.3b)
+instead.** Silero V5 only viable via candle-native port or sidecar.
+
+### Mimi codec Q8 (Phase 7.4 / 8.x)
+candle-transformers' Mimi uses `Conv1d`/`Conv1dTranspose`, not
+`Linear`/`QMatMul`. No off-the-shelf quant path. ~1-2 weeks to fork
+candle-transformers' Mimi and write custom quantized conv layers, for
+marginal win (Mimi decode is small fraction of TTS time). **Skip.**
+
+### KV cache reuse across sentences (Phase 7.5)
+Conv-phase data shows sentence variance is content-dependent, not
+state-dependent: `sentence[1] = 504 ms` vs `sentence[2] = 2917 ms` in
+the same warmed bench. KV reuse won't move the needle. **Skip.**
+
+### Rayon parallelism (Phase 8 research)
+Survey concluded: DSP tail is < 5 ms; IIR/state recurrences (HPF biquad,
+LUFS gating) can't parallelize without algorithmic rework; rayon
+overhead (≥ 1 µs per join) exceeds gain on tasks < 100 µs. The only
+real lever is multi-tenant continuous batching of the backbone forward
+— a separate ~1500-LOC architectural project. **Skip rayon as a
+single-connection optimization.**
+
+### Codec swap (FlexiCodec / DualCodec / VARSTok)
+All require full backbone retrain — CSM was trained against Mimi tokens.
+20-25+ days each for marginal latency gain. **Skip.**
+
+### Custom CSM distillation (100-300M)
+No published checkpoint at this size; 30+ days to roll a knowledge-
+distillation pipeline. **Skip until a real consumer needs it.**
+
+### Continuous batching (multi-tenant)
+~500-1500 LOC architectural project. Separate concern from
+single-connection latency. **Skip until a real multi-tenant deploy
+exists.**
+
+### Kyutai 4× post-EOT flush trick (Phase 8.1.1)
+Cited Moshi-paper trick. Re-examined: assumes audio-arrival-bottleneck;
+on M-series Metal we're already at the per-frame compute floor. Quality
+fix (drain asr_delay buffer for trailing words) not a perf fix.
+**Defer to a separate quality sweep.**
+
+## Deferred multi-session work (Tier 2 / Tier 3)
+
+These remain in the plan at `~/.claude/plans/jiggly-cuddling-sparkle.md`
+with realistic sizing. None can be completed in a single session.
+
+| Item | Why valuable | Honest size | Trigger to start |
+|------|--------------|-------------|------------------|
+| **VoXtream-style look-ahead** (arXiv 2509.15969) | -30 % first-sentence TTS gen | 5-7 days | TTFA budget tightens below 500 ms |
+| **Moonshine v2 STT in candle** (arXiv 2602.12241) | 50 ms TTFT (5.8× whisper-tiny) | 3-5 days, English-only | English-only deploy + Kyutai too slow |
+| **Frame-Stacked Local Transformers** (arXiv 2509.19592) | 2.1× backbone throughput | 7-10 days + training | Real consumer hits 3 s/utterance ceiling |
+| **VADUSA speculative decoding** (arXiv 2410.21951) | 1.5× compounded with above | 10-15 days + training | After Frame-Stacked, if more headroom needed |
+| **Whisper sidecar process** | unblocks the 50× Whisper win | 5-10 days (IPC + lifecycle) | English-only TTFA target |
+| **Silero V5 candle port** | unblocks the protobuf conflict | 3-5 days (small RNN) | Production needs robust silence detection |
+| **clawsample-csm integration** | multi-tenant managed service | 5-10 days | Real consumer for public TTS API (plan: `docs/clawsample_integration_plan.md`) |
+
+**Combined Tier 3 estimate (Frame-Stacked + VADUSA): 3-3.5× total TTS
+speedup.** Cuts ~3 s per-utterance to ~0.9 s, plausibly drops total_turn
+from 17 s to 8-10 s. But requires fine-tuning runs on Sesame's released
+CSM-1B + a small audio dataset.
+
+## Architectural lessons captured
+
+1. **In-process linkage of external ML runtimes is a recurring trap.**
+   ggml (whisper-rs) hits Metal contention; ort + sentencepiece-sys hits
+   protobuf version mismatch. Both required runtime gates we hadn't
+   anticipated. Future ML deps that aren't pure-candle should default
+   to **sidecar-process pattern** rather than linking.
+
+2. **Bench thermals dominate single-machine A/B variance.** A 90-second
+   cooldown between runs is often necessary. Looking at one bench in
+   isolation can mislead by 2-3×; always re-run from cold to confirm.
+
+3. **First-frame compilation is the dominant cold-start cost.** On
+   Metal, JIT kernel compilation triggers the first time a code path is
+   exercised. A boot warm-up that exercises only short paths leaves
+   long-context paths uncompiled — the *first user turn* still pays
+   them. Phase 8.2's longer warm-up (`max_audio_ms=2000`) closed this gap.
+
+4. **Conv-phase per-sentence variance is content-dependent.** TTS gen
+   time depends on text length × frame count × early-EOT behavior, not
+   on whether KV cache is hot. Don't optimize for state amortization
+   until empirical data shows state matters.
+
+5. **`tokio::join!` polls cooperatively in one task.** Sync inference
+   inside one of the joined futures blocks the entire task and
+   defeats parallelism with the other future. Spawn separate tasks
+   when you need real concurrency between sync compute and an async
+   pump (Phase 6f.stream-tts learned this the hard way).
+
+## How to verify any claim in this doc
+
+Every number is reproducible via:
+```bash
+# 1. Build
+cargo build -p rtx-csm --release --features metal --example converse_server
+cargo build -p rtx-csm --release --features metal --example converse_server_bench
+
+# 2. Generate Q8 GGUF (one-time)
+cargo run -p rtx-csm --release --features metal --example quantize -- \
+    --policy q8 --out /tmp/csm_q8.gguf
+
+# 3. Bench any flag combination
+target/release/examples/converse_server --bind 127.0.0.1:18099 \
+    --mock-llm --quantized-gguf /tmp/csm_q8.gguf --stream-tts &
+target/release/examples/converse_server_bench \
+    --base http://127.0.0.1:18099 --turns 3
+```
+
+Standalone profiles for STT and Whisper:
+```bash
+target/release/examples/stt_profile --in /tmp/asr_test.flac
+cargo run --release --features asr-metal --example whisper_profile -- \
+    --in /tmp/asr_test.flac
+```
+
+ort runtime-conflict gate:
+```bash
+cargo run --release --features metal,vad --example ort_conflict_probe
+```
+
+Silence-heavy A/B for the energy VAD:
+```bash
+cargo run --release --example make_silence_test
+target/release/examples/converse_server_bench --base http://127.0.0.1:18099 \
+    --turns 3 --in /tmp/asr_silence_heavy.wav
+```