rtx-csm: Phase 6f.warmup — pre-warm Generator at server boot

Issues one throwaway Generator.generate() call after model load + before
accepting connections. Pays Metal kernel compilation + first-frame KV
cache init up-front so the first real user turn doesn't carry that
overhead.

A/B (mock LLM, FP CSM, M-series Metal, 10.43s LibriSpeech in):
  no warm-up  sentence[0] tts_gen=5679ms  total=24818ms
  with warm-up sentence[0] tts_gen=4272ms total=18654ms

Warm-up cost at boot: 1417ms. Direct saving on first-sentence TTS gen:
~1.4s. The warm-up pays for itself on the first turn and is amortized
to zero across the server's lifetime.

Failures during warm-up are logged at warn level and the server boots
anyway — first turn falls back to the original cold-start behavior.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-04-27 04:04:27 -07:00
co-authored by Claude Opus 4.7
parent c1ce6e388c
commit abc07ffd7e
@@ -441,6 +441,33 @@ async fn main() -> Result<()> {
}; };
tracing::info!("models loaded"); tracing::info!("models loaded");
// Warm up the Generator: do one throwaway generate() so Metal kernel
// compilation + first-frame KV cache init happen now, not on the
// first user turn. Empirically this eliminates ~5-6s of dead time
// from the first-sentence TTS gen latency. Cost is paid once at
// boot. The throwaway audio is discarded.
let warmup_t = std::time::Instant::now();
{
let opts = GenerateOptions {
max_audio_ms: 200,
seed: 0,
..GenerateOptions::default()
};
match generator.generate("Hi.", cli.speaker, &[], opts) {
Ok(_) => {
tracing::info!(
"generator warm-up complete ({}ms)",
warmup_t.elapsed().as_millis()
);
}
Err(e) => {
tracing::warn!(
"generator warm-up failed (non-fatal, first turn will be slower): {e}"
);
}
}
}
let auth_token = cli let auth_token = cli
.auth_token .auth_token
.or_else(|| std::env::var("RTX_AUTH_TOKEN").ok()); .or_else(|| std::env::var("RTX_AUTH_TOKEN").ok());