osobhandClaude Opus 4.7
15dd3575d4
A new model crate at crates/models/rtx-csm implementing end-to-end inference, quantization, and fine-tuning for Sesame's Conversational Speech Model (CSM-1B). Built on candle 0.9 + Kyutai Mimi codec. Key capabilities: - Inference (FP F16 on Metal, F32 on CPU, BF16 on CUDA) - Quantized inference (Q8_0 / Q4_K_M GGUF, ~3x speedup, ~50% memory) - Streaming Mimi decode with proper StreamTensor state machine - In-context voice cloning via SpeakerProfile - Classifier-Free Guidance (Koel-TTS recipe) - Long-form chunked generation with rolling context - Audio post-processing (HPF + declick + EBU R128 LUFS) - Text input normalization (brackets, times, unicode, length caps) - Frame-level repetition guard (loop-escape) - Top-k + top-p sampling - LoRA fine-tuning end-to-end (training + inference, on FP and Q8 bases) - In-process Whisper ASR via whisper-rs (under --features asr) - Standalone TTS HTTP server (Axum) - Bench harness with manifest export + per-prompt WER Phases delivered: quantization, ASR/WER eval, LoRA voice cloning, HTTP service. AudioSeal/WavLM/Unmute remain as documented future work. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
31 lines
937 B
Rust
31 lines
937 B
Rust
use crate::error::Result;
|
|
use candle_core::{Device, Tensor};
|
|
|
|
pub fn pick_device() -> Result<Device> {
|
|
#[cfg(feature = "cuda")]
|
|
if let Ok(d) = Device::new_cuda(0) {
|
|
return Ok(d);
|
|
}
|
|
#[cfg(feature = "metal")]
|
|
if let Ok(d) = Device::new_metal(0) {
|
|
return Ok(d);
|
|
}
|
|
Ok(Device::Cpu)
|
|
}
|
|
|
|
/// Root-mean-square error between two equal-length f32 waveforms.
|
|
pub fn rms(a: &[f32], b: &[f32]) -> f32 {
|
|
debug_assert_eq!(a.len(), b.len());
|
|
let n = a.len() as f32;
|
|
let sum_sq: f32 = a.iter().zip(b).map(|(x, y)| (x - y).powi(2)).sum();
|
|
(sum_sq / n).sqrt()
|
|
}
|
|
|
|
/// CSM EOT signal: every codebook in the frame is zero. Caller must gate this
|
|
/// to frame_idx >= 1 to avoid spurious EOT on the first frame.
|
|
pub fn all_zero_codebooks(frame: &Tensor) -> Result<bool> {
|
|
// frame shape: (1, num_codebooks) i64
|
|
let v = frame.flatten_all()?.to_vec1::<i64>()?;
|
|
Ok(v.iter().all(|x| *x == 0))
|
|
}
|