osobhandClaude Opus 4.7
15dd3575d4
A new model crate at crates/models/rtx-csm implementing end-to-end inference, quantization, and fine-tuning for Sesame's Conversational Speech Model (CSM-1B). Built on candle 0.9 + Kyutai Mimi codec. Key capabilities: - Inference (FP F16 on Metal, F32 on CPU, BF16 on CUDA) - Quantized inference (Q8_0 / Q4_K_M GGUF, ~3x speedup, ~50% memory) - Streaming Mimi decode with proper StreamTensor state machine - In-context voice cloning via SpeakerProfile - Classifier-Free Guidance (Koel-TTS recipe) - Long-form chunked generation with rolling context - Audio post-processing (HPF + declick + EBU R128 LUFS) - Text input normalization (brackets, times, unicode, length caps) - Frame-level repetition guard (loop-escape) - Top-k + top-p sampling - LoRA fine-tuning end-to-end (training + inference, on FP and Q8 bases) - In-process Whisper ASR via whisper-rs (under --features asr) - Standalone TTS HTTP server (Axum) - Bench harness with manifest export + per-prompt WER Phases delivered: quantization, ASR/WER eval, LoRA voice cloning, HTTP service. AudioSeal/WavLM/Unmute remain as documented future work. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
52 lines
1.1 KiB
Rust
52 lines
1.1 KiB
Rust
use thiserror::Error;
|
|
|
|
#[derive(Debug, Error)]
|
|
pub enum CsmError {
|
|
#[error("io: {0}")]
|
|
Io(#[from] std::io::Error),
|
|
|
|
#[error("hf-hub: {0}")]
|
|
Hub(#[from] hf_hub::api::sync::ApiError),
|
|
|
|
#[error("candle: {0}")]
|
|
Candle(#[from] candle_core::Error),
|
|
|
|
#[error("tokenizers: {0}")]
|
|
Tokenizer(String),
|
|
|
|
#[error("safetensors: {0}")]
|
|
SafeTensors(#[from] safetensors::SafeTensorError),
|
|
|
|
#[error("hound: {0}")]
|
|
Hound(#[from] hound::Error),
|
|
|
|
#[error("symphonia: {0}")]
|
|
Symphonia(#[from] symphonia::core::errors::Error),
|
|
|
|
#[error("rubato: {0}")]
|
|
Rubato(String),
|
|
|
|
#[error("shape mismatch: {0}")]
|
|
Shape(String),
|
|
|
|
#[error("context overflow: {used}/{max} tokens")]
|
|
ContextOverflow { used: usize, max: usize },
|
|
|
|
#[error("generation reached audio EOT")]
|
|
EotReached,
|
|
|
|
#[error("config: {0}")]
|
|
Config(String),
|
|
|
|
#[error("other: {0}")]
|
|
Other(#[from] anyhow::Error),
|
|
}
|
|
|
|
pub type Result<T> = std::result::Result<T, CsmError>;
|
|
|
|
impl From<tokenizers::Error> for CsmError {
|
|
fn from(e: tokenizers::Error) -> Self {
|
|
Self::Tokenizer(e.to_string())
|
|
}
|
|
}
|