osobhandClaude Opus 4.7
15dd3575d4
A new model crate at crates/models/rtx-csm implementing end-to-end inference, quantization, and fine-tuning for Sesame's Conversational Speech Model (CSM-1B). Built on candle 0.9 + Kyutai Mimi codec. Key capabilities: - Inference (FP F16 on Metal, F32 on CPU, BF16 on CUDA) - Quantized inference (Q8_0 / Q4_K_M GGUF, ~3x speedup, ~50% memory) - Streaming Mimi decode with proper StreamTensor state machine - In-context voice cloning via SpeakerProfile - Classifier-Free Guidance (Koel-TTS recipe) - Long-form chunked generation with rolling context - Audio post-processing (HPF + declick + EBU R128 LUFS) - Text input normalization (brackets, times, unicode, length caps) - Frame-level repetition guard (loop-escape) - Top-k + top-p sampling - LoRA fine-tuning end-to-end (training + inference, on FP and Q8 bases) - In-process Whisper ASR via whisper-rs (under --features asr) - Standalone TTS HTTP server (Axum) - Bench harness with manifest export + per-prompt WER Phases delivered: quantization, ASR/WER eval, LoRA voice cloning, HTTP service. AudioSeal/WavLM/Unmute remain as documented future work. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
33 lines
835 B
Rust
33 lines
835 B
Rust
//! Llama-3.2 BPE tokenizer wrapper.
|
|
//!
|
|
//! CSM speaker is encoded inline as text "[<id>]" then concatenated with the
|
|
//! utterance, no special speaker token id.
|
|
|
|
use crate::error::Result;
|
|
use std::path::Path;
|
|
use tokenizers::Tokenizer;
|
|
|
|
pub struct CsmTokenizer {
|
|
inner: Tokenizer,
|
|
}
|
|
|
|
impl CsmTokenizer {
|
|
pub fn from_file<P: AsRef<Path>>(path: P) -> Result<Self> {
|
|
let inner = Tokenizer::from_file(path.as_ref())?;
|
|
Ok(Self { inner })
|
|
}
|
|
|
|
pub fn format_segment(speaker: u32, text: &str) -> String {
|
|
format!("[{speaker}]{text}")
|
|
}
|
|
|
|
pub fn encode(&self, text: &str) -> Result<Vec<u32>> {
|
|
let enc = self.inner.encode(text, false)?;
|
|
Ok(enc.get_ids().to_vec())
|
|
}
|
|
|
|
pub fn decode(&self, ids: &[u32]) -> Result<String> {
|
|
Ok(self.inner.decode(ids, false)?)
|
|
}
|
|
}
|