rustytorch

515 Commits 1 Branch 0 Tags

2 Commits

Include renames

Author	SHA1	Message	Date
osobhandClaude Opus 4.7	b4a7133ffb	rtx-csm: Mimi.reload() — workaround state-leak in cumulative encode loop Mimi's transformer carries an internal position counter across encode() calls that the upstream `reset_state()` does NOT fully clear. When TrainingDataset::load_from_manifest encodes ~80-100 clips back-to-back during data prep, the counter overflows the 8192-position buffer and panics with `narrow invalid args [8192, 32]`. src/mimi.rs: - cache the safetensors path on Mimi at construction - new Mimi.reload() drops the inner Model and rebuilds from the cached path (~200 ms on Metal) src/training.rs: - call generator.mimi.reload() every 50 clips during load_from_manifest. Adds ~1 s overhead on a 200-clip corpus (4 reloads × ~200 ms) vs the alternative of a hard panic. - reset_state() before each encode in TrainingExample::from_audio is kept (still useful to clear streaming chunk state). Found while running the end-to-end personal-voice training pipeline on a 20-minute YouTube source: the bug surfaces around clip 86 when Mimi's transformer hits position 8181+. Filtering to short clips alone didn't help — the cumulative state grows even with sub-12-second inputs. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>	2026-04-28 10:57:25 -07:00
osobhandClaude Opus 4.7	15dd3575d4	Add rtx-csm: Rust-native port of Sesame CSM-1B with LoRA voice cloning A new model crate at crates/models/rtx-csm implementing end-to-end inference, quantization, and fine-tuning for Sesame's Conversational Speech Model (CSM-1B). Built on candle 0.9 + Kyutai Mimi codec. Key capabilities: - Inference (FP F16 on Metal, F32 on CPU, BF16 on CUDA) - Quantized inference (Q8_0 / Q4_K_M GGUF, ~3x speedup, ~50% memory) - Streaming Mimi decode with proper StreamTensor state machine - In-context voice cloning via SpeakerProfile - Classifier-Free Guidance (Koel-TTS recipe) - Long-form chunked generation with rolling context - Audio post-processing (HPF + declick + EBU R128 LUFS) - Text input normalization (brackets, times, unicode, length caps) - Frame-level repetition guard (loop-escape) - Top-k + top-p sampling - LoRA fine-tuning end-to-end (training + inference, on FP and Q8 bases) - In-process Whisper ASR via whisper-rs (under --features asr) - Standalone TTS HTTP server (Axum) - Bench harness with manifest export + per-prompt WER Phases delivered: quantization, ASR/WER eval, LoRA voice cloning, HTTP service. AudioSeal/WavLM/Unmute remain as documented future work. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>	2026-04-25 18:33:57 -07:00

Author

SHA1

Message

Date

osobhandClaude Opus 4.7

b4a7133ffb

rtx-csm: Mimi.reload() — workaround state-leak in cumulative encode loop

Mimi's transformer carries an internal position counter across encode()
calls that the upstream `reset_state()` does NOT fully clear. When
TrainingDataset::load_from_manifest encodes ~80-100 clips back-to-back
during data prep, the counter overflows the 8192-position buffer and
panics with `narrow invalid args [8192, 32]`.

src/mimi.rs:
  - cache the safetensors path on Mimi at construction
  - new Mimi.reload() drops the inner Model and rebuilds from the
    cached path (~200 ms on Metal)

src/training.rs:
  - call generator.mimi.reload() every 50 clips during
    load_from_manifest. Adds ~1 s overhead on a 200-clip corpus
    (4 reloads × ~200 ms) vs the alternative of a hard panic.
  - reset_state() before each encode in TrainingExample::from_audio
    is kept (still useful to clear streaming chunk state).

Found while running the end-to-end personal-voice training pipeline on
a 20-minute YouTube source: the bug surfaces around clip 86 when
Mimi's transformer hits position 8181+. Filtering to short clips
alone didn't help — the cumulative state grows even with sub-12-second
inputs.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

2026-04-28 10:57:25 -07:00

osobhandClaude Opus 4.7

15dd3575d4

Add rtx-csm: Rust-native port of Sesame CSM-1B with LoRA voice cloning

A new model crate at crates/models/rtx-csm implementing end-to-end
inference, quantization, and fine-tuning for Sesame's Conversational
Speech Model (CSM-1B). Built on candle 0.9 + Kyutai Mimi codec.

Key capabilities:
- Inference (FP F16 on Metal, F32 on CPU, BF16 on CUDA)
- Quantized inference (Q8_0 / Q4_K_M GGUF, ~3x speedup, ~50% memory)
- Streaming Mimi decode with proper StreamTensor state machine
- In-context voice cloning via SpeakerProfile
- Classifier-Free Guidance (Koel-TTS recipe)
- Long-form chunked generation with rolling context
- Audio post-processing (HPF + declick + EBU R128 LUFS)
- Text input normalization (brackets, times, unicode, length caps)
- Frame-level repetition guard (loop-escape)
- Top-k + top-p sampling
- LoRA fine-tuning end-to-end (training + inference, on FP and Q8 bases)
- In-process Whisper ASR via whisper-rs (under --features asr)
- Standalone TTS HTTP server (Axum)
- Bench harness with manifest export + per-prompt WER

Phases delivered: quantization, ASR/WER eval, LoRA voice cloning, HTTP
service. AudioSeal/WavLM/Unmute remain as documented future work.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

2026-04-25 18:33:57 -07:00