rustyverse/rustytorch
2
0
Fork 0
Code Issues Pull Requests Actions 6 Packages Projects Releases Wiki Activity
414 Commits 1 Branch 0 Tags
96a7f1c70042140bdc984b020138b80a3e280e69
Commit Graph
5 Commits
Author SHA1 Message Date
osobhandClaude Opus 4.7 a5cedfb46a rtx-csm: emotional_speech_guide — CREMA-D vs RAVDESS firdhokk verdict 
8-gen bench (4 emotions × 2 corpora) at seed=42 against firdhokk
Whisper-LV3:

  target    RAVDESS              CREMA-D
  happy     happy (0.999) ✓      happy (0.999) ✓
  angry     neutral (0.92)       sad (0.99)
  fearful   happy (0.998)        fearful (0.984) ✓
  sad       angry (0.99)         fearful (0.99)

CREMA-D 2/4 vs RAVDESS 1/4. Larger / more naturalistic corpus
produces more class-pure fearful direction. Neither corpus solves
angry or sad — recipe shifts into 'vague expressivity' rather than
class-specific corners.

Practical: prefer CREMA-D when available; A/B both per emotion if
class precision matters.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
 2026-04-30 00:01:02 -07:00
Omar Sobh 6a03aeba61 fix(cuda+csm): P0 rtx-backend-cuda compile fix + P2 rtx-csm clippy cleanup (#9) 
Co-authored-by: Omar Sobh <[email protected]>
Co-committed-by: Omar Sobh <[email protected]>
 2026-04-30 05:24:58 +00:00
osobhandClaude Opus 4.7 9161b32a91 rtx-csm: per-request watermark message in tts_server 
Watermarker trait gains embed_with_message(audio, message) with a
default impl forwarding to embed (no-op for watermarkers without a
payload). AudioSealWatermarker overrides to use the requested message
instead of self.message; ResampledWatermarker forwards through the
resample dance.

TtsRequest gains optional watermark_message: Option<String> (decimal or
0xHEX). Useful for clawsample to tag each generation with a unique ID
(e.g. job_id mod 0x10000) for audit trails. When omitted, falls back
to the server-startup --audioseal-message default.

Verified end-to-end: override "0xBEEF" -> detect 0xBEEF (mean_presence
0.9995, 16/16 bits). Default fallback also decodes correctly.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
 2026-04-26 04:01:30 -07:00
osobhandClaude Opus 4.7 938b54a2b0 rtx-csm: AudioSeal watermark — Rust port end-to-end 
SEANet generator + detector matching `facebook/audioseal` reference layout
(weight_norm-merged via pure-Rust pickle reader). Verified on real CSM
speech: mean_presence=0.9943, 16/16 message bits decoded.

- src/audioseal.rs: SeanetEncoder (4-stage strided downsample, 2-layer
  LSTM bottleneck at 512 channels, 128-dim projection), MsgProcessor
  (16-bit message via embedding sum + broadcast-add), SeanetDecoder,
  Generator (encoder+msg+decoder), Detector (encoder + single 320×
  reverse_convolution + 1×1 head). Padding mirrors audiocraft
  _get_extra_padding_for_conv1d exactly.
- src/audioseal_convert.rs: candle_core::pickle reads .pth directly;
  merge_weight_norm computes g*v/‖v‖ over all axes except 0; writes
  flat safetensors keyed identically to what Generator/Detector read.
- examples/audioseal_inspect.rs: dumps tensor keys + shapes.
- examples/audioseal_convert.rs: HF download + convert CLI.
- examples/audioseal_demo.rs: load + embed + detect on real WAV or
  synthetic burst, optionally writes watermarked WAV.
- audio_io.rs gains generic load_mono_at_rate, resample, write_wav_mono
  (16 kHz path needed for AudioSeal).

12 new unit tests + 2 converter tests; 63 lib tests total green.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
 2026-04-25 19:26:13 -07:00
osobhandClaude Opus 4.7 15dd3575d4 Add rtx-csm: Rust-native port of Sesame CSM-1B with LoRA voice cloning 
A new model crate at crates/models/rtx-csm implementing end-to-end
inference, quantization, and fine-tuning for Sesame's Conversational
Speech Model (CSM-1B). Built on candle 0.9 + Kyutai Mimi codec.

Key capabilities:
- Inference (FP F16 on Metal, F32 on CPU, BF16 on CUDA)
- Quantized inference (Q8_0 / Q4_K_M GGUF, ~3x speedup, ~50% memory)
- Streaming Mimi decode with proper StreamTensor state machine
- In-context voice cloning via SpeakerProfile
- Classifier-Free Guidance (Koel-TTS recipe)
- Long-form chunked generation with rolling context
- Audio post-processing (HPF + declick + EBU R128 LUFS)
- Text input normalization (brackets, times, unicode, length caps)
- Frame-level repetition guard (loop-escape)
- Top-k + top-p sampling
- LoRA fine-tuning end-to-end (training + inference, on FP and Q8 bases)
- In-process Whisper ASR via whisper-rs (under --features asr)
- Standalone TTS HTTP server (Axum)
- Bench harness with manifest export + per-prompt WER

Phases delivered: quantization, ASR/WER eval, LoRA voice cloning, HTTP
service. AudioSeal/WavLM/Unmute remain as documented future work.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
 2026-04-25 18:33:57 -07:00