Commit Graph
4 Commits
Author SHA1 Message Date
osobhandClaude Opus 4.7 9161b32a91 rtx-csm: per-request watermark message in tts_server
Watermarker trait gains embed_with_message(audio, message) with a
default impl forwarding to embed (no-op for watermarkers without a
payload). AudioSealWatermarker overrides to use the requested message
instead of self.message; ResampledWatermarker forwards through the
resample dance.

TtsRequest gains optional watermark_message: Option<String> (decimal or
0xHEX). Useful for clawsample to tag each generation with a unique ID
(e.g. job_id mod 0x10000) for audit trails. When omitted, falls back
to the server-startup --audioseal-message default.

Verified end-to-end: override "0xBEEF" -> detect 0xBEEF (mean_presence
0.9995, 16/16 bits). Default fallback also decodes correctly.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
2026-04-26 04:01:30 -07:00
osobhandClaude Opus 4.7 c3eddc8873 rtx-csm: tts_server /v1/tts_stream — chunked PCM streaming endpoint
Streams 16-bit little-endian PCM (24 kHz mono) as Mimi produces chunks.
Wraps Generator::generate_streaming via spawn_blocking + tokio::sync::mpsc
bridge into an axum Body::from_stream response.

Same JSON request format as /v1/tts; Content-Type is
audio/L16; rate=24000; channels=1 per RFC 2586.

First-byte (first audio chunk) latency on Metal: ~880 ms vs ~6 s wall
for the non-streaming /v1/tts path — 6.8x faster perceived UX, the
difference between "the app froze" and "the app started speaking."

Caveat: streaming endpoint does NOT apply post-processing or the inline
watermarker (those operate on the full utterance). For watermarked
output use /v1/tts. A chunked AudioSeal port is the natural follow-up
for streaming watermarking.

Adds futures-util as a dev-dependency for the Stream trait.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
2026-04-26 03:42:50 -07:00
osobhandClaude Opus 4.7 3f61cee136 rtx-csm: tts_server full pipeline — TTS + watermark + speaker
Extends the HTTP service with three new endpoints exposing AudioSeal
detection and WavLM-SV speaker scoring alongside the existing TTS:

  GET  /health
  POST /v1/tts                    audio/wav (24 kHz mono)
  POST /v1/detect      [audio]    JSON { mean_presence, message_hex }
  POST /v1/speaker_embed [audio]  JSON { embedding: [512 floats] }
  POST /v1/speaker_compare [a+b]  JSON { cosine }

Wires the inline watermarker into /v1/tts when --audioseal-* flags are
set: every TTS response is auto-watermarked through the
ResampledWatermarker (24 kHz <-> 16 kHz) adapter.

Verified end-to-end on Metal:
  /health -> ok
  /v1/tts -> 200, 145964 bytes (3s @ 24kHz)
  /v1/detect -> mean_presence=0.998 on watermarked output
  /v1/speaker_embed -> 512-d float vector
  /v1/speaker_compare a==b -> cosine 1.0000001

axum gains the "multipart" feature for audio uploads.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
2026-04-26 03:27:50 -07:00
osobhandClaude Opus 4.7 15dd3575d4 Add rtx-csm: Rust-native port of Sesame CSM-1B with LoRA voice cloning
A new model crate at crates/models/rtx-csm implementing end-to-end
inference, quantization, and fine-tuning for Sesame's Conversational
Speech Model (CSM-1B). Built on candle 0.9 + Kyutai Mimi codec.

Key capabilities:
- Inference (FP F16 on Metal, F32 on CPU, BF16 on CUDA)
- Quantized inference (Q8_0 / Q4_K_M GGUF, ~3x speedup, ~50% memory)
- Streaming Mimi decode with proper StreamTensor state machine
- In-context voice cloning via SpeakerProfile
- Classifier-Free Guidance (Koel-TTS recipe)
- Long-form chunked generation with rolling context
- Audio post-processing (HPF + declick + EBU R128 LUFS)
- Text input normalization (brackets, times, unicode, length caps)
- Frame-level repetition guard (loop-escape)
- Top-k + top-p sampling
- LoRA fine-tuning end-to-end (training + inference, on FP and Q8 bases)
- In-process Whisper ASR via whisper-rs (under --features asr)
- Standalone TTS HTTP server (Axum)
- Bench harness with manifest export + per-prompt WER

Phases delivered: quantization, ASR/WER eval, LoRA voice cloning, HTTP
service. AudioSeal/WavLM/Unmute remain as documented future work.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
2026-04-25 18:33:57 -07:00