Generic LLM client abstraction for the conversational stack:
- LlmClient trait with generate_stream(messages, config) -> TokenStream.
Default generate() impl folds the stream for non-streaming callers.
- ChatMessage / Role / GenConfig types with sensible defaults.
- OpenAiCompatibleClient: HTTP impl with SSE streaming. Works against
OpenAI, Z.AI, vLLM, llama.cpp's HTTP server, LiteLLM — any endpoint
serving the Chat Completions schema.
- examples/llm_chat: demo CLI that prints token-by-token to stdout
with TTFT + total-time + char-count metrics.
Promotes tokio + reqwest + futures-util to regular dependencies (no
longer dev-only) so the trait is part of the public library surface.
Adds async-trait + eventsource-stream for the SSE streaming.
3 unit tests (constructors, role serialization, default config); 82
lib tests total green.
Phase 6 progress:
- 6a Kyutai STT: integration scaffolded; output bridge needs Python
reference diff (deferred)
- 6b LLM client: shipped (this commit)
- 6c session glue (axum WS + duplex audio loop): next
- 6d productionization: deferred
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Integrates the moshi crate (0.6.4, candle 0.9.1) for streaming STT.
Module + demo + custom config for kyutai/stt-1b-en_fr. Model loads
cleanly, LM forward pass advances (model_step_idx increments correctly),
but word events don't yet emit on a 10s CSM speech sample.
What works:
- moshi 0.6.4 added as dependency (candle 0.9.1, version-compatible)
- src/stt.rs wraps moshi::asr::State + moshi::lm + moshi::mimi
- Stt::load_default downloads kyutai/stt-1b-en_fr (~3 GB) from HF
- Custom config_stt_1b_en_fr() matching the released checkpoint:
d_model=2048, num_layers=16, dim_feedforward=8192 (moshi's SwiGLU
hidden = 11/4 * d_model = 5632 — verified vs safetensors), text vocab
8001/8000, audio vocab 2049, 32 codebooks, no depformer
- AsrEvent enum + From<moshi::asr::AsrMsg> conversion
- examples/stt_demo.rs streams a WAV through the pipeline
- 2 unit tests for AsrEvent conversion
What needs more work:
- Word emission: 0 words detected on 10s of clean CSM speech, even
though LM forward advances every frame. Likely culprits:
a) asr_delay_in_tokens 6 vs HF stt_config.audio_delay_seconds=0.5
(6.25 frames). Off-by-one possible.
b) Sentencepiece detok not yet wired (tokens emitted but text=None).
c) Subtle weight-key remap differences between moshi's expected
layout and the released checkpoint that don't trip a shape check.
d) renormalize/audio preprocessing mismatch.
Next step (Phase 6a polish): compare against the official
delayed-streams-modeling/scripts/stt_from_file_pytorch.py reference to
identify the missing piece. The integration framework is sound; only
the final LM-output-to-text-event step needs work.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Streams 16-bit little-endian PCM (24 kHz mono) as Mimi produces chunks.
Wraps Generator::generate_streaming via spawn_blocking + tokio::sync::mpsc
bridge into an axum Body::from_stream response.
Same JSON request format as /v1/tts; Content-Type is
audio/L16; rate=24000; channels=1 per RFC 2586.
First-byte (first audio chunk) latency on Metal: ~880 ms vs ~6 s wall
for the non-streaming /v1/tts path — 6.8x faster perceived UX, the
difference between "the app froze" and "the app started speaking."
Caveat: streaming endpoint does NOT apply post-processing or the inline
watermarker (those operate on the full utterance). For watermarked
output use /v1/tts. A chunked AudioSeal port is the natural follow-up
for streaming watermarking.
Adds futures-util as a dev-dependency for the Stream trait.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Extends the HTTP service with three new endpoints exposing AudioSeal
detection and WavLM-SV speaker scoring alongside the existing TTS:
GET /health
POST /v1/tts audio/wav (24 kHz mono)
POST /v1/detect [audio] JSON { mean_presence, message_hex }
POST /v1/speaker_embed [audio] JSON { embedding: [512 floats] }
POST /v1/speaker_compare [a+b] JSON { cosine }
Wires the inline watermarker into /v1/tts when --audioseal-* flags are
set: every TTS response is auto-watermarked through the
ResampledWatermarker (24 kHz <-> 16 kHz) adapter.
Verified end-to-end on Metal:
/health -> ok
/v1/tts -> 200, 145964 bytes (3s @ 24kHz)
/v1/detect -> mean_presence=0.998 on watermarked output
/v1/speaker_embed -> 512-d float vector
/v1/speaker_compare a==b -> cosine 1.0000001
axum gains the "multipart" feature for audio uploads.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Two real bugs found via code inspection against HF source:
1. candle's .gelu() is the tanh approximation; PyTorch's default 'gelu'
activation (used in WavLM via ACT2FN['gelu']) is the exact erf-based
version. Switched all 3 sites (feature extractor convs, pos_conv,
FFN) from .gelu() to .gelu_erf() to match the reference.
2. gru_rel_pos_const lookup used vb.pp("name").get(shape, "") which
resolves to "<prefix>.name." (trailing dot) and fails to find the
tensor. The .or_else(|_| zeros) silently swallowed the failure,
leaving all 12 layers' gating constants at zero instead of the
trained values. Fixed to attn.get(shape, "gru_rel_pos_const") which
resolves correctly.
examples/wavlm_sv_inspect.rs: utility for sanity-checking specific
tensors inside converted safetensors (e.g. layer_weights).
Same-content same-speaker cosine: 0.9985 -> 0.9963 (≈unchanged).
Cross-content same-speaker cosine: 0.4882 -> 0.4118 (still drifting).
Phase 5d (Python reference comparison) remains the gate for
identifying the residual numerical drift.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Single CLI ties together every capability shipped this session:
text -> CSM-1B (with optional LoRA) -> post-process (HPF/declick/LUFS)
-> AudioSeal watermark embed -> AudioSeal detect verify -> WavLM-SV
speaker embedding + optional reference scoring.
Verified on Metal: 4s speech generated + watermarked + detected
(mean_presence=0.9999, 16/16 bits decoded) + 512-d speaker embedding
extracted in ~30s.
Cross-content same-speaker cosine sits around 0.49 vs 0.998 for
same-content same-speaker — suggests the WavLM-SV port may leak content
into the speaker embedding more than the HF reference. Phase 5d numerical
parity work (Python sidecar comparison) would tighten this.
This is the canonical usage example for downstream callers
(clawsample-csm etc.) — copy the structure.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
End-to-end watermarker driver that handles any source sample rate by
resampling to AudioSeal's 16 kHz native, embedding, then resampling back.
Tested on 10s of real CSM 24 kHz speech: mean_presence=0.9988 detection,
12/16 message bits round-trip (4-bit erosion from double resample).
- examples/audioseal_apply.rs: --in/--out/--source-rate/--message; loads
source via audio_io::load_mono_at_rate, calls AudioSealWatermarker
through the public Watermarker trait, verifies via in-process detect.
- Fix bit-match counter overflow in audioseal_demo.rs and audioseal_apply.rs.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
