Parity check tool runs HF reference + Rust port on the same audio pair
and prints verdict. Verified: same-utterance HF<->Rust embedding cosine
= 0.997/0.999, well within the >0.99 tolerance gate.
Re-interpretation: the earlier cross-content same-speaker cosine of
0.41 was NOT a port bug. HF gives 0.37 on the exact same pair. CSM-1B
"speaker 0" is genuinely stochastic across generations. Same-content
same-speaker pair: HF 0.989, Rust 0.996.
Remaining +/-0.04 cosine delta is accumulated FP noise across the long
forward pass (CNN -> 12 transformer layers -> 5 TDNN -> stat pool).
For cosine-based speaker verification this is functionally equivalent.
WavLM-SV port: production-ready. Phase 5 (a, b, c, d) all shipped.
scripts/.gitignore excludes the .venv from version control.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Watermarker trait gains embed_with_message(audio, message) with a
default impl forwarding to embed (no-op for watermarkers without a
payload). AudioSealWatermarker overrides to use the requested message
instead of self.message; ResampledWatermarker forwards through the
resample dance.
TtsRequest gains optional watermark_message: Option<String> (decimal or
0xHEX). Useful for clawsample to tag each generation with a unique ID
(e.g. job_id mod 0x10000) for audit trails. When omitted, falls back
to the server-startup --audioseal-message default.
Verified end-to-end: override "0xBEEF" -> detect 0xBEEF (mean_presence
0.9995, 16/16 bits). Default fallback also decodes correctly.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Streams 16-bit little-endian PCM (24 kHz mono) as Mimi produces chunks.
Wraps Generator::generate_streaming via spawn_blocking + tokio::sync::mpsc
bridge into an axum Body::from_stream response.
Same JSON request format as /v1/tts; Content-Type is
audio/L16; rate=24000; channels=1 per RFC 2586.
First-byte (first audio chunk) latency on Metal: ~880 ms vs ~6 s wall
for the non-streaming /v1/tts path — 6.8x faster perceived UX, the
difference between "the app froze" and "the app started speaking."
Caveat: streaming endpoint does NOT apply post-processing or the inline
watermarker (those operate on the full utterance). For watermarked
output use /v1/tts. A chunked AudioSeal port is the natural follow-up
for streaming watermarking.
Adds futures-util as a dev-dependency for the Stream trait.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Extends the HTTP service with three new endpoints exposing AudioSeal
detection and WavLM-SV speaker scoring alongside the existing TTS:
GET /health
POST /v1/tts audio/wav (24 kHz mono)
POST /v1/detect [audio] JSON { mean_presence, message_hex }
POST /v1/speaker_embed [audio] JSON { embedding: [512 floats] }
POST /v1/speaker_compare [a+b] JSON { cosine }
Wires the inline watermarker into /v1/tts when --audioseal-* flags are
set: every TTS response is auto-watermarked through the
ResampledWatermarker (24 kHz <-> 16 kHz) adapter.
Verified end-to-end on Metal:
/health -> ok
/v1/tts -> 200, 145964 bytes (3s @ 24kHz)
/v1/detect -> mean_presence=0.998 on watermarked output
/v1/speaker_embed -> 512-d float vector
/v1/speaker_compare a==b -> cosine 1.0000001
axum gains the "multipart" feature for audio uploads.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Two real bugs found via code inspection against HF source:
1. candle's .gelu() is the tanh approximation; PyTorch's default 'gelu'
activation (used in WavLM via ACT2FN['gelu']) is the exact erf-based
version. Switched all 3 sites (feature extractor convs, pos_conv,
FFN) from .gelu() to .gelu_erf() to match the reference.
2. gru_rel_pos_const lookup used vb.pp("name").get(shape, "") which
resolves to "<prefix>.name." (trailing dot) and fails to find the
tensor. The .or_else(|_| zeros) silently swallowed the failure,
leaving all 12 layers' gating constants at zero instead of the
trained values. Fixed to attn.get(shape, "gru_rel_pos_const") which
resolves correctly.
examples/wavlm_sv_inspect.rs: utility for sanity-checking specific
tensors inside converted safetensors (e.g. layer_weights).
Same-content same-speaker cosine: 0.9985 -> 0.9963 (≈unchanged).
Cross-content same-speaker cosine: 0.4882 -> 0.4118 (still drifting).
Phase 5d (Python reference comparison) remains the gate for
identifying the residual numerical drift.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Adds the long-form analogue of generate_to_wav. generate_long previously
returned raw PCM and bypassed the Generator-bound watermarker hook,
meaning long-form output skipped watermarking entirely if installed.
generate_long_to_wav mirrors generate_to_wav exactly:
chunked-generation -> post-process -> watermark -> WAV write.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
- scripts/wavlm_sv_parity.py: Python-side reference embedder. Loads HF
WavLMForXVector + Wav2Vec2FeatureExtractor and dumps a JSON fingerprint
(cosine + per-utterance norm + first/last 8 elements) for comparison.
- examples/wavlm_sv_demo gains --parity-json flag emitting the same
fingerprint structure on the Rust side.
Once the user has a Python env with transformers + torch installed,
running both produces side-by-side JSON files for diffing — first-pass
sanity check on whether our port matches HF numerically. We can't run
the Python side from this Rust shell.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Single CLI ties together every capability shipped this session:
text -> CSM-1B (with optional LoRA) -> post-process (HPF/declick/LUFS)
-> AudioSeal watermark embed -> AudioSeal detect verify -> WavLM-SV
speaker embedding + optional reference scoring.
Verified on Metal: 4s speech generated + watermarked + detected
(mean_presence=0.9999, 16/16 bits decoded) + 512-d speaker embedding
extracted in ~30s.
Cross-content same-speaker cosine sits around 0.49 vs 0.998 for
same-content same-speaker — suggests the WavLM-SV port may leak content
into the speaker embedding more than the HF reference. Phase 5d numerical
parity work (Python sidecar comparison) would tighten this.
This is the canonical usage example for downstream callers
(clawsample-csm etc.) — copy the structure.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Wraps the WavLM-SV port from Phase 5c. Full 100M-param X-vector head
running in-process; previously returned a typed error.
- WavLmSimilarity::load(path, device) loads converted safetensors.
- WavLmSimilarity::embed(samples) caches a 512-d embedding for repeat
comparisons.
- score(a, b) embeds both inputs and cosines them.
- Module docs updated; SpectralCentroidSimilarity kept as a weak-baseline
check.
Caller-facing change: any code using the SpeakerSimilarity trait now
gets a real speaker model with one constructor swap.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
A single \`generate\` invocation now produces a watermarked WAV when
AudioSeal weights are passed via CLI. End-to-end verified on real CSM
speech: mean_presence=1.0000, 16/16 message bits decoded.
- Generator gains \`watermarker: Option<Box<dyn Watermarker>>\` slot;
\`generate_to_wav\` runs \`wm.embed(&pcm)\` after post-process, before
WAV write. Field is Send+Sync so the existing Arc<Mutex<Generator>>
tts_server pattern still works.
- watermark.rs ships ResampledWatermarker<W> adapter for handling rate
mismatches (CSM 24 kHz ↔ AudioSeal 16 kHz). Output length is normalized
to input length so it's a transparent drop-in.
- examples/generate.rs gains --watermark-generator/--watermark-detector/
--watermark-message flags. Loads AudioSeal, wraps in resampler, installs.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
End-to-end watermarker driver that handles any source sample rate by
resampling to AudioSeal's 16 kHz native, embedding, then resampling back.
Tested on 10s of real CSM 24 kHz speech: mean_presence=0.9988 detection,
12/16 message bits round-trip (4-bit erosion from double resample).
- examples/audioseal_apply.rs: --in/--out/--source-rate/--message; loads
source via audio_io::load_mono_at_rate, calls AudioSealWatermarker
through the public Watermarker trait, verifies via in-process detect.
- Fix bit-match counter overflow in audioseal_demo.rs and audioseal_apply.rs.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
The encoder-gradient path through the decoder was transposing the decoder
before the matmul, producing `[batch, d_model] × [d_sae, d_model]` — a
shape mismatch for every batch > 1. The decoder is stored as
`[d_model, d_sae]`, so `recon_grad @ decoder` is already the right shape
(and matches the comment at the call site, which reads
"recon_grad @ decoder @ d_relu").
All 9 existing `sae::tests` still pass. Omni-Cortex's `LatentDictionary`
now trains correctly on batches larger than 1.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
- rtx-metal: fix MetalError import in sparse/conversion.rs non-macOS stub
- rtx-onnx: update session.rs and tensor_bridge.rs for ort 2.x API changes
- rtx-fusion: fix Cargo.toml package name
- rtx-hub: fix discovery.rs type mismatch
- Full workspace (80+ crates) now compiles clean on Linux
osobhandClaude Opus 4.7