Python sidecar for scoring TTS outputs against the firdhokk
Whisper-LV3 SER classifier (sanity-verified non-saturated, 3/5
correct on RAVDESS ground-truth).
Replaces the in-process emotion2vec_plus_base path which collapses
to 'Surprised' on every input (documented in
emotional_speech_guide.md and quality_eval.rs caveat).
Reads JSONL with {gen_wav, target_emotion} rows; writes JSONL with
top_emotion, top_prob, target_prob, match (bool), and the full
8-class probability distribution.
Class set is firdhokk's 7 (no calm — calm aliases to neutral on
input). Excited aliases to happy.
Smoke-verified on the 4 prior decoder-route outputs (Amini ctx,
seed=42, recipe defaults):
happy → neutral (0.80) ✗
angry → happy (0.999) ✗
fearful → fearful (0.68) ✓
sad → fearful (0.998) ✗ (sad↔fearful confusion)
Top-1 match: 1/4 — confirms the gap documented in
emotional_speech_guide.md 'Known Limitations'.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Original picker used min(WER), tie-broken by max(cosine). Fragile:
on emotion=surprised it picked "You can." (2 words, WER 0.93) over
"...Today I want to share something" (13 words, WER 1.00) because
WER weights all errors uniformly — terse-and-mostly-wrong beats
long-and-mostly-right.
New scoring:
score = WER + (1.0 if words(transcript) < 5 else 0)
sort_by(score, -cosine)
Verified on existing benches:
surprised: now picks seed=100 ("...Today I want to share something
with...", 13 words, score 1.0) over seed=7 ("You can.",
2 words, score 1.929 with +1 length penalty).
calm: still picks seed=100 (full transcript revealed: "It's a
good reflection. Not that that. I want to share
something with you that I've been thinking about." —
near-verbatim! the earlier 55-char display had been
truncating it).
disgust: all 3 candidates score ~1.93 (no seed has > 5 words,
all get the length penalty); picker honestly admits
none is good rather than picking a fake winner.
Worth noting: the calm seed=100 case is ANOTHER near-verbatim
single-shot result we missed in the previous bench because the
display truncation hid the full transcript content.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Capstone consumer interface composing every piece shipped today:
fetch_audio.sh → audio_to_manifest → pick_context.sh →
emotional_speech_n.sh (N-seed picker, decoder route)
Caches fetch + manifest by URL hash so re-runs with the same
--workdir skip the slow steps. Defaults to the Phase 9 recipe:
target=decoder, scale=1.0, layers [2,3], cfg=linear:3.0:1.0:25,
5-seed roll with the lowest-WER winner picked.
End-to-end smoke test (cached Carlini source, prompt "Today I want
to share..."):
picker auto-selected: nicholas_carlini...spk0.0078.wav (10.78 s)
seed 42 (winner): cos 0.974, WER 0.143 ⭐
"But Jason, today I want to share something
with you that I h"
seed 100: cos 0.986, WER 0.286
"It ties upon a share something with you that
I have been thi"
seed 7: cos 0.862, WER 1.000
"Let me think, let him out."
The auto-picker chose spk0 (Carlini himself) where manual selection
earlier in the day grabbed spk1 (the announcer) — so the automated
pipeline is also a slight context-selection improvement.
Three sub-second-WER results recorded over the day:
- WER 0.071 Amini imperative prompt (manual)
- WER 0.125 Amini original prompt (manual)
- WER 0.143 Carlini auto-picked spk0 (this commit, end-to-end)
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Wrapper now supports --target backbone|decoder. Decoder is the
default (Phase 9 finding: it preserves text fidelity that backbone
destroys). Per-(target, emotion) defaults:
decoder: scale 1.0, layers [2,3] (last two of 4)
backbone: scale 0.2-0.3 (per-emotion), layers [8,10,12]
emotional_speech_n.sh's existing passthrough already forwards
--target through to this wrapper unchanged.
Bench all 4 emotions on the decoder route, seed 7, recipe defaults:
emotion cos WER transcript
happy 0.65 0.93 "I've been happy cycling and beat..."
angry 0.78 2.29 "And of course, coming first, Vern is..."
fearful 0.73 0.86 "I'm not eye sensing when that's a mile."
sad 0.67 0.93 "- I'm actually off my night. I'll take
something. - All right..."
Sad — the previously-unsolvable emotion on the backbone (model
resisted at every tested scale 0.15-0.3) — produces real fluent
English on the decoder route. The word "happy" surfaces in the
happy output. All 4 emotions produce coherent speech: no music
tokens, no premature EOT, no gibberish. WER stays in the 0.86-2.3
range, comparable to baseline-with-CFG.
The decoder route subsumes everything the backbone route was
trying to do and unlocks the failure case it couldn't reach.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Productionizes yesterday's seed-variance finding. Wraps
emotional_speech.sh, rolls a list of seeds, scores each via
quality_eval (speaker_cosine + Moonshine WER), and copies the
lowest-WER candidate to --out. Defaults to 5 seeds; pass
--seeds 42,7 for cheaper runs.
Tie-breaking is `min(WER), then -max(cosine)` — text fidelity
takes precedence over speaker character because user-typed text
should be rendered verbatim, while voice character is only
secondary on top of context conditioning. Failed generations
(short clips that get the -1 cosine sentinel) sort to the bottom.
Smoke run on the canonical "Today I want to share..." prompt:
seed 7 → cos 0.845, WER 0.714 "Today, today I want to share..." ← picked
seed 100 → cos -1, WER 1.000 "It is." (premature EOT)
seed 42 → cos 0.916, WER 2.000 "The police are, if you're..." (drift)
Cost: N × single-shot cost. The recipe being unreliable per-seed
is the whole reason this wrapper exists — pay the multiplier in
exchange for a reliably-best output.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Capstone wrapper for the Phase 9 recipe (Selective CFG + RAVDESS
steering + mid-layer subset). Picks the steering scale automatically
from an empirical per-emotion map:
- happy: 0.30
- angry: 0.20
- fearful: 0.20
- sad: 0.20 (note: sad is unreliable — see below)
These came from a follow-up sweep after the multi-emotion demo
revealed the recipe is emotion-sensitive: scale 0.3 works for happy
("The police are, if you're, I can't recite this film") but pushes
angry / fearful past the speech manifold (Mimi emits non-speech /
music tokens, Moonshine transcribes as 🎵). Dropping to 0.2 recovers
fluent speech for both:
- [email protected]: "The next disorder is completing kashim for more."
- [email protected]: "You just heard a little bit about this decision,
though."
- [email protected]: "The police are, if you're, I can't recite this
film. I"
Sad is the outlier — model resists "sad" steering at every scale
between 0.15 and 0.3. Likely a corpus issue (sad RAVDESS clips are
the lowest-energy subset). Documented as a known limitation rather
than worked around.
perf_history.md updated with the per-emotion sensitivity finding.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Adds a clean labeled-emotion data path now that the auto-tagged
manifests have proven inadequate for steering extraction.
scripts/build_ravdess_manifest.sh: parses the RAVDESS speech-only
distribution (Audio_Speech_Actors_01-24.zip from Zenodo, 208 MB) into
a manifest.jsonl with proper {neutral, calm, happy, sad, angry,
fearful, disgust, surprised} labels and the two canonical statements
("Kids are talking by the door", "Dogs are sitting by the door"). 1440
clips, balanced 192/emotion (96 neutral — RAVDESS lacks the 'strong'
intensity for neutral).
examples/steering_extract Mimi reload-every-10: the streaming state
counter overflows 8192 frames after ~80 encodes even with
reset_state(). Same fix training/audio_to_manifest/converse_server use
(commit 0568dd3); now applied here too.
A/B with angry-vs-neutral steering @ 50 samples per pool, matched
against an angry RAVDESS reference clip:
case cos WER transcript
baseline 0.55 0.92 "It is a very important thing to do."
[email protected] 0.61 1.00 "You"
[email protected] 0.74 1.00 "So" ← best speaker_cosine
[email protected] 0.62 1.00 "You"
[email protected] 0.60 1.00 "You"
[email protected] 0.64 6.15 "the Lord, the Lord, the Lord..."
Speaker cosine 0.55 → 0.74 with mid-layer steering at scale 0.5 — a
35 % jump, the largest empirical gain we've measured. The model is
clearly migrating toward the angry actor's voice character. Side
effect: premature EOT (output reduces to one or two words). Likely
because RAVDESS clips themselves are very short ("Kids are talking
by the door", ~3s) so the steering biases toward terse outputs.
That's a corpus-shape artifact, not a code bug — different emotion
corpora with longer utterances should fix it.
Tightest single result of Sprint 2.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Sprint 1 of the post-research roadmap. TTSDS2 (arXiv 2506.19441) was
the original target but its install is broken on Python 3.12 + modern
torchaudio (deprecated `torchaudio.sox_effects`, `pyannote.audio` 3.1
calls removed `set_audio_backend`, `openai-whisper==20240927` needs
`pkg_resources`). Pivoted to a Rust-native foundation we already own
end-to-end: WavLM-SV + Moonshine + amplitude.
`examples/quality_eval` consumes a JSONL of `(ref_wav, gen_wav,
ref_text)` rows and emits per-row metrics:
- speaker_cosine via WavLM-SV (microsoft/wavlm-base-plus-sv)
- wer via Moonshine v2 transcript vs ref_text (Levenshtein on
lowercased / punctuation-stripped tokens)
- gen_peak_db, gen_rms_db (full-band amplitude of gen_wav)
`scripts/i2d_loop.sh` implements I2D (arXiv 2603.24430): synth N
times feeding each output back as the next iteration's context, score
all iterations with quality_eval, emit a TSV degradation curve.
Smoke-tested:
- quality_eval on the picker A/B set independently confirms the
picker — bottom-context (score 0) → WER 0.55, top-context
(score 2.0) → WER 0.18 (3× worse without picker filter).
- i2d_loop with 3 iterations on Amini context shows clean
collapse: cos 0.84 → 0.58, WER 0.5 → 1.0 by iter 1.
Foundation for Sprint 2 emotion-steering A/B comparisons.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
End-to-end voice clone: fetch_audio.sh → audio_to_manifest →
pick_context.sh → examples/generate. Caches each step by URL hash so
re-runs with the same --workdir skip the slow fetch and
diarize/transcribe stages.
Smoke-tested with cached fetch + manifest. Picks the highest-scoring
context clip across all (manifest, speaker) groups, hands it plus the
target text to generate via the new repeatable --context-wav /
--context-text pairs.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
A/B test 2026-04-29 across mit_2024intro / mcc / carlini manifests
showed the binary peak threshold (≤ -3 dBFS = +0.5) failed to
differentiate hot clips against each other: an mcc clip with input
peak=-1.47 dBFS scored same as one at -3.5 dBFS, and the model output
tracked input amplitude.
Replace with a linear penalty: 0.5 at peak ≤ -9 dBFS, ramping to 0 at
peak = 0 dBFS, clamped. mcc spk0 now produces graduated scores
(1.63 / 1.58 / 1.56 / 1.51) instead of a 1.5 plateau, reordering the
top selection toward the cleaner-peak clip.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
scripts/pick_context.sh ranks manifest clips by suitability for CSM-1B's
--context-wav conditioning (duration sweet spot 10-13.5s, RMS -25 to
-15 dB, peak ≤ -3 dBFS) and groups by (manifest_stem, speaker_id) since
diarizer labels are per-file.
examples/generate.rs now accepts repeatable --context-wav and
--context-text pairs, zipped into Vec<Segment> for Generator::generate.
Validates equal counts at runtime.
Smoke-tested with two 10-13s spk0 clips from the MIT-2024 manifest.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Wrapper that pulls audio from any yt-dlp-supported URL (YouTube,
LibriVox, archive.org, podcast feeds) and converts to the 24 kHz mono
16-bit PCM format examples/audio_to_manifest ingests. Slugifies the
output filename so manifest paths stay shell-safe.
Prints the next-step audio_to_manifest command with all the right
flags (--auto-emotion-tag --use-emotion2vec --stage audiobook), so a
new user can copy-paste the printed line straight into a terminal.
Requires external tools (yt-dlp, ffmpeg); install on macOS via
`brew install yt-dlp ffmpeg`.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Parity check tool runs HF reference + Rust port on the same audio pair
and prints verdict. Verified: same-utterance HF<->Rust embedding cosine
= 0.997/0.999, well within the >0.99 tolerance gate.
Re-interpretation: the earlier cross-content same-speaker cosine of
0.41 was NOT a port bug. HF gives 0.37 on the exact same pair. CSM-1B
"speaker 0" is genuinely stochastic across generations. Same-content
same-speaker pair: HF 0.989, Rust 0.996.
Remaining +/-0.04 cosine delta is accumulated FP noise across the long
forward pass (CNN -> 12 transformer layers -> 5 TDNN -> stat pool).
For cosine-based speaker verification this is functionally equivalent.
WavLM-SV port: production-ready. Phase 5 (a, b, c, d) all shipped.
scripts/.gitignore excludes the .venv from version control.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
- scripts/wavlm_sv_parity.py: Python-side reference embedder. Loads HF
WavLMForXVector + Wav2Vec2FeatureExtractor and dumps a JSON fingerprint
(cosine + per-utterance norm + first/last 8 elements) for comparison.
- examples/wavlm_sv_demo gains --parity-json flag emitting the same
fingerprint structure on the Rust side.
Once the user has a Python env with transformers + torch installed,
running both produces side-by-side JSON files for diffing — first-pass
sanity check on whether our port matches HF numerically. We can't run
the Python side from this Rust shell.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
