The canonical voice loop now lives in zeroclaw-channel-voice
(`~/projects/zeroclaw/crates/zeroclaw-channel-voice`, binary
`voice_server`). It routes the LLM path through zeroclaw's agent
runtime — multi-turn history, tools, memory, provider routing —
instead of the OpenAI-compatible direct path here.
Same WS wire protocol so `examples/converse_client.rs` drives both;
no client-side migration needed.
This binary is intentionally kept buildable for:
1. Reproducing perf_history.md Phase 8.10 benches.
2. Standalone (no-agent) use when zeroclaw isn't desired.
Module doc + main() startup banner updated to point at the new home.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
8-gen bench (4 emotions × 2 corpora) at seed=42 against firdhokk
Whisper-LV3:
target RAVDESS CREMA-D
happy happy (0.999) ✓ happy (0.999) ✓
angry neutral (0.92) sad (0.99)
fearful happy (0.998) fearful (0.984) ✓
sad angry (0.99) fearful (0.99)
CREMA-D 2/4 vs RAVDESS 1/4. Larger / more naturalistic corpus
produces more class-pure fearful direction. Neither corpus solves
angry or sad — recipe shifts into 'vague expressivity' rather than
class-specific corners.
Practical: prefer CREMA-D when available; A/B both per emotion if
class precision matters.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Python sidecar for scoring TTS outputs against the firdhokk
Whisper-LV3 SER classifier (sanity-verified non-saturated, 3/5
correct on RAVDESS ground-truth).
Replaces the in-process emotion2vec_plus_base path which collapses
to 'Surprised' on every input (documented in
emotional_speech_guide.md and quality_eval.rs caveat).
Reads JSONL with {gen_wav, target_emotion} rows; writes JSONL with
top_emotion, top_prob, target_prob, match (bool), and the full
8-class probability distribution.
Class set is firdhokk's 7 (no calm — calm aliases to neutral on
input). Excited aliases to happy.
Smoke-verified on the 4 prior decoder-route outputs (Amini ctx,
seed=42, recipe defaults):
happy → neutral (0.80) ✗
angry → happy (0.999) ✗
fearful → fearful (0.68) ✓
sad → fearful (0.998) ✗ (sad↔fearful confusion)
Top-1 match: 1/4 — confirms the gap documented in
emotional_speech_guide.md 'Known Limitations'.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Wired up firdhokk/speech-emotion-recognition-with-openai-whisper-large-v3
as a working alternative to the broken emotion2vec_plus_base. Sanity
verified on real RAVDESS clips: 3/5 correct, 2/5 near-miss (happy↔
surprised, sad↔fearful). Probabilities are NOT saturated — the
classifier actually distinguishes per-input.
Then scored our 4 decoder-route outputs (Amini context, seed=42,
recipe defaults) and found that **only fearful registers as the
intended class**:
target verdict conf
happy neutral 0.80 ✗ (steering produces neutral output)
angry happy 0.999 ✗ (high-arousal cross-class)
fearful fearful 0.68 ✓
sad fearful 0.998 ✗ (sad↔fearful confusion)
Honest framing: the recipe shifts speaker character toward an
expressive-sounding direction (cosine evidence) and preserves text
(decoder vs backbone) but does NOT produce class-distinct emotion.
The metric stack we used through Phase 9 (cosine + WER) couldn't
see this gap because it measures voice fidelity and text rendering,
not emotion class.
Hypothesized fixes (not yet tested):
- CREMA-D extraction (91 actors vs RAVDESS 24) for class-purer
steering vectors
- Mixed backbone+decoder steering (backbone for prosody)
- EmoNet classifier (TTS-aware, may give different verdicts)
Doc'd in emotional_speech_guide.md as a known limitation. Closes
out an honest scientific picture: today's work successfully ports
the architectural finding (decoder route preserves text), but
class-precise emotion control remains unsolved.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Wires emotion2vec into quality_eval so per-row metrics include
target_emotion_prob, top_emotion, top_emotion_prob. Pure inference,
optional via --emotion2vec / --target-emotion flags.
Critical empirical finding documented in code + user guide: the
emotion2vec_plus_base checkpoint classifies every input as
"Surprised" with prob ≥ 0.99, INCLUDING ground-truth RAVDESS clips
with explicit emotion labels. Real angry-RAVDESS → "Surprised"
(0.9999999). Real neutral-RAVDESS → "Surprised" (0.9999996).
The metric implementation is correct (matches the trait's
EmotionDetector::classify code path with same per-utterance zero-
mean unit-variance normalization); the underlying classifier
collapses to a dominant class on most input — likely the same
"9→5 fold collapse" the project already documented in the data-
labeling path.
Practical implication: target_emotion_prob is near-zero for almost
every (target, output) that isn't "surprised", so it can't be used
as a picker score. The emotion2vec metric still works as a
diagnostic ("did the model produce something that classifies as
audio at all?") but not as a generation-quality validator.
Doc'd in:
- examples/quality_eval.rs CLI doc (caveat block on --emotion2vec)
- docs/emotional_speech_guide.md (Known limitations section with
full sanity-check table)
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Brief addition to the recipe section explaining the WER + length
floor scoring used by emotional_speech_n.sh (committed in b7b267b).
Validates that the new scoring preserves canonical winners on
happy and calm while flipping surprised to the long-and-correct
candidate.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Original picker used min(WER), tie-broken by max(cosine). Fragile:
on emotion=surprised it picked "You can." (2 words, WER 0.93) over
"...Today I want to share something" (13 words, WER 1.00) because
WER weights all errors uniformly — terse-and-mostly-wrong beats
long-and-mostly-right.
New scoring:
score = WER + (1.0 if words(transcript) < 5 else 0)
sort_by(score, -cosine)
Verified on existing benches:
surprised: now picks seed=100 ("...Today I want to share something
with...", 13 words, score 1.0) over seed=7 ("You can.",
2 words, score 1.929 with +1 length penalty).
calm: still picks seed=100 (full transcript revealed: "It's a
good reflection. Not that that. I want to share
something with you that I've been thinking about." —
near-verbatim! the earlier 55-char display had been
truncating it).
disgust: all 3 candidates score ~1.93 (no seed has > 5 words,
all get the length penalty); picker honestly admits
none is good rather than picking a fake winner.
Worth noting: the calm seed=100 case is ANOTHER near-verbatim
single-shot result we missed in the previous bench because the
display truncation hid the full transcript content.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
3-seed picker run (7, 42, 100) on calm/disgust/surprised refines the
single-shot characterization:
calm: winner seed=100, WER 0.64
"It's a good reflection. Not that that. I want to
share somet..."
(recipe lands the prompt — single-shot at seed 42 only
produced hesitation markers; the picker found a seed
with actual content)
disgust: no reliable seed
(all 3 seeds WER ≥ 0.93; likely RAVDESS corpus issue —
disgust clips are low-energy / acoustically close to
neutral. Try CREMA-D or ESD for this emotion.)
surprised: picker chose seed=7 (WER 0.93, short "You can.") over
seed=100 (WER 1.0, "...Today I want to share something")
— WER weighting issue: deletions and insertions count
uniformly, so terse-but-mostly-wrong beat long-and-
mostly-right. Manual selection or weighting WER less
heavily would help here.
Updated per-emotion table marks disgust as ✗ (corpus limitation),
surprised as ⚠ (picker scoring artifact), calm as ✓ (works with
N-seed picker).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Extracted decoder steering vectors for the remaining 3 RAVDESS
emotions (calm, disgust, surprised). Single-seed bench at
seed=42 on Amini context, decoder route, recipe defaults:
emotion cos_ctx WER transcript
calm 0.70 0.86 "I'm sorry. Um, I don't know."
(natural hesitation markers — the
recipe produces semantically-emotion-
matched content, not just acoustic
shift)
disgust 0.81 0.93 "For that, that..." (truncated)
surprised 0.95 2.57 "Too couple, sorry, and that's saying,
even a premier and super driver..."
(long rambling; voice migrates well,
text drifts)
All 7 RAVDESS emotions now produce coherent English on the decoder
route — calm is solid first-shot, disgust truncates, surprised
rambles. Roll N seeds via emotional_speech_n.sh for the latter two.
emotional_speech_guide.md updated with the per-emotion table now
covering all 7. Voice character preservation (cos vs context > 0.7)
holds for every emotion in the pack.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Phase 9 produced 25 commits and a complex pipeline; perf_history is
the engineering log but new users coming to this cold need a clean
"how do I make CSM speak with emotion" handbook.
Sections:
- What this gets you (single-shot WER 0.07–0.21, voice cosine ≥ 0.95)
- One-liner quickstart (RAVDESS download → extract → use)
- The recipe explained — every flag and why it's there
- Per-emotion notes (works/best-seed/caveats per emotion)
- When it works / when it doesn't
- Troubleshooting (music tokens, premature EOT, repetition, etc.)
- Architecture cheat sheet (backbone=semantic, decoder=acoustic)
References perf_history.md for the full empirical log; this doc is
the user-facing distillation.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Capstone consumer interface composing every piece shipped today:
fetch_audio.sh → audio_to_manifest → pick_context.sh →
emotional_speech_n.sh (N-seed picker, decoder route)
Caches fetch + manifest by URL hash so re-runs with the same
--workdir skip the slow steps. Defaults to the Phase 9 recipe:
target=decoder, scale=1.0, layers [2,3], cfg=linear:3.0:1.0:25,
5-seed roll with the lowest-WER winner picked.
End-to-end smoke test (cached Carlini source, prompt "Today I want
to share..."):
picker auto-selected: nicholas_carlini...spk0.0078.wav (10.78 s)
seed 42 (winner): cos 0.974, WER 0.143 ⭐
"But Jason, today I want to share something
with you that I h"
seed 100: cos 0.986, WER 0.286
"It ties upon a share something with you that
I have been thi"
seed 7: cos 0.862, WER 1.000
"Let me think, let him out."
The auto-picker chose spk0 (Carlini himself) where manual selection
earlier in the day grabbed spk1 (the announcer) — so the automated
pipeline is also a slight context-selection improvement.
Three sub-second-WER results recorded over the day:
- WER 0.071 Amini imperative prompt (manual)
- WER 0.125 Amini original prompt (manual)
- WER 0.143 Carlini auto-picked spk0 (this commit, end-to-end)
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Cross-emotion × Carlini context at Amini-best seeds reveals the
"magic combo" doesn't fully transfer:
happy@42 Amini WER 0.21 → Carlini WER **0.071** (transfers!)
angry@100 Amini WER 0.93 → Carlini WER 1.21 (URL drift)
fearful@7 Amini WER 0.86 → Carlini WER 1.00 ("Screw it")
sad@7 Amini WER 0.93 → Carlini WER 1.00 (no transcript)
Only happy@42 cleanly generalizes across contexts. The previous
"context-robust" claim was too strong — the (emotion, seed, context)
interaction matters. Cosine vs context stays high for angry (0.95)
even when text drifts, so voice character preservation is the more
robust property than text fidelity.
Honest production interface: `emotional_speech_n.sh` rolling 5 seeds
per (context, prompt). The single-shot recipe lands well only when
all dimensions align, but the picker absorbs the variance.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
3-context bench at happy@seed=42 decoder recipe (Amini / MCC /
Carlini). Cosine measured vs the context wav (does recipe preserve
input voice character?), WER vs prompt:
context cos_ctx WER transcript
amini 0.972 0.21 "All right, today I want to share
something with you that I've been
thinking about."
mcc 0.58 0.93 "You" (sub-speaker mismatch)
carlini 0.958 0.071 ⭐ "So today I want to share something
with you that I have been thinking
about."
Carlini's WER 0.071 is the new single-shot best of Phase 9. Only
prefix "So" added to the verbatim prompt. Cos vs context > 0.95 on
the two working contexts means the recipe preserves speaker
character of the reference — does NOT impose RAVDESS speaker
identity on every output.
The recipe is context-robust on speaker identities the picker
selects correctly. McConaughey failed because we picked the
manifest's spk1 (likely the Oscars announcer), not McConaughey
himself. That's a context-selection issue, not a recipe issue.
Empirical capstone: single-shot near-verbatim emotional speech
with preserved voice character is achievable.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
3-prompt × 2-condition bench at happy@seed=42 decoder recipe vs
no-steering baseline. The recipe holds across prompts and produces
the lowest WER recorded across all Phase 9 experiments:
prompt base happy decoder
Today I want to… 0.52 / 0.93 "You" 0.82 / 0.21 ⭐
Have you ever… 0.72 / 1.39 drift 0.77 / 1.00 "With blames"
Weather has been… 0.73 / 1.88 ♪♪♪ 0.69 / 0.125 ⭐ "That the
weather has been
absolutely beautiful
this mor"
The imperative prompt's WER 0.125 is the lowest recorded.
Improvement vs baseline ranges 1.4× to 15× lower WER. The
no-steering baseline produced literal singing tokens (♪♪) on the
weather prompt, suggesting CSM's CFG-only path is fragile on
prompts the model "dislikes."
Empirical conclusion: decoder route + RAVDESS happy steering at
seed 42, scale 1.0, layers [2,3] is a reproducible recipe, not a
single-prompt anomaly. N-seed picker still the right consumer
interface, but this single configuration alone reaches
near-publishable quality on multiple prompts.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Records the breakthrough single-shot result: happy@seed=42,
decoder route, scale 1.0, layers [2,3] →
"All right, today I want to share something with you tha"
(WER 0.21, cos 0.82). Closest-to-perfect single-condition
result of the entire Phase 9 sprint.
Per-emotion seed winners diverge:
happy=42, angry=100, fearful=7, sad=7
No universal best seed exists; this validates emotional_speech_n.sh
as the production interface (rolls multiple, picks lowest WER).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Wrapper now supports --target backbone|decoder. Decoder is the
default (Phase 9 finding: it preserves text fidelity that backbone
destroys). Per-(target, emotion) defaults:
decoder: scale 1.0, layers [2,3] (last two of 4)
backbone: scale 0.2-0.3 (per-emotion), layers [8,10,12]
emotional_speech_n.sh's existing passthrough already forwards
--target through to this wrapper unchanged.
Bench all 4 emotions on the decoder route, seed 7, recipe defaults:
emotion cos WER transcript
happy 0.65 0.93 "I've been happy cycling and beat..."
angry 0.78 2.29 "And of course, coming first, Vern is..."
fearful 0.73 0.86 "I'm not eye sensing when that's a mile."
sad 0.67 0.93 "- I'm actually off my night. I'll take
something. - All right..."
Sad — the previously-unsolvable emotion on the backbone (model
resisted at every tested scale 0.15-0.3) — produces real fluent
English on the decoder route. The word "happy" surfaces in the
happy output. All 4 emotions produce coherent speech: no music
tokens, no premature EOT, no gibberish. WER stays in the 0.86-2.3
range, comparable to baseline-with-CFG.
The decoder route subsumes everything the backbone route was
trying to do and unlocks the failure case it couldn't reach.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Mirror of --steering-layers for the decoder: comma-separated layer
indices to actually steer (decoder has 4 layers; useful subsets are
[3], [2,3], [1,2]).
Sweep at seed 7, [email protected]:
[0,1,2,3] cos 0.86 WER 0.93 "I'm not that tall ×3" ← repetition
[3] cos 0.58 WER 0.86 "I'll be off and offense..."
[2,3] cos 0.80 WER 1.43 "My daughter, Penny Ryan, and I have…"
[0,1] cos 0.82 WER 1.57 "And I'll check on them..."
[1,2] cos 0.81 WER 1.43 "I'm going to call him an X-Man..."
The repetition is specific to all-layers-at-once steering. Any 2-layer
subset eliminates it while preserving most of the cosine boost. Same
pattern as the backbone's [8,10,12] finding: partial perturbation
lets the unsteered layers act as a stabilizing prior.
[2,3] (decoder last 2) is the new recommended recipe — best cosine
of the no-repetition subsets and the longest fluent transcript.
Documented in docs/perf_history.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
3-seed × 2-condition ([email protected] alone vs +CFG) bench plus a 4-step
scale sweep. Captures the honest tradeoff:
- Backbone steering destroys word content (semantic gibberish).
- Decoder steering preserves coherent English BUT produces
repetition or premature EOT.
Neither produces single-shot production-quality emotional speech;
emotional_speech_n.sh (N-seed picker, lowest-WER wins) remains the
right consumer interface — it doesn't care which failure mode
generated the bad samples, just discards them by metric.
Decoder vector magnitudes are ~10× smaller than backbone (norm 0.85
at deepest layer vs 14.9), so the apparent useful scale window is
~10× higher (0.5-1.0 instead of 0.2-0.3).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Adds capture_decoder_activations on Model and ModelBackend, plus
--target-module decoder|backbone on examples/steering_extract.
Decoder mode runs a text-only prompt through the backbone (no
capture), Mimi-encodes the audio separately to grab the middle
frame's c0 token, teacher-forces that c0, and captures one mean-
pooled-over-seq vector per decoder layer. Result: 4 layers ×
1024 embed dim per call, much faster than backbone capture
(text-only prompts are short).
A/B with the canonical "Today I want to share..." prompt at seed
7 (the previously-identified low-WER seed):
case cos WER transcript
baseline 0.76 0.36 "And today I want to share something some
funnel distraits" (high baseline at this
seed)
[email protected] 0.75 5.00 "Today, I want to share some needs of my
prey" (backbone destroys content)
[email protected] 0.86 0.93 "I'm not that tall. I'm not that tall."
(fluent but repetitive — biggest cos)
[email protected] 0.61 2.57 over-steered
[email protected] 0.61 2.14 broken
[email protected] is the largest speaker_cosine boost we've measured AND
produces clean English. Backbone steering at the same seed destroyed
content fidelity. This validates the architectural hypothesis: the
backbone carries semantic content (what the model says), the depth
decoder carries acoustic detail (how it sounds). Steering the
decoder shifts voice character without disturbing word content the
way backbone steering does.
Open issues: [email protected] produces repetitive output ("I'm not that
tall" three times). Likely lower scale (~0.5) plus the existing
repetition guard would fix it; left for follow-up.
Decoder vector magnitudes are ~10× smaller than backbone (norm 0.85
at deepest layer vs 14.9), so the appropriate scale is ~10× higher
than the backbone recipe (1.0 vs 0.1-0.3).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Adds set_decoder_steering on Model + Generator and
--decoder-steering-vec / --decoder-steering-scale on examples/generate.
The decoder is already a LlamaModel under the hood, so the existing
LayerSteering hook in csm_fork::Layer::forward applies as-is — only
the public surface needed wiring.
Architectural hypothesis being tested: backbone carries semantic
content (what the model says), depth decoder carries acoustic detail
(how it sounds). Backbone steering shifts character at the cost of
text fidelity (Sprint 2 finding); decoder steering should shift
prosody/timbre without disturbing word content.
Smoke test with random Gaussian decoder vectors (4 layers × 1024
embed_dim, stddev 0.1, scale 0.5):
case cos WER transcript
baseline 0.72 1.0 "No."
backbone 0.83 1.4 "That's for on-beat for bee..."
decoder 0.76 1.0 "So" (premature EOT)
both 0.81 3.0 "I'm going to go to the next one..."
Decoder steering DOES alter output (cosine 0.72 → 0.76, transcript
changes) but random vectors trigger premature EOT — same pattern as
random backbone vectors. The infrastructure works; getting the real
emotion-from-acoustic-codebooks signal needs decoder activation
capture, which the current Model::capture_backbone_activations
doesn't do (it captures the backbone forward only).
Decoder capture is the next-session item. With it we can extract
real per-emotion decoder vectors from RAVDESS and test the
hypothesis properly.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Productionizes yesterday's seed-variance finding. Wraps
emotional_speech.sh, rolls a list of seeds, scores each via
quality_eval (speaker_cosine + Moonshine WER), and copies the
lowest-WER candidate to --out. Defaults to 5 seeds; pass
--seeds 42,7 for cheaper runs.
Tie-breaking is `min(WER), then -max(cosine)` — text fidelity
takes precedence over speaker character because user-typed text
should be rendered verbatim, while voice character is only
secondary on top of context conditioning. Failed generations
(short clips that get the -1 cosine sentinel) sort to the bottom.
Smoke run on the canonical "Today I want to share..." prompt:
seed 7 → cos 0.845, WER 0.714 "Today, today I want to share..." ← picked
seed 100 → cos -1, WER 1.000 "It is." (premature EOT)
seed 42 → cos 0.916, WER 2.000 "The police are, if you're..." (drift)
Cost: N × single-shot cost. The recipe being unreliable per-seed
is the whole reason this wrapper exists — pay the multiplier in
exchange for a reliably-best output.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
3-seed × 3-prompt reproducibility bench on [email protected] reveals that the
recipe shifts speaker character reliably but produces high text-
fidelity variance:
seed 42: WER 2.00 "The police are, if you're, I can't recite..."
seed 7: WER 0.71 "Today, today I want to share..." ← near-verbatim
seed 100: 0.32 s premature EOT
Cross-prompt at seed 42 drifts uniformly across 3 prompts. Speaker
cosine is consistently elevated; text content is roll-the-dice.
Documenting this as the honest characterization rather than overclaim
the single-seed Sprint 2 results. Practical recipe: roll N seeds,
pick lowest-WER output.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Capstone wrapper for the Phase 9 recipe (Selective CFG + RAVDESS
steering + mid-layer subset). Picks the steering scale automatically
from an empirical per-emotion map:
- happy: 0.30
- angry: 0.20
- fearful: 0.20
- sad: 0.20 (note: sad is unreliable — see below)
These came from a follow-up sweep after the multi-emotion demo
revealed the recipe is emotion-sensitive: scale 0.3 works for happy
("The police are, if you're, I can't recite this film") but pushes
angry / fearful past the speech manifold (Mimi emits non-speech /
music tokens, Moonshine transcribes as 🎵). Dropping to 0.2 recovers
fluent speech for both:
- [email protected]: "The next disorder is completing kashim for more."
- [email protected]: "You just heard a little bit about this decision,
though."
- [email protected]: "The police are, if you're, I can't recite this
film. I"
Sad is the outlier — model resists "sad" steering at every scale
between 0.15 and 0.3. Likely a corpus issue (sad RAVDESS clips are
the lowest-energy subset). Documented as a known limitation rather
than worked around.
perf_history.md updated with the per-emotion sensitivity finding.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Sweep of 4 schedules × 4 prompts produced concrete data on schedule
sensitivity. step:3.0:1.5:12 (the Sprint 3 winner) catastrophically
fails dense prompts: lecture-style input → 0.08 s of audio (one
frame). step:2.0:1.0:8 produced the best single shot — near-verbatim
question rendering "Well, it had stem-wondered. Have you ever
wondered why we sometimes hear voices the way we do?" — but tanked
the lecture prompt (2.4 s "You").
linear:3.0:1.0:25 is the only schedule that's never the best AND
never the worst — graceful degradation across all four prompt
categories. Updates the recommended recipe in perf_history.md
(formerly step:3.0:1.5:12).
quality_eval: skip WavLM-SV scoring on clips shorter than 0.25 s
(emit -1 sentinel) — WavLM-SV's TDNN front-end requires a few
hundred samples and crashed mid-sweep on the 0.08 s clip. Now the
eval emits a row instead of bailing on the whole batch.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Adds a clean labeled-emotion data path now that the auto-tagged
manifests have proven inadequate for steering extraction.
scripts/build_ravdess_manifest.sh: parses the RAVDESS speech-only
distribution (Audio_Speech_Actors_01-24.zip from Zenodo, 208 MB) into
a manifest.jsonl with proper {neutral, calm, happy, sad, angry,
fearful, disgust, surprised} labels and the two canonical statements
("Kids are talking by the door", "Dogs are sitting by the door"). 1440
clips, balanced 192/emotion (96 neutral — RAVDESS lacks the 'strong'
intensity for neutral).
examples/steering_extract Mimi reload-every-10: the streaming state
counter overflows 8192 frames after ~80 encodes even with
reset_state(). Same fix training/audio_to_manifest/converse_server use
(commit 0568dd3); now applied here too.
A/B with angry-vs-neutral steering @ 50 samples per pool, matched
against an angry RAVDESS reference clip:
case cos WER transcript
baseline 0.55 0.92 "It is a very important thing to do."
[email protected] 0.61 1.00 "You"
[email protected] 0.74 1.00 "So" ← best speaker_cosine
[email protected] 0.62 1.00 "You"
[email protected] 0.60 1.00 "You"
[email protected] 0.64 6.15 "the Lord, the Lord, the Lord..."
Speaker cosine 0.55 → 0.74 with mid-layer steering at scale 0.5 — a
35 % jump, the largest empirical gain we've measured. The model is
clearly migrating toward the angry actor's voice character. Side
effect: premature EOT (output reduces to one or two words). Likely
because RAVDESS clips themselves are very short ("Kids are talking
by the door", ~3s) so the steering biases toward terse outputs.
That's a corpus-shape artifact, not a code bug — different emotion
corpora with longer utterances should fix it.
Tightest single result of Sprint 2.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Per-frame CFG scale schedule (arXiv 2509.19668, Zheng & Maleki). Pure
inference-time. Standard CFG uses one fixed scale for the whole
sequence; this lets the scale vary across frames so early frames
(speaker character) get full CFG and later frames (text adherence)
get a lower scale.
What lands:
- src/cfg_schedule.rs: CfgSchedule enum (Constant / Step /
LinearRamp), scale_at(frame_idx), parser for CLI form
`step:E:L:T | linear:S:E:R | const:X`. 6 unit tests.
- src/generator.rs: GenerateOptions::cfg_schedule (takes precedence
over legacy cfg_scale; fixed-f64 path is preserved as
Constant(s) for back-compat). Generation loop reads
schedule.scale_at(frame_idx) and passes per-frame to
generate_frame_cfg.
- examples/generate.rs: --cfg-schedule, --cfg-scale, --enable-cfg
flags. Loading via load_csm_1b_with_cfg when --enable-cfg.
A/B with 6s output on Amini context, prompt about Selective CFG:
case cos WER transcript
no-CFG baseline 0.944 1.50 "Okay, the M.U. worked..." (off)
const:2.0 0.854 0.92 "On the right side." (short)
step:3.0:1.5:12 0.938 1.00 "The officer for the selective
C.F.D. paper recommends" (best)
linear:3.0:1.0:25 0.854 1.08 "On the surface..." (off)
Step schedule produces the transcript closest to the input ("the
selective CFG paper recommends..."). WER stays at 1.0 because
Moonshine doesn't know "CFG" as a word, but qualitatively this is
the only one that's coherently following the prompt. Speaker cosine
stays ≈ baseline (0.94) instead of dropping to 0.85 like the
constant and linear cases.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Closes the loop on Phase A (apply hook). Adds:
- LlamaModel capture buffer + start/take_capture API. Pushes one
mean-pooled-over-seq activation per layer into a per-call Vec when
active. Steering apply runs first, so captures reflect post-steering
state when both are on (extractor disables steering for the duration
of the call to capture baseline activations).
- Model::capture_backbone_activations: teacher-forced forward over a
built prompt, returns per-layer (embed_dim,) activation tensors.
- ModelBackend passthrough; FP-only (Quantized errors out).
- examples/steering_extract: reads emotion-labeled JSONL, accumulates
per-emotion sums on CPU f32, writes per-layer
(mean(target) - mean(baseline)) as `layer_<i>_steering` safetensors.
Smoke run on carlini2 manifest (excited vs surprised, 10 samples each):
- Vector norms grow monotonically with depth (layer 0: 2.25, layer 15:
12.61) — consistent with deeper layers carrying richer
emotion/style signal.
- Loaded into examples/generate at scales 0.5/1.0/2.0; quality_eval
shows WER hits 1.0 immediately. This is the EmoSteer paper's warning
("large α may produce unintelligible speech") triggering at small
α — diagnosis: the corpus is the problem, not the infrastructure.
The emotion_tag labels in our existing manifests are noisy
(emotion2vec output on lecture audio collapses to [surprised] /
[excited] without a clean neutral pool), and 10 samples per pool
is well short of the paper's 1000/emotion.
What this validates:
- End-to-end extraction → save → load → apply pathway works.
- quality_eval (Sprint 1) cleanly catches the regression — the metric
foundation does its job.
What's next (a future session):
- Real emotion-labeled dataset (CREMA-D, ESD, RAVDESS) for proper
pools with a true neutral baseline.
- Layer-subset experiments (paper steers layers 1,6,11,16,21 of 32;
for our 16-layer backbone the analogue is roughly 1, 4, 8, 12).
- Listening test alongside the metric numbers.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Adds the apply hook for ActAdd-style activation steering on the Llama
backbone. Inspired by EmoSteer-TTS (arXiv 2508.03543), but adapted: the
paper is flow-matching-specific (DiT layers, 32 CFM steps, per-token
attribution search via mel synthesis), none of which apply to CSM's
autoregressive Llama-over-Mimi-tokens. What's portable is the
underlying difference-in-means construction with residual-stream
addition — the standard ActAdd / contrastive-steering pattern.
What lands:
- src/steering.rs: LayerSteering type, per-layer (1, embed_dim) tensors,
global scale, safetensors load with keys `layer_<i>_steering`. Three
unit tests covering empty/no-op, dimension validation, and apply math.
- src/csm_fork.rs LlamaModel: optional `steering: Option<LayerSteering>`
field, applied after every layer's forward inside the for-loop. Adds
~3 LOC to the hot path; gated by the Option so unsteered generation
has zero cost beyond a None check.
- src/csm_fork.rs Model::set_backbone_steering: installs steering only
on the conditional backbone (cfg_backbone is intentionally left
un-steered so CFG correctly subtracts an unsteered baseline).
- src/generator.rs Generator::set_steering: errors on quantized
backend (only FP supported for now).
- examples/generate.rs: --steering-vec / --steering-scale flags.
- examples/steering_random.rs: smoke helper that writes random Gaussian
vectors so the apply path can be exercised end-to-end before the
real corpus extractor lands. Box-Muller via seeded rand to avoid an
extra rand_distr dep.
Smoke test (16-layer random Gaussian, stddev=0.05, scale=0.5):
- baseline (no steering, same seed/text): 3.04 s @ RMS -19.5 dB
- steered (random vectors): 1.84 s @ RMS -16.2 dB,
EOT triggered earlier
Output clearly differs — pathway is wired correctly. Random vectors
aren't musically meaningful; that's Phase B.
Phase B (next session): corpus extractor that runs forward passes over
emotion-labeled audio (we already have audio_to_manifest emitting
emotion_tag rows), captures per-layer post-residual activations, and
computes the difference-in-means between emotion_X and neutral pools.
Then A/B with quality_eval.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Sprint 1 of the post-research roadmap. TTSDS2 (arXiv 2506.19441) was
the original target but its install is broken on Python 3.12 + modern
torchaudio (deprecated `torchaudio.sox_effects`, `pyannote.audio` 3.1
calls removed `set_audio_backend`, `openai-whisper==20240927` needs
`pkg_resources`). Pivoted to a Rust-native foundation we already own
end-to-end: WavLM-SV + Moonshine + amplitude.
`examples/quality_eval` consumes a JSONL of `(ref_wav, gen_wav,
ref_text)` rows and emits per-row metrics:
- speaker_cosine via WavLM-SV (microsoft/wavlm-base-plus-sv)
- wer via Moonshine v2 transcript vs ref_text (Levenshtein on
lowercased / punctuation-stripped tokens)
- gen_peak_db, gen_rms_db (full-band amplitude of gen_wav)
`scripts/i2d_loop.sh` implements I2D (arXiv 2603.24430): synth N
times feeding each output back as the next iteration's context, score
all iterations with quality_eval, emit a TSV degradation curve.
Smoke-tested:
- quality_eval on the picker A/B set independently confirms the
picker — bottom-context (score 0) → WER 0.55, top-context
(score 2.0) → WER 0.18 (3× worse without picker filter).
- i2d_loop with 3 iterations on Amini context shows clean
collapse: cos 0.84 → 0.58, WER 0.5 → 1.0 by iter 1.
Foundation for Sprint 2 emotion-steering A/B comparisons.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
End-to-end voice clone: fetch_audio.sh → audio_to_manifest →
pick_context.sh → examples/generate. Caches each step by URL hash so
re-runs with the same --workdir skip the slow fetch and
diarize/transcribe stages.
Smoke-tested with cached fetch + manifest. Picks the highest-scoring
context clip across all (manifest, speaker) groups, hands it plus the
target text to generate via the new repeatable --context-wav /
--context-text pairs.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Phase 8.1.1 quality fix from the perf plan. Tokens emitted at LM step
`t` correspond to audio frame `t - ASR_DELAY_FRAMES` (6 frames /
0.48 s), so when a caller stops feeding audio without trailing
silence the last few words trail off — they're still inside the
delay pipeline.
finish() now steps ASR_DELAY_FRAMES additional silent frames after
handling any partial sub-frame buffer, giving the LM the chance to
emit those buffered tokens. Cost: 7 extra step_pcm calls per turn.
Verified end-to-end via stt_demo on a mid-utterance trim of the
LibriSpeech reference clip:
pre-flush: 11 words ("...turnips and carrots and bruised")
post-flush: 13 words ("...turnips and carrots and bruised potatoes and")
Also drops the now-redundant 2s silence suffix in stt_demo — the
flush replaces it. Affects converse_server's real-time end-of-turn
path where suffix padding wasn't possible.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
A/B test 2026-04-29 across mit_2024intro / mcc / carlini manifests
showed the binary peak threshold (≤ -3 dBFS = +0.5) failed to
differentiate hot clips against each other: an mcc clip with input
peak=-1.47 dBFS scored same as one at -3.5 dBFS, and the model output
tracked input amplitude.
Replace with a linear penalty: 0.5 at peak ≤ -9 dBFS, ramping to 0 at
peak = 0 dBFS, clamped. mcc spk0 now produces graduated scores
(1.63 / 1.58 / 1.56 / 1.51) instead of a 1.5 plateau, reordering the
top selection toward the cleaner-peak clip.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
scripts/pick_context.sh ranks manifest clips by suitability for CSM-1B's
--context-wav conditioning (duration sweet spot 10-13.5s, RMS -25 to
-15 dB, peak ≤ -3 dBFS) and groups by (manifest_stem, speaker_id) since
diarizer labels are per-file.
examples/generate.rs now accepts repeatable --context-wav and
--context-text pairs, zipped into Vec<Segment> for Generator::generate.
Validates equal counts at runtime.
Smoke-tested with two 10-13s spk0 clips from the MIT-2024 manifest.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Empirical: with 25-clip cadence the cumulative state still tipped over
once at position 8102 mid-window during a 167-clip lora_eval load.
Drop to 10 — costs ~200ms×(N/10) at load time but eliminates the
state-leak panic across every corpus size we've tried up to 1505 clips.
Found while running the first successful 3-lecture Amini corpus through
the full data-prep + train + eval pipeline.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Three real bugs found while running a YouTube → train → eval pipeline
end-to-end on real corpora:
1. emotion2vec was producing near-constant logits regardless of input.
Per config.yaml `normalize: true` — data2vec2/emotion2vec expects
per-utterance zero-mean unit-variance normalization on the raw
waveform before the local_encoder. Added inside the EmotionDetector
trait impl so all callers get it.
Verified empirically: 4 different audio inputs (Carlini talk,
audience question, McConaughey speech) now produce different argmax
classes. Before fix: all 4 produced identical logits.
2. The 9→5 emotion fold was collapsing every real-world clip to
[excited]. happy / surprised / other all mapped to Excited covered
~95% of natural speech. Replaced with a direct 9-class identity
mapping; EmotionLabel gained Disgusted, Fearful, Happy, Surprised,
Unk variants. Now: 132 [surprised] + 12 [excited] across the
Carlini corpus instead of 144 [excited].
3. lora_train_emotional --peak-lr / --epochs flags. The canned 3-stage
recipe over-fits on small (~100 clip) corpora at extended rank 8;
users need to tune. (The recipe stays as defaults; flags are pure
overrides.)
Plus diagnostic: examples/emotion2vec_probe — feed real audio files
into emotion2vec and dump per-class logits. Used to find bug #1.
Lib suite still 131/131 (the test that locked the 9→5 fold updated
to lock the new identity mapping).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Mimi's transformer carries an internal position counter across encode()
calls that the upstream `reset_state()` does NOT fully clear. When
TrainingDataset::load_from_manifest encodes ~80-100 clips back-to-back
during data prep, the counter overflows the 8192-position buffer and
panics with `narrow invalid args [8192, 32]`.
src/mimi.rs:
- cache the safetensors path on Mimi at construction
- new Mimi.reload() drops the inner Model and rebuilds from the
cached path (~200 ms on Metal)
src/training.rs:
- call generator.mimi.reload() every 50 clips during
load_from_manifest. Adds ~1 s overhead on a 200-clip corpus
(4 reloads × ~200 ms) vs the alternative of a hard panic.
- reset_state() before each encode in TrainingExample::from_audio
is kept (still useful to clear streaming chunk state).
Found while running the end-to-end personal-voice training pipeline on
a 20-minute YouTube source: the bug surfaces around clip 86 when
Mimi's transformer hits position 8181+. Filtering to short clips
alone didn't help — the cumulative state grows even with sub-12-second
inputs.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Wrapper that pulls audio from any yt-dlp-supported URL (YouTube,
LibriVox, archive.org, podcast feeds) and converts to the 24 kHz mono
16-bit PCM format examples/audio_to_manifest ingests. Slugifies the
output filename so manifest paths stay shell-safe.
Prints the next-step audio_to_manifest command with all the right
flags (--auto-emotion-tag --use-emotion2vec --stage audiobook), so a
new user can copy-paste the printed line straight into a terminal.
Requires external tools (yt-dlp, ffmpeg); install on macOS via
`brew install yt-dlp ffmpeg`.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Word-level forced alignment shipped. Phase 13.9 complete.
viterbi_align(log_probs, tokens, blank_id, vocab) — standard CTC
forced-alignment trellis: states alternate [blank, t0, blank, t1, ...,
tN, blank] (length 2N+1), at each frame stay/advance/ε-skip-blank-
between-different-tokens (canonical CTC ε-skip rule correctly forbids
skipping blank between SAME tokens), max-likelihood path recovered via
backptr table.
transcript_to_token_ids — text → CTC token ids; runs of spaces collapse
to | separator; unknown chars → <unk>.
group_into_words — fold adjacent non-| AlignedToken into AlignedWord
with carried frame_start/frame_end.
frame_to_ms — 50 Hz frame grid → ms (20 ms/frame at conv stride 320).
examples/wav2vec2_smoke --align <target> wires it end-to-end:
forced-aligns a known transcript and prints (word, start_ms, end_ms).
Verified on Metal: 10.42 s LibriSpeech audio, first 8 words →
HE 560-640 HOPED 720-960 THERE 1000-1140
WOULD 1180-1320 BE 1360-2240 STEW 2980-4720
FOR 5300-6000 DINNER 7040-8540
viterbi alignment in 0 ms (39 tokens). Boundaries match audio.
4 new unit tests:
- transcript_to_token_ids_handles_spaces_and_unknowns
- viterbi_align_recovers_obvious_alignment
- group_into_words_splits_on_separator
- frame_to_ms_50hz_grid
Lib suite 131/131 (was 127, +4).
Phase 13.9 complete (slices 1+2+3+4). Crate now ships full English
ASR + word-level forced alignment in pure candle — no whisper.cpp,
no ort, no Python. Data-prep can cut long audio at exact word
boundaries before feeding into the Phase 12.3 curriculum trainer.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Full port of facebook/wav2vec2-base-960h (94.4 M params, MIT) closing
the WhisperX-class word-alignment gap from the audio-ML survey. Same
staged-scaffolding pattern that worked for emotion2vec — but landed
slices 1+2+3 in one session.
src/wav2vec2.rs ships:
- Wav2Vec2Config::base_960h
- FeatureExtractor — 7 Conv1d (1→512, total stride 320). Layer 0
uses GroupNorm with num_groups=num_channels=512 (HF's wav2vec2
feat_extract_norm: "group"). Critical: state-dict key is
layer_norm.* but the OP is GroupNorm — loading as LayerNorm
produces empty CTC output.
- FeatureProjection — LayerNorm(512) + Linear(512→768)
- ConvPosEmbedding — kernel 128 grouped Conv1d, materialized at
load time from upstream weight_g + weight_v (fairseq's weight_norm
on dim=2; eps-guarded division for numerical stability)
- Block — POST-norm transformer with separate Q/K/V (vs emotion2vec's
fused QKV), uses (B*H, T, D) Metal 3D-matmul workaround from
Phase 8.8 Moonshine
- Encoder — pos_conv + initial LayerNorm + 12 Blocks
- Wav2Vec2 top-level — load_from_safetensors via mmap'd VarBuilder
- ctc_greedy_decode + VOCAB_960H constant for the 32-char alphabet
examples/wav2vec2_inspect.rs (slice 1): dumps tensor layout + config
examples/wav2vec2_smoke.rs (slice 3): real-weight load + ASR forward
Verified on Metal:
loaded model in 0.28 s
forward in 9 ms for 10.42 s audio (~1150× realtime)
transcript: "HE HOPED THERE WOULD BE STEW FOR DINNER TURNIPS AND
CARROTS AND BRUISED POTATOES AND FAT MUTTON PIECES TO
BE LADLED OUT IN THICK PEPPERED FLOWER FAT AND SAUCE"
Numerical parity with upstream Python — the FLOWER-for-FLOUR typo is
the known wav2vec2-base-960h failure mode, matches HF reference exactly.
7 new unit tests; lib suite 127/127 (was 120).
Slice 4 remaining: Viterbi forced alignment given known transcript,
to emit (token, frame_start_ms, frame_end_ms) for word-boundary cuts.
The ASR path itself is now production-ready.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Port complete. The Phase 13.3 prosody-rule placeholder is now
retire-able by setting one CLI flag — the real candle-ported
emotion2vec_plus_base classifier slots in behind the same
EmotionDetector trait the placeholder used.
impl EmotionDetector for Emotion2Vec — builds (1, 1, T) tensor on
stored device, runs forward, argmaxes the 9 logits, maps to the
5-bucket label via Classifier::tag_for_class. Empty input
short-circuits to Neutral.
Emotion2Vec struct gained a `device` field so the trait impl can
build tensors without an out-of-band handle. new() / load_from_pickle()
threaded through; existing tests + smoke binary updated.
audio_to_manifest --use-emotion2vec — pairs with --auto-emotion-tag
to swap ProsodyDetector for Emotion2Vec, boxed as
Box<dyn EmotionDetector> so the call site is unchanged.
converse_server --use-emotion2vec — same pattern; built once at boot
and stored in Shared as Box<dyn EmotionDetector + Send + Sync>.
~150 ms/turn forward cost vs <1 ms for prosody, but actually runs
SOTA SER. Removed redundant reactive_emotion: bool field — the
Option<Box<dyn>> already encodes the same state.
Verified end-to-end on Metal:
- audio_to_manifest --use-emotion2vec on 2-speaker concat → both
tagged [excited] (prosody had said [neutral] on same input)
- converse_server --quantized-gguf … --lora … --reactive-emotion
--use-emotion2vec boots, 1 bench turn 0 errors, /metrics shows
reactive_emotion_total{label="excited"} 1 — same tag
audio_to_manifest produced. Cross-consumer consistency.
Phase 13.8 complete (slices 1+2+3+4 shipped). The emotional-voice
stack now has a real, trained, candle-ported SER classifier with
no Python sidecar, no ort, no whisper.cpp.
Lib suite 120/120.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Slice 2 complete + slice 3 collapsed in. The full candle port loads
real upstream weights and runs forward in 160 ms.
RelativePositionalEncoder — 5 grouped Conv1d (kernel 19, groups 16,
same-padding via pad 9) with GELU between. Output is added back to
input as a positional bias. Pickle keys
relative_positional_encoder.{1..=5}.0.weight/bias (1-based indexing,
no .0.*).
Emotion2Vec top-level — wires LocalEncoder → ProjectFeatures →
RelPosEnc → ContextEncoder → MainEncoder → mean-pool → Classifier.
The proj.* classifier head lives at the state-dict root (not under
d2v_model.), so the constructor uses vb directly there.
Emotion2Vec::load_from_pickle uses VarBuilder::from_pth_with_state
to descend into the fairseq-style nested checkpoint via the "model"
key. One-shot loader; all 185 upstream tensor keys must map onto
candle params of matching shape — and they do.
examples/emotion2vec_smoke.rs — full pipeline integration test:
downloads (or reuses cached) emotion2vec_plus_base from HF, loads it
into candle, runs forward on 2 s of synthetic audio, prints all 9
raw logits + argmax + the 9→5 bucket fold.
Verified on Metal:
loaded model in 0.19 s
forward in 160 ms
9 logits all finite (50-290 range, expected for raw classifier)
argmax: class 7 (surprised) → 5-bucket [excited]
Mechanical correctness end-to-end. Semantic accuracy on real
emotional speech lands in slice 4 (EmotionDetector trait impl +
swap into audio_to_manifest + converse_server reactive-emotion path).
2 new unit tests:
- relative_positional_encoder_preserves_shape
- emotion2vec_random_init_end_to_end_shape
Lib suite 120/120 (was 118, +2).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Three composition wrappers over the Block from slice 2b.
ContextEncoder — 4 Blocks + final LayerNorm
(context_encoder.blocks.0..3 + context_encoder.norm). Acts as a prenet
between ProjectFeatures and the main encoder.
MainEncoder — 8 Blocks, no final LayerNorm. Confirmed via inspector:
all 96 d2v_model.blocks.* tensors live inside numbered blocks; there's
no d2v_model.norm. Pre-norm pattern's per-block norm2 keeps residuals
conditioned without a global tail norm.
Classifier — single Linear 768→9 (proj.weight/proj.bias at the top of
the state dict, NOT under d2v_model.). Includes tag_for_class(idx)
that folds the 9 fine-grained model classes (angry/disgusted/fearful/
happy/neutral/other/sad/surprised/<unk>) into the 5-bucket label set
the Phase 13.3 EmotionDetector trait already uses:
- 0 angry → Angry
- 1 disgusted, 2 fearful,
6 sad → Sad (low valence)
- 3 happy, 5 other,
7 surprised → Excited (high arousal)
- 4 neutral, 8 <unk> → Neutral
4 new tests:
- context_encoder_chains_4_blocks_with_final_norm
- main_encoder_chains_8_blocks_no_final_norm
- classifier_emits_9_logits
- classifier_class_to_emotion_label_mapping (locks the 9→5 fold)
Lib suite 118/118 (was 114, +4 new).
Slice 2 remaining: relative_positional_encoder (5 Conv1d, the conv
positional bias) + top-level Emotion2Vec + .pt pickle loader. Then
slice 3 = forward pass + numerical parity check vs upstream Python.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Two more candle modules toward the full port.
ProjectFeatures — LayerNorm(512) → Linear(512→768). Sits between
LocalEncoder and the transformer. Pickle layout matches upstream:
project_features.1.* is the LayerNorm, project_features.2.* is the
Linear. Both have learnable affine params; the .1 LN is NOT just an
eps constant.
Block — pre-norm fused-QKV transformer block, the workhorse for both
ContextEncoder (4 instances) and MainEncoder (8 instances). Pickle
keys per block: norm1, attn.qkv (fused 768→2304), attn.proj,
norm2, mlp.fc1, mlp.fc2. GELU MLP activation. No positional encoding
inside the block — the conv-based positional bias lives at the encoder
boundary.
Attention uses the (B*H, T, D) 3D collapse-before-matmul Metal
workaround we shipped for Phase 8.8 Moonshine — candle's 4D batched
matmul still has the shape-mismatch bug.
2 new unit tests:
- project_features_shape_check: (1, 50, 512) → (1, 50, 768)
- block_residual_shape_check: random (2, 8, 768) → same shape AND
all values finite (catches softmax NaN / attention overflow)
Lib suite 114/114 (was 112, +2 new).
Remaining within slice 2: relative_positional_encoder (5 Conv1d),
ContextEncoder (4 Blocks), MainEncoder (8 Blocks + LN), Classifier
(Linear 768→9), top-level Emotion2Vec + pickle .pt loader.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Closes the gap where converse_server (Phase 12.4) supported LoRA but
the simpler HTTP-only tts_server didn't. Same flag set
(--lora / --lora-rank / --lora-alpha / --extended-lora) and same
apply_lora_adapter shared helper. Combines with --quantized-gguf
(Phase 12.6) for Q8 + voice clone over plain HTTP.
Verified end-to-end on Metal: tts_server --quantized-gguf … --lora …
boots, injects LoRA into the quantized backbone (q=16 k=16 v=16 o=16
+ MLP w1/w2/w3 = 224 tensors), listens. Single HTTP POST /v1/tts
returned 200 OK with a 146KB 24kHz mono WAV. Lib suite 110/110.
tts_server is now the simplest production deploy for a personalized
voice: HTTP-only, no STT/LLM overhead, LoRA + Q8 + AudioSeal/SilentCipher
+ WavLM-SV all available behind one binary.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
