Adds capture_decoder_activations on Model and ModelBackend, plus
--target-module decoder|backbone on examples/steering_extract.
Decoder mode runs a text-only prompt through the backbone (no
capture), Mimi-encodes the audio separately to grab the middle
frame's c0 token, teacher-forces that c0, and captures one mean-
pooled-over-seq vector per decoder layer. Result: 4 layers ×
1024 embed dim per call, much faster than backbone capture
(text-only prompts are short).
A/B with the canonical "Today I want to share..." prompt at seed
7 (the previously-identified low-WER seed):
case cos WER transcript
baseline 0.76 0.36 "And today I want to share something some
funnel distraits" (high baseline at this
seed)
[email protected] 0.75 5.00 "Today, I want to share some needs of my
prey" (backbone destroys content)
[email protected] 0.86 0.93 "I'm not that tall. I'm not that tall."
(fluent but repetitive — biggest cos)
[email protected] 0.61 2.57 over-steered
[email protected] 0.61 2.14 broken
[email protected] is the largest speaker_cosine boost we've measured AND
produces clean English. Backbone steering at the same seed destroyed
content fidelity. This validates the architectural hypothesis: the
backbone carries semantic content (what the model says), the depth
decoder carries acoustic detail (how it sounds). Steering the
decoder shifts voice character without disturbing word content the
way backbone steering does.
Open issues: [email protected] produces repetitive output ("I'm not that
tall" three times). Likely lower scale (~0.5) plus the existing
repetition guard would fix it; left for follow-up.
Decoder vector magnitudes are ~10× smaller than backbone (norm 0.85
at deepest layer vs 14.9), so the appropriate scale is ~10× higher
than the backbone recipe (1.0 vs 0.1-0.3).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Adds a clean labeled-emotion data path now that the auto-tagged
manifests have proven inadequate for steering extraction.
scripts/build_ravdess_manifest.sh: parses the RAVDESS speech-only
distribution (Audio_Speech_Actors_01-24.zip from Zenodo, 208 MB) into
a manifest.jsonl with proper {neutral, calm, happy, sad, angry,
fearful, disgust, surprised} labels and the two canonical statements
("Kids are talking by the door", "Dogs are sitting by the door"). 1440
clips, balanced 192/emotion (96 neutral — RAVDESS lacks the 'strong'
intensity for neutral).
examples/steering_extract Mimi reload-every-10: the streaming state
counter overflows 8192 frames after ~80 encodes even with
reset_state(). Same fix training/audio_to_manifest/converse_server use
(commit 0568dd3); now applied here too.
A/B with angry-vs-neutral steering @ 50 samples per pool, matched
against an angry RAVDESS reference clip:
case cos WER transcript
baseline 0.55 0.92 "It is a very important thing to do."
[email protected] 0.61 1.00 "You"
[email protected] 0.74 1.00 "So" ← best speaker_cosine
[email protected] 0.62 1.00 "You"
[email protected] 0.60 1.00 "You"
[email protected] 0.64 6.15 "the Lord, the Lord, the Lord..."
Speaker cosine 0.55 → 0.74 with mid-layer steering at scale 0.5 — a
35 % jump, the largest empirical gain we've measured. The model is
clearly migrating toward the angry actor's voice character. Side
effect: premature EOT (output reduces to one or two words). Likely
because RAVDESS clips themselves are very short ("Kids are talking
by the door", ~3s) so the steering biases toward terse outputs.
That's a corpus-shape artifact, not a code bug — different emotion
corpora with longer utterances should fix it.
Tightest single result of Sprint 2.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Closes the loop on Phase A (apply hook). Adds:
- LlamaModel capture buffer + start/take_capture API. Pushes one
mean-pooled-over-seq activation per layer into a per-call Vec when
active. Steering apply runs first, so captures reflect post-steering
state when both are on (extractor disables steering for the duration
of the call to capture baseline activations).
- Model::capture_backbone_activations: teacher-forced forward over a
built prompt, returns per-layer (embed_dim,) activation tensors.
- ModelBackend passthrough; FP-only (Quantized errors out).
- examples/steering_extract: reads emotion-labeled JSONL, accumulates
per-emotion sums on CPU f32, writes per-layer
(mean(target) - mean(baseline)) as `layer_<i>_steering` safetensors.
Smoke run on carlini2 manifest (excited vs surprised, 10 samples each):
- Vector norms grow monotonically with depth (layer 0: 2.25, layer 15:
12.61) — consistent with deeper layers carrying richer
emotion/style signal.
- Loaded into examples/generate at scales 0.5/1.0/2.0; quality_eval
shows WER hits 1.0 immediately. This is the EmoSteer paper's warning
("large α may produce unintelligible speech") triggering at small
α — diagnosis: the corpus is the problem, not the infrastructure.
The emotion_tag labels in our existing manifests are noisy
(emotion2vec output on lecture audio collapses to [surprised] /
[excited] without a clean neutral pool), and 10 samples per pool
is well short of the paper's 1000/emotion.
What this validates:
- End-to-end extraction → save → load → apply pathway works.
- quality_eval (Sprint 1) cleanly catches the regression — the metric
foundation does its job.
What's next (a future session):
- Real emotion-labeled dataset (CREMA-D, ESD, RAVDESS) for proper
pools with a true neutral baseline.
- Layer-subset experiments (paper steers layers 1,6,11,16,21 of 32;
for our 16-layer backbone the analogue is roughly 1, 4, 8, 12).
- Listening test alongside the metric numbers.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
redclawsystems