Adds capture_decoder_activations on Model and ModelBackend, plus
--target-module decoder|backbone on examples/steering_extract.
Decoder mode runs a text-only prompt through the backbone (no
capture), Mimi-encodes the audio separately to grab the middle
frame's c0 token, teacher-forces that c0, and captures one mean-
pooled-over-seq vector per decoder layer. Result: 4 layers ×
1024 embed dim per call, much faster than backbone capture
(text-only prompts are short).
A/B with the canonical "Today I want to share..." prompt at seed
7 (the previously-identified low-WER seed):
case cos WER transcript
baseline 0.76 0.36 "And today I want to share something some
funnel distraits" (high baseline at this
seed)
[email protected] 0.75 5.00 "Today, I want to share some needs of my
prey" (backbone destroys content)
[email protected] 0.86 0.93 "I'm not that tall. I'm not that tall."
(fluent but repetitive — biggest cos)
[email protected] 0.61 2.57 over-steered
[email protected] 0.61 2.14 broken
[email protected] is the largest speaker_cosine boost we've measured AND
produces clean English. Backbone steering at the same seed destroyed
content fidelity. This validates the architectural hypothesis: the
backbone carries semantic content (what the model says), the depth
decoder carries acoustic detail (how it sounds). Steering the
decoder shifts voice character without disturbing word content the
way backbone steering does.
Open issues: [email protected] produces repetitive output ("I'm not that
tall" three times). Likely lower scale (~0.5) plus the existing
repetition guard would fix it; left for follow-up.
Decoder vector magnitudes are ~10× smaller than backbone (norm 0.85
at deepest layer vs 14.9), so the appropriate scale is ~10× higher
than the backbone recipe (1.0 vs 0.1-0.3).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Closes the loop on Phase A (apply hook). Adds:
- LlamaModel capture buffer + start/take_capture API. Pushes one
mean-pooled-over-seq activation per layer into a per-call Vec when
active. Steering apply runs first, so captures reflect post-steering
state when both are on (extractor disables steering for the duration
of the call to capture baseline activations).
- Model::capture_backbone_activations: teacher-forced forward over a
built prompt, returns per-layer (embed_dim,) activation tensors.
- ModelBackend passthrough; FP-only (Quantized errors out).
- examples/steering_extract: reads emotion-labeled JSONL, accumulates
per-emotion sums on CPU f32, writes per-layer
(mean(target) - mean(baseline)) as `layer_<i>_steering` safetensors.
Smoke run on carlini2 manifest (excited vs surprised, 10 samples each):
- Vector norms grow monotonically with depth (layer 0: 2.25, layer 15:
12.61) — consistent with deeper layers carrying richer
emotion/style signal.
- Loaded into examples/generate at scales 0.5/1.0/2.0; quality_eval
shows WER hits 1.0 immediately. This is the EmoSteer paper's warning
("large α may produce unintelligible speech") triggering at small
α — diagnosis: the corpus is the problem, not the infrastructure.
The emotion_tag labels in our existing manifests are noisy
(emotion2vec output on lecture audio collapses to [surprised] /
[excited] without a clean neutral pool), and 10 samples per pool
is well short of the paper's 1000/emotion.
What this validates:
- End-to-end extraction → save → load → apply pathway works.
- quality_eval (Sprint 1) cleanly catches the regression — the metric
foundation does its job.
What's next (a future session):
- Real emotion-labeled dataset (CREMA-D, ESD, RAVDESS) for proper
pools with a true neutral baseline.
- Layer-subset experiments (paper steers layers 1,6,11,16,21 of 32;
for our 16-layer backbone the analogue is roughly 1, 4, 8, 12).
- Listening test alongside the metric numbers.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Omar Sobh