Adds capture_decoder_activations on Model and ModelBackend, plus
--target-module decoder|backbone on examples/steering_extract.
Decoder mode runs a text-only prompt through the backbone (no
capture), Mimi-encodes the audio separately to grab the middle
frame's c0 token, teacher-forces that c0, and captures one mean-
pooled-over-seq vector per decoder layer. Result: 4 layers ×
1024 embed dim per call, much faster than backbone capture
(text-only prompts are short).
A/B with the canonical "Today I want to share..." prompt at seed
7 (the previously-identified low-WER seed):
case cos WER transcript
baseline 0.76 0.36 "And today I want to share something some
funnel distraits" (high baseline at this
seed)
[email protected] 0.75 5.00 "Today, I want to share some needs of my
prey" (backbone destroys content)
[email protected] 0.86 0.93 "I'm not that tall. I'm not that tall."
(fluent but repetitive — biggest cos)
[email protected] 0.61 2.57 over-steered
[email protected] 0.61 2.14 broken
[email protected] is the largest speaker_cosine boost we've measured AND
produces clean English. Backbone steering at the same seed destroyed
content fidelity. This validates the architectural hypothesis: the
backbone carries semantic content (what the model says), the depth
decoder carries acoustic detail (how it sounds). Steering the
decoder shifts voice character without disturbing word content the
way backbone steering does.
Open issues: [email protected] produces repetitive output ("I'm not that
tall" three times). Likely lower scale (~0.5) plus the existing
repetition guard would fix it; left for follow-up.
Decoder vector magnitudes are ~10× smaller than backbone (norm 0.85
at deepest layer vs 14.9), so the appropriate scale is ~10× higher
than the backbone recipe (1.0 vs 0.1-0.3).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Adds set_decoder_steering on Model + Generator and
--decoder-steering-vec / --decoder-steering-scale on examples/generate.
The decoder is already a LlamaModel under the hood, so the existing
LayerSteering hook in csm_fork::Layer::forward applies as-is — only
the public surface needed wiring.
Architectural hypothesis being tested: backbone carries semantic
content (what the model says), depth decoder carries acoustic detail
(how it sounds). Backbone steering shifts character at the cost of
text fidelity (Sprint 2 finding); decoder steering should shift
prosody/timbre without disturbing word content.
Smoke test with random Gaussian decoder vectors (4 layers × 1024
embed_dim, stddev 0.1, scale 0.5):
case cos WER transcript
baseline 0.72 1.0 "No."
backbone 0.83 1.4 "That's for on-beat for bee..."
decoder 0.76 1.0 "So" (premature EOT)
both 0.81 3.0 "I'm going to go to the next one..."
Decoder steering DOES alter output (cosine 0.72 → 0.76, transcript
changes) but random vectors trigger premature EOT — same pattern as
random backbone vectors. The infrastructure works; getting the real
emotion-from-acoustic-codebooks signal needs decoder activation
capture, which the current Model::capture_backbone_activations
doesn't do (it captures the backbone forward only).
Decoder capture is the next-session item. With it we can extract
real per-emotion decoder vectors from RAVDESS and test the
hypothesis properly.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Closes the loop on Phase A (apply hook). Adds:
- LlamaModel capture buffer + start/take_capture API. Pushes one
mean-pooled-over-seq activation per layer into a per-call Vec when
active. Steering apply runs first, so captures reflect post-steering
state when both are on (extractor disables steering for the duration
of the call to capture baseline activations).
- Model::capture_backbone_activations: teacher-forced forward over a
built prompt, returns per-layer (embed_dim,) activation tensors.
- ModelBackend passthrough; FP-only (Quantized errors out).
- examples/steering_extract: reads emotion-labeled JSONL, accumulates
per-emotion sums on CPU f32, writes per-layer
(mean(target) - mean(baseline)) as `layer_<i>_steering` safetensors.
Smoke run on carlini2 manifest (excited vs surprised, 10 samples each):
- Vector norms grow monotonically with depth (layer 0: 2.25, layer 15:
12.61) — consistent with deeper layers carrying richer
emotion/style signal.
- Loaded into examples/generate at scales 0.5/1.0/2.0; quality_eval
shows WER hits 1.0 immediately. This is the EmoSteer paper's warning
("large α may produce unintelligible speech") triggering at small
α — diagnosis: the corpus is the problem, not the infrastructure.
The emotion_tag labels in our existing manifests are noisy
(emotion2vec output on lecture audio collapses to [surprised] /
[excited] without a clean neutral pool), and 10 samples per pool
is well short of the paper's 1000/emotion.
What this validates:
- End-to-end extraction → save → load → apply pathway works.
- quality_eval (Sprint 1) cleanly catches the regression — the metric
foundation does its job.
What's next (a future session):
- Real emotion-labeled dataset (CREMA-D, ESD, RAVDESS) for proper
pools with a true neutral baseline.
- Layer-subset experiments (paper steers layers 1,6,11,16,21 of 32;
for our 16-layer backbone the analogue is roughly 1, 4, 8, 12).
- Listening test alongside the metric numbers.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Adds the apply hook for ActAdd-style activation steering on the Llama
backbone. Inspired by EmoSteer-TTS (arXiv 2508.03543), but adapted: the
paper is flow-matching-specific (DiT layers, 32 CFM steps, per-token
attribution search via mel synthesis), none of which apply to CSM's
autoregressive Llama-over-Mimi-tokens. What's portable is the
underlying difference-in-means construction with residual-stream
addition — the standard ActAdd / contrastive-steering pattern.
What lands:
- src/steering.rs: LayerSteering type, per-layer (1, embed_dim) tensors,
global scale, safetensors load with keys `layer_<i>_steering`. Three
unit tests covering empty/no-op, dimension validation, and apply math.
- src/csm_fork.rs LlamaModel: optional `steering: Option<LayerSteering>`
field, applied after every layer's forward inside the for-loop. Adds
~3 LOC to the hot path; gated by the Option so unsteered generation
has zero cost beyond a None check.
- src/csm_fork.rs Model::set_backbone_steering: installs steering only
on the conditional backbone (cfg_backbone is intentionally left
un-steered so CFG correctly subtracts an unsteered baseline).
- src/generator.rs Generator::set_steering: errors on quantized
backend (only FP supported for now).
- examples/generate.rs: --steering-vec / --steering-scale flags.
- examples/steering_random.rs: smoke helper that writes random Gaussian
vectors so the apply path can be exercised end-to-end before the
real corpus extractor lands. Box-Muller via seeded rand to avoid an
extra rand_distr dep.
Smoke test (16-layer random Gaussian, stddev=0.05, scale=0.5):
- baseline (no steering, same seed/text): 3.04 s @ RMS -19.5 dB
- steered (random vectors): 1.84 s @ RMS -16.2 dB,
EOT triggered earlier
Output clearly differs — pathway is wired correctly. Random vectors
aren't musically meaningful; that's Phase B.
Phase B (next session): corpus extractor that runs forward passes over
emotion-labeled audio (we already have audio_to_manifest emitting
emotion_tag rows), captures per-layer post-residual activations, and
computes the difference-in-means between emotion_X and neutral pools.
Then A/B with quality_eval.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Both backbones (FP csm_fork + Q8 csm_quantized) now expose 7 LoRA hooks
per layer: q/k/v/o on attention plus gate/up/down (Llama w1/w3/w2) on
the SwiGLU MLP. LoraConfig::default() still returns q+v only (backward
compat for existing trained adapters); LoraConfig::extended() returns
the full 7-module set. lora_train + lora_finetune_step take a
--extended-lora flag.
Verified end-to-end on Metal: injection across all 16 backbone layers
× 7 modules = 224 adapter Vars, 5.6M trainable params (~6.6× q+v alone,
still tiny vs the 1B base). Step-0 loss matches the q+v baseline
exactly (B=0 init is also a no-op for the new hooks). Forward + backward
+ AdamW + refresh_lora cycle runs without errors.
LoRA test suite: 9 pass (added config_extended_targets_full_attn_and_mlp);
full lib suite still 92/92.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Omar Sobh