rtx-csm: depth-decoder steering API
Adds set_decoder_steering on Model + Generator and --decoder-steering-vec / --decoder-steering-scale on examples/generate. The decoder is already a LlamaModel under the hood, so the existing LayerSteering hook in csm_fork::Layer::forward applies as-is — only the public surface needed wiring. Architectural hypothesis being tested: backbone carries semantic content (what the model says), depth decoder carries acoustic detail (how it sounds). Backbone steering shifts character at the cost of text fidelity (Sprint 2 finding); decoder steering should shift prosody/timbre without disturbing word content. Smoke test with random Gaussian decoder vectors (4 layers × 1024 embed_dim, stddev 0.1, scale 0.5): case cos WER transcript baseline 0.72 1.0 "No." backbone 0.83 1.4 "That's for on-beat for bee..." decoder 0.76 1.0 "So" (premature EOT) both 0.81 3.0 "I'm going to go to the next one..." Decoder steering DOES alter output (cosine 0.72 → 0.76, transcript changes) but random vectors trigger premature EOT — same pattern as random backbone vectors. The infrastructure works; getting the real emotion-from-acoustic-codebooks signal needs decoder activation capture, which the current Model::capture_backbone_activations doesn't do (it captures the backbone forward only). Decoder capture is the next-session item. With it we can extract real per-emotion decoder vectors from RAVDESS and test the hypothesis properly. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
This commit is contained in:
@@ -793,6 +793,28 @@ impl Model {
|
||||
self.backbone.layers.len()
|
||||
}
|
||||
|
||||
/// Install activation-steering vectors on the depth decoder. The
|
||||
/// decoder generates the acoustic codebooks (c1..N-1) given a
|
||||
/// sampled c0 + the backbone hidden state — so steering here should
|
||||
/// affect prosody/timbre without changing word-level content. The
|
||||
/// decoder for CSM-1B is Llama100M (4 layers × 1024 embed_dim);
|
||||
/// vectors must match that shape.
|
||||
pub fn set_decoder_steering(&mut self, steering: Option<crate::steering::LayerSteering>) {
|
||||
self.decoder.set_steering(steering);
|
||||
}
|
||||
|
||||
pub fn decoder_num_layers(&self) -> usize {
|
||||
self.decoder.layers.len()
|
||||
}
|
||||
|
||||
pub fn decoder_embed_dim(&self) -> usize {
|
||||
LlamaConfig::from_flavor(self.config.decoder_flavor).embed_dim
|
||||
}
|
||||
|
||||
pub fn backbone_embed_dim(&self) -> usize {
|
||||
LlamaConfig::from_flavor(self.config.backbone_flavor).embed_dim
|
||||
}
|
||||
|
||||
/// Teacher-forced forward through the backbone over a built-prompt
|
||||
/// `(tokens, mask)` and return one mean-pooled-over-seq activation
|
||||
/// vector per layer. Used by `examples/steering_extract` to derive
|
||||
|
||||
Reference in New Issue
Block a user