rtx-csm: perf_history — decoder steering empirical characterization

3-seed × 2-condition ([email protected] alone vs +CFG) bench plus a 4-step
scale sweep. Captures the honest tradeoff:

- Backbone steering destroys word content (semantic gibberish).
- Decoder steering preserves coherent English BUT produces
  repetition or premature EOT.

Neither produces single-shot production-quality emotional speech;
emotional_speech_n.sh (N-seed picker, lowest-WER wins) remains the
right consumer interface — it doesn't care which failure mode
generated the bad samples, just discards them by metric.

Decoder vector magnitudes are ~10× smaller than backbone (norm 0.85
at deepest layer vs 14.9), so the apparent useful scale window is
~10× higher (0.5-1.0 instead of 0.2-0.3).

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-04-29 14:21:06 -07:00
co-authored by Claude Opus 4.7
parent 99a7e53aa8
commit 68ecae1b08
@@ -160,6 +160,55 @@ shipped on the FP backbone; quantized path unchanged.
"biased random sampling toward emotional voice" than to "biased random sampling toward emotional voice" than to
"deterministic emotional rendering." Reproducible voice cloning "deterministic emotional rendering." Reproducible voice cloning
(`--context-wav` alone) is much more deterministic than this. (`--context-wav` alone) is much more deterministic than this.
#### Decoder steering — architectural validation, different failure mode
The `--target-module decoder` extraction (commit `99a7e53`)
captures activations at the depth decoder (Llama100M, 4 layers ×
1024 embed dim) rather than the backbone. The decoder generates
codebooks 1..N-1 (acoustic detail) given a sampled c0 (semantic) +
the backbone hidden state — so the architectural prediction is that
decoder steering should shift voice character without disturbing
word content.
A/B at seed 7 confirms the prediction: backbone steering destroys
text ("some needs of my prey"), decoder steering preserves it
("I'm not that tall"). Vector magnitudes are ~10× smaller at the
decoder (norm 0.85 deepest vs 14.9 backbone), so apparent scales
are higher.
Multi-seed at decoder@1.0:
seed 7: cos 0.86, "I'm not that tall. I'm not that tall." ×3
seed 42: cos 0.65, "Okay." (premature EOT)
seed 100: cos 0.76, "I have to do that. I have to do that." ×4
Same decoder@1.0 + Selective CFG `linear:3.0:1.0:25`:
seed 7: cos 0.67, "I'm going to make a whole lot of the time…"
seed 42: cos 0.52, "We have to come back to the world." *(clean!)*
seed 100: cos -1, "And?" (premature EOT)
Scale sweep at seed 7:
dec@0.3: cos 0.56 "You" (premature EOT)
dec@0.5: cos 0.82 "Don't the bad thing." (no repetition, short)
dec@0.7: cos 0.77 "No, I have been. I want to be in. I thought…"
dec@1.0: cos 0.86 "I'm not that tall." ×3 (best cos, repetitive)
Honest framing of what decoder steering changes vs backbone:
| | backbone | decoder |
|--|----------|---------|
| voice character | ↑↑ | ↑↑ |
| text fidelity | ↓↓ destroys words | ↑ preserves coherent English |
| failure mode | semantic gibberish ("prey", "Lord") | repetition / premature EOT |
| scale window | 0.2-0.3 | 0.5-1.0 |
Both fail differently; neither produces production-quality emotional
speech in single-shot. The N-seed picker (`emotional_speech_n.sh`)
is still the right consumer interface — it doesn't care which
failure mode produced the bad samples, just sorts by WER.
- *RAVDESS corpus shape biases output length*: clips are ~3 s - *RAVDESS corpus shape biases output length*: clips are ~3 s
("Kids are talking by the door") so the steering biases the ("Kids are talking by the door") so the steering biases the
model toward terse outputs (one or two words). Longer-utterance model toward terse outputs (one or two words). Longer-utterance