rtx-csm: perf_history — decoder steering empirical characterization
3-seed × 2-condition ([email protected] alone vs +CFG) bench plus a 4-step scale sweep. Captures the honest tradeoff: - Backbone steering destroys word content (semantic gibberish). - Decoder steering preserves coherent English BUT produces repetition or premature EOT. Neither produces single-shot production-quality emotional speech; emotional_speech_n.sh (N-seed picker, lowest-WER wins) remains the right consumer interface — it doesn't care which failure mode generated the bad samples, just discards them by metric. Decoder vector magnitudes are ~10× smaller than backbone (norm 0.85 at deepest layer vs 14.9), so the apparent useful scale window is ~10× higher (0.5-1.0 instead of 0.2-0.3). Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
This commit is contained in:
@@ -160,6 +160,55 @@ shipped on the FP backbone; quantized path unchanged.
|
|||||||
"biased random sampling toward emotional voice" than to
|
"biased random sampling toward emotional voice" than to
|
||||||
"deterministic emotional rendering." Reproducible voice cloning
|
"deterministic emotional rendering." Reproducible voice cloning
|
||||||
(`--context-wav` alone) is much more deterministic than this.
|
(`--context-wav` alone) is much more deterministic than this.
|
||||||
|
|
||||||
|
#### Decoder steering — architectural validation, different failure mode
|
||||||
|
|
||||||
|
The `--target-module decoder` extraction (commit `99a7e53`)
|
||||||
|
captures activations at the depth decoder (Llama100M, 4 layers ×
|
||||||
|
1024 embed dim) rather than the backbone. The decoder generates
|
||||||
|
codebooks 1..N-1 (acoustic detail) given a sampled c0 (semantic) +
|
||||||
|
the backbone hidden state — so the architectural prediction is that
|
||||||
|
decoder steering should shift voice character without disturbing
|
||||||
|
word content.
|
||||||
|
|
||||||
|
A/B at seed 7 confirms the prediction: backbone steering destroys
|
||||||
|
text ("some needs of my prey"), decoder steering preserves it
|
||||||
|
("I'm not that tall"). Vector magnitudes are ~10× smaller at the
|
||||||
|
decoder (norm 0.85 deepest vs 14.9 backbone), so apparent scales
|
||||||
|
are higher.
|
||||||
|
|
||||||
|
Multi-seed at decoder@1.0:
|
||||||
|
|
||||||
|
seed 7: cos 0.86, "I'm not that tall. I'm not that tall." ×3
|
||||||
|
seed 42: cos 0.65, "Okay." (premature EOT)
|
||||||
|
seed 100: cos 0.76, "I have to do that. I have to do that." ×4
|
||||||
|
|
||||||
|
Same decoder@1.0 + Selective CFG `linear:3.0:1.0:25`:
|
||||||
|
|
||||||
|
seed 7: cos 0.67, "I'm going to make a whole lot of the time…"
|
||||||
|
seed 42: cos 0.52, "We have to come back to the world." *(clean!)*
|
||||||
|
seed 100: cos -1, "And?" (premature EOT)
|
||||||
|
|
||||||
|
Scale sweep at seed 7:
|
||||||
|
|
||||||
|
dec@0.3: cos 0.56 "You" (premature EOT)
|
||||||
|
dec@0.5: cos 0.82 "Don't the bad thing." (no repetition, short)
|
||||||
|
dec@0.7: cos 0.77 "No, I have been. I want to be in. I thought…"
|
||||||
|
dec@1.0: cos 0.86 "I'm not that tall." ×3 (best cos, repetitive)
|
||||||
|
|
||||||
|
Honest framing of what decoder steering changes vs backbone:
|
||||||
|
|
||||||
|
| | backbone | decoder |
|
||||||
|
|--|----------|---------|
|
||||||
|
| voice character | ↑↑ | ↑↑ |
|
||||||
|
| text fidelity | ↓↓ destroys words | ↑ preserves coherent English |
|
||||||
|
| failure mode | semantic gibberish ("prey", "Lord") | repetition / premature EOT |
|
||||||
|
| scale window | 0.2-0.3 | 0.5-1.0 |
|
||||||
|
|
||||||
|
Both fail differently; neither produces production-quality emotional
|
||||||
|
speech in single-shot. The N-seed picker (`emotional_speech_n.sh`)
|
||||||
|
is still the right consumer interface — it doesn't care which
|
||||||
|
failure mode produced the bad samples, just sorts by WER.
|
||||||
- *RAVDESS corpus shape biases output length*: clips are ~3 s
|
- *RAVDESS corpus shape biases output length*: clips are ~3 s
|
||||||
("Kids are talking by the door") so the steering biases the
|
("Kids are talking by the door") so the steering biases the
|
||||||
model toward terse outputs (one or two words). Longer-utterance
|
model toward terse outputs (one or two words). Longer-utterance
|
||||||
|
|||||||
Reference in New Issue
Block a user