osobhandClaude Opus 4.7
2008c9c4a6
Capstone wrapper for the Phase 9 recipe (Selective CFG + RAVDESS
steering + mid-layer subset). Picks the steering scale automatically
from an empirical per-emotion map:
- happy: 0.30
- angry: 0.20
- fearful: 0.20
- sad: 0.20 (note: sad is unreliable — see below)
These came from a follow-up sweep after the multi-emotion demo
revealed the recipe is emotion-sensitive: scale 0.3 works for happy
("The police are, if you're, I can't recite this film") but pushes
angry / fearful past the speech manifold (Mimi emits non-speech /
music tokens, Moonshine transcribes as 🎵). Dropping to 0.2 recovers
fluent speech for both:
- [email protected]: "The next disorder is completing kashim for more."
- [email protected]: "You just heard a little bit about this decision,
though."
- [email protected]: "The police are, if you're, I can't recite this
film. I"
Sad is the outlier — model resists "sad" steering at every scale
between 0.15 and 0.3. Likely a corpus issue (sad RAVDESS clips are
the lowest-energy subset). Documented as a known limitation rather
than worked around.
perf_history.md updated with the per-emotion sensitivity finding.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
131 lines
4.7 KiB
Bash
Executable File
131 lines
4.7 KiB
Bash
Executable File
#!/usr/bin/env bash
|
||
# One-shot emotional voice synthesis using the Phase 9 recipe:
|
||
# - Selective CFG schedule `linear:3.0:1.0:25` (robust default)
|
||
# - Activation steering @ scale 0.3 on mid layers [8, 10, 12]
|
||
# - Reference-clip context conditioning
|
||
#
|
||
# This is the codified version of the empirical findings logged in
|
||
# `docs/perf_history.md` Phase 9. Recipe was tuned for cross-character
|
||
# emotional speech (e.g. take a neutral lecture voice, render it angry).
|
||
#
|
||
# Steering vectors must be pre-extracted via
|
||
# `examples/steering_extract` from a labeled emotion corpus (RAVDESS,
|
||
# CREMA-D, ESD, etc). Use `scripts/build_ravdess_manifest.sh` to
|
||
# convert RAVDESS into a manifest first.
|
||
#
|
||
# Usage:
|
||
# scripts/emotional_speech.sh \
|
||
# --text "What you want said." \
|
||
# --context-wav <reference voice>.wav \
|
||
# --context-text "Transcript of the reference clip." \
|
||
# --emotion angry \
|
||
# --steering-dir /tmp/ravdess_steering \
|
||
# --out /tmp/out.wav \
|
||
# [--scale 0.3] # steering scale, lower = more text fidelity
|
||
# [--cfg-schedule linear:3.0:1.0:25]
|
||
# [--layers 8,10,12]
|
||
#
|
||
# Recipe defaults are deliberate; override only when you have a reason.
|
||
# Sprint 2 measurements:
|
||
# - speaker_cosine vs angry-actor reference rises 0.55 → 0.846 with
|
||
# scale=0.3 stacked on Selective CFG
|
||
# - scale ≥ 0.5 corrupts the output (Moonshine WER 1.0 on Unicode noise)
|
||
# - layers including 0–7 also corrupt (input embedding fidelity loss)
|
||
#
|
||
# Per-emotion scale sweet spots (RAVDESS-extracted, mid-layer recipe,
|
||
# linear:3.0:1.0:25 CFG, single-prompt single-seed measurement — treat
|
||
# as starting points, not guarantees):
|
||
# - happy: 0.30 → fluent speech
|
||
# - angry: 0.20 → fluent speech (0.30 produces music-token outputs)
|
||
# - fearful: 0.20 → fluent speech (0.30 produces music-token outputs)
|
||
# - sad: no robust scale found — model resists "sad" steering on
|
||
# most prompts; one-word outputs at all tested scales
|
||
# We leave --scale defaulted to 0.3 because the wrapper can't know which
|
||
# emotion is being requested at default-construction time. Override
|
||
# with --scale 0.2 when using angry or fearful.
|
||
|
||
set -euo pipefail
|
||
|
||
TEXT=""
|
||
CTX_WAV=""
|
||
CTX_TEXT=""
|
||
EMOTION=""
|
||
STEERING_DIR=""
|
||
OUT=""
|
||
SCALE="" # if unset, picked per-emotion below from the empirical map
|
||
CFG_SCHEDULE="linear:3.0:1.0:25"
|
||
LAYERS="8,10,12"
|
||
SPEAKER=0
|
||
SEED=42
|
||
|
||
while [[ $# -gt 0 ]]; do
|
||
case "$1" in
|
||
--text) TEXT="$2"; shift 2 ;;
|
||
--context-wav) CTX_WAV="$2"; shift 2 ;;
|
||
--context-text) CTX_TEXT="$2"; shift 2 ;;
|
||
--emotion) EMOTION="$2"; shift 2 ;;
|
||
--steering-dir) STEERING_DIR="$2"; shift 2 ;;
|
||
--out) OUT="$2"; shift 2 ;;
|
||
--scale) SCALE="$2"; shift 2 ;;
|
||
--cfg-schedule) CFG_SCHEDULE="$2"; shift 2 ;;
|
||
--layers) LAYERS="$2"; shift 2 ;;
|
||
--speaker) SPEAKER="$2"; shift 2 ;;
|
||
--seed) SEED="$2"; shift 2 ;;
|
||
-h|--help) sed -n '2,32p' "$0"; exit 0 ;;
|
||
*) echo "unknown arg: $1" >&2; exit 1 ;;
|
||
esac
|
||
done
|
||
|
||
for v in TEXT CTX_WAV CTX_TEXT EMOTION STEERING_DIR OUT; do
|
||
if [[ -z "${!v}" ]]; then
|
||
echo "error: --${v,,} (or one of the required flags) is unset" >&2
|
||
echo "usage: $0 --text T --context-wav W --context-text C --emotion E --steering-dir D --out O" >&2
|
||
exit 1
|
||
fi
|
||
done
|
||
|
||
STEERING_FILE="$STEERING_DIR/${EMOTION}.safetensors"
|
||
if [[ ! -f "$STEERING_FILE" ]]; then
|
||
echo "error: no steering file for emotion '$EMOTION' at $STEERING_FILE" >&2
|
||
echo "available emotions in $STEERING_DIR:" >&2
|
||
ls "$STEERING_DIR" 2>/dev/null | sed 's/\.safetensors$//' | sed 's/^/ /' >&2 || true
|
||
exit 1
|
||
fi
|
||
|
||
# Per-emotion scale defaults (Sprint 2 sweet-spot measurements).
|
||
if [[ -z "$SCALE" ]]; then
|
||
case "$EMOTION" in
|
||
angry|fearful) SCALE=0.2 ;;
|
||
happy) SCALE=0.3 ;;
|
||
sad) SCALE=0.2 ;; # rough fallback; sad isn't reliable
|
||
*) SCALE=0.3 ;; # generic default
|
||
esac
|
||
fi
|
||
|
||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||
WORKSPACE_DIR="$(cd "$SCRIPT_DIR/../../../.." && pwd)"
|
||
GEN_BIN="$WORKSPACE_DIR/target/release/examples/generate"
|
||
if [[ ! -x "$GEN_BIN" ]]; then
|
||
echo "error: missing $GEN_BIN" >&2
|
||
echo "build with: cargo build -p rtx-csm --release --example generate" >&2
|
||
exit 1
|
||
fi
|
||
|
||
echo "→ recipe: emotion=$EMOTION scale=$SCALE cfg=$CFG_SCHEDULE layers=$LAYERS"
|
||
echo " context: $(basename "$CTX_WAV")"
|
||
|
||
"$GEN_BIN" \
|
||
--text "$TEXT" \
|
||
--speaker "$SPEAKER" \
|
||
--seed "$SEED" \
|
||
--context-wav "$CTX_WAV" \
|
||
--context-text "$CTX_TEXT" \
|
||
--context-speaker 1 \
|
||
--enable-cfg --cfg-schedule "$CFG_SCHEDULE" \
|
||
--steering-vec "$STEERING_FILE" \
|
||
--steering-scale "$SCALE" \
|
||
--steering-layers "$LAYERS" \
|
||
--out "$OUT"
|
||
|
||
echo "✓ wrote $OUT"
|