osobhandClaude Opus 4.7
808e79dc40
Adds a clean labeled-emotion data path now that the auto-tagged
manifests have proven inadequate for steering extraction.
scripts/build_ravdess_manifest.sh: parses the RAVDESS speech-only
distribution (Audio_Speech_Actors_01-24.zip from Zenodo, 208 MB) into
a manifest.jsonl with proper {neutral, calm, happy, sad, angry,
fearful, disgust, surprised} labels and the two canonical statements
("Kids are talking by the door", "Dogs are sitting by the door"). 1440
clips, balanced 192/emotion (96 neutral — RAVDESS lacks the 'strong'
intensity for neutral).
examples/steering_extract Mimi reload-every-10: the streaming state
counter overflows 8192 frames after ~80 encodes even with
reset_state(). Same fix training/audio_to_manifest/converse_server use
(commit 0568dd3); now applied here too.
A/B with angry-vs-neutral steering @ 50 samples per pool, matched
against an angry RAVDESS reference clip:
case cos WER transcript
baseline 0.55 0.92 "It is a very important thing to do."
[email protected] 0.61 1.00 "You"
[email protected] 0.74 1.00 "So" ← best speaker_cosine
[email protected] 0.62 1.00 "You"
[email protected] 0.60 1.00 "You"
[email protected] 0.64 6.15 "the Lord, the Lord, the Lord..."
Speaker cosine 0.55 → 0.74 with mid-layer steering at scale 0.5 — a
35 % jump, the largest empirical gain we've measured. The model is
clearly migrating toward the angry actor's voice character. Side
effect: premature EOT (output reduces to one or two words). Likely
because RAVDESS clips themselves are very short ("Kids are talking
by the door", ~3s) so the steering biases toward terse outputs.
That's a corpus-shape artifact, not a code bug — different emotion
corpora with longer utterances should fix it.
Tightest single result of Sprint 2.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
64 lines
2.1 KiB
Bash
Executable File
64 lines
2.1 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Convert a RAVDESS speech-only extraction directory tree into a
|
|
# rtx-csm manifest.jsonl with proper emotion_tag, transcript, and
|
|
# speaker fields (parsed from the filename encoding).
|
|
#
|
|
# RAVDESS download (208 MB, no auth):
|
|
# curl -L -O https://zenodo.org/records/1188976/files/Audio_Speech_Actors_01-24.zip
|
|
# unzip Audio_Speech_Actors_01-24.zip -d /tmp/ravdess
|
|
#
|
|
# Filename convention (e.g. `03-01-04-01-02-01-12.wav`):
|
|
# modality (01=AV, 02=video, 03=audio-only)
|
|
# vocal channel (01=speech, 02=song)
|
|
# emotion (01=neutral, 02=calm, 03=happy, 04=sad, 05=angry, 06=fearful, 07=disgust, 08=surprised)
|
|
# intensity (01=normal, 02=strong; neutral has no strong)
|
|
# statement (01="Kids are talking by the door", 02="Dogs are sitting by the door")
|
|
# repetition (01, 02)
|
|
# actor (01..24; odd = male, even = female)
|
|
#
|
|
# Output schema matches what `examples/steering_extract` expects:
|
|
# {"wav": "<abs path>", "transcript": "...", "emotion_tag": "[neutral]",
|
|
# "speaker": 0|1}
|
|
#
|
|
# Usage:
|
|
# scripts/build_ravdess_manifest.sh /tmp/ravdess > /tmp/ravdess/manifest.jsonl
|
|
|
|
set -euo pipefail
|
|
|
|
ROOT="${1:?usage: $0 <ravdess_root>}"
|
|
if [[ ! -d "$ROOT" ]]; then
|
|
echo "not a directory: $ROOT" >&2
|
|
exit 1
|
|
fi
|
|
|
|
# zsh's `read` doesn't have -a, so spawn bash explicitly.
|
|
bash -c '
|
|
ROOT="$1"
|
|
for wav in "$ROOT"/Actor_*/*.wav; do
|
|
base=$(basename "$wav" .wav)
|
|
IFS="-" read -ra parts <<< "$base"
|
|
emo_code="${parts[2]}"
|
|
stmt_code="${parts[4]}"
|
|
actor="${parts[6]}"
|
|
case "$emo_code" in
|
|
01) e="neutral" ;;
|
|
02) e="calm" ;;
|
|
03) e="happy" ;;
|
|
04) e="sad" ;;
|
|
05) e="angry" ;;
|
|
06) e="fearful" ;;
|
|
07) e="disgust" ;;
|
|
08) e="surprised" ;;
|
|
*) e="unknown" ;;
|
|
esac
|
|
case "$stmt_code" in
|
|
01) t="Kids are talking by the door." ;;
|
|
02) t="Dogs are sitting by the door." ;;
|
|
*) t="" ;;
|
|
esac
|
|
spk=$((10#$actor % 2))
|
|
jq -nc --arg w "$wav" --arg t "$t" --arg e "$e" --argjson s "$spk" \
|
|
"{wav: \$w, transcript: \$t, emotion_tag: (\"[\" + \$e + \"]\"), speaker: \$s}"
|
|
done
|
|
' _ "$ROOT"
|