8-gen bench (4 emotions × 2 corpora) at seed=42 against firdhokk
Whisper-LV3:
target RAVDESS CREMA-D
happy happy (0.999) ✓ happy (0.999) ✓
angry neutral (0.92) sad (0.99)
fearful happy (0.998) fearful (0.984) ✓
sad angry (0.99) fearful (0.99)
CREMA-D 2/4 vs RAVDESS 1/4. Larger / more naturalistic corpus
produces more class-pure fearful direction. Neither corpus solves
angry or sad — recipe shifts into 'vague expressivity' rather than
class-specific corners.
Practical: prefer CREMA-D when available; A/B both per emotion if
class precision matters.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Closes the train→generate→evaluate cycle. Users can now produce a hard
quality number for any adapter without listening manually.
evaluate_held_out runs teacher-forced forward_loss over a JSONL manifest
(same format as 12.3 trainer). Uses apply_emotion_hint so eval prompts
match training prompts. Frame sampling is seed-controlled — identical
seeds across runs score the same frames in the same order, which is what
makes a base-vs-LoRA A/B fair.
EvalRow + EvalSummary types, both serde-Serialize for JSON output.
examples/lora_eval.rs wraps it: --eval-manifest --report [--lora].
Documented usage: run twice with the same seed, diff the summary blocks.
Verified end-to-end on the existing 3-row curriculum manifest with
seed=42, frames-per-example=4: LoRA shifted mean/median/p90 loss
directionally in its favor (-0.011/-0.018/-0.007). Tiny because the
test adapter only saw ~10 training steps, but the eval signal is real
and the A/B path is wired.
Lib suite 99/99.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
