8-gen bench (4 emotions × 2 corpora) at seed=42 against firdhokk
Whisper-LV3:
target RAVDESS CREMA-D
happy happy (0.999) ✓ happy (0.999) ✓
angry neutral (0.92) sad (0.99)
fearful happy (0.998) fearful (0.984) ✓
sad angry (0.99) fearful (0.99)
CREMA-D 2/4 vs RAVDESS 1/4. Larger / more naturalistic corpus
produces more class-pure fearful direction. Neither corpus solves
angry or sad — recipe shifts into 'vague expressivity' rather than
class-specific corners.
Practical: prefer CREMA-D when available; A/B both per emotion if
class precision matters.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Empirical: with 25-clip cadence the cumulative state still tipped over
once at position 8102 mid-window during a 167-clip lora_eval load.
Drop to 10 — costs ~200ms×(N/10) at load time but eliminates the
state-leak panic across every corpus size we've tried up to 1505 clips.
Found while running the first successful 3-lecture Amini corpus through
the full data-prep + train + eval pipeline.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Mimi's transformer carries an internal position counter across encode()
calls that the upstream `reset_state()` does NOT fully clear. When
TrainingDataset::load_from_manifest encodes ~80-100 clips back-to-back
during data prep, the counter overflows the 8192-position buffer and
panics with `narrow invalid args [8192, 32]`.
src/mimi.rs:
- cache the safetensors path on Mimi at construction
- new Mimi.reload() drops the inner Model and rebuilds from the
cached path (~200 ms on Metal)
src/training.rs:
- call generator.mimi.reload() every 50 clips during
load_from_manifest. Adds ~1 s overhead on a 200-clip corpus
(4 reloads × ~200 ms) vs the alternative of a hard panic.
- reset_state() before each encode in TrainingExample::from_audio
is kept (still useful to clear streaming chunk state).
Found while running the end-to-end personal-voice training pipeline on
a 20-minute YouTube source: the bug surfaces around clip 86 when
Mimi's transformer hits position 8181+. Filtering to short clips
alone didn't help — the cumulative state grows even with sub-12-second
inputs.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Closes the train→generate→evaluate cycle. Users can now produce a hard
quality number for any adapter without listening manually.
evaluate_held_out runs teacher-forced forward_loss over a JSONL manifest
(same format as 12.3 trainer). Uses apply_emotion_hint so eval prompts
match training prompts. Frame sampling is seed-controlled — identical
seeds across runs score the same frames in the same order, which is what
makes a base-vs-LoRA A/B fair.
EvalRow + EvalSummary types, both serde-Serialize for JSON output.
examples/lora_eval.rs wraps it: --eval-manifest --report [--lora].
Documented usage: run twice with the same seed, diff the summary blocks.
Verified end-to-end on the existing 3-row curriculum manifest with
seed=42, frames-per-example=4: LoRA shifted mean/median/p90 loss
directionally in its favor (-0.011/-0.018/-0.007). Tiny because the
test adapter only saw ~10 training steps, but the eval signal is real
and the A/B path is wired.
Lib suite 99/99.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Trained adapters now embed (rank, alpha, target_modules, crate_version)
as JSON in safetensors __metadata__["rtx_csm_lora"]. apply_lora_adapter
reads it at load time so users no longer have to remember matching
--lora-rank/--lora-alpha/--extended-lora flags from training.
LoraAdapterMetadata::is_extended() heuristic: target_modules contains
any MLP path or output_proj or k_proj. Handles both the canonical
extended() preset and future custom configs that overlap it.
apply_lora_adapter rank/alpha/extended params became Option<_>
(None = use file metadata, Some = override). Both callers updated.
save_lora_adapter_with_metadata is the new path used by both trainers;
the plain save_lora_adapter still exists for the metadata-less case
(per-stage curriculum snapshots).
safetensors dep bumped 0.4 → 0.7 to match candle 0.9's transitive pin
so candle's Tensor: View impl is in scope for serialize_to_file
(candle's own save wrapper hardcodes the metadata arg to None).
Backward compat: pre-12.5 adapters load fine when explicit CLI flags
are passed; auto-detect path is skipped silently.
Verified end-to-end: trained adapter saved with metadata,
`generate --lora <path>` (no other flags) auto-detected
rank=8 alpha=16 extended=true and applied. Older metadata-less adapter
still loaded with explicit flags. 3 new unit tests; lib suite 99/99.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Closes the train→generate loop the lora_train comment has promised
since Phase 3 ("forthcoming --lora flag on generate").
apply_lora_adapter(generator, path, rank, alpha, extended, device)
shared helper in src/training.rs wraps add_lora_to_backbone +
load_lora_adapter + refresh_lora. Both examples/generate and
examples/converse_server now call it instead of inlining their own
versions, and both now take --extended-lora to opt into Phase 12.1
coverage. Classic q+v adapters still load without the flag.
lora_train.rs now prints the exact `--lora <path> --lora-rank N
--lora-alpha N [--extended-lora]` command-line you need to apply the
trained adapter at inference, replacing the (forthcoming) message.
End-to-end verified: a Phase 12.3 curriculum-trained adapter loaded
into generate with identical seed/text produces different audio
(92KB vs 61KB, EOT @ frame 24 vs 16) — confirming the adapter takes
effect through to the sampled output. The 3-utterance smoke adapter
hasn't learned anything meaningful but the wiring is sound.
Phase 12 emotional voice stack now complete end-to-end:
12.1 capacity → 12.2 control tokens → 12.3 curriculum → 12.4 inference.
Lib suite 96/96.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Closes the personal_voice_training_guide.md §4 stack: capacity (12.1) +
control tokens (12.2) + multi-stage curriculum (this).
TrainingExample gains emotion_tag + stage. Trainer::train applies the
tag via the same apply_emotion_hint helper inference uses (now
pub(crate)) — training and inference must use identical prefix
formatting or the adapter won't transfer.
TrainingDataset::load_from_manifest reads JSONL
`{wav, transcript, emotion_tag?, stage?, speaker?}` rows; wav paths
resolve relative to manifest dir.
CurriculumStage + CurriculumTrainer run N stages sequentially against a
shared VarMap. Per stage: filter by ex.stage label, build a transient
sub-dataset, run Trainer, save snapshot if requested. The "*" stage
name is a global catch-all.
examples/lora_train_emotional.rs wraps the canonical 3-stage recipe:
audiobook (3 ep × lr 1e-4) → podcast (1 ep × lr 3e-5) → va (1 ep ×
lr 1e-5). --extended-lora recommended (FFN is the prosodic-style
carrier per the guide).
Verified end-to-end on Metal: 3-row manifest → all 3 stages execute,
checkpoints + final adapter written, prompt-token lengths varied by
emotion-tag length (9 vs 11 for different tags) confirming the tag
flowed through the training tokenization. Lib suite 96/96.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
