8-gen bench (4 emotions × 2 corpora) at seed=42 against firdhokk
Whisper-LV3:
target RAVDESS CREMA-D
happy happy (0.999) ✓ happy (0.999) ✓
angry neutral (0.92) sad (0.99)
fearful happy (0.998) fearful (0.984) ✓
sad angry (0.99) fearful (0.99)
CREMA-D 2/4 vs RAVDESS 1/4. Larger / more naturalistic corpus
produces more class-pure fearful direction. Neither corpus solves
angry or sad — recipe shifts into 'vague expressivity' rather than
class-specific corners.
Practical: prefer CREMA-D when available; A/B both per emotion if
class precision matters.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Word-level forced alignment shipped. Phase 13.9 complete.
viterbi_align(log_probs, tokens, blank_id, vocab) — standard CTC
forced-alignment trellis: states alternate [blank, t0, blank, t1, ...,
tN, blank] (length 2N+1), at each frame stay/advance/ε-skip-blank-
between-different-tokens (canonical CTC ε-skip rule correctly forbids
skipping blank between SAME tokens), max-likelihood path recovered via
backptr table.
transcript_to_token_ids — text → CTC token ids; runs of spaces collapse
to | separator; unknown chars → <unk>.
group_into_words — fold adjacent non-| AlignedToken into AlignedWord
with carried frame_start/frame_end.
frame_to_ms — 50 Hz frame grid → ms (20 ms/frame at conv stride 320).
examples/wav2vec2_smoke --align <target> wires it end-to-end:
forced-aligns a known transcript and prints (word, start_ms, end_ms).
Verified on Metal: 10.42 s LibriSpeech audio, first 8 words →
HE 560-640 HOPED 720-960 THERE 1000-1140
WOULD 1180-1320 BE 1360-2240 STEW 2980-4720
FOR 5300-6000 DINNER 7040-8540
viterbi alignment in 0 ms (39 tokens). Boundaries match audio.
4 new unit tests:
- transcript_to_token_ids_handles_spaces_and_unknowns
- viterbi_align_recovers_obvious_alignment
- group_into_words_splits_on_separator
- frame_to_ms_50hz_grid
Lib suite 131/131 (was 127, +4).
Phase 13.9 complete (slices 1+2+3+4). Crate now ships full English
ASR + word-level forced alignment in pure candle — no whisper.cpp,
no ort, no Python. Data-prep can cut long audio at exact word
boundaries before feeding into the Phase 12.3 curriculum trainer.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Full port of facebook/wav2vec2-base-960h (94.4 M params, MIT) closing
the WhisperX-class word-alignment gap from the audio-ML survey. Same
staged-scaffolding pattern that worked for emotion2vec — but landed
slices 1+2+3 in one session.
src/wav2vec2.rs ships:
- Wav2Vec2Config::base_960h
- FeatureExtractor — 7 Conv1d (1→512, total stride 320). Layer 0
uses GroupNorm with num_groups=num_channels=512 (HF's wav2vec2
feat_extract_norm: "group"). Critical: state-dict key is
layer_norm.* but the OP is GroupNorm — loading as LayerNorm
produces empty CTC output.
- FeatureProjection — LayerNorm(512) + Linear(512→768)
- ConvPosEmbedding — kernel 128 grouped Conv1d, materialized at
load time from upstream weight_g + weight_v (fairseq's weight_norm
on dim=2; eps-guarded division for numerical stability)
- Block — POST-norm transformer with separate Q/K/V (vs emotion2vec's
fused QKV), uses (B*H, T, D) Metal 3D-matmul workaround from
Phase 8.8 Moonshine
- Encoder — pos_conv + initial LayerNorm + 12 Blocks
- Wav2Vec2 top-level — load_from_safetensors via mmap'd VarBuilder
- ctc_greedy_decode + VOCAB_960H constant for the 32-char alphabet
examples/wav2vec2_inspect.rs (slice 1): dumps tensor layout + config
examples/wav2vec2_smoke.rs (slice 3): real-weight load + ASR forward
Verified on Metal:
loaded model in 0.28 s
forward in 9 ms for 10.42 s audio (~1150× realtime)
transcript: "HE HOPED THERE WOULD BE STEW FOR DINNER TURNIPS AND
CARROTS AND BRUISED POTATOES AND FAT MUTTON PIECES TO
BE LADLED OUT IN THICK PEPPERED FLOWER FAT AND SAUCE"
Numerical parity with upstream Python — the FLOWER-for-FLOUR typo is
the known wav2vec2-base-960h failure mode, matches HF reference exactly.
7 new unit tests; lib suite 127/127 (was 120).
Slice 4 remaining: Viterbi forced alignment given known transcript,
to emit (token, frame_start_ms, frame_end_ms) for word-boundary cuts.
The ASR path itself is now production-ready.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
