8-gen bench (4 emotions × 2 corpora) at seed=42 against firdhokk
Whisper-LV3:
target RAVDESS CREMA-D
happy happy (0.999) ✓ happy (0.999) ✓
angry neutral (0.92) sad (0.99)
fearful happy (0.998) fearful (0.984) ✓
sad angry (0.99) fearful (0.99)
CREMA-D 2/4 vs RAVDESS 1/4. Larger / more naturalistic corpus
produces more class-pure fearful direction. Neither corpus solves
angry or sad — recipe shifts into 'vague expressivity' rather than
class-specific corners.
Practical: prefer CREMA-D when available; A/B both per emotion if
class precision matters.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Full encoder-decoder Moonshine v2 working end-to-end on candle 0.9 +
Metal. Loads HF safetensors, runs through every transformer block, and
produces real logits.
Components added to src/moonshine.rs:
CrossAttention MHA with K/V from encoder output (no causal mask)
DecoderSelfAttention MHA with causal mask, partial RoPE on q/k
DecoderMlp SwiGLU: fused fc1 [2304, 288] split gate+up,
silu(gate) * up, fc2 [288, 1152] back to hidden
DecoderLayer Pre-LN self-attn + Pre-LN cross-attn + Pre-LN MLP
Decoder token embed -> 6 layers -> final LN -> tied LM head
load_full() convenience: returns (Encoder, Decoder)
Smoke test verifies end-to-end:
encoder forward : 1 ms (cached after warm-up)
decoder forward : 85 ms (1 token, prefill mode)
logits shape : (1, 1, 32768)
logit max abs : 30.66 (real signal, not zeros)
argmax token_id : 379 (non-trivial prediction; eos=2)
Implementation notes:
- Same (B*H, T, D) 3D matmul pattern as encoder to dodge candle's 4D
Metal matmul shape-mismatch bug.
- LM head tied to decoder.embed_tokens.weight (cached on Decoder for
fast forward; logits = hidden @ embed.T).
- Causal mask is a (T, T) -inf upper-triangular added to scores
before softmax.
- Decoder final LN tensor is `decoder.norm.weight` (NOT
`decoder.layer_norm.weight` — encoder uses the latter naming).
- No KV cache yet: this is prefill mode. Phase 8.8 will add the
streaming-generation loop with cache + tokenizer.
NOT yet verified: numerical parity vs HF Python reference. The token
predicted (id=379) looks plausible for silent-mostly audio, but a
parity check is still needed (Phase 8.9). Architecture appears
correct based on shape + signal sanity.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Full encoder forward path: conv stem -> 6 transformer layers -> final
LayerNorm. Loads HF safetensors, runs end-to-end on Metal.
Components added to src/moonshine.rs:
RotaryCache partial RoPE (32 of 36 head_dim, theta=10000)
EncoderAttention MHA (8 heads, no bias), partial RoPE on q/k
EncoderMlp 288 -> 1152 -> 288 with bias, GELU(erf) activation
EncoderLayer Pre-LN attn + Pre-LN MLP (LayerNorm weight-only)
Encoder stem + 6 layers + final LayerNorm
load_encoder() VarBuilder convenience for the standalone smoke
Smoke test (`examples/moonshine_smoke`) verified end-to-end:
input (1, 1, 160000) -> output (1, 415, 288)
forward: 132 ms (10 s of audio at 0.013x realtime)
max abs: 6.67 (signal preserved, not zeros)
Implementation notes captured in the diff:
- candle Metal 4D batched matmul had shape-mismatch issues for our
(B, H, T, D) pattern. Switched to (B*H, T, D) 3D form which is
unambiguous and avoids the kernel bug.
- LayerNorm is weight-only (no bias tensors in safetensors); we
construct LayerNorm with a zeros bias to satisfy candle's API.
- rotary_dim = floor(head_dim * 0.9 / 2) * 2 = 32 (must be even).
The remaining 4 head_dim channels pass through unchanged via
`narrow + cat` on dim 3.
Numerical parity vs HF Python reference is NOT yet verified — that's
the next bounded chunk (Phase 8.7). Shape + signal correctness are
verified by the smoke test.
Next: decoder transformer block (self-attn + cross-attn + SwiGLU).
~3-4 h of focused work.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
