# emotion2vec_plus_base candle port — design notes

Inspector: `cargo run -p rtx-csm --release --example emotion2vec_inspect -- --key model`

Repo: <https://huggingface.co/emotion2vec/emotion2vec_plus_base> (1.12 GB
`model.pt`, ~93 M actual params; the rest is optimizer state).

## Architecture (from `config.yaml` + 185 inspected tensors)

```
audio 16 kHz (1, T)
 → local_encoder              7 × Conv1d (1→512 chans, ~T/320 stride product)
                              feature_encoder_spec:
                                [(512, 10, 5)] + [(512, 3, 2)]×4 + [(512,2,2)]×2
 → project_features           Linear 512 → 768
 → relative_positional_encoder 5 × Conv1d (768→48, kernel=19)
                              conv_pos_depth=5, conv_pos_width=95,
                              conv_pos_groups=16
 → context_encoder            4-layer transformer (prenet_depth=4)
 → blocks.0..7                8-layer main transformer (depth=8)
                              embed_dim=768, num_heads=12, mlp_ratio=4.0
                              fused QKV (qkv.weight: 2304×768)
 → norm                       LayerNorm (768)
 → mean-pool over time
 → proj                       Linear 768 → 9
 → softmax
```

Block structure (matches both `context_encoder.blocks.*` and `blocks.*`):
- `norm1` (LayerNorm 768)
- `attn.qkv` (Linear 768→2304, fused QKV)
- `attn.proj` (Linear 768→768)
- `norm2` (LayerNorm 768)
- `mlp.fc1` (Linear 768→3072)
- `mlp.fc2` (Linear 3072→768)

## 9 output classes

From `tokens.txt` (Chinese/English bilingual labels):
0=angry 1=disgusted 2=fearful 3=happy 4=neutral 5=other 6=sad 7=surprised
8=`<unk>`

## ALiBi vs RoPE

Config says `use_alibi_encoder: true`, `num_alibi_heads: 12`,
`learned_alibi_scale_per_head: true`. The "relative_positional_encoder"
above is a Conv1d-based positional bias (similar to wav2vec2's conv pos
embedding), not learned ALiBi slopes. Need to inspect the
`learned_alibi_scale` parameter values when porting (likely under a
different prefix not yet surfaced).

## Slicing plan (multi-session)

- ✅ **Slice 1 (Phase 13.8, this commit)**: inspector + design notes
- ⏳ **Slice 2 (~half-day)**: candle module scaffolding —
  `LocalEncoder` (Conv1d stack) + `Block` + `ContextEncoder` +
  `MainEncoder` + `Classifier`, with `from_pickle` loaders mapping the
  dotted keys to candle Linear/Conv1d/LayerNorm. No forward yet.
- ⏳ **Slice 3 (~half-day)**: forward pass on synthetic 16 kHz audio,
  verify shapes match through every stage, smoke test against a known
  emotion clip.
- ⏳ **Slice 4 (~hour)**: implement `EmotionDetector` for the new model
  and swap into `audio_to_manifest --auto-emotion-tag` and
  `converse_server --reactive-emotion`. Phase 13.3 prosody-rule
  placeholder retired.

## Architectural overlap with existing in-crate ports

- `wavlm_sv.rs` (Phase 5d) — shares the conv feature extractor pattern,
  but uses different positional encoding (relative buckets instead of
  ALiBi/conv). Block structure differs (gated relative-pos attn vs
  fused QKV here).
- The fused QKV here is closer to vanilla ViT than to the in-crate
  WavLM blocks, so the port is more from-scratch than a refactor.