Files

T

osobhandClaude Opus 4.7 a77b22d47c rtx-csm: Phase 13.8 — emotion2vec_plus_base port, slice 1 (inspector)

First slice of the multi-session port that will replace the Phase 13.3
prosody-rule SER placeholder with a real classifier via the existing
EmotionDetector trait.

examples/emotion2vec_inspect.rs:
  - downloads model.pt + config.yaml + tokens.txt from
    emotion2vec/emotion2vec_plus_base on HF Hub
  - descends fairseq-style nested checkpoint via --key model
  - dumps all 185 tensors with shapes/dtypes + per-prefix summary
  - uses pickle::read_pth_tensor_info, same pattern as audioseal_inspect

Architecture confirmed (full notes in docs/emotion2vec_port_notes.md):
  - 93 M params, F32 (the 1.12 GB file is mostly optimizer state)
  - local_encoder: 7 Conv1d layers (wav2vec2 feature extractor:
    [(512,10,5)] + [(512,3,2)]×4 + [(512,2,2)]×2, T → T/320)
  - project_features: Linear 512 → 768
  - relative_positional_encoder: 5 Conv1d layers (kernel 19)
  - context_encoder: 4-layer transformer prenet (prenet_depth=4)
  - blocks.0..7: 8-layer main transformer (depth=8, embed_dim=768,
    12 heads, mlp_ratio=4, fused QKV qkv.weight=[2304, 768])
  - proj: Linear 768 → 9 (angry/disgusted/fearful/happy/neutral/other/
    sad/surprised/<unk>)

Slicing plan (remaining):
  Slice 2 (~half-day): candle module scaffolding + from_pickle loaders
  Slice 3 (~half-day): forward pass + shape verification
  Slice 4 (~hour): EmotionDetector impl + swap into audio_to_manifest
                  and converse_server

Lib suite 110/110.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

2026-04-27 23:14:06 -07:00

3.1 KiB

Raw Blame History

emotion2vec_plus_base candle port — design notes

Inspector: cargo run -p rtx-csm --release --example emotion2vec_inspect -- --key model

Repo: https://huggingface.co/emotion2vec/emotion2vec_plus_base (1.12 GB model.pt, ~93 M actual params; the rest is optimizer state).

Architecture (from `config.yaml` + 185 inspected tensors)

audio 16 kHz (1, T)
 → local_encoder              7 × Conv1d (1→512 chans, ~T/320 stride product)
                              feature_encoder_spec:
                                [(512, 10, 5)] + [(512, 3, 2)]×4 + [(512,2,2)]×2
 → project_features           Linear 512 → 768
 → relative_positional_encoder 5 × Conv1d (768→48, kernel=19)
                              conv_pos_depth=5, conv_pos_width=95,
                              conv_pos_groups=16
 → context_encoder            4-layer transformer (prenet_depth=4)
 → blocks.0..7                8-layer main transformer (depth=8)
                              embed_dim=768, num_heads=12, mlp_ratio=4.0
                              fused QKV (qkv.weight: 2304×768)
 → norm                       LayerNorm (768)
 → mean-pool over time
 → proj                       Linear 768 → 9
 → softmax

Block structure (matches both context_encoder.blocks.* and blocks.*):

norm1 (LayerNorm 768)
attn.qkv (Linear 768→2304, fused QKV)
attn.proj (Linear 768→768)
norm2 (LayerNorm 768)
mlp.fc1 (Linear 768→3072)
mlp.fc2 (Linear 3072→768)

9 output classes

From tokens.txt (Chinese/English bilingual labels): 0=angry 1=disgusted 2=fearful 3=happy 4=neutral 5=other 6=sad 7=surprised 8=<unk>

ALiBi vs RoPE

Config says use_alibi_encoder: true, num_alibi_heads: 12, learned_alibi_scale_per_head: true. The "relative_positional_encoder" above is a Conv1d-based positional bias (similar to wav2vec2's conv pos embedding), not learned ALiBi slopes. Need to inspect the learned_alibi_scale parameter values when porting (likely under a different prefix not yet surfaced).

Slicing plan (multi-session)

✅ Slice 1 (Phase 13.8, this commit): inspector + design notes
⏳ Slice 2 (~half-day): candle module scaffolding — LocalEncoder (Conv1d stack) + Block + ContextEncoder + MainEncoder + Classifier, with from_pickle loaders mapping the dotted keys to candle Linear/Conv1d/LayerNorm. No forward yet.
⏳ Slice 3 (~half-day): forward pass on synthetic 16 kHz audio, verify shapes match through every stage, smoke test against a known emotion clip.
⏳ Slice 4 (~hour): implement EmotionDetector for the new model and swap into audio_to_manifest --auto-emotion-tag and converse_server --reactive-emotion. Phase 13.3 prosody-rule placeholder retired.

Architectural overlap with existing in-crate ports

wavlm_sv.rs (Phase 5d) — shares the conv feature extractor pattern, but uses different positional encoding (relative buckets instead of ALiBi/conv). Block structure differs (gated relative-pos attn vs fused QKV here).
The fused QKV here is closer to vanilla ViT than to the in-crate WavLM blocks, so the port is more from-scratch than a refactor.

3.1 KiB Raw Blame History Unescape Escape