rustyverse/rustytorch
2
0
Fork 0
Code Issues Pull Requests Actions 28 Packages Projects Releases Wiki Activity
431 Commits 1 Branch 0 Tags
0dc95a8de5eb08a44407d7f4de07979cd11d9b18
Commit Graph
6 Commits
Author SHA1 Message Date
osobhandClaude Opus 4.7 a5cedfb46a rtx-csm: emotional_speech_guide — CREMA-D vs RAVDESS firdhokk verdict 
8-gen bench (4 emotions × 2 corpora) at seed=42 against firdhokk
Whisper-LV3:

  target    RAVDESS              CREMA-D
  happy     happy (0.999) ✓      happy (0.999) ✓
  angry     neutral (0.92)       sad (0.99)
  fearful   happy (0.998)        fearful (0.984) ✓
  sad       angry (0.99)         fearful (0.99)

CREMA-D 2/4 vs RAVDESS 1/4. Larger / more naturalistic corpus
produces more class-pure fearful direction. Neither corpus solves
angry or sad — recipe shifts into 'vague expressivity' rather than
class-specific corners.

Practical: prefer CREMA-D when available; A/B both per emotion if
class precision matters.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
 2026-04-30 00:01:02 -07:00
Omar Sobh 6a03aeba61 fix(cuda+csm): P0 rtx-backend-cuda compile fix + P2 rtx-csm clippy cleanup (#9) 
Co-authored-by: Omar Sobh <[email protected]>
Co-committed-by: Omar Sobh <[email protected]>
 2026-04-30 05:24:58 +00:00
osobhandClaude Opus 4.7 e61bd70e03 rtx-csm: WavLM-SV bug fixes — gelu_erf + gru_rel_pos_const loading 
Two real bugs found via code inspection against HF source:

1. candle's .gelu() is the tanh approximation; PyTorch's default 'gelu'
   activation (used in WavLM via ACT2FN['gelu']) is the exact erf-based
   version. Switched all 3 sites (feature extractor convs, pos_conv,
   FFN) from .gelu() to .gelu_erf() to match the reference.

2. gru_rel_pos_const lookup used vb.pp("name").get(shape, "") which
   resolves to "<prefix>.name." (trailing dot) and fails to find the
   tensor. The .or_else(|_| zeros) silently swallowed the failure,
   leaving all 12 layers' gating constants at zero instead of the
   trained values. Fixed to attn.get(shape, "gru_rel_pos_const") which
   resolves correctly.

examples/wavlm_sv_inspect.rs: utility for sanity-checking specific
tensors inside converted safetensors (e.g. layer_weights).

Same-content same-speaker cosine: 0.9985 -> 0.9963 (≈unchanged).
Cross-content same-speaker cosine: 0.4882 -> 0.4118 (still drifting).
Phase 5d (Python reference comparison) remains the gate for
identifying the residual numerical drift.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
 2026-04-25 22:00:59 -07:00
osobhandClaude Opus 4.7 d1dba7a05c rtx-csm: WavLM-SV converter + end-to-end speaker similarity 
Phase 5c — pure-Rust converter for microsoft/wavlm-base-plus-sv with
auto-detecting weight_norm merger; verified on real 100M-param weights:
load + embed + cosine-similarity round-trip works on Metal.

- wavlm_sv_convert.rs: candle_core::pickle reads pytorch_model.bin
  directly. merge_weight_norm_auto picks the kept dim from g's shape
  (dim=0 for AudioSeal SEANet, dim=2 for WavLM pos_conv_embed). Skips
  classifier.*/objective.* (train-only AMSoftmax head).
- examples/wavlm_sv_convert: HF download + convert CLI. Verified output:
  1 weight_norm pair merged + 261 passthrough + 3 skipped = 262 tensors.
- examples/wavlm_sv_demo: load + embed pair of WAVs + cosine similarity.
- examples/audioseal_inspect: gains --which wavlm-sv variant for key
  discovery.
- hub.rs: REPO_WAVLM_SV + resolve_wavlm_sv() helper.
- wavlm_sv::XVectorHead bug fix: layer_weights is top-level, not nested
  under prefix.

Verified end-to-end on Metal: cosine sim 0.9985 on same-speaker pair
(CSM vs CSM-watermarked, 10s @ 24 kHz resampled to 16 kHz). Numerical
parity vs HF reference is Phase 5d.

3 converter tests + 12 wavlm_sv tests; 78 lib tests total green.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
 2026-04-25 19:57:52 -07:00
osobhandClaude Opus 4.7 3165b9339b rtx-csm: WavLM transformer encoder with gated rel-pos attention 
Phase 5b — full implementation of WavLMEncoderLayer.forward, replacing
the Phase 5a stub. With random weights the encoder is no longer a no-op
(verified: L2(output - input) > 1e-6 in unit test).

- relative_position_bucket helper: T5-style bidirectional bucketing
  matching HF _relative_positions_bucket. 320 buckets, 800 max distance,
  half/half split with linear inner / log-spaced outer.
- WavLmEncoderLayer::compute_position_bias (layer 0 only): builds (T,T)
  bucket index, embedding-looks up rel_attn_embed, permutes to
  (num_heads, T, T) matching HF's compute_bias output.
- WavLmEncoderLayer::gated_position_bias: HF gating math verbatim —
  Linear(head_dim → 8), reshape (..., 2, 4) sum, sigmoid, chunk to
  gate_a/gate_b, compute gate_a * (gate_b * gru_rel_pos_const - 1) + 2,
  broadcast-multiply position_bias.
- WavLmEncoderLayer::attention: multi-head self-attention with the
  gated bias added to scores before softmax. Standard 1/sqrt(d) scale.
- WavLmEncoderLayer::forward_with_bias returns (output, position_bias)
  so Encoder::forward_all_layers can thread bias from layer 0 through
  layers 1-11 (HF's has_relative_position_bias=(i==0) pattern).
- 3 new tests bring wavlm_sv to 12 tests; 75 lib tests total green.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
 2026-04-25 19:48:56 -07:00
osobhandClaude Opus 4.7 90e53b3c0c rtx-csm: WavLM-Base+ SV scaffold for speaker similarity 
Phase 5a — architectural skeleton for the microsoft/wavlm-base-plus-sv
reference, drop-in replacement for the SpectralCentroidSimilarity weak
baseline in speaker_sim.rs.

Modules in src/wavlm_sv.rs (~600 LOC):
- FeatureExtractor: 7-layer Conv1d, 320× downsample, GroupNorm at layer
  0 (num_groups=num_channels=512), GELU activations.
- FeatureProjection: LayerNorm + Linear 512→768.
- PosConv: Conv1d(768, 768, k=128, groups=16, pad=64) + GELU; SamePad
  strips trailing frame for even kernel.
- WavLmEncoderLayer: struct shape complete (Q/K/V/out projections, pre-
  attention LN, FFN intermediate/output, final LN, gru_rel_pos_const +
  gru_rel_pos_linear, optional rel_attn_embed at layer 0). forward() is
  a STUB; Phase 5b implements gated rel-pos attention.
- Encoder: 12 stacked layers, returns Vec<Tensor> of 13 hidden states.
- Tdnn: dilated unfold + Linear(in*kernel, out) — matches HF impl.
- XVectorHead: softmax-weighted layer sum + projector 768→512 + 5 TDNN
  layers (kernels [5,3,3,1,1] dilations [1,2,3,1,1]) + statistics pool
  + 3000→512 embedding projection.
- WavLmSv top-level + zero-mean unit-variance normalize + cosine
  similarity helper for verification scoring.

9 shape-correctness tests; 72 lib tests total green.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
 2026-04-25 19:42:59 -07:00