osobhandClaude Opus 4.7
9c1894ec73
Slice 2 complete + slice 3 collapsed in. The full candle port loads
real upstream weights and runs forward in 160 ms.
RelativePositionalEncoder — 5 grouped Conv1d (kernel 19, groups 16,
same-padding via pad 9) with GELU between. Output is added back to
input as a positional bias. Pickle keys
relative_positional_encoder.{1..=5}.0.weight/bias (1-based indexing,
no .0.*).
Emotion2Vec top-level — wires LocalEncoder → ProjectFeatures →
RelPosEnc → ContextEncoder → MainEncoder → mean-pool → Classifier.
The proj.* classifier head lives at the state-dict root (not under
d2v_model.), so the constructor uses vb directly there.
Emotion2Vec::load_from_pickle uses VarBuilder::from_pth_with_state
to descend into the fairseq-style nested checkpoint via the "model"
key. One-shot loader; all 185 upstream tensor keys must map onto
candle params of matching shape — and they do.
examples/emotion2vec_smoke.rs — full pipeline integration test:
downloads (or reuses cached) emotion2vec_plus_base from HF, loads it
into candle, runs forward on 2 s of synthetic audio, prints all 9
raw logits + argmax + the 9→5 bucket fold.
Verified on Metal:
loaded model in 0.19 s
forward in 160 ms
9 logits all finite (50-290 range, expected for raw classifier)
argmax: class 7 (surprised) → 5-bucket [excited]
Mechanical correctness end-to-end. Semantic accuracy on real
emotional speech lands in slice 4 (EmotionDetector trait impl +
swap into audio_to_manifest + converse_server reactive-emotion path).
2 new unit tests:
- relative_positional_encoder_preserves_shape
- emotion2vec_random_init_end_to_end_shape
Lib suite 120/120 (was 118, +2).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
87 lines
2.7 KiB
Rust
87 lines
2.7 KiB
Rust
//! Phase 13.8 — slice 2d smoke test.
|
|
//!
|
|
//! Loads the real `emotion2vec/emotion2vec_plus_base` pickle from HF Hub,
|
|
//! runs a synthetic-audio forward pass, and dumps the 9 raw class logits
|
|
//! plus the argmax label. This is the first real-weight integration of
|
|
//! the candle port: every key in the upstream state dict must map to a
|
|
//! candle param of matching shape, or `load_from_pickle` will fail.
|
|
//!
|
|
//! Usage:
|
|
//! ```bash
|
|
//! cargo run -p rtx-csm --release --features metal --example emotion2vec_smoke
|
|
//! ```
|
|
|
|
use anyhow::{Context, Result};
|
|
use candle_core::{Device, Tensor};
|
|
use hf_hub::api::sync::Api;
|
|
use rtx_csm::emotion2vec::{Classifier, Emotion2Vec};
|
|
|
|
const REPO: &str = "emotion2vec/emotion2vec_plus_base";
|
|
const PT_FILE: &str = "model.pt";
|
|
|
|
fn main() -> Result<()> {
|
|
tracing_subscriber::fmt().init();
|
|
let device = if candle_core::utils::metal_is_available() {
|
|
Device::new_metal(0)?
|
|
} else {
|
|
Device::Cpu
|
|
};
|
|
eprintln!("device: {device:?}");
|
|
|
|
// Resolve the cached .pt (or download on first run).
|
|
let api = Api::new().context("hf_hub init")?;
|
|
let path = api
|
|
.model(REPO.to_string())
|
|
.get(PT_FILE)
|
|
.with_context(|| format!("download {PT_FILE} from {REPO}"))?;
|
|
eprintln!("pickle: {}", path.display());
|
|
|
|
// Load the candle port from the real pickle.
|
|
let load_t = std::time::Instant::now();
|
|
let model = Emotion2Vec::load_from_pickle(&path, &device)?;
|
|
eprintln!("loaded model in {:.2}s", load_t.elapsed().as_secs_f32());
|
|
|
|
// Synthesize 2 seconds of low-amplitude noise as a smoke input.
|
|
// (Real evaluation in slice 3 will use a known-emotion clip.)
|
|
let audio = Tensor::randn(0f32, 0.05, (1, 1, 32_000), &device)?;
|
|
|
|
let fwd_t = std::time::Instant::now();
|
|
let logits = model.forward(&audio)?;
|
|
eprintln!("forward in {:.0} ms", fwd_t.elapsed().as_millis());
|
|
|
|
let logits_vec = logits.flatten_all()?.to_vec1::<f32>()?;
|
|
let labels = [
|
|
"angry",
|
|
"disgusted",
|
|
"fearful",
|
|
"happy",
|
|
"neutral",
|
|
"other",
|
|
"sad",
|
|
"surprised",
|
|
"<unk>",
|
|
];
|
|
println!();
|
|
println!("=== logits (raw, before softmax) ===");
|
|
for (i, l) in logits_vec.iter().enumerate() {
|
|
println!(" {}: {:>9.4} ({})", i, l, labels[i]);
|
|
}
|
|
|
|
// Argmax → emotion label.
|
|
let argmax = logits_vec
|
|
.iter()
|
|
.enumerate()
|
|
.max_by(|(_, a), (_, b)| a.partial_cmp(b).unwrap_or(std::cmp::Ordering::Equal))
|
|
.map(|(i, _)| i)
|
|
.unwrap_or(0);
|
|
let bucket = Classifier::tag_for_class(argmax as u32);
|
|
println!();
|
|
println!(
|
|
"argmax: class {argmax} ({}) → 5-bucket label {}",
|
|
labels[argmax],
|
|
bucket.as_tag()
|
|
);
|
|
|
|
Ok(())
|
|
}
|