Promotes the proven attention set-encoder smoke test (PR #4,
tape_train_smoke.rs) into a reusable rtx-transformers layer for Supervised
Memory Training. The teacher maps a window of past tokens to a fixed-size
memory M via embed → self-attention → residual → mean-pool → memory
projection, with a decoder head supervised by predict-the-future MSE so that
M becomes a sufficient statistic of the past.
- Trains end-to-end on Autodiff<CpuBackend> (the gradient-correct real backend
from PR #3), with a self-contained deterministic host-side Adam.
- Time-parallel by construction (one window → one memory, no recurrence to
unroll) — this is the oracle whose trajectory the recurrent Mamba updater is
later behaviorally cloned against, so the recurrent net never needs BPTT.
- Exposes named_params/set_named_params so the caller (omni-think's
PredictiveStateTeacher facade) owns safetensors persistence + BLAKE3 sealing.
- Adds rtx-backend + rtx-backend-cpu deps (the tape needs a concrete backend).
Tests: teacher trains (loss >5x drop), encode is deterministic + fixed-size,
params round-trip. fmt + clippy(-D warnings) clean on the new module.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
End-to-end training on `Autodiff<CpuBackend>` (forward → backward →
extract grads → SGD → repeat), guarding the gradient-correctness fixes:
- `mlp_trains_and_loss_decreases`: 2-layer MLP (matmul + gelu), loss 8.65 → 0.39.
- `attention_set_encoder_learns_window_mean`: a minimal attention set-encoder
(the SMT predictive-state teacher shape) learns to predict its input window's
per-dim mean, loss 0.27 → ~0.0. The embedding fans out to Q/K/V and the
residual (4 uses), so this also regression-guards the fan-out gradient
accumulation fix inside a real attention block.
All ops used are gradient-checked in `tape_cpu_gradcheck.rs`.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The decorator autograd (`Autodiff<B>`) had never been gradient-checked
against a real tensor backend — the entire test suite runs on a shape-only
`MockBackend` whose ops return their input, so they validate graph structure
but never gradient values. Running it through `CpuBackend` for the first time
(new `tests/tape_cpu_gradcheck.rs`, finite-difference checks) surfaced three
bugs that made the tape unusable for training; this fixes all three.
1. Double-free / UB in the dimension-erasure cast. The backward ops cast a
tensor to its runtime const-generic dimension via
`mem::transmute_copy::<_, TensorPrimitive<N>>(&src)` in ~100 sites. That
bit-copies the owned `Vec` without forgetting the source, so two values own
one buffer → double-free on any heap-backed backend (and Stacked-Borrows UB
from the typed pun). Replaced every site with a single `into_dim` helper
that is now **fully safe** — it round-trips through `to_data`/`from_data`
and rebuilds the shape with `array::from_fn`, no `unsafe` at all. (This is
why the whole repo previously bypassed the tape with analytic backward.)
2. Fan-out gradients were silently dropped. `accumulate_gradients` was a stub
that returned one path and discarded the other, and `AutodiffTensor::clone`
minted a fresh `TensorId`. Together, reusing a tensor (residuals,
`mul(s, s)`, shared Q/K/V — universal in transformers) split its gradient
across two ids and summed neither, yielding a fraction of the true value.
`accumulate_gradients` now sums via `B::add`; `clone` preserves the id so
fan-out paths collide on one sink.
3. Softmax backward panicked. `SoftmaxBackward` / `stable_softmax_backward`
subtracted a keep-dim row-sum from the full-shape grad, but the elementwise
backends assert equal shapes (no broadcasting). Added `broadcast_along_dim`
to tile the row-sum to full width first.
Verified: `tape_cpu_gradcheck` (matmul, fan-out add·mul, softmax) passes with
rel-err < 2e-2 vs central differences; full `rtx-autograd` suite green (263
passed, 0 failed); lib clippy `-D warnings` clean.
Known follow-up (out of scope): `cargo miri test` still aborts on a
Stacked-Borrows / integer-to-pointer violation inside `rtx-backend-cpu`'s
buffer internals — a grad-free `from_data`+`add`+`sum` probe reproduces the
identical error, so it is pre-existing backend UB, not an autograd issue.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Persist accumulated WIP across rtx-tensor / rtx-transformers / rtx-runtime.
Two related feature groups:
GPU enablement (unblocks the Phase-3 spec §8 CUDA Mamba path):
- rtx-tensor: `Tensor::cuda_device_ptr()` + storage GPU-buffer accessors
(`raw_ptr.rs`) expose the raw CUdeviceptr that kernel launches need —
the "rtx-tensor GPU memory access API" the Mamba CUDA kernels were
blocked on.
- rtx-transformers: `MambaBlock::forward_cuda` runs the four linear
projections through cuBLAS on-device (in/out/x/dt_proj), keeping the
selective scan + conv1d + activations on CPU; dispatched automatically
from `forward` when on a CUDA device under the `cuda` feature. Updated
`mamba_cuda_kernels.rs` accordingly.
- supporting plumbing in rtx-runtime stream/bridge and rtx-tensor
storage/conversion/concatenation/creation + rtx-flash-attention.
Linear algebra (rtx-tensor):
- `linalg/matrix_exp.rs`: real matrix exponential via scaling-and-squaring
with a degree-13 Padé approximant (Higham 2005), f64 internally.
- `complex/linalg.rs`: complex matmul/adjoint, Hermitian eigendecomposition
(`ComplexEigenResult`), and the complex matrix exponential, nalgebra-backed.
- tests for both.
Builds verified on the CPU path (`cargo check -p rtx-tensor -p rtx-transformers
-p rtx-runtime -p rtx-flash-attention` clean). The `cuda` feature and the
rtx-backend-cuda NVCC build remain unbuildable on this host (CUDA/glibc header
mismatch) — pre-existing and unrelated to these changes.
End-to-end demonstration that training the real selective-scan Mamba buys
a genuine temporal-modeling win — the payoff of M1–M3.
Task: next-token prediction on a multi-regime sequence (x[t] = μ_regime +
noise). Predicting x[t+1] inside a regime requires integrating recent
history to average out the noise — a memoryless model can't.
Result (same sequence, all three):
persistence (memoryless) MSE = 0.0895
untrained Mamba MSE = 0.2167
trained Mamba (400 Adam) MSE = 0.0002
The trained backbone integrates history to de-noise the regime mean,
beating the memoryless persistence baseline by ~450× and improving
~1000× over its untrained self. (Single-sequence fit: demonstrates the
SSM's temporal-modeling capacity, not held-out generalization.)
This replaces the old non-result ("random Mamba 6.5% vs linear 48%") with
a real "trained SSM exploits temporal structure" demonstration.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Plain randn(0,1) init made A=-exp(A_log) and Δ wildly large, overflowing
the real exp(Δ·A) scan to NaN (the stub never cared — it discarded these
weights). new()/new_seeded() now share a build() with canonical S6 init:
- A_log = ln(1..=d_state) ⇒ A = -(1..=d_state), bounded
- dt_bias so softplus(dt_bias) ≈ 0.01 (small, stable Δ; near-identity
scan at init — intentional for gradient flow)
- D = 1, zero conv bias, projections scaled by 1/√fan_in (capped 0.5)
This fixes the NaN that broke omni-cortex's d231 action-conditioned
predictor training (now green). Seeded determinism preserved.
Tests: active_block helper (Δ overridden to ≈0.69) exercises the
scan-active regime so the liveness check can observe each weight; the
training test asserts a seed-varying backbone weight (conv1d_weight)
moves. All 6 selective-scan tests green incl. the finite-diff grad check.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Demonstrates end-to-end trainability of the real selective scan. A
self-contained Adam loop (the rtx-transformers AdamOptimizer has no
public gradient setter — the spec permits a bespoke loop) fits a teacher
block's output on a fixed input: forward → MSE → analytic backward →
Adam step → rebuild. Over 200 steps the loss drops >50% and the backbone
weight A_log moves, confirming gradients actually train the model (not
just the head). All 6 selective-scan tests green.
The production AdamOptimizer can be wired once it exposes a gradient
setter; the M2 backward already returns grads in its HashMap shape.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Hand-written VJP of the selective-scan forward (Phase-3 spec §5/§7):
MambaBlock::backward(x, d_out) -> per-parameter gradients keyed by
persistence name, summed over the batch. Differentiates the scan
analytically (reverse-time recurrence over the cached h trajectory)
rather than through the immature rtx-tensor autograd tape.
Covers every parameter: in_proj, conv1d_weight, conv1d_bias, A_log
(via A=-exp(A_log) ⇒ dA_log = dA·A), x_proj, dt_proj, dt_bias, D,
out_proj. Adds stable sigmoid_f32 / silu_grad_f32 helpers.
New test analytic_gradients_match_finite_differences: on a small
well-conditioned instance, ≥30 sampled grad elements across all 9
params match central finite differences within (5e-3 + 5e-2·|fd|).
All 5 selective-scan tests green.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The MambaBlock forward was a stub — SelectiveScan::forward passed input
through, discretize returned zeros, conv1d was a no-op, and B/C were
randn per call, leaving conv1d_weight/dt_proj/A_log as dead weights with
zero temporal mixing. This implements a genuine S6 selective-scan:
- New params: x_proj [d_inner, dt_rank+2*d_state] (data-dependent dt,B,C),
dt_bias [d_inner], D [d_inner] (skip). Added to new()/new_seeded() and
the persistence contract (persistence_tensors/from_persistence_tensors).
- Real forward (CPU f32, looped — backbone is small): in_proj -> causal
depthwise conv1d -> SiLU -> x_proj->(dt,B,C) -> delta=softplus(dt.dt_proj
+dt_bias) -> A=-exp(A_log) -> sequential scan h=dA.h+dBu, y=sum C.h + D.u
-> gate by SiLU(z) -> out_proj. Residual moved OUT (canonical).
Numerically-stable silu_f32/softplus_f32 helpers.
- The scan runs inline (not via the immature rtx-tensor autograd tape);
the analytic backward lands in M2 per docs/phase3_real_ssm_spec.md.
New tests/real_selective_scan.rs (4 cases, all green): liveness (each
formerly-dead weight now moves the output), causality (no future
leakage), seeded determinism, and finite/non-constant output.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Add 10 new tests exercising previously uncovered GenericTensor operations:
leaky_relu, elu, gt_scalar, var, var_dim, conv2d (identity kernel,
stride, grad propagation), max_pool2d, and avg_pool2d. Also add
test_utils.rs module for shared test helpers.
SDLC-Coverage: rtx-tensor generic/tensor.rs 19.6% → targeting ~40%+
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Two coordinated additions for the omni-cortex D249/D250 work:
rtx-tensor: Tensor::randn_seeded(shape, device, seed) — like
randn() but routes through StdRng::seed_from_u64(seed) so two
calls with (shape, device, seed) produce bit-exact identical
tensors. CPU is the canonical generator; GPU calls go via to_device
transfer. Required for reproducible model init.
rtx-transformers: MambaBlock gains:
- new_seeded(config, device, seed) — every internal weight tensor
initialised via randn_seeded() with per-tensor SplitMix64-derived
seeds. Two calls with the same (config, device, seed) → bit-
exact identical block.
- persistence_tensors() -> Vec<(&'static str, &Tensor)> — read-only
view of the six (or seven, with conv_bias) internal weight
tensors with canonical names (in_proj, conv1d_weight,
conv1d_bias?, A_log, dt_proj, out_proj). Stable across versions
so safetensors round-trip works.
- from_persistence_tensors(config, device, HashMap<String, Tensor>)
— rebuild a MambaBlock from a name → tensor map. Validates each
tensor's shape against the config and surfaces clean errors on
mismatch (so wrong-DIM safetensors loads fail explicitly).
These three primitives together give omni-cortex's D249 (operator-
seeded determinism) and D250 (safetensors round-trip + BLAKE3 hash
pin) clean library hooks without exposing MambaBlock's private
fields.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Closes the LoRA inference path that was previously stubbed. Two new
public APIs in rtx-csm:
1. lora::load_lora_set_from_safetensors(path, config) -> LoraSet
Reads a trained adapter file (produced by training::
save_lora_adapter[_with_metadata]). Pairs the .lora_a / .lora_b
tensors by base-weight prefix into LoraAdapter entries.
2. lora::merge_into_safetensors(base, lora, scale, output)
Reads the base CSM safetensors, folds in the LoRA deltas at the
given scale (typically alpha/rank from training), writes a merged
safetensors. Original dtype preserved (F16 on Metal, BF16 on
CUDA, F32 on CPU). Tensors LoRA doesn't target are passed
through unchanged.
3. Generator::load_csm_1b_from_path(path, device)
Variant of load_csm_1b that takes an explicit weights path
instead of going through the HF cache. Mimi + tokenizer still
resolve via the hub. This is the path consumers use to load a
merged checkpoint.
MergeReport struct restructured to expose merged/skipped/passthrough
counts so callers can verify the adapter actually targeted weights.
The previous typed-error test is replaced with a missing-base-file
test that exercises the real code path.
Used by zeroclaw-channel-voice's `--lora-adapter` flag to bake a
LoRA adapter into a per-process merged checkpoint at boot, with
zero per-inference overhead.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Two additions for downstream voice-channel prosody / quality wiring:
1. Converse::with_pre_sentence_hook
New PreSentenceHook (Box<dyn FnMut(&mut Generator, &str) -> Result<()>>)
that fires before each sentence's synth call. Receives mutable
access to the underlying Generator + the sentence text — lets
callers apply per-sentence steering (e.g. emotion shifts mid-reply)
without touching crate internals. Wired into both `synthesize` and
`synthesize_streaming` paths.
2. PostProcess streaming split
- New StreamingHpfState — stateful biquad whose IIR taps carry
across chunk boundaries so streaming HPF doesn't click at chunk
joins. Identical filter coefficients to the one-shot path.
- PostProcess::apply_chunk_safe(samples, hpf_state) — HPF + declick
per chunk, no LUFS (needs full utterance).
- PostProcess::apply_lufs(samples, sample_rate) -> Result<f32> —
full-utterance loudness gain, returns the linear gain applied
so streaming pipelines can compensate retroactively if needed.
- compute_lufs_gain helper extracted from loudness_normalize.
Used by zeroclaw-channel-voice for the Maya-gap-closure pack.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
ring_allreduce() contained its own simulation that multiplied each gradient
value by world_size (to fake an AllReduce sum), bypassing the ProcessGroup
dispatch entirely. This meant the overlapped synchronization path never used
NCCL or RNCCL, even when those features were compiled in.
Replace the hand-rolled simulation with a call to
self.process_group.allreduce(tensor, ReduceOp::Sum) so the overlapped path
uses the same backend as synchronize_gradients_sequential. The communication
latency sleep is kept for benchmarking purposes.
Add test_ring_allreduce_matches_sequential_path to verify both paths produce
identical gradient values under CPU simulation.
Closes#10
Without a voice anchor, CSM-1B picks a different speaker each turn
and drifts mid-sentence on longer outputs (high-pitch squeaks,
female/male swap mid-utterance). The fix is the standard CSM
speaker-prompt pattern: pass a Segment with reference audio + its
transcript as context to every generate() call.
Previously Converse::synthesize and synthesize_streaming hardcoded
`&[]` for the context arg. Add a `context: Vec<Segment>` field on
Converse plus a builder method:
let conv = Converse::new(&llm, &mut gen)
.with_context(vec![Segment::new(0, transcript, audio)]);
Both synth paths now pass `&self.context` instead of `&[]`. Empty
context (default) keeps prior behavior.
Verified end-to-end with zeroclaw-channel-voice + macOS `say`-
generated reference: same input now produces deterministic-length
output across turns (2.64s vs. previously varying 6/19/38s) and the
voice matches the seed throughout.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
The canonical voice loop now lives in zeroclaw-channel-voice
(`~/projects/zeroclaw/crates/zeroclaw-channel-voice`, binary
`voice_server`). It routes the LLM path through zeroclaw's agent
runtime — multi-turn history, tools, memory, provider routing —
instead of the OpenAI-compatible direct path here.
Same WS wire protocol so `examples/converse_client.rs` drives both;
no client-side migration needed.
This binary is intentionally kept buildable for:
1. Reproducing perf_history.md Phase 8.10 benches.
2. Standalone (no-agent) use when zeroclaw isn't desired.
Module doc + main() startup banner updated to point at the new home.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
8-gen bench (4 emotions × 2 corpora) at seed=42 against firdhokk
Whisper-LV3:
target RAVDESS CREMA-D
happy happy (0.999) ✓ happy (0.999) ✓
angry neutral (0.92) sad (0.99)
fearful happy (0.998) fearful (0.984) ✓
sad angry (0.99) fearful (0.99)
CREMA-D 2/4 vs RAVDESS 1/4. Larger / more naturalistic corpus
produces more class-pure fearful direction. Neither corpus solves
angry or sad — recipe shifts into 'vague expressivity' rather than
class-specific corners.
Practical: prefer CREMA-D when available; A/B both per emotion if
class precision matters.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Python sidecar for scoring TTS outputs against the firdhokk
Whisper-LV3 SER classifier (sanity-verified non-saturated, 3/5
correct on RAVDESS ground-truth).
Replaces the in-process emotion2vec_plus_base path which collapses
to 'Surprised' on every input (documented in
emotional_speech_guide.md and quality_eval.rs caveat).
Reads JSONL with {gen_wav, target_emotion} rows; writes JSONL with
top_emotion, top_prob, target_prob, match (bool), and the full
8-class probability distribution.
Class set is firdhokk's 7 (no calm — calm aliases to neutral on
input). Excited aliases to happy.
Smoke-verified on the 4 prior decoder-route outputs (Amini ctx,
seed=42, recipe defaults):
happy → neutral (0.80) ✗
angry → happy (0.999) ✗
fearful → fearful (0.68) ✓
sad → fearful (0.998) ✗ (sad↔fearful confusion)
Top-1 match: 1/4 — confirms the gap documented in
emotional_speech_guide.md 'Known Limitations'.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
P2 fixes:
- Run cargo clippy --fix to auto-fix ~20 warnings (format strings, closures, etc.)
- Fix post.rs: &mut Vec<f32> -> &mut [f32] (prefer slice over Vec reference)
- Fix prompt.rs, longform.rs, training.rs: add blank lines before doc list items
- Fix diarize.rs: replace loop with slice.fill(true) for cleaner code
- Add #[allow(clippy::needless_range_loop)] where i is used in arithmetic
- Add #[allow(clippy::too_many_arguments)] to functions needing builder refactor
Down from 37 warnings to 0 warnings in rtx-csm.
P0 fixes:
- Add [[test]] required-features=["cuda"] to Cargo.toml so backend_parity_tests
only compile when the cuda feature is enabled (avoids E0425 compile errors)
- Add missing Lazy/OnceCell/RwLock/Arc/HashMap imports to device.rs
- Add missing Arc import to tensor.rs
- Add missing Backend/BoolU8/DeviceId/DeviceOps imports to lib.rs
All 16 backend parity tests now pass with --features cuda.
Wired up firdhokk/speech-emotion-recognition-with-openai-whisper-large-v3
as a working alternative to the broken emotion2vec_plus_base. Sanity
verified on real RAVDESS clips: 3/5 correct, 2/5 near-miss (happy↔
surprised, sad↔fearful). Probabilities are NOT saturated — the
classifier actually distinguishes per-input.
Then scored our 4 decoder-route outputs (Amini context, seed=42,
recipe defaults) and found that **only fearful registers as the
intended class**:
target verdict conf
happy neutral 0.80 ✗ (steering produces neutral output)
angry happy 0.999 ✗ (high-arousal cross-class)
fearful fearful 0.68 ✓
sad fearful 0.998 ✗ (sad↔fearful confusion)
Honest framing: the recipe shifts speaker character toward an
expressive-sounding direction (cosine evidence) and preserves text
(decoder vs backbone) but does NOT produce class-distinct emotion.
The metric stack we used through Phase 9 (cosine + WER) couldn't
see this gap because it measures voice fidelity and text rendering,
not emotion class.
Hypothesized fixes (not yet tested):
- CREMA-D extraction (91 actors vs RAVDESS 24) for class-purer
steering vectors
- Mixed backbone+decoder steering (backbone for prosody)
- EmoNet classifier (TTS-aware, may give different verdicts)
Doc'd in emotional_speech_guide.md as a known limitation. Closes
out an honest scientific picture: today's work successfully ports
the architectural finding (decoder route preserves text), but
class-precise emotion control remains unsolved.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Wires emotion2vec into quality_eval so per-row metrics include
target_emotion_prob, top_emotion, top_emotion_prob. Pure inference,
optional via --emotion2vec / --target-emotion flags.
Critical empirical finding documented in code + user guide: the
emotion2vec_plus_base checkpoint classifies every input as
"Surprised" with prob ≥ 0.99, INCLUDING ground-truth RAVDESS clips
with explicit emotion labels. Real angry-RAVDESS → "Surprised"
(0.9999999). Real neutral-RAVDESS → "Surprised" (0.9999996).
The metric implementation is correct (matches the trait's
EmotionDetector::classify code path with same per-utterance zero-
mean unit-variance normalization); the underlying classifier
collapses to a dominant class on most input — likely the same
"9→5 fold collapse" the project already documented in the data-
labeling path.
Practical implication: target_emotion_prob is near-zero for almost
every (target, output) that isn't "surprised", so it can't be used
as a picker score. The emotion2vec metric still works as a
diagnostic ("did the model produce something that classifies as
audio at all?") but not as a generation-quality validator.
Doc'd in:
- examples/quality_eval.rs CLI doc (caveat block on --emotion2vec)
- docs/emotional_speech_guide.md (Known limitations section with
full sanity-check table)
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Brief addition to the recipe section explaining the WER + length
floor scoring used by emotional_speech_n.sh (committed in b7b267b).
Validates that the new scoring preserves canonical winners on
happy and calm while flipping surprised to the long-and-correct
candidate.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Original picker used min(WER), tie-broken by max(cosine). Fragile:
on emotion=surprised it picked "You can." (2 words, WER 0.93) over
"...Today I want to share something" (13 words, WER 1.00) because
WER weights all errors uniformly — terse-and-mostly-wrong beats
long-and-mostly-right.
New scoring:
score = WER + (1.0 if words(transcript) < 5 else 0)
sort_by(score, -cosine)
Verified on existing benches:
surprised: now picks seed=100 ("...Today I want to share something
with...", 13 words, score 1.0) over seed=7 ("You can.",
2 words, score 1.929 with +1 length penalty).
calm: still picks seed=100 (full transcript revealed: "It's a
good reflection. Not that that. I want to share
something with you that I've been thinking about." —
near-verbatim! the earlier 55-char display had been
truncating it).
disgust: all 3 candidates score ~1.93 (no seed has > 5 words,
all get the length penalty); picker honestly admits
none is good rather than picking a fake winner.
Worth noting: the calm seed=100 case is ANOTHER near-verbatim
single-shot result we missed in the previous bench because the
display truncation hid the full transcript content.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
3-seed picker run (7, 42, 100) on calm/disgust/surprised refines the
single-shot characterization:
calm: winner seed=100, WER 0.64
"It's a good reflection. Not that that. I want to
share somet..."
(recipe lands the prompt — single-shot at seed 42 only
produced hesitation markers; the picker found a seed
with actual content)
disgust: no reliable seed
(all 3 seeds WER ≥ 0.93; likely RAVDESS corpus issue —
disgust clips are low-energy / acoustically close to
neutral. Try CREMA-D or ESD for this emotion.)
surprised: picker chose seed=7 (WER 0.93, short "You can.") over
seed=100 (WER 1.0, "...Today I want to share something")
— WER weighting issue: deletions and insertions count
uniformly, so terse-but-mostly-wrong beat long-and-
mostly-right. Manual selection or weighting WER less
heavily would help here.
Updated per-emotion table marks disgust as ✗ (corpus limitation),
surprised as ⚠ (picker scoring artifact), calm as ✓ (works with
N-seed picker).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Extracted decoder steering vectors for the remaining 3 RAVDESS
emotions (calm, disgust, surprised). Single-seed bench at
seed=42 on Amini context, decoder route, recipe defaults:
emotion cos_ctx WER transcript
calm 0.70 0.86 "I'm sorry. Um, I don't know."
(natural hesitation markers — the
recipe produces semantically-emotion-
matched content, not just acoustic
shift)
disgust 0.81 0.93 "For that, that..." (truncated)
surprised 0.95 2.57 "Too couple, sorry, and that's saying,
even a premier and super driver..."
(long rambling; voice migrates well,
text drifts)
All 7 RAVDESS emotions now produce coherent English on the decoder
route — calm is solid first-shot, disgust truncates, surprised
rambles. Roll N seeds via emotional_speech_n.sh for the latter two.
emotional_speech_guide.md updated with the per-emotion table now
covering all 7. Voice character preservation (cos vs context > 0.7)
holds for every emotion in the pack.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Phase 9 produced 25 commits and a complex pipeline; perf_history is
the engineering log but new users coming to this cold need a clean
"how do I make CSM speak with emotion" handbook.
Sections:
- What this gets you (single-shot WER 0.07–0.21, voice cosine ≥ 0.95)
- One-liner quickstart (RAVDESS download → extract → use)
- The recipe explained — every flag and why it's there
- Per-emotion notes (works/best-seed/caveats per emotion)
- When it works / when it doesn't
- Troubleshooting (music tokens, premature EOT, repetition, etc.)
- Architecture cheat sheet (backbone=semantic, decoder=acoustic)
References perf_history.md for the full empirical log; this doc is
the user-facing distillation.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Capstone consumer interface composing every piece shipped today:
fetch_audio.sh → audio_to_manifest → pick_context.sh →
emotional_speech_n.sh (N-seed picker, decoder route)
Caches fetch + manifest by URL hash so re-runs with the same
--workdir skip the slow steps. Defaults to the Phase 9 recipe:
target=decoder, scale=1.0, layers [2,3], cfg=linear:3.0:1.0:25,
5-seed roll with the lowest-WER winner picked.
End-to-end smoke test (cached Carlini source, prompt "Today I want
to share..."):
picker auto-selected: nicholas_carlini...spk0.0078.wav (10.78 s)
seed 42 (winner): cos 0.974, WER 0.143 ⭐
"But Jason, today I want to share something
with you that I h"
seed 100: cos 0.986, WER 0.286
"It ties upon a share something with you that
I have been thi"
seed 7: cos 0.862, WER 1.000
"Let me think, let him out."
The auto-picker chose spk0 (Carlini himself) where manual selection
earlier in the day grabbed spk1 (the announcer) — so the automated
pipeline is also a slight context-selection improvement.
Three sub-second-WER results recorded over the day:
- WER 0.071 Amini imperative prompt (manual)
- WER 0.125 Amini original prompt (manual)
- WER 0.143 Carlini auto-picked spk0 (this commit, end-to-end)
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Cross-emotion × Carlini context at Amini-best seeds reveals the
"magic combo" doesn't fully transfer:
happy@42 Amini WER 0.21 → Carlini WER **0.071** (transfers!)
angry@100 Amini WER 0.93 → Carlini WER 1.21 (URL drift)
fearful@7 Amini WER 0.86 → Carlini WER 1.00 ("Screw it")
sad@7 Amini WER 0.93 → Carlini WER 1.00 (no transcript)
Only happy@42 cleanly generalizes across contexts. The previous
"context-robust" claim was too strong — the (emotion, seed, context)
interaction matters. Cosine vs context stays high for angry (0.95)
even when text drifts, so voice character preservation is the more
robust property than text fidelity.
Honest production interface: `emotional_speech_n.sh` rolling 5 seeds
per (context, prompt). The single-shot recipe lands well only when
all dimensions align, but the picker absorbs the variance.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
3-context bench at happy@seed=42 decoder recipe (Amini / MCC /
Carlini). Cosine measured vs the context wav (does recipe preserve
input voice character?), WER vs prompt:
context cos_ctx WER transcript
amini 0.972 0.21 "All right, today I want to share
something with you that I've been
thinking about."
mcc 0.58 0.93 "You" (sub-speaker mismatch)
carlini 0.958 0.071 ⭐ "So today I want to share something
with you that I have been thinking
about."
Carlini's WER 0.071 is the new single-shot best of Phase 9. Only
prefix "So" added to the verbatim prompt. Cos vs context > 0.95 on
the two working contexts means the recipe preserves speaker
character of the reference — does NOT impose RAVDESS speaker
identity on every output.
The recipe is context-robust on speaker identities the picker
selects correctly. McConaughey failed because we picked the
manifest's spk1 (likely the Oscars announcer), not McConaughey
himself. That's a context-selection issue, not a recipe issue.
Empirical capstone: single-shot near-verbatim emotional speech
with preserved voice character is achievable.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
3-prompt × 2-condition bench at happy@seed=42 decoder recipe vs
no-steering baseline. The recipe holds across prompts and produces
the lowest WER recorded across all Phase 9 experiments:
prompt base happy decoder
Today I want to… 0.52 / 0.93 "You" 0.82 / 0.21 ⭐
Have you ever… 0.72 / 1.39 drift 0.77 / 1.00 "With blames"
Weather has been… 0.73 / 1.88 ♪♪♪ 0.69 / 0.125 ⭐ "That the
weather has been
absolutely beautiful
this mor"
The imperative prompt's WER 0.125 is the lowest recorded.
Improvement vs baseline ranges 1.4× to 15× lower WER. The
no-steering baseline produced literal singing tokens (♪♪) on the
weather prompt, suggesting CSM's CFG-only path is fragile on
prompts the model "dislikes."
Empirical conclusion: decoder route + RAVDESS happy steering at
seed 42, scale 1.0, layers [2,3] is a reproducible recipe, not a
single-prompt anomaly. N-seed picker still the right consumer
interface, but this single configuration alone reaches
near-publishable quality on multiple prompts.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Records the breakthrough single-shot result: happy@seed=42,
decoder route, scale 1.0, layers [2,3] →
"All right, today I want to share something with you tha"
(WER 0.21, cos 0.82). Closest-to-perfect single-condition
result of the entire Phase 9 sprint.
Per-emotion seed winners diverge:
happy=42, angry=100, fearful=7, sad=7
No universal best seed exists; this validates emotional_speech_n.sh
as the production interface (rolls multiple, picks lowest WER).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Wrapper now supports --target backbone|decoder. Decoder is the
default (Phase 9 finding: it preserves text fidelity that backbone
destroys). Per-(target, emotion) defaults:
decoder: scale 1.0, layers [2,3] (last two of 4)
backbone: scale 0.2-0.3 (per-emotion), layers [8,10,12]
emotional_speech_n.sh's existing passthrough already forwards
--target through to this wrapper unchanged.
Bench all 4 emotions on the decoder route, seed 7, recipe defaults:
emotion cos WER transcript
happy 0.65 0.93 "I've been happy cycling and beat..."
angry 0.78 2.29 "And of course, coming first, Vern is..."
fearful 0.73 0.86 "I'm not eye sensing when that's a mile."
sad 0.67 0.93 "- I'm actually off my night. I'll take
something. - All right..."
Sad — the previously-unsolvable emotion on the backbone (model
resisted at every tested scale 0.15-0.3) — produces real fluent
English on the decoder route. The word "happy" surfaces in the
happy output. All 4 emotions produce coherent speech: no music
tokens, no premature EOT, no gibberish. WER stays in the 0.86-2.3
range, comparable to baseline-with-CFG.
The decoder route subsumes everything the backbone route was
trying to do and unlocks the failure case it couldn't reach.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Mirror of --steering-layers for the decoder: comma-separated layer
indices to actually steer (decoder has 4 layers; useful subsets are
[3], [2,3], [1,2]).
Sweep at seed 7, [email protected]:
[0,1,2,3] cos 0.86 WER 0.93 "I'm not that tall ×3" ← repetition
[3] cos 0.58 WER 0.86 "I'll be off and offense..."
[2,3] cos 0.80 WER 1.43 "My daughter, Penny Ryan, and I have…"
[0,1] cos 0.82 WER 1.57 "And I'll check on them..."
[1,2] cos 0.81 WER 1.43 "I'm going to call him an X-Man..."
The repetition is specific to all-layers-at-once steering. Any 2-layer
subset eliminates it while preserving most of the cosine boost. Same
pattern as the backbone's [8,10,12] finding: partial perturbation
lets the unsteered layers act as a stabilizing prior.
[2,3] (decoder last 2) is the new recommended recipe — best cosine
of the no-repetition subsets and the longest fluent transcript.
Documented in docs/perf_history.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
3-seed × 2-condition ([email protected] alone vs +CFG) bench plus a 4-step
scale sweep. Captures the honest tradeoff:
- Backbone steering destroys word content (semantic gibberish).
- Decoder steering preserves coherent English BUT produces
repetition or premature EOT.
Neither produces single-shot production-quality emotional speech;
emotional_speech_n.sh (N-seed picker, lowest-WER wins) remains the
right consumer interface — it doesn't care which failure mode
generated the bad samples, just discards them by metric.
Decoder vector magnitudes are ~10× smaller than backbone (norm 0.85
at deepest layer vs 14.9), so the apparent useful scale window is
~10× higher (0.5-1.0 instead of 0.2-0.3).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Adds capture_decoder_activations on Model and ModelBackend, plus
--target-module decoder|backbone on examples/steering_extract.
Decoder mode runs a text-only prompt through the backbone (no
capture), Mimi-encodes the audio separately to grab the middle
frame's c0 token, teacher-forces that c0, and captures one mean-
pooled-over-seq vector per decoder layer. Result: 4 layers ×
1024 embed dim per call, much faster than backbone capture
(text-only prompts are short).
A/B with the canonical "Today I want to share..." prompt at seed
7 (the previously-identified low-WER seed):
case cos WER transcript
baseline 0.76 0.36 "And today I want to share something some
funnel distraits" (high baseline at this
seed)
[email protected] 0.75 5.00 "Today, I want to share some needs of my
prey" (backbone destroys content)
[email protected] 0.86 0.93 "I'm not that tall. I'm not that tall."
(fluent but repetitive — biggest cos)
[email protected] 0.61 2.57 over-steered
[email protected] 0.61 2.14 broken
[email protected] is the largest speaker_cosine boost we've measured AND
produces clean English. Backbone steering at the same seed destroyed
content fidelity. This validates the architectural hypothesis: the
backbone carries semantic content (what the model says), the depth
decoder carries acoustic detail (how it sounds). Steering the
decoder shifts voice character without disturbing word content the
way backbone steering does.
Open issues: [email protected] produces repetitive output ("I'm not that
tall" three times). Likely lower scale (~0.5) plus the existing
repetition guard would fix it; left for follow-up.
Decoder vector magnitudes are ~10× smaller than backbone (norm 0.85
at deepest layer vs 14.9), so the appropriate scale is ~10× higher
than the backbone recipe (1.0 vs 0.1-0.3).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Adds set_decoder_steering on Model + Generator and
--decoder-steering-vec / --decoder-steering-scale on examples/generate.
The decoder is already a LlamaModel under the hood, so the existing
LayerSteering hook in csm_fork::Layer::forward applies as-is — only
the public surface needed wiring.
Architectural hypothesis being tested: backbone carries semantic
content (what the model says), depth decoder carries acoustic detail
(how it sounds). Backbone steering shifts character at the cost of
text fidelity (Sprint 2 finding); decoder steering should shift
prosody/timbre without disturbing word content.
Smoke test with random Gaussian decoder vectors (4 layers × 1024
embed_dim, stddev 0.1, scale 0.5):
case cos WER transcript
baseline 0.72 1.0 "No."
backbone 0.83 1.4 "That's for on-beat for bee..."
decoder 0.76 1.0 "So" (premature EOT)
both 0.81 3.0 "I'm going to go to the next one..."
Decoder steering DOES alter output (cosine 0.72 → 0.76, transcript
changes) but random vectors trigger premature EOT — same pattern as
random backbone vectors. The infrastructure works; getting the real
emotion-from-acoustic-codebooks signal needs decoder activation
capture, which the current Model::capture_backbone_activations
doesn't do (it captures the backbone forward only).
Decoder capture is the next-session item. With it we can extract
real per-emotion decoder vectors from RAVDESS and test the
hypothesis properly.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Productionizes yesterday's seed-variance finding. Wraps
emotional_speech.sh, rolls a list of seeds, scores each via
quality_eval (speaker_cosine + Moonshine WER), and copies the
lowest-WER candidate to --out. Defaults to 5 seeds; pass
--seeds 42,7 for cheaper runs.
Tie-breaking is `min(WER), then -max(cosine)` — text fidelity
takes precedence over speaker character because user-typed text
should be rendered verbatim, while voice character is only
secondary on top of context conditioning. Failed generations
(short clips that get the -1 cosine sentinel) sort to the bottom.
Smoke run on the canonical "Today I want to share..." prompt:
seed 7 → cos 0.845, WER 0.714 "Today, today I want to share..." ← picked
seed 100 → cos -1, WER 1.000 "It is." (premature EOT)
seed 42 → cos 0.916, WER 2.000 "The police are, if you're..." (drift)
Cost: N × single-shot cost. The recipe being unreliable per-seed
is the whole reason this wrapper exists — pay the multiplier in
exchange for a reliably-best output.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
3-seed × 3-prompt reproducibility bench on [email protected] reveals that the
recipe shifts speaker character reliably but produces high text-
fidelity variance:
seed 42: WER 2.00 "The police are, if you're, I can't recite..."
seed 7: WER 0.71 "Today, today I want to share..." ← near-verbatim
seed 100: 0.32 s premature EOT
Cross-prompt at seed 42 drifts uniformly across 3 prompts. Speaker
cosine is consistently elevated; text content is roll-the-dice.
Documenting this as the honest characterization rather than overclaim
the single-seed Sprint 2 results. Practical recipe: roll N seeds,
pick lowest-WER output.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Capstone wrapper for the Phase 9 recipe (Selective CFG + RAVDESS
steering + mid-layer subset). Picks the steering scale automatically
from an empirical per-emotion map:
- happy: 0.30
- angry: 0.20
- fearful: 0.20
- sad: 0.20 (note: sad is unreliable — see below)
These came from a follow-up sweep after the multi-emotion demo
revealed the recipe is emotion-sensitive: scale 0.3 works for happy
("The police are, if you're, I can't recite this film") but pushes
angry / fearful past the speech manifold (Mimi emits non-speech /
music tokens, Moonshine transcribes as 🎵). Dropping to 0.2 recovers
fluent speech for both:
- [email protected]: "The next disorder is completing kashim for more."
- [email protected]: "You just heard a little bit about this decision,
though."
- [email protected]: "The police are, if you're, I can't recite this
film. I"
Sad is the outlier — model resists "sad" steering at every scale
between 0.15 and 0.3. Likely a corpus issue (sad RAVDESS clips are
the lowest-energy subset). Documented as a known limitation rather
than worked around.
perf_history.md updated with the per-emotion sensitivity finding.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
osobhandClaude Opus 4.8