96a7f1c70042140bdc984b020138b80a3e280e69
100
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
3e3e8819f6 |
rtx-tensor + rtx-transformers: deterministic init + Mamba weight persistence
Two coordinated additions for the omni-cortex D249/D250 work:
rtx-tensor: Tensor::randn_seeded(shape, device, seed) — like
randn() but routes through StdRng::seed_from_u64(seed) so two
calls with (shape, device, seed) produce bit-exact identical
tensors. CPU is the canonical generator; GPU calls go via to_device
transfer. Required for reproducible model init.
rtx-transformers: MambaBlock gains:
- new_seeded(config, device, seed) — every internal weight tensor
initialised via randn_seeded() with per-tensor SplitMix64-derived
seeds. Two calls with the same (config, device, seed) → bit-
exact identical block.
- persistence_tensors() -> Vec<(&'static str, &Tensor)> — read-only
view of the six (or seven, with conv_bias) internal weight
tensors with canonical names (in_proj, conv1d_weight,
conv1d_bias?, A_log, dt_proj, out_proj). Stable across versions
so safetensors round-trip works.
- from_persistence_tensors(config, device, HashMap<String, Tensor>)
— rebuild a MambaBlock from a name → tensor map. Validates each
tensor's shape against the config and surfaces clean errors on
mismatch (so wrong-DIM safetensors loads fail explicitly).
These three primitives together give omni-cortex's D249 (operator-
seeded determinism) and D250 (safetensors round-trip + BLAKE3 hash
pin) clean library hooks without exposing MambaBlock's private
fields.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
||
|
osobhandClaude Opus 4.7
|
7e576e8d69 |
rtx-csm: implement LoRA merge + load-from-path
Closes the LoRA inference path that was previously stubbed. Two new public APIs in rtx-csm: 1. lora::load_lora_set_from_safetensors(path, config) -> LoraSet Reads a trained adapter file (produced by training:: save_lora_adapter[_with_metadata]). Pairs the .lora_a / .lora_b tensors by base-weight prefix into LoraAdapter entries. 2. lora::merge_into_safetensors(base, lora, scale, output) Reads the base CSM safetensors, folds in the LoRA deltas at the given scale (typically alpha/rank from training), writes a merged safetensors. Original dtype preserved (F16 on Metal, BF16 on CUDA, F32 on CPU). Tensors LoRA doesn't target are passed through unchanged. 3. Generator::load_csm_1b_from_path(path, device) Variant of load_csm_1b that takes an explicit weights path instead of going through the HF cache. Mimi + tokenizer still resolve via the hub. This is the path consumers use to load a merged checkpoint. MergeReport struct restructured to expose merged/skipped/passthrough counts so callers can verify the adapter actually targeted weights. The previous typed-error test is replaced with a missing-base-file test that exercises the real code path. Used by zeroclaw-channel-voice's `--lora-adapter` flag to bake a LoRA adapter into a per-process merged checkpoint at boot, with zero per-inference overhead. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
osobhandClaude Opus 4.7
|
8a50a1efcd |
rtx-csm: pre-sentence hook + streaming-safe post-process
Two additions for downstream voice-channel prosody / quality wiring:
1. Converse::with_pre_sentence_hook
New PreSentenceHook (Box<dyn FnMut(&mut Generator, &str) -> Result<()>>)
that fires before each sentence's synth call. Receives mutable
access to the underlying Generator + the sentence text — lets
callers apply per-sentence steering (e.g. emotion shifts mid-reply)
without touching crate internals. Wired into both `synthesize` and
`synthesize_streaming` paths.
2. PostProcess streaming split
- New StreamingHpfState — stateful biquad whose IIR taps carry
across chunk boundaries so streaming HPF doesn't click at chunk
joins. Identical filter coefficients to the one-shot path.
- PostProcess::apply_chunk_safe(samples, hpf_state) — HPF + declick
per chunk, no LUFS (needs full utterance).
- PostProcess::apply_lufs(samples, sample_rate) -> Result<f32> —
full-utterance loudness gain, returns the linear gain applied
so streaming pipelines can compensate retroactively if needed.
- compute_lufs_gain helper extracted from loudness_normalize.
Used by zeroclaw-channel-voice for the Maya-gap-closure pack.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
||
|
osobhandClaude Opus 4.7
|
bf5e86c549 |
rtx-csm: Converse::with_context for persistent speaker prompt
Without a voice anchor, CSM-1B picks a different speaker each turn
and drifts mid-sentence on longer outputs (high-pitch squeaks,
female/male swap mid-utterance). The fix is the standard CSM
speaker-prompt pattern: pass a Segment with reference audio + its
transcript as context to every generate() call.
Previously Converse::synthesize and synthesize_streaming hardcoded
`&[]` for the context arg. Add a `context: Vec<Segment>` field on
Converse plus a builder method:
let conv = Converse::new(&llm, &mut gen)
.with_context(vec![Segment::new(0, transcript, audio)]);
Both synth paths now pass `&self.context` instead of `&[]`. Empty
context (default) keeps prior behavior.
Verified end-to-end with zeroclaw-channel-voice + macOS `say`-
generated reference: same input now produces deterministic-length
output across turns (2.64s vs. previously varying 6/19/38s) and the
voice matches the seed throughout.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
||
|
osobhandClaude Opus 4.7
|
fff1b7acd5 |
rtx-csm: converse_server — deprecation note pointing at zeroclaw-channel-voice
The canonical voice loop now lives in zeroclaw-channel-voice (`~/projects/zeroclaw/crates/zeroclaw-channel-voice`, binary `voice_server`). It routes the LLM path through zeroclaw's agent runtime — multi-turn history, tools, memory, provider routing — instead of the OpenAI-compatible direct path here. Same WS wire protocol so `examples/converse_client.rs` drives both; no client-side migration needed. This binary is intentionally kept buildable for: 1. Reproducing perf_history.md Phase 8.10 benches. 2. Standalone (no-agent) use when zeroclaw isn't desired. Module doc + main() startup banner updated to point at the new home. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
osobhandClaude Opus 4.7
|
a5cedfb46a |
rtx-csm: emotional_speech_guide — CREMA-D vs RAVDESS firdhokk verdict
8-gen bench (4 emotions × 2 corpora) at seed=42 against firdhokk Whisper-LV3: target RAVDESS CREMA-D happy happy (0.999) ✓ happy (0.999) ✓ angry neutral (0.92) sad (0.99) fearful happy (0.998) fearful (0.984) ✓ sad angry (0.99) fearful (0.99) CREMA-D 2/4 vs RAVDESS 1/4. Larger / more naturalistic corpus produces more class-pure fearful direction. Neither corpus solves angry or sad — recipe shifts into 'vague expressivity' rather than class-specific corners. Practical: prefer CREMA-D when available; A/B both per emotion if class precision matters. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
osobhandClaude Opus 4.7
|
f4d8268381 |
rtx-csm: scripts/classify_emotion.py — firdhokk SER sidecar
Python sidecar for scoring TTS outputs against the firdhokk
Whisper-LV3 SER classifier (sanity-verified non-saturated, 3/5
correct on RAVDESS ground-truth).
Replaces the in-process emotion2vec_plus_base path which collapses
to 'Surprised' on every input (documented in
emotional_speech_guide.md and quality_eval.rs caveat).
Reads JSONL with {gen_wav, target_emotion} rows; writes JSONL with
top_emotion, top_prob, target_prob, match (bool), and the full
8-class probability distribution.
Class set is firdhokk's 7 (no calm — calm aliases to neutral on
input). Excited aliases to happy.
Smoke-verified on the 4 prior decoder-route outputs (Amini ctx,
seed=42, recipe defaults):
happy → neutral (0.80) ✗
angry → happy (0.999) ✗
fearful → fearful (0.68) ✓
sad → fearful (0.998) ✗ (sad↔fearful confusion)
Top-1 match: 1/4 — confirms the gap documented in
emotional_speech_guide.md 'Known Limitations'.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
||
|
osobhandClaude Opus 4.7
|
8045ba79d4 |
rtx-csm: emotional_speech_guide — firdhokk classifier reveals emotion gap
Wired up firdhokk/speech-emotion-recognition-with-openai-whisper-large-v3 as a working alternative to the broken emotion2vec_plus_base. Sanity verified on real RAVDESS clips: 3/5 correct, 2/5 near-miss (happy↔ surprised, sad↔fearful). Probabilities are NOT saturated — the classifier actually distinguishes per-input. Then scored our 4 decoder-route outputs (Amini context, seed=42, recipe defaults) and found that **only fearful registers as the intended class**: target verdict conf happy neutral 0.80 ✗ (steering produces neutral output) angry happy 0.999 ✗ (high-arousal cross-class) fearful fearful 0.68 ✓ sad fearful 0.998 ✗ (sad↔fearful confusion) Honest framing: the recipe shifts speaker character toward an expressive-sounding direction (cosine evidence) and preserves text (decoder vs backbone) but does NOT produce class-distinct emotion. The metric stack we used through Phase 9 (cosine + WER) couldn't see this gap because it measures voice fidelity and text rendering, not emotion class. Hypothesized fixes (not yet tested): - CREMA-D extraction (91 actors vs RAVDESS 24) for class-purer steering vectors - Mixed backbone+decoder steering (backbone for prosody) - EmoNet classifier (TTS-aware, may give different verdicts) Doc'd in emotional_speech_guide.md as a known limitation. Closes out an honest scientific picture: today's work successfully ports the architectural finding (decoder route preserves text), but class-precise emotion control remains unsolved. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
osobhandClaude Opus 4.7
|
9ebac71784 |
rtx-csm: scripts/build_crema_d_manifest.sh — CREMA-D corpus support
CREMA-D (7442 clips × 91 actors × 6 emotions × 12 sentences) — larger and more naturalistic than RAVDESS (1440 × 24 × 8 × 2). Free, no registration, sparse-cloneable from GitHub. Filename-encoded labels parsed via case statement (bash 3.2 compatible — no associative arrays). Verified: 7442 rows balanced 1271 each of angry/disgust/fearful/ happy/sad + 1087 neutral. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
osobhandClaude Opus 4.7
|
b9c1e3b300 |
rtx-csm: quality_eval — emotion2vec metric (with broken-classifier caveat)
Wires emotion2vec into quality_eval so per-row metrics include
target_emotion_prob, top_emotion, top_emotion_prob. Pure inference,
optional via --emotion2vec / --target-emotion flags.
Critical empirical finding documented in code + user guide: the
emotion2vec_plus_base checkpoint classifies every input as
"Surprised" with prob ≥ 0.99, INCLUDING ground-truth RAVDESS clips
with explicit emotion labels. Real angry-RAVDESS → "Surprised"
(0.9999999). Real neutral-RAVDESS → "Surprised" (0.9999996).
The metric implementation is correct (matches the trait's
EmotionDetector::classify code path with same per-utterance zero-
mean unit-variance normalization); the underlying classifier
collapses to a dominant class on most input — likely the same
"9→5 fold collapse" the project already documented in the data-
labeling path.
Practical implication: target_emotion_prob is near-zero for almost
every (target, output) that isn't "surprised", so it can't be used
as a picker score. The emotion2vec metric still works as a
diagnostic ("did the model produce something that classifies as
audio at all?") but not as a generation-quality validator.
Doc'd in:
- examples/quality_eval.rs CLI doc (caveat block on --emotion2vec)
- docs/emotional_speech_guide.md (Known limitations section with
full sanity-check table)
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
||
|
osobhandClaude Opus 4.7
|
cc4c84f2cd |
rtx-csm: emotional_speech_guide — document composite picker score
Brief addition to the recipe section explaining the WER + length
floor scoring used by emotional_speech_n.sh (committed in
|
||
|
osobhandClaude Opus 4.7
|
b7b267b701 |
rtx-csm: emotional_speech_n.sh — composite score with length floor
Original picker used min(WER), tie-broken by max(cosine). Fragile:
on emotion=surprised it picked "You can." (2 words, WER 0.93) over
"...Today I want to share something" (13 words, WER 1.00) because
WER weights all errors uniformly — terse-and-mostly-wrong beats
long-and-mostly-right.
New scoring:
score = WER + (1.0 if words(transcript) < 5 else 0)
sort_by(score, -cosine)
Verified on existing benches:
surprised: now picks seed=100 ("...Today I want to share something
with...", 13 words, score 1.0) over seed=7 ("You can.",
2 words, score 1.929 with +1 length penalty).
calm: still picks seed=100 (full transcript revealed: "It's a
good reflection. Not that that. I want to share
something with you that I've been thinking about." —
near-verbatim! the earlier 55-char display had been
truncating it).
disgust: all 3 candidates score ~1.93 (no seed has > 5 words,
all get the length penalty); picker honestly admits
none is good rather than picking a fake winner.
Worth noting: the calm seed=100 case is ANOTHER near-verbatim
single-shot result we missed in the previous bench because the
display truncation hid the full transcript content.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
||
|
osobhandClaude Opus 4.7
|
8190b26e93 |
rtx-csm: emotional_speech_guide — N-seed bench for new 3 emotions
3-seed picker run (7, 42, 100) on calm/disgust/surprised refines the
single-shot characterization:
calm: winner seed=100, WER 0.64
"It's a good reflection. Not that that. I want to
share somet..."
(recipe lands the prompt — single-shot at seed 42 only
produced hesitation markers; the picker found a seed
with actual content)
disgust: no reliable seed
(all 3 seeds WER ≥ 0.93; likely RAVDESS corpus issue —
disgust clips are low-energy / acoustically close to
neutral. Try CREMA-D or ESD for this emotion.)
surprised: picker chose seed=7 (WER 0.93, short "You can.") over
seed=100 (WER 1.0, "...Today I want to share something")
— WER weighting issue: deletions and insertions count
uniformly, so terse-but-mostly-wrong beat long-and-
mostly-right. Manual selection or weighting WER less
heavily would help here.
Updated per-emotion table marks disgust as ✗ (corpus limitation),
surprised as ⚠ (picker scoring artifact), calm as ✓ (works with
N-seed picker).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
||
|
osobhandClaude Opus 4.7
|
e6f8d77328 |
rtx-csm: 7-emotion RAVDESS pack — calm/disgust/surprised benched
Extracted decoder steering vectors for the remaining 3 RAVDESS
emotions (calm, disgust, surprised). Single-seed bench at
seed=42 on Amini context, decoder route, recipe defaults:
emotion cos_ctx WER transcript
calm 0.70 0.86 "I'm sorry. Um, I don't know."
(natural hesitation markers — the
recipe produces semantically-emotion-
matched content, not just acoustic
shift)
disgust 0.81 0.93 "For that, that..." (truncated)
surprised 0.95 2.57 "Too couple, sorry, and that's saying,
even a premier and super driver..."
(long rambling; voice migrates well,
text drifts)
All 7 RAVDESS emotions now produce coherent English on the decoder
route — calm is solid first-shot, disgust truncates, surprised
rambles. Roll N seeds via emotional_speech_n.sh for the latter two.
emotional_speech_guide.md updated with the per-emotion table now
covering all 7. Voice character preservation (cos vs context > 0.7)
holds for every emotion in the pack.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
||
|
osobhandClaude Opus 4.7
|
f022e14f34 |
rtx-csm: docs/emotional_speech_guide.md — user-facing handbook
Phase 9 produced 25 commits and a complex pipeline; perf_history is the engineering log but new users coming to this cold need a clean "how do I make CSM speak with emotion" handbook. Sections: - What this gets you (single-shot WER 0.07–0.21, voice cosine ≥ 0.95) - One-liner quickstart (RAVDESS download → extract → use) - The recipe explained — every flag and why it's there - Per-emotion notes (works/best-seed/caveats per emotion) - When it works / when it doesn't - Troubleshooting (music tokens, premature EOT, repetition, etc.) - Architecture cheat sheet (backbone=semantic, decoder=acoustic) References perf_history.md for the full empirical log; this doc is the user-facing distillation. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
osobhandClaude Opus 4.7
|
1b0205f098 |
rtx-csm: emotional_clone.sh — URL → emotional voice clone in one command
Capstone consumer interface composing every piece shipped today: fetch_audio.sh → audio_to_manifest → pick_context.sh → emotional_speech_n.sh (N-seed picker, decoder route) Caches fetch + manifest by URL hash so re-runs with the same --workdir skip the slow steps. Defaults to the Phase 9 recipe: target=decoder, scale=1.0, layers [2,3], cfg=linear:3.0:1.0:25, 5-seed roll with the lowest-WER winner picked. End-to-end smoke test (cached Carlini source, prompt "Today I want to share..."): picker auto-selected: nicholas_carlini...spk0.0078.wav (10.78 s) seed 42 (winner): cos 0.974, WER 0.143 ⭐ "But Jason, today I want to share something with you that I h" seed 100: cos 0.986, WER 0.286 "It ties upon a share something with you that I have been thi" seed 7: cos 0.862, WER 1.000 "Let me think, let him out." The auto-picker chose spk0 (Carlini himself) where manual selection earlier in the day grabbed spk1 (the announcer) — so the automated pipeline is also a slight context-selection improvement. Three sub-second-WER results recorded over the day: - WER 0.071 Amini imperative prompt (manual) - WER 0.125 Amini original prompt (manual) - WER 0.143 Carlini auto-picked spk0 (this commit, end-to-end) Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
osobhandClaude Opus 4.7
|
6e22507994 |
rtx-csm: perf_history — per-emotion seed × context interaction
Cross-emotion × Carlini context at Amini-best seeds reveals the
"magic combo" doesn't fully transfer:
happy@42 Amini WER 0.21 → Carlini WER **0.071** (transfers!)
angry@100 Amini WER 0.93 → Carlini WER 1.21 (URL drift)
fearful@7 Amini WER 0.86 → Carlini WER 1.00 ("Screw it")
sad@7 Amini WER 0.93 → Carlini WER 1.00 (no transcript)
Only happy@42 cleanly generalizes across contexts. The previous
"context-robust" claim was too strong — the (emotion, seed, context)
interaction matters. Cosine vs context stays high for angry (0.95)
even when text drifts, so voice character preservation is the more
robust property than text fidelity.
Honest production interface: `emotional_speech_n.sh` rolling 5 seeds
per (context, prompt). The single-shot recipe lands well only when
all dimensions align, but the picker absorbs the variance.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
||
|
osobhandClaude Opus 4.7
|
48c9fdd5f2 |
rtx-csm: perf_history — cross-context validation, WER 0.071 new best
3-context bench at happy@seed=42 decoder recipe (Amini / MCC /
Carlini). Cosine measured vs the context wav (does recipe preserve
input voice character?), WER vs prompt:
context cos_ctx WER transcript
amini 0.972 0.21 "All right, today I want to share
something with you that I've been
thinking about."
mcc 0.58 0.93 "You" (sub-speaker mismatch)
carlini 0.958 0.071 ⭐ "So today I want to share something
with you that I have been thinking
about."
Carlini's WER 0.071 is the new single-shot best of Phase 9. Only
prefix "So" added to the verbatim prompt. Cos vs context > 0.95 on
the two working contexts means the recipe preserves speaker
character of the reference — does NOT impose RAVDESS speaker
identity on every output.
The recipe is context-robust on speaker identities the picker
selects correctly. McConaughey failed because we picked the
manifest's spk1 (likely the Oscars announcer), not McConaughey
himself. That's a context-selection issue, not a recipe issue.
Empirical capstone: single-shot near-verbatim emotional speech
with preserved voice character is achievable.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
||
|
osobhandClaude Opus 4.7
|
d0ab272804 |
rtx-csm: perf_history — cross-prompt validation, WER 0.125 best of day
3-prompt × 2-condition bench at happy@seed=42 decoder recipe vs no-steering baseline. The recipe holds across prompts and produces the lowest WER recorded across all Phase 9 experiments: prompt base happy decoder Today I want to… 0.52 / 0.93 "You" 0.82 / 0.21 ⭐ Have you ever… 0.72 / 1.39 drift 0.77 / 1.00 "With blames" Weather has been… 0.73 / 1.88 ♪♪♪ 0.69 / 0.125 ⭐ "That the weather has been absolutely beautiful this mor" The imperative prompt's WER 0.125 is the lowest recorded. Improvement vs baseline ranges 1.4× to 15× lower WER. The no-steering baseline produced literal singing tokens (♪♪) on the weather prompt, suggesting CSM's CFG-only path is fragile on prompts the model "dislikes." Empirical conclusion: decoder route + RAVDESS happy steering at seed 42, scale 1.0, layers [2,3] is a reproducible recipe, not a single-prompt anomaly. N-seed picker still the right consumer interface, but this single configuration alone reaches near-publishable quality on multiple prompts. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
osobhandClaude Opus 4.7
|
942de8ef49 |
rtx-csm: perf_history — decoder 4-emotion × 3-seed capstone
Records the breakthrough single-shot result: happy@seed=42, decoder route, scale 1.0, layers [2,3] → "All right, today I want to share something with you tha" (WER 0.21, cos 0.82). Closest-to-perfect single-condition result of the entire Phase 9 sprint. Per-emotion seed winners diverge: happy=42, angry=100, fearful=7, sad=7 No universal best seed exists; this validates emotional_speech_n.sh as the production interface (rolls multiple, picks lowest WER). Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
osobhandClaude Opus 4.7
|
2e50af3a28 |
rtx-csm: emotional_speech.sh — decoder route + sad finally works
Wrapper now supports --target backbone|decoder. Decoder is the
default (Phase 9 finding: it preserves text fidelity that backbone
destroys). Per-(target, emotion) defaults:
decoder: scale 1.0, layers [2,3] (last two of 4)
backbone: scale 0.2-0.3 (per-emotion), layers [8,10,12]
emotional_speech_n.sh's existing passthrough already forwards
--target through to this wrapper unchanged.
Bench all 4 emotions on the decoder route, seed 7, recipe defaults:
emotion cos WER transcript
happy 0.65 0.93 "I've been happy cycling and beat..."
angry 0.78 2.29 "And of course, coming first, Vern is..."
fearful 0.73 0.86 "I'm not eye sensing when that's a mile."
sad 0.67 0.93 "- I'm actually off my night. I'll take
something. - All right..."
Sad — the previously-unsolvable emotion on the backbone (model
resisted at every tested scale 0.15-0.3) — produces real fluent
English on the decoder route. The word "happy" surfaces in the
happy output. All 4 emotions produce coherent speech: no music
tokens, no premature EOT, no gibberish. WER stays in the 0.86-2.3
range, comparable to baseline-with-CFG.
The decoder route subsumes everything the backbone route was
trying to do and unlocks the failure case it couldn't reach.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
||
|
osobhandClaude Opus 4.7
|
f642132f4a |
rtx-csm: --decoder-steering-layers — layer subset fixes repetition
Mirror of --steering-layers for the decoder: comma-separated layer indices to actually steer (decoder has 4 layers; useful subsets are [3], [2,3], [1,2]). Sweep at seed 7, [email protected]: [0,1,2,3] cos 0.86 WER 0.93 "I'm not that tall ×3" ← repetition [3] cos 0.58 WER 0.86 "I'll be off and offense..." [2,3] cos 0.80 WER 1.43 "My daughter, Penny Ryan, and I have…" [0,1] cos 0.82 WER 1.57 "And I'll check on them..." [1,2] cos 0.81 WER 1.43 "I'm going to call him an X-Man..." The repetition is specific to all-layers-at-once steering. Any 2-layer subset eliminates it while preserving most of the cosine boost. Same pattern as the backbone's [8,10,12] finding: partial perturbation lets the unsteered layers act as a stabilizing prior. [2,3] (decoder last 2) is the new recommended recipe — best cosine of the no-repetition subsets and the longest fluent transcript. Documented in docs/perf_history.md. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
osobhandClaude Opus 4.7
|
68ecae1b08 |
rtx-csm: perf_history — decoder steering empirical characterization
3-seed × 2-condition ([email protected] alone vs +CFG) bench plus a 4-step scale sweep. Captures the honest tradeoff: - Backbone steering destroys word content (semantic gibberish). - Decoder steering preserves coherent English BUT produces repetition or premature EOT. Neither produces single-shot production-quality emotional speech; emotional_speech_n.sh (N-seed picker, lowest-WER wins) remains the right consumer interface — it doesn't care which failure mode generated the bad samples, just discards them by metric. Decoder vector magnitudes are ~10× smaller than backbone (norm 0.85 at deepest layer vs 14.9), so the apparent useful scale window is ~10× higher (0.5-1.0 instead of 0.2-0.3). Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
osobhandClaude Opus 4.7
|
99a7e53aa8 |
rtx-csm: decoder activation capture — architectural hypothesis validated
Adds capture_decoder_activations on Model and ModelBackend, plus
--target-module decoder|backbone on examples/steering_extract.
Decoder mode runs a text-only prompt through the backbone (no
capture), Mimi-encodes the audio separately to grab the middle
frame's c0 token, teacher-forces that c0, and captures one mean-
pooled-over-seq vector per decoder layer. Result: 4 layers ×
1024 embed dim per call, much faster than backbone capture
(text-only prompts are short).
A/B with the canonical "Today I want to share..." prompt at seed
7 (the previously-identified low-WER seed):
case cos WER transcript
baseline 0.76 0.36 "And today I want to share something some
funnel distraits" (high baseline at this
seed)
[email protected] 0.75 5.00 "Today, I want to share some needs of my
prey" (backbone destroys content)
[email protected] 0.86 0.93 "I'm not that tall. I'm not that tall."
(fluent but repetitive — biggest cos)
[email protected] 0.61 2.57 over-steered
[email protected] 0.61 2.14 broken
[email protected] is the largest speaker_cosine boost we've measured AND
produces clean English. Backbone steering at the same seed destroyed
content fidelity. This validates the architectural hypothesis: the
backbone carries semantic content (what the model says), the depth
decoder carries acoustic detail (how it sounds). Steering the
decoder shifts voice character without disturbing word content the
way backbone steering does.
Open issues: [email protected] produces repetitive output ("I'm not that
tall" three times). Likely lower scale (~0.5) plus the existing
repetition guard would fix it; left for follow-up.
Decoder vector magnitudes are ~10× smaller than backbone (norm 0.85
at deepest layer vs 14.9), so the appropriate scale is ~10× higher
than the backbone recipe (1.0 vs 0.1-0.3).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
||
|
osobhandClaude Opus 4.7
|
a1fa72d151 |
rtx-csm: depth-decoder steering API
Adds set_decoder_steering on Model + Generator and --decoder-steering-vec / --decoder-steering-scale on examples/generate. The decoder is already a LlamaModel under the hood, so the existing LayerSteering hook in csm_fork::Layer::forward applies as-is — only the public surface needed wiring. Architectural hypothesis being tested: backbone carries semantic content (what the model says), depth decoder carries acoustic detail (how it sounds). Backbone steering shifts character at the cost of text fidelity (Sprint 2 finding); decoder steering should shift prosody/timbre without disturbing word content. Smoke test with random Gaussian decoder vectors (4 layers × 1024 embed_dim, stddev 0.1, scale 0.5): case cos WER transcript baseline 0.72 1.0 "No." backbone 0.83 1.4 "That's for on-beat for bee..." decoder 0.76 1.0 "So" (premature EOT) both 0.81 3.0 "I'm going to go to the next one..." Decoder steering DOES alter output (cosine 0.72 → 0.76, transcript changes) but random vectors trigger premature EOT — same pattern as random backbone vectors. The infrastructure works; getting the real emotion-from-acoustic-codebooks signal needs decoder activation capture, which the current Model::capture_backbone_activations doesn't do (it captures the backbone forward only). Decoder capture is the next-session item. With it we can extract real per-emotion decoder vectors from RAVDESS and test the hypothesis properly. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
osobhandClaude Opus 4.7
|
5a9d37bf4f |
rtx-csm: emotional_speech_n.sh — N-seed pick-best wrapper
Productionizes yesterday's seed-variance finding. Wraps emotional_speech.sh, rolls a list of seeds, scores each via quality_eval (speaker_cosine + Moonshine WER), and copies the lowest-WER candidate to --out. Defaults to 5 seeds; pass --seeds 42,7 for cheaper runs. Tie-breaking is `min(WER), then -max(cosine)` — text fidelity takes precedence over speaker character because user-typed text should be rendered verbatim, while voice character is only secondary on top of context conditioning. Failed generations (short clips that get the -1 cosine sentinel) sort to the bottom. Smoke run on the canonical "Today I want to share..." prompt: seed 7 → cos 0.845, WER 0.714 "Today, today I want to share..." ← picked seed 100 → cos -1, WER 1.000 "It is." (premature EOT) seed 42 → cos 0.916, WER 2.000 "The police are, if you're..." (drift) Cost: N × single-shot cost. The recipe being unreliable per-seed is the whole reason this wrapper exists — pay the multiplier in exchange for a reliably-best output. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
osobhandClaude Opus 4.7
|
406c437d70 |
rtx-csm: perf_history — recipe seed variance caveat
3-seed × 3-prompt reproducibility bench on [email protected] reveals that the recipe shifts speaker character reliably but produces high text- fidelity variance: seed 42: WER 2.00 "The police are, if you're, I can't recite..." seed 7: WER 0.71 "Today, today I want to share..." ← near-verbatim seed 100: 0.32 s premature EOT Cross-prompt at seed 42 drifts uniformly across 3 prompts. Speaker cosine is consistently elevated; text content is roll-the-dice. Documenting this as the honest characterization rather than overclaim the single-seed Sprint 2 results. Practical recipe: roll N seeds, pick lowest-WER output. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
osobhandClaude Opus 4.7
|
2008c9c4a6 |
rtx-csm: emotional_speech.sh — per-emotion scale defaults
Capstone wrapper for the Phase 9 recipe (Selective CFG + RAVDESS
steering + mid-layer subset). Picks the steering scale automatically
from an empirical per-emotion map:
- happy: 0.30
- angry: 0.20
- fearful: 0.20
- sad: 0.20 (note: sad is unreliable — see below)
These came from a follow-up sweep after the multi-emotion demo
revealed the recipe is emotion-sensitive: scale 0.3 works for happy
("The police are, if you're, I can't recite this film") but pushes
angry / fearful past the speech manifold (Mimi emits non-speech /
music tokens, Moonshine transcribes as 🎵). Dropping to 0.2 recovers
fluent speech for both:
- [email protected]: "The next disorder is completing kashim for more."
- [email protected]: "You just heard a little bit about this decision,
though."
- [email protected]: "The police are, if you're, I can't recite this
film. I"
Sad is the outlier — model resists "sad" steering at every scale
between 0.15 and 0.3. Likely a corpus issue (sad RAVDESS clips are
the lowest-energy subset). Documented as a known limitation rather
than worked around.
perf_history.md updated with the per-emotion sensitivity finding.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
||
|
osobhandClaude Opus 4.7
|
29f4b6fc56 |
rtx-csm: CFG schedule × prompt sweep — robust default is linear
Sweep of 4 schedules × 4 prompts produced concrete data on schedule sensitivity. step:3.0:1.5:12 (the Sprint 3 winner) catastrophically fails dense prompts: lecture-style input → 0.08 s of audio (one frame). step:2.0:1.0:8 produced the best single shot — near-verbatim question rendering "Well, it had stem-wondered. Have you ever wondered why we sometimes hear voices the way we do?" — but tanked the lecture prompt (2.4 s "You"). linear:3.0:1.0:25 is the only schedule that's never the best AND never the worst — graceful degradation across all four prompt categories. Updates the recommended recipe in perf_history.md (formerly step:3.0:1.5:12). quality_eval: skip WavLM-SV scoring on clips shorter than 0.25 s (emit -1 sentinel) — WavLM-SV's TDNN front-end requires a few hundred samples and crashed mid-sweep on the 0.08 s clip. Now the eval emits a row instead of bailing on the whole batch. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
osobhandClaude Opus 4.7
|
38f586cfa3 |
rtx-csm: docs/perf_history — Phase 9 emotion/quality/control summary
Records all of today's three-paper sprint: Sprint 1 (eval foundation), Sprint 2 A/B + follow-ups (steering apply, extract, layer-subset, RAVDESS), Sprint 3 (Selective CFG). Captures the composition finding (step CFG + RAVDESS mid-layer steering @ scale 0.3 → speaker_cosine 0.846, largest cross-character migration we've measured) and the recommended invocation recipe. Documents what was rejected and why (TTSDS2 install hell, EmoSteer's flow-matching-specific algorithm). Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
osobhandClaude Opus 4.7
|
808e79dc40 |
rtx-csm: RAVDESS-derived steering — first real emotion signal
Adds a clean labeled-emotion data path now that the auto-tagged
manifests have proven inadequate for steering extraction.
scripts/build_ravdess_manifest.sh: parses the RAVDESS speech-only
distribution (Audio_Speech_Actors_01-24.zip from Zenodo, 208 MB) into
a manifest.jsonl with proper {neutral, calm, happy, sad, angry,
fearful, disgust, surprised} labels and the two canonical statements
("Kids are talking by the door", "Dogs are sitting by the door"). 1440
clips, balanced 192/emotion (96 neutral — RAVDESS lacks the 'strong'
intensity for neutral).
examples/steering_extract Mimi reload-every-10: the streaming state
counter overflows 8192 frames after ~80 encodes even with
reset_state(). Same fix training/audio_to_manifest/converse_server use
(commit
|
||
|
osobhandClaude Opus 4.7
|
af6c246ef6 |
rtx-csm: layer-subset steering — recovers Sprint 2 fluency
EmoSteer-TTS (arXiv 2508.03543) targets only a spaced subset of
middle-to-deep DiT layers (1, 6, 11, 16, 21 of 32) rather than every
layer. Previously our LayerSteering applied all 16 vectors which —
combined with the noisy emotion-label corpus from Phase B — destroyed
output fluency at scale 0.5.
Adds:
- LayerSteering::restrict_to_layers(&[usize]) — clears every vector
whose index isn't in the allowlist. Plus active_layers() inspector
and a unit test.
- examples/generate --steering-layers 8,10,12 — comma-separated CLI
flag that runs restrict_to_layers after load.
A/B with the existing excited-vs-surprised vectors at scale=0.5
across five layer subsets:
case cos WER transcript
baseline 0.45 0.92 "It is a very important thing to do."
all16 0.37 1.00 "© transcript Emily Beynon" (broken)
[8,10,12] 0.63 0.54 "I want to talk about something." ✓
[4,8,12] 0.42 1.00 "Oh, my God." (broken — layer 4 too early)
[12-15] 0.48 0.77 "I want to have fun with that."
The mid-layer subset is the clear winner — highest speaker cosine,
lowest WER, transcript closest to the prompt ("Today I want to talk
about something genuinely important..."). Including layer 4 destroys
output fluency even at scale 0.5, validating the paper's avoidance of
shallow layers. Pure-deep is between mid and broken.
This unblocks Phase B's empirical validation: even with the noisy
auto-tagged corpus, the extracted vectors produce meaningful steering
when applied to the right layers. A real labeled emotion dataset
should compound from here.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
||
|
osobhandClaude Opus 4.7
|
6b69fb68c7 |
rtx-csm: Sprint 3 — Selective CFG schedule (step / linear / const)
Per-frame CFG scale schedule (arXiv 2509.19668, Zheng & Maleki). Pure
inference-time. Standard CFG uses one fixed scale for the whole
sequence; this lets the scale vary across frames so early frames
(speaker character) get full CFG and later frames (text adherence)
get a lower scale.
What lands:
- src/cfg_schedule.rs: CfgSchedule enum (Constant / Step /
LinearRamp), scale_at(frame_idx), parser for CLI form
`step:E:L:T | linear:S:E:R | const:X`. 6 unit tests.
- src/generator.rs: GenerateOptions::cfg_schedule (takes precedence
over legacy cfg_scale; fixed-f64 path is preserved as
Constant(s) for back-compat). Generation loop reads
schedule.scale_at(frame_idx) and passes per-frame to
generate_frame_cfg.
- examples/generate.rs: --cfg-schedule, --cfg-scale, --enable-cfg
flags. Loading via load_csm_1b_with_cfg when --enable-cfg.
A/B with 6s output on Amini context, prompt about Selective CFG:
case cos WER transcript
no-CFG baseline 0.944 1.50 "Okay, the M.U. worked..." (off)
const:2.0 0.854 0.92 "On the right side." (short)
step:3.0:1.5:12 0.938 1.00 "The officer for the selective
C.F.D. paper recommends" (best)
linear:3.0:1.0:25 0.854 1.08 "On the surface..." (off)
Step schedule produces the transcript closest to the input ("the
selective CFG paper recommends..."). WER stays at 1.0 because
Moonshine doesn't know "CFG" as a word, but qualitatively this is
the only one that's coherently following the prompt. Speaker cosine
stays ≈ baseline (0.94) instead of dropping to 0.85 like the
constant and linear cases.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
||
|
osobhandClaude Opus 4.7
|
3a67e4aa50 |
rtx-csm: Sprint 2 Phase B — steering vector extractor
Closes the loop on Phase A (apply hook). Adds:
- LlamaModel capture buffer + start/take_capture API. Pushes one
mean-pooled-over-seq activation per layer into a per-call Vec when
active. Steering apply runs first, so captures reflect post-steering
state when both are on (extractor disables steering for the duration
of the call to capture baseline activations).
- Model::capture_backbone_activations: teacher-forced forward over a
built prompt, returns per-layer (embed_dim,) activation tensors.
- ModelBackend passthrough; FP-only (Quantized errors out).
- examples/steering_extract: reads emotion-labeled JSONL, accumulates
per-emotion sums on CPU f32, writes per-layer
(mean(target) - mean(baseline)) as `layer_<i>_steering` safetensors.
Smoke run on carlini2 manifest (excited vs surprised, 10 samples each):
- Vector norms grow monotonically with depth (layer 0: 2.25, layer 15:
12.61) — consistent with deeper layers carrying richer
emotion/style signal.
- Loaded into examples/generate at scales 0.5/1.0/2.0; quality_eval
shows WER hits 1.0 immediately. This is the EmoSteer paper's warning
("large α may produce unintelligible speech") triggering at small
α — diagnosis: the corpus is the problem, not the infrastructure.
The emotion_tag labels in our existing manifests are noisy
(emotion2vec output on lecture audio collapses to [surprised] /
[excited] without a clean neutral pool), and 10 samples per pool
is well short of the paper's 1000/emotion.
What this validates:
- End-to-end extraction → save → load → apply pathway works.
- quality_eval (Sprint 1) cleanly catches the regression — the metric
foundation does its job.
What's next (a future session):
- Real emotion-labeled dataset (CREMA-D, ESD, RAVDESS) for proper
pools with a true neutral baseline.
- Layer-subset experiments (paper steers layers 1,6,11,16,21 of 32;
for our 16-layer backbone the analogue is roughly 1, 4, 8, 12).
- Listening test alongside the metric numbers.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
||
|
osobhandClaude Opus 4.7
|
aa274f2210 |
rtx-csm: Sprint 2 Phase A — activation steering API
Adds the apply hook for ActAdd-style activation steering on the Llama
backbone. Inspired by EmoSteer-TTS (arXiv 2508.03543), but adapted: the
paper is flow-matching-specific (DiT layers, 32 CFM steps, per-token
attribution search via mel synthesis), none of which apply to CSM's
autoregressive Llama-over-Mimi-tokens. What's portable is the
underlying difference-in-means construction with residual-stream
addition — the standard ActAdd / contrastive-steering pattern.
What lands:
- src/steering.rs: LayerSteering type, per-layer (1, embed_dim) tensors,
global scale, safetensors load with keys `layer_<i>_steering`. Three
unit tests covering empty/no-op, dimension validation, and apply math.
- src/csm_fork.rs LlamaModel: optional `steering: Option<LayerSteering>`
field, applied after every layer's forward inside the for-loop. Adds
~3 LOC to the hot path; gated by the Option so unsteered generation
has zero cost beyond a None check.
- src/csm_fork.rs Model::set_backbone_steering: installs steering only
on the conditional backbone (cfg_backbone is intentionally left
un-steered so CFG correctly subtracts an unsteered baseline).
- src/generator.rs Generator::set_steering: errors on quantized
backend (only FP supported for now).
- examples/generate.rs: --steering-vec / --steering-scale flags.
- examples/steering_random.rs: smoke helper that writes random Gaussian
vectors so the apply path can be exercised end-to-end before the
real corpus extractor lands. Box-Muller via seeded rand to avoid an
extra rand_distr dep.
Smoke test (16-layer random Gaussian, stddev=0.05, scale=0.5):
- baseline (no steering, same seed/text): 3.04 s @ RMS -19.5 dB
- steered (random vectors): 1.84 s @ RMS -16.2 dB,
EOT triggered earlier
Output clearly differs — pathway is wired correctly. Random vectors
aren't musically meaningful; that's Phase B.
Phase B (next session): corpus extractor that runs forward passes over
emotion-labeled audio (we already have audio_to_manifest emitting
emotion_tag rows), captures per-layer post-residual activations, and
computes the difference-in-means between emotion_X and neutral pools.
Then A/B with quality_eval.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
||
|
osobhandClaude Opus 4.7
|
62d360ba90 |
rtx-csm: Sprint 1 eval foundation — quality_eval + I2D loop
Sprint 1 of the post-research roadmap. TTSDS2 (arXiv 2506.19441) was
the original target but its install is broken on Python 3.12 + modern
torchaudio (deprecated `torchaudio.sox_effects`, `pyannote.audio` 3.1
calls removed `set_audio_backend`, `openai-whisper==20240927` needs
`pkg_resources`). Pivoted to a Rust-native foundation we already own
end-to-end: WavLM-SV + Moonshine + amplitude.
`examples/quality_eval` consumes a JSONL of `(ref_wav, gen_wav,
ref_text)` rows and emits per-row metrics:
- speaker_cosine via WavLM-SV (microsoft/wavlm-base-plus-sv)
- wer via Moonshine v2 transcript vs ref_text (Levenshtein on
lowercased / punctuation-stripped tokens)
- gen_peak_db, gen_rms_db (full-band amplitude of gen_wav)
`scripts/i2d_loop.sh` implements I2D (arXiv 2603.24430): synth N
times feeding each output back as the next iteration's context, score
all iterations with quality_eval, emit a TSV degradation curve.
Smoke-tested:
- quality_eval on the picker A/B set independently confirms the
picker — bottom-context (score 0) → WER 0.55, top-context
(score 2.0) → WER 0.18 (3× worse without picker filter).
- i2d_loop with 3 iterations on Amini context shows clean
collapse: cos 0.84 → 0.58, WER 0.5 → 1.0 by iter 1.
Foundation for Sprint 2 emotion-steering A/B comparisons.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
||
|
osobhandClaude Opus 4.7
|
fac338ad74 |
rtx-csm: scripts/clone_voice.sh — URL → cloned utterance wrapper
End-to-end voice clone: fetch_audio.sh → audio_to_manifest → pick_context.sh → examples/generate. Caches each step by URL hash so re-runs with the same --workdir skip the slow fetch and diarize/transcribe stages. Smoke-tested with cached fetch + manifest. Picks the highest-scoring context clip across all (manifest, speaker) groups, hands it plus the target text to generate via the new repeatable --context-wav / --context-text pairs. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
osobhandClaude Opus 4.7
|
adc9784646 |
rtx-csm: Stt::finish() — drain asr_delay buffer at end-of-stream
Phase 8.1.1 quality fix from the perf plan. Tokens emitted at LM step
`t` correspond to audio frame `t - ASR_DELAY_FRAMES` (6 frames /
0.48 s), so when a caller stops feeding audio without trailing
silence the last few words trail off — they're still inside the
delay pipeline.
finish() now steps ASR_DELAY_FRAMES additional silent frames after
handling any partial sub-frame buffer, giving the LM the chance to
emit those buffered tokens. Cost: 7 extra step_pcm calls per turn.
Verified end-to-end via stt_demo on a mid-utterance trim of the
LibriSpeech reference clip:
pre-flush: 11 words ("...turnips and carrots and bruised")
post-flush: 13 words ("...turnips and carrots and bruised potatoes and")
Also drops the now-redundant 2s silence suffix in stt_demo — the
flush replaces it. Affects converse_server's real-time end-of-turn
path where suffix padding wasn't possible.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
||
|
osobhandClaude Opus 4.7
|
9317047ae6 |
rtx-csm: pick_context.sh — linear peak penalty (A/B fix)
A/B test 2026-04-29 across mit_2024intro / mcc / carlini manifests showed the binary peak threshold (≤ -3 dBFS = +0.5) failed to differentiate hot clips against each other: an mcc clip with input peak=-1.47 dBFS scored same as one at -3.5 dBFS, and the model output tracked input amplitude. Replace with a linear penalty: 0.5 at peak ≤ -9 dBFS, ramping to 0 at peak = 0 dBFS, clamped. mcc spk0 now produces graduated scores (1.63 / 1.58 / 1.56 / 1.51) instead of a 1.5 plateau, reordering the top selection toward the cleaner-peak clip. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
osobhandClaude Opus 4.7
|
1ae8bfc74f |
rtx-csm: voice-cloning UX — context picker + multi-clip context
scripts/pick_context.sh ranks manifest clips by suitability for CSM-1B's --context-wav conditioning (duration sweet spot 10-13.5s, RMS -25 to -15 dB, peak ≤ -3 dBFS) and groups by (manifest_stem, speaker_id) since diarizer labels are per-file. examples/generate.rs now accepts repeatable --context-wav and --context-text pairs, zipped into Vec<Segment> for Generator::generate. Validates equal counts at runtime. Smoke-tested with two 10-13s spk0 clips from the MIT-2024 manifest. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
osobhandClaude Opus 4.7
|
0568dd3653 |
rtx-csm: drop Mimi reload cadence to every-10 clips
Empirical: with 25-clip cadence the cumulative state still tipped over once at position 8102 mid-window during a 167-clip lora_eval load. Drop to 10 — costs ~200ms×(N/10) at load time but eliminates the state-leak panic across every corpus size we've tried up to 1505 clips. Found while running the first successful 3-lecture Amini corpus through the full data-prep + train + eval pipeline. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
osobhandClaude Opus 4.7
|
d42aba0f1b |
rtx-csm: emotion2vec input normalization + 9-class direct mapping
Three real bugs found while running a YouTube → train → eval pipeline end-to-end on real corpora: 1. emotion2vec was producing near-constant logits regardless of input. Per config.yaml `normalize: true` — data2vec2/emotion2vec expects per-utterance zero-mean unit-variance normalization on the raw waveform before the local_encoder. Added inside the EmotionDetector trait impl so all callers get it. Verified empirically: 4 different audio inputs (Carlini talk, audience question, McConaughey speech) now produce different argmax classes. Before fix: all 4 produced identical logits. 2. The 9→5 emotion fold was collapsing every real-world clip to [excited]. happy / surprised / other all mapped to Excited covered ~95% of natural speech. Replaced with a direct 9-class identity mapping; EmotionLabel gained Disgusted, Fearful, Happy, Surprised, Unk variants. Now: 132 [surprised] + 12 [excited] across the Carlini corpus instead of 144 [excited]. 3. lora_train_emotional --peak-lr / --epochs flags. The canned 3-stage recipe over-fits on small (~100 clip) corpora at extended rank 8; users need to tune. (The recipe stays as defaults; flags are pure overrides.) Plus diagnostic: examples/emotion2vec_probe — feed real audio files into emotion2vec and dump per-class logits. Used to find bug #1. Lib suite still 131/131 (the test that locked the 9→5 fold updated to lock the new identity mapping). Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
osobhandClaude Opus 4.7
|
b4a7133ffb |
rtx-csm: Mimi.reload() — workaround state-leak in cumulative encode loop
Mimi's transformer carries an internal position counter across encode()
calls that the upstream `reset_state()` does NOT fully clear. When
TrainingDataset::load_from_manifest encodes ~80-100 clips back-to-back
during data prep, the counter overflows the 8192-position buffer and
panics with `narrow invalid args [8192, 32]`.
src/mimi.rs:
- cache the safetensors path on Mimi at construction
- new Mimi.reload() drops the inner Model and rebuilds from the
cached path (~200 ms on Metal)
src/training.rs:
- call generator.mimi.reload() every 50 clips during
load_from_manifest. Adds ~1 s overhead on a 200-clip corpus
(4 reloads × ~200 ms) vs the alternative of a hard panic.
- reset_state() before each encode in TrainingExample::from_audio
is kept (still useful to clear streaming chunk state).
Found while running the end-to-end personal-voice training pipeline on
a 20-minute YouTube source: the bug surfaces around clip 86 when
Mimi's transformer hits position 8181+. Filtering to short clips
alone didn't help — the cumulative state grows even with sub-12-second
inputs.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
||
|
osobhandClaude Opus 4.7
|
7e37e45df7 |
rtx-csm: scripts/fetch_audio.sh — yt-dlp + ffmpeg → 24kHz mono WAV
Wrapper that pulls audio from any yt-dlp-supported URL (YouTube, LibriVox, archive.org, podcast feeds) and converts to the 24 kHz mono 16-bit PCM format examples/audio_to_manifest ingests. Slugifies the output filename so manifest paths stay shell-safe. Prints the next-step audio_to_manifest command with all the right flags (--auto-emotion-tag --use-emotion2vec --stage audiobook), so a new user can copy-paste the printed line straight into a terminal. Requires external tools (yt-dlp, ffmpeg); install on macOS via `brew install yt-dlp ffmpeg`. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
osobhandClaude Opus 4.7
|
83654b677b |
rtx-csm: Phase 13.9 — wav2vec2 slice 4 (CTC Viterbi forced alignment)
Word-level forced alignment shipped. Phase 13.9 complete. viterbi_align(log_probs, tokens, blank_id, vocab) — standard CTC forced-alignment trellis: states alternate [blank, t0, blank, t1, ..., tN, blank] (length 2N+1), at each frame stay/advance/ε-skip-blank- between-different-tokens (canonical CTC ε-skip rule correctly forbids skipping blank between SAME tokens), max-likelihood path recovered via backptr table. transcript_to_token_ids — text → CTC token ids; runs of spaces collapse to | separator; unknown chars → <unk>. group_into_words — fold adjacent non-| AlignedToken into AlignedWord with carried frame_start/frame_end. frame_to_ms — 50 Hz frame grid → ms (20 ms/frame at conv stride 320). examples/wav2vec2_smoke --align <target> wires it end-to-end: forced-aligns a known transcript and prints (word, start_ms, end_ms). Verified on Metal: 10.42 s LibriSpeech audio, first 8 words → HE 560-640 HOPED 720-960 THERE 1000-1140 WOULD 1180-1320 BE 1360-2240 STEW 2980-4720 FOR 5300-6000 DINNER 7040-8540 viterbi alignment in 0 ms (39 tokens). Boundaries match audio. 4 new unit tests: - transcript_to_token_ids_handles_spaces_and_unknowns - viterbi_align_recovers_obvious_alignment - group_into_words_splits_on_separator - frame_to_ms_50hz_grid Lib suite 131/131 (was 127, +4). Phase 13.9 complete (slices 1+2+3+4). Crate now ships full English ASR + word-level forced alignment in pure candle — no whisper.cpp, no ort, no Python. Data-prep can cut long audio at exact word boundaries before feeding into the Phase 12.3 curriculum trainer. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
osobhandClaude Opus 4.7
|
209279c13e |
rtx-csm: Phase 13.9 — wav2vec2 candle port (slices 1+2+3, real ASR working)
Full port of facebook/wav2vec2-base-960h (94.4 M params, MIT) closing
the WhisperX-class word-alignment gap from the audio-ML survey. Same
staged-scaffolding pattern that worked for emotion2vec — but landed
slices 1+2+3 in one session.
src/wav2vec2.rs ships:
- Wav2Vec2Config::base_960h
- FeatureExtractor — 7 Conv1d (1→512, total stride 320). Layer 0
uses GroupNorm with num_groups=num_channels=512 (HF's wav2vec2
feat_extract_norm: "group"). Critical: state-dict key is
layer_norm.* but the OP is GroupNorm — loading as LayerNorm
produces empty CTC output.
- FeatureProjection — LayerNorm(512) + Linear(512→768)
- ConvPosEmbedding — kernel 128 grouped Conv1d, materialized at
load time from upstream weight_g + weight_v (fairseq's weight_norm
on dim=2; eps-guarded division for numerical stability)
- Block — POST-norm transformer with separate Q/K/V (vs emotion2vec's
fused QKV), uses (B*H, T, D) Metal 3D-matmul workaround from
Phase 8.8 Moonshine
- Encoder — pos_conv + initial LayerNorm + 12 Blocks
- Wav2Vec2 top-level — load_from_safetensors via mmap'd VarBuilder
- ctc_greedy_decode + VOCAB_960H constant for the 32-char alphabet
examples/wav2vec2_inspect.rs (slice 1): dumps tensor layout + config
examples/wav2vec2_smoke.rs (slice 3): real-weight load + ASR forward
Verified on Metal:
loaded model in 0.28 s
forward in 9 ms for 10.42 s audio (~1150× realtime)
transcript: "HE HOPED THERE WOULD BE STEW FOR DINNER TURNIPS AND
CARROTS AND BRUISED POTATOES AND FAT MUTTON PIECES TO
BE LADLED OUT IN THICK PEPPERED FLOWER FAT AND SAUCE"
Numerical parity with upstream Python — the FLOWER-for-FLOUR typo is
the known wav2vec2-base-960h failure mode, matches HF reference exactly.
7 new unit tests; lib suite 127/127 (was 120).
Slice 4 remaining: Viterbi forced alignment given known transcript,
to emit (token, frame_start_ms, frame_end_ms) for word-boundary cuts.
The ASR path itself is now production-ready.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
||
|
osobhandClaude Opus 4.7
|
f50233a9cc |
rtx-csm: Phase 13.8 — emotion2vec port, slice 4 (EmotionDetector + integration)
Port complete. The Phase 13.3 prosody-rule placeholder is now
retire-able by setting one CLI flag — the real candle-ported
emotion2vec_plus_base classifier slots in behind the same
EmotionDetector trait the placeholder used.
impl EmotionDetector for Emotion2Vec — builds (1, 1, T) tensor on
stored device, runs forward, argmaxes the 9 logits, maps to the
5-bucket label via Classifier::tag_for_class. Empty input
short-circuits to Neutral.
Emotion2Vec struct gained a `device` field so the trait impl can
build tensors without an out-of-band handle. new() / load_from_pickle()
threaded through; existing tests + smoke binary updated.
audio_to_manifest --use-emotion2vec — pairs with --auto-emotion-tag
to swap ProsodyDetector for Emotion2Vec, boxed as
Box<dyn EmotionDetector> so the call site is unchanged.
converse_server --use-emotion2vec — same pattern; built once at boot
and stored in Shared as Box<dyn EmotionDetector + Send + Sync>.
~150 ms/turn forward cost vs <1 ms for prosody, but actually runs
SOTA SER. Removed redundant reactive_emotion: bool field — the
Option<Box<dyn>> already encodes the same state.
Verified end-to-end on Metal:
- audio_to_manifest --use-emotion2vec on 2-speaker concat → both
tagged [excited] (prosody had said [neutral] on same input)
- converse_server --quantized-gguf … --lora … --reactive-emotion
--use-emotion2vec boots, 1 bench turn 0 errors, /metrics shows
reactive_emotion_total{label="excited"} 1 — same tag
audio_to_manifest produced. Cross-consumer consistency.
Phase 13.8 complete (slices 1+2+3+4 shipped). The emotional-voice
stack now has a real, trained, candle-ported SER classifier with
no Python sidecar, no ort, no whisper.cpp.
Lib suite 120/120.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
||
|
osobhandClaude Opus 4.7
|
9c1894ec73 |
rtx-csm: Phase 13.8 — emotion2vec port, slice 2d (real weights load + forward)
Slice 2 complete + slice 3 collapsed in. The full candle port loads
real upstream weights and runs forward in 160 ms.
RelativePositionalEncoder — 5 grouped Conv1d (kernel 19, groups 16,
same-padding via pad 9) with GELU between. Output is added back to
input as a positional bias. Pickle keys
relative_positional_encoder.{1..=5}.0.weight/bias (1-based indexing,
no .0.*).
Emotion2Vec top-level — wires LocalEncoder → ProjectFeatures →
RelPosEnc → ContextEncoder → MainEncoder → mean-pool → Classifier.
The proj.* classifier head lives at the state-dict root (not under
d2v_model.), so the constructor uses vb directly there.
Emotion2Vec::load_from_pickle uses VarBuilder::from_pth_with_state
to descend into the fairseq-style nested checkpoint via the "model"
key. One-shot loader; all 185 upstream tensor keys must map onto
candle params of matching shape — and they do.
examples/emotion2vec_smoke.rs — full pipeline integration test:
downloads (or reuses cached) emotion2vec_plus_base from HF, loads it
into candle, runs forward on 2 s of synthetic audio, prints all 9
raw logits + argmax + the 9→5 bucket fold.
Verified on Metal:
loaded model in 0.19 s
forward in 160 ms
9 logits all finite (50-290 range, expected for raw classifier)
argmax: class 7 (surprised) → 5-bucket [excited]
Mechanical correctness end-to-end. Semantic accuracy on real
emotional speech lands in slice 4 (EmotionDetector trait impl +
swap into audio_to_manifest + converse_server reactive-emotion path).
2 new unit tests:
- relative_positional_encoder_preserves_shape
- emotion2vec_random_init_end_to_end_shape
Lib suite 120/120 (was 118, +2).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
||
|
osobhandClaude Opus 4.7
|
491f51565a |
rtx-csm: Phase 13.8 — emotion2vec port, slice 2c (Classifier + Encoders)
Three composition wrappers over the Block from slice 2b.
ContextEncoder — 4 Blocks + final LayerNorm
(context_encoder.blocks.0..3 + context_encoder.norm). Acts as a prenet
between ProjectFeatures and the main encoder.
MainEncoder — 8 Blocks, no final LayerNorm. Confirmed via inspector:
all 96 d2v_model.blocks.* tensors live inside numbered blocks; there's
no d2v_model.norm. Pre-norm pattern's per-block norm2 keeps residuals
conditioned without a global tail norm.
Classifier — single Linear 768→9 (proj.weight/proj.bias at the top of
the state dict, NOT under d2v_model.). Includes tag_for_class(idx)
that folds the 9 fine-grained model classes (angry/disgusted/fearful/
happy/neutral/other/sad/surprised/<unk>) into the 5-bucket label set
the Phase 13.3 EmotionDetector trait already uses:
- 0 angry → Angry
- 1 disgusted, 2 fearful,
6 sad → Sad (low valence)
- 3 happy, 5 other,
7 surprised → Excited (high arousal)
- 4 neutral, 8 <unk> → Neutral
4 new tests:
- context_encoder_chains_4_blocks_with_final_norm
- main_encoder_chains_8_blocks_no_final_norm
- classifier_emits_9_logits
- classifier_class_to_emotion_label_mapping (locks the 9→5 fold)
Lib suite 118/118 (was 114, +4 new).
Slice 2 remaining: relative_positional_encoder (5 Conv1d, the conv
positional bias) + top-level Emotion2Vec + .pt pickle loader. Then
slice 3 = forward pass + numerical parity check vs upstream Python.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
||
|
osobhandClaude Opus 4.7
|
90f1b4667c |
rtx-csm: Phase 13.8 — emotion2vec port, slice 2b (ProjectFeatures + Block)
Two more candle modules toward the full port.
ProjectFeatures — LayerNorm(512) → Linear(512→768). Sits between
LocalEncoder and the transformer. Pickle layout matches upstream:
project_features.1.* is the LayerNorm, project_features.2.* is the
Linear. Both have learnable affine params; the .1 LN is NOT just an
eps constant.
Block — pre-norm fused-QKV transformer block, the workhorse for both
ContextEncoder (4 instances) and MainEncoder (8 instances). Pickle
keys per block: norm1, attn.qkv (fused 768→2304), attn.proj,
norm2, mlp.fc1, mlp.fc2. GELU MLP activation. No positional encoding
inside the block — the conv-based positional bias lives at the encoder
boundary.
Attention uses the (B*H, T, D) 3D collapse-before-matmul Metal
workaround we shipped for Phase 8.8 Moonshine — candle's 4D batched
matmul still has the shape-mismatch bug.
2 new unit tests:
- project_features_shape_check: (1, 50, 512) → (1, 50, 768)
- block_residual_shape_check: random (2, 8, 768) → same shape AND
all values finite (catches softmax NaN / attention overflow)
Lib suite 114/114 (was 112, +2 new).
Remaining within slice 2: relative_positional_encoder (5 Conv1d),
ContextEncoder (4 Blocks), MainEncoder (8 Blocks + LN), Classifier
(Linear 768→9), top-level Emotion2Vec + pickle .pt loader.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
||
|
osobhandClaude Opus 4.7
|
ce00e48acb |
rtx-csm: Phase 13.8 — emotion2vec port, slice 2a (LocalEncoder)
First module of the candle port. src/emotion2vec.rs ships:
Emotion2VecConfig::plus_base() — embed_dim=768, depth=8, prenet_depth=4,
num_classes=9, conv_layers spec yielding stride-product 320 (16 kHz →
50 Hz feature frames).
LocalConvBlock — one Conv1d → LayerNorm → GELU block. Weight key layout
matches the upstream pickle exactly: .0.weight for the conv (no bias),
.2.1.weight/bias for the LayerNorm (upstream wraps it as
Sequential(Conv1d, Dropout, Sequential(TransposeLast, LayerNorm,
TransposeLast), GELU); we collapse dropout / transposes since they're
inactive at inference / handled inline via .transpose(1,2)).
LocalEncoder — 7-block stack, channels 1→512, time shrinks by stride
product. Output shape (B, 512, T/320) — for 16 kHz input that's a
50 Hz feature frame rate.
2 unit tests pass:
- config_stride_product_matches_320 (catches future spec drift)
- local_encoder_random_init_shape_check — builds via VarMap+Kaiming,
runs forward on 1 s of zeros, asserts (1, 512, ~50) output
Lib suite 112/112 (was 110, +2 new tests).
Remaining within slice 2: project_features, relative_positional_encoder,
Block (fused-QKV), ContextEncoder (4 prenet blocks), MainEncoder
(8 main blocks), Classifier, and top-level Emotion2Vec with .pt loader.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
||
|
osobhandClaude Opus 4.7
|
a77b22d47c |
rtx-csm: Phase 13.8 — emotion2vec_plus_base port, slice 1 (inspector)
First slice of the multi-session port that will replace the Phase 13.3
prosody-rule SER placeholder with a real classifier via the existing
EmotionDetector trait.
examples/emotion2vec_inspect.rs:
- downloads model.pt + config.yaml + tokens.txt from
emotion2vec/emotion2vec_plus_base on HF Hub
- descends fairseq-style nested checkpoint via --key model
- dumps all 185 tensors with shapes/dtypes + per-prefix summary
- uses pickle::read_pth_tensor_info, same pattern as audioseal_inspect
Architecture confirmed (full notes in docs/emotion2vec_port_notes.md):
- 93 M params, F32 (the 1.12 GB file is mostly optimizer state)
- local_encoder: 7 Conv1d layers (wav2vec2 feature extractor:
[(512,10,5)] + [(512,3,2)]×4 + [(512,2,2)]×2, T → T/320)
- project_features: Linear 512 → 768
- relative_positional_encoder: 5 Conv1d layers (kernel 19)
- context_encoder: 4-layer transformer prenet (prenet_depth=4)
- blocks.0..7: 8-layer main transformer (depth=8, embed_dim=768,
12 heads, mlp_ratio=4, fused QKV qkv.weight=[2304, 768])
- proj: Linear 768 → 9 (angry/disgusted/fearful/happy/neutral/other/
sad/surprised/<unk>)
Slicing plan (remaining):
Slice 2 (~half-day): candle module scaffolding + from_pickle loaders
Slice 3 (~half-day): forward pass + shape verification
Slice 4 (~hour): EmotionDetector impl + swap into audio_to_manifest
and converse_server
Lib suite 110/110.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
||
|
osobhandClaude Opus 4.7
|
a4fca2b126 |
rtx-csm: Phase 13.7 — --lora support in tts_server
Closes the gap where converse_server (Phase 12.4) supported LoRA but the simpler HTTP-only tts_server didn't. Same flag set (--lora / --lora-rank / --lora-alpha / --extended-lora) and same apply_lora_adapter shared helper. Combines with --quantized-gguf (Phase 12.6) for Q8 + voice clone over plain HTTP. Verified end-to-end on Metal: tts_server --quantized-gguf … --lora … boots, injects LoRA into the quantized backbone (q=16 k=16 v=16 o=16 + MLP w1/w2/w3 = 224 tensors), listens. Single HTTP POST /v1/tts returned 200 OK with a 146KB 24kHz mono WAV. Lib suite 110/110. tts_server is now the simplest production deploy for a personalized voice: HTTP-only, no STT/LLM overhead, LoRA + Q8 + AudioSeal/SilentCipher + WavLM-SV all available behind one binary. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
osobhandClaude Opus 4.7
|
01081eb45a |
rtx-csm: Phase 13.6 — emotion metrics for converse_server
Production observability on the Phase 13.4/13.5 stack. Adds Prometheus
counters to the existing /metrics endpoint:
rtx_csm_reactive_emotion_calls_total — detector invocations
rtx_csm_reactive_emotion_total{label=...} — 5 buckets (neutral,
calm, sad, angry,
excited)
rtx_csm_emotion_aware_llm_applied_total — LLM augmentations
that actually fired
Verified end-to-end on Metal: server with Q8 + LoRA + reactive-emotion
+ emotion-aware-llm, 1 bench turn → /metrics shows
calls=1, calm=1 (all other labels 0), llm_applied=1. Lib suite 110/110.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
||
|
osobhandClaude Opus 4.7
|
273a92dcb9 |
rtx-csm: Phase 13.5 — emotion-aware LLM prompting
Composes Phase 13.4 reactive emotion with LLM message construction so
the assistant's RESPONSE TEXT adapts to detected user tone, not just
TTS prosody.
--emotion-aware-llm flag (requires --reactive-emotion). When a non-
Neutral label is detected, the LLM-facing copy of the user message is
augmented with `\n\n[user audio tone: {label} — adjust your response
in tone and content to match]`. Per-turn only — the augmentation lives
in history_clone, never in the persistent history, so subsequent turns
aren't biased by stale signals.
Verified end-to-end: server with Q8 + LoRA + reactive-emotion +
emotion-aware-llm booted, bench turn completed 0 errors. Mock LLM
echoed back the augmented text (taking ~44s of TTS), confirming the
annotation reached the LLM. Lib suite 110/110.
The full reactive voice loop now adapts both prosody (TTS emotion_hint)
AND content (LLM annotated user message) to detected user tone. Both
paths flow through the same EmotionDetector trait, so when emotion2vec_
plus_base lands the placeholder swaps cleanly.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
||
|
osobhandClaude Opus 4.7
|
091fe6bd91 |
rtx-csm: Phase 13.4 — reactive emotion in converse_server
The "cry-back when user sounds sad" feature from personal_voice_training_ guide.md §5. Composes everything from today's session: Phase 13.3 prosody-rule SER feeds Phase 12.2 emotion_hint plumbing, run per turn inside the production voice loop. --reactive-emotion flag. Per turn, between STT finalization and the LLM/TTS opts construction: 1. Trim the 2 s silence pad off user_audio_24k 2. audio_io::resample 24 → 16 kHz (rubato, already in deps) 3. ProsodyDetector::default().classify() over the speech buffer 4. If non-Neutral → use the tag as the turn's emotion_hint 5. Otherwise fall back to the static --emotion-hint EmotionDetector trait means a future emotion2vec_plus_base candle port slots in here without changing this code path. End-to-end verified: Q8 + LoRA + reactive-emotion server, single bench turn through WS, 0 errors, server logged `reactive-emotion: detected [calm]` and used it as the response's emotion_hint. Lib suite 110/110. Architecture now demonstrates the full Maya-class reactive voice loop: user audio → STT → ProsodyDetector → LLM → CSM TTS with matching tag → assistant responds in matching emotional register. All in-crate, sub-2s TTFA combo preserved. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
osobhandClaude Opus 4.7
|
6d59251c51 |
rtx-nn / rtx-multimodal: cargo fmt reformatting
Pure formatting changes across rtx-nn (conv_transpose1d, conv/mod, rnn/lstm) and rtx-multimodal (audio/generation, audio/source_separation): multi-line braces, trailing commas, import ordering. No logic changes. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
osobhandClaude Opus 4.7
|
ed4e5e4b85 |
rtx-csm: Phase 13.3 — prosody-rule SER baseline + --auto-emotion-tag
Closes the third gap from the audio-ML Rust ecosystem survey:
speech emotion recognition. Honest scope — this is a hand-tuned
placeholder, not a real classifier. The trait makes a future
emotion2vec_plus_base candle port a one-line swap.
src/ser.rs (~330 LOC):
- EmotionDetector trait
- ProsodyDetector impl: autocorrelation F0 (65-400 Hz, voiced via
autocorr peak ratio) + RMS + voiced-ratio aggregation
- 5 buckets compatible with Phase 12.2 emotion-hint format:
[neutral] [calm] [sad] [angry] [excited]
- 6 unit tests (autocorr accuracy on a pure tone, silence handling,
sad/excited/neutral edge cases, tag-format invariant)
audio_to_manifest gains --auto-emotion-tag: classifies each diarized
clip and writes the resolved label into the manifest row's
emotion_tag. Static --emotion-tag stays as a fallback.
End-to-end verified: 2-speaker concat → both clips classified
[neutral] (correct — synthetic CSM samples are prosodically flat).
Manifest round-trips through lora_train_emotional unchanged.
Lib suite 110/110 (6 new SER tests). Pure-DSP, zero ML deps, zero
runtime risk.
The data-prep pipeline is now end-to-end auto-labeled in-crate:
audio_to_manifest --auto-emotion-tag raw.wav → manifest.jsonl
→ lora_train_emotional → lora_eval → converse_server with --lora
Zero Python, zero ort, zero whisper.cpp.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
||
|
osobhandClaude Opus 4.7
|
2feb5c9d67 |
rtx-csm: Phase 13.2 — audio→manifest pipeline (no Python sidecar)
Single command turns raw audio (podcast/audiobook/conversation) into a
training-ready JSONL manifest + per-segment clip wavs.
audio_to_manifest:
1. Diarize (Phase 13.1: Silero V5 + WavLM-SV + clustering)
2. Per segment: slice audio + Moonshine encode/decode → transcript
3. Write `<stem>.spk{N}.{idx:04}.wav` + manifest.jsonl
Manifest rows match ManifestRow exactly (Phase 12.3), so the output
flows directly into lora_train_emotional / load_from_manifest.
Knobs: --segments-json (reuse precomputed diar), --emotion-tag and
--stage applied uniformly, --min-transcript-chars filters ASR failures,
plus all Phase 13.1 diarization knobs.
DiarizedSegment gained serde::Deserialize for the segments-json
reuse path.
Verified end-to-end: 2-speaker concat → 2 segments diarized in 192 ms
→ Moonshine transcribed → 2 manifest rows + 2 clips written → round-
trips through lora_train_emotional cleanly (LoRA injected with extended
coverage, adapter saved with embedded metadata, lib suite 104/104).
The full no-Python data-prep loop now reads:
audio_to_manifest raw.wav → manifest.jsonl
lora_train_emotional manifest.jsonl → voice.safetensors
lora_eval base + lora for A/B
generate / converse_server with --lora voice.safetensors
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
||
|
osobhandClaude Opus 4.7
|
73d38ed290 |
rtx-csm: Phase 13.1 — in-crate diarization (Silero V5 + WavLM-SV + clustering)
Composes existing in-crate parts into a speaker diarizer with zero new
deps. Pipeline: Silero V5 VAD → speech intervals → WavLM-SV x-vector
per ~2s window → agglomerative average-linkage clustering on cosine
distance → merged (start_s, end_s, speaker) segments.
src/diarize.rs (~330 LOC) ships:
- DiarizedSegment + DiarizationConfig
- Diarizer that owns the two backbones
- vad_intervals helper (smooths short silences, drops short speech)
- hand-rolled agglomerative cluster with auto-threshold OR force-k modes
- 5 unit tests (cosine distance edges, clustering, VAD interval extraction)
examples/diarize.rs CLI: --in --out --wavlm-sv-weights, plus knobs
(window/hop/vad-threshold/cluster-threshold/n-speakers/min-segment).
JSON output is consumable by ffmpeg/sox for downstream slicing.
Verified end-to-end on Metal:
- Single-speaker 10.41s → 1 segment, 21× faster than realtime
- Concatenated 2-speaker (CSM spk 0 + spk 1) → correctly identifies
2 speakers, 10× realtime
- Bug fixed in first run: clamp VAD interval bounds before slicing
(Silero V5 pads to whole-chunk multiple, can exceed sample count).
Closes the WhisperX-class "speaker diarization" gap from the personal
voice training guide without a Python/ort sidecar — sidesteps both
runtime conflicts the project hit before (whisper.cpp/ggml in Phase 7.6,
ort/protobuf in Phase 8.1.3). ~80% of pyannote-community-1 fidelity,
which is fine for data prep.
Lib suite 104/104 (5 new tests).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
||
|
osobhandClaude Opus 4.7
|
f7b6deac1a |
rtx-csm: Phase 12.7 — lora_eval held-out quality harness
Closes the train→generate→evaluate cycle. Users can now produce a hard quality number for any adapter without listening manually. evaluate_held_out runs teacher-forced forward_loss over a JSONL manifest (same format as 12.3 trainer). Uses apply_emotion_hint so eval prompts match training prompts. Frame sampling is seed-controlled — identical seeds across runs score the same frames in the same order, which is what makes a base-vs-LoRA A/B fair. EvalRow + EvalSummary types, both serde-Serialize for JSON output. examples/lora_eval.rs wraps it: --eval-manifest --report [--lora]. Documented usage: run twice with the same seed, diff the summary blocks. Verified end-to-end on the existing 3-row curriculum manifest with seed=42, frames-per-example=4: LoRA shifted mean/median/p90 loss directionally in its favor (-0.011/-0.018/-0.007). Tiny because the test adapter only saw ~10 training steps, but the eval signal is real and the A/B path is wired. Lib suite 99/99. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
osobhandClaude Opus 4.7
|
48df43810a |
rtx-csm: Phase 12.6 — unblock LoRA-on-Q8 in converse_server
Remove the `--lora and --quantized-gguf cannot combine` bail. That guard was wallpaper from before Phase 12.1 added LoRA hooks to csm_quantized.rs. The shared apply_lora_adapter helper routes through the model.rs wrapper which dispatches to either backbone, so FP and Q8 paths are equivalent from the call site's perspective. Verified end-to-end: - generate --quantized-gguf … --lora … runs with extended LoRA on the quantized backbone, auto-detects metadata via Phase 12.5, produces audio. - converse_server --quantized-gguf … --lora … --stream-tts boots, warms up, listens; converse_server_bench --turns 1 completes cleanly (0 errors, tts_per_utterance=2610ms, e2e_first_audio=3657ms). CLI flag docstring updated to advertise the now-combined behaviour. This is the production-deployable combo: sub-2s TTFA Q8 + personalized voice from Phase 12.x training. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
osobhandClaude Opus 4.7
|
32a811904a |
rtx-csm: Phase 12.5 — self-describing LoRA adapters
Trained adapters now embed (rank, alpha, target_modules, crate_version) as JSON in safetensors __metadata__["rtx_csm_lora"]. apply_lora_adapter reads it at load time so users no longer have to remember matching --lora-rank/--lora-alpha/--extended-lora flags from training. LoraAdapterMetadata::is_extended() heuristic: target_modules contains any MLP path or output_proj or k_proj. Handles both the canonical extended() preset and future custom configs that overlap it. apply_lora_adapter rank/alpha/extended params became Option<_> (None = use file metadata, Some = override). Both callers updated. save_lora_adapter_with_metadata is the new path used by both trainers; the plain save_lora_adapter still exists for the metadata-less case (per-stage curriculum snapshots). safetensors dep bumped 0.4 → 0.7 to match candle 0.9's transitive pin so candle's Tensor: View impl is in scope for serialize_to_file (candle's own save wrapper hardcodes the metadata arg to None). Backward compat: pre-12.5 adapters load fine when explicit CLI flags are passed; auto-detect path is skipped silently. Verified end-to-end: trained adapter saved with metadata, `generate --lora <path>` (no other flags) auto-detected rank=8 alpha=16 extended=true and applied. Older metadata-less adapter still loaded with explicit flags. 3 new unit tests; lib suite 99/99. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
osobhandClaude Opus 4.7
|
9d4eabc773 |
rtx-csm: Phase 12.4 — inference-time LoRA loading + extended-lora flag
Closes the train→generate loop the lora_train comment has promised
since Phase 3 ("forthcoming --lora flag on generate").
apply_lora_adapter(generator, path, rank, alpha, extended, device)
shared helper in src/training.rs wraps add_lora_to_backbone +
load_lora_adapter + refresh_lora. Both examples/generate and
examples/converse_server now call it instead of inlining their own
versions, and both now take --extended-lora to opt into Phase 12.1
coverage. Classic q+v adapters still load without the flag.
lora_train.rs now prints the exact `--lora <path> --lora-rank N
--lora-alpha N [--extended-lora]` command-line you need to apply the
trained adapter at inference, replacing the (forthcoming) message.
End-to-end verified: a Phase 12.3 curriculum-trained adapter loaded
into generate with identical seed/text produces different audio
(92KB vs 61KB, EOT @ frame 24 vs 16) — confirming the adapter takes
effect through to the sampled output. The 3-utterance smoke adapter
hasn't learned anything meaningful but the wiring is sound.
Phase 12 emotional voice stack now complete end-to-end:
12.1 capacity → 12.2 control tokens → 12.3 curriculum → 12.4 inference.
Lib suite 96/96.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
||
|
osobhandClaude Opus 4.7
|
510cd7011c |
rtx-csm: Phase 12.3 — curriculum LoRA trainer for emotional fine-tunes
Closes the personal_voice_training_guide.md §4 stack: capacity (12.1) +
control tokens (12.2) + multi-stage curriculum (this).
TrainingExample gains emotion_tag + stage. Trainer::train applies the
tag via the same apply_emotion_hint helper inference uses (now
pub(crate)) — training and inference must use identical prefix
formatting or the adapter won't transfer.
TrainingDataset::load_from_manifest reads JSONL
`{wav, transcript, emotion_tag?, stage?, speaker?}` rows; wav paths
resolve relative to manifest dir.
CurriculumStage + CurriculumTrainer run N stages sequentially against a
shared VarMap. Per stage: filter by ex.stage label, build a transient
sub-dataset, run Trainer, save snapshot if requested. The "*" stage
name is a global catch-all.
examples/lora_train_emotional.rs wraps the canonical 3-stage recipe:
audiobook (3 ep × lr 1e-4) → podcast (1 ep × lr 3e-5) → va (1 ep ×
lr 1e-5). --extended-lora recommended (FFN is the prosodic-style
carrier per the guide).
Verified end-to-end on Metal: 3-row manifest → all 3 stages execute,
checkpoints + final adapter written, prompt-token lengths varied by
emotion-tag length (9 vs 11 for different tags) confirming the tag
flowed through the training tokenization. Lib suite 96/96.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
||
|
osobhandClaude Opus 4.7
|
9023684d38 |
rtx-csm: Phase 12.2 — emotion control token plumbing
GenerateOptions gains emotion_hint: Option<String>. After text normalization, the hint (if Some) is prepended as `<tag> <text>` so the Llama BPE tokenizer encodes it as ordinary tokens. Plumbed through generate, generate_streaming, and generate_with_profile (via delegation), plus a `--emotion-hint` flag on examples/generate and a Shared field + CLI flag on examples/converse_server (per-turn ConverseOptions). GenerateOptions lost Copy because Option<String> isn't Copy; updated the four callers that depended on it (bench, longform, converse synth + stream) to .clone() the opts at the call site. Cheap — the struct is small and clones are per-turn, not per-frame. On the un-adapted base this is a no-op cosmetic prefix. The point is to unlock Phase 12.1-fine-tuned adapters: train with `[whisper] X` paired with whispered audio, and the adapter learns the tag→prosody mapping at inference time. Verified end-to-end: --emotion-hint "[whisper]" --max-audio-ms 3000 produced a valid 24kHz mono WAV through tokenizer → backbone → Mimi with no panics. Lib suite 96/96 (added 4 apply_emotion_hint unit tests). Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
osobhandClaude Opus 4.7
|
5423136bae |
rtx-csm: Phase 12.1 — extend LoRA coverage q+v → full attn + MLP
Both backbones (FP csm_fork + Q8 csm_quantized) now expose 7 LoRA hooks per layer: q/k/v/o on attention plus gate/up/down (Llama w1/w3/w2) on the SwiGLU MLP. LoraConfig::default() still returns q+v only (backward compat for existing trained adapters); LoraConfig::extended() returns the full 7-module set. lora_train + lora_finetune_step take a --extended-lora flag. Verified end-to-end on Metal: injection across all 16 backbone layers × 7 modules = 224 adapter Vars, 5.6M trainable params (~6.6× q+v alone, still tiny vs the 1B base). Step-0 loss matches the q+v baseline exactly (B=0 init is also a no-op for the new hooks). Forward + backward + AdamW + refresh_lora cycle runs without errors. LoRA test suite: 9 pass (added config_extended_targets_full_attn_and_mlp); full lib suite still 92/92. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
osobhandClaude Opus 4.7
|
a0dbe7caf4 |
rtx-csm: docs — personal-use emotional voice training guide
The hobby-project sibling of maya_finetune_analysis.md. For the case
where you want the most emotionally responsive voice ever, for personal
use only (not distributed, not commercial), using all media formats
legitimately accessible to one person.
Key practical guidance:
Best ROI sources: audiobooks (200-500 hr easy) > podcasts (100 hr per
weekend) > anime/game VA reels (the gold mine for extreme emotional
range) > YouTube > movies. Skip Reddit clips. SKIP TTS-synthesized
data (mode collapse).
Data prep pipeline (specific tool choices):
yt-dlp -> Demucs v4 htdemucs_ft -> Silero V5 VAD (we have the
candle port already) -> WhisperX (the right answer for
diarization+ASR+alignment, don't roll your own) -> DNSMOS quality
gate -> single-speaker filter -> resample 24 kHz -> Mimi tokenize
Emotion labeling: emotion2vec+ as primary auto-tagger, GPT-4o or
Claude as LLM-as-judge for the 5-10% you'll actually train on
(~$50/100hr), hand-label 200 clips for Cohen's kappa validation.
Plus implicit conditioning on previous-turn audio (what Sesame
likely did). Do BOTH.
Training recipe (100-200 hr corpus, single A100/H100):
- LoRA: extend from q+v to q,k,v,o + MLP gate/up/down. r=32-64.
- Curriculum: audiobooks (3 ep clean) -> podcasts (1 ep) -> VA/
movies (1 ep, lower LR). Prevents messy data destabilizing
acoustic priors.
- One LoRA, multiple emotion control tokens. Per-emotion LoRAs
can't switch fast enough at inference.
- 5-10% mix-in of EmoV-DB/ESD/MEAD/RAVDESS. Not more.
Reality check:
60-120 focused hr -> "clearly better than base CSM in your domain"
300+ hr -> "genuinely beats Maya for me"
Biggest trap: spending 80% of time on data, 15% on training infra,
5% on actually listening critically. Listening is where the model
gets good. Set a rule: every checkpoint, 20 prompts + notes.
Second trap: training on TTS-synthesized data. Mode collapse.
Where motivation dies: hour 40 when WhisperX diarization fails on
a podcast and you spend a Saturday debugging pyannote.
Going BEYOND Maya:
- GoEmotions 28-label taxonomy + V/A continuous (5x5 = 25 pseudo)
- Multi-persona via 512-d persona embeddings (YourTTS pattern)
- Reactive emotion: emotion2vec+ on user audio at inference,
feed as control token. ~50 ms latency. Feasible today.
Concrete Phase 12 candidates (bounded codable items, NOT the data
collection itself):
1. Extend rtx-csm LoRA coverage q+v -> k,o,MLP (~1-2 hr)
2. Wire WhisperX as scripts/ data-prep step (Python sidecar)
3. emotion2vec+ via ort sidecar, JSON labels
4. Emotion control token plumbing in Generator::generate
5. Curriculum trainer examples/lora_train_emotional.rs
Papers cited: CosyVoice 2, Voicebox, NaturalSpeech 3, emotion2vec+,
Spirit-LM. Tools: yt-dlp, Demucs v4, WhisperX, pyannote 3.x, Silero V5,
DNSMOS, GoEmotions taxonomy.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
||
|
osobhandClaude Opus 4.7
|
fa8d7e9eb2 |
rtx-csm: research — emotional corpora vs Hollywood for CSM fine-tune
Extended docs/maya_finetune_analysis.md with the user's "train on
Hollywood scenes that denote emotions" angle. The instinct is sound;
the legal pitfalls are severe; legitimate alternatives exist.
Headline:
DON'T train on Hollywood movies. Copyright + right-of-publicity is
uninsurable for a shipped product (NYT v OpenAI / Andersen v
Stability / RIAA v Suno / Johansson v OpenAI "Sky" all 2024-2026).
Source separation works technically; the law doesn't.
Legitimate corpora that capture the same "actors performing emotion"
property (matrix added to the doc):
Commercial-clean (use these):
EmoV-DB 7 hr / 4 spk CC-BY 4.0 — explicit laughs/yawns
CREMA-D 5 hr / 91 spk ODC-By 1.0 — read but emotion-tagged
DailyTalk 20 hr / 2 spk CC-BY-SA — dyadic conversational
LAION Emo Speech ~5000 hr CC-BY 4.0 — but provenance risk
Hume Prosody proprietary paid commercial
Research-only (skip for shipped product):
Expresso (Meta) 47 hr / 4 spk CC-BY-NC — best quality
IEMOCAP 12 hr / 10 spk academic — best emotional range
MELD (Friends) 13 hr Warner Bros — audio is copyrighted
RAVDESS, ESD small/medium research
New "Path D" recipe added:
Stage 1 (~6 hr GPU on H100):
- EmoV-DB + CREMA-D combined (~12 hr, commercial-clean)
- LoRA r=8 α=16 on q+v +k+o + decoder cross-attn
- 3 epochs, lr 1e-4 cosine, bf16
Stage 2 (~weekend, 5-10 hr recording):
- One voice actor improvising LLM-prompted dialogue
- Stage-2 LoRA r=16 on the stage-1 checkpoint
Per the corpus research: gets ~70% of Maya's emotional
expressiveness, ~30% of her personality. The single-speaker stage 2
is the "uncanny news anchor doing feelings" -> "specific persona"
overlay. Crucially this is a WEEKEND with one actor, not the 40-hr
studio sprint Sesame did.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
||
|
osobhandClaude Opus 4.7
|
406780e194 |
rtx-csm: research — Maya personality fine-tune analysis
Honest research into what Sesame likely did to fine-tune the open
CSM-1B base into the deployed Maya/Miles persona. Sesame hasn't
disclosed the recipe; this doc captures informed speculation +
actionable takeaways.
Key findings:
1. Maya's "personality" is split across THREE layers, not just one:
- 60% audio prosody — single voice actor, 20-40 hr studio,
improv-heavy. Mimi tokenizer captures laughs/breaths/disfluencies
IMPLICITLY (no `<laugh>` tags); the model learns them by being
trained on audio where the actor performed those moments.
- 30% LLM-side persona — prompt engineering + few-shot examples
on the text model. NOT a voice-model property at all.
- 10% conversational dynamics — VAD + endpointing + barge-in +
streaming TTS. We're already at parity here.
2. Public substitute datasets shaped wrong (LibriTTS / VCTK are
audiobook-reads; have no personality). Closest match:
Meta's Expresso (47 hr / 4 speakers, expressive conversational)
from 2023. Worth investigating if we ever pursue real Maya-class
prosody.
3. Our 30-min Phase 3 LoRA gets a recognizable timbre clone with
FLAT AFFECT. Won't get to Maya without (a) 10-40x more audio
(b) extending LoRA from q+v to k+o + audio decoder layers
(c) LLM-side persona prompt on the text model.
Three concrete next chunks captured:
A. LLM-side persona prompt (~30 min, biggest ROI/minute)
B. Extended LoRA coverage (~1-2 hours)
C. Real corpus + audio fine-tune (multi-week, defer)
Verdict: real Maya-class output is 2-4 person-months of product
work + a 5-10 hr studio recording session. The IP gap is real and
not closeable with documentation alone. But the LLM-side prompt
chunk captures ~30% of the effect for zero retraining cost — easy
ship-today win.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
||
|
osobhandClaude Opus 4.7
|
237216a31f |
rtx-csm: Phase 11 — Silero V5 VAD pure-candle port (closes 8.1.3 deferred)
Phase 8.1.3 deferred Silero V5 VAD because the only Rust crate path (`voice_activity_detector` via `ort`) collides with `sentencepiece-sys` on protobuf 3.14 vs 3.21 and panics at process startup. This commit closes that gap with a NATIVE candle port. Direct port from `Snakers4/silero-vad/src/silero_vad/tinygrad_model.py` (71 LOC reference). Architecture: stft_conv Conv1d(1, 258, k=256, s=128) no bias conv1 Conv1d(129, 128, k=3, p=1) conv2 Conv1d(128, 64, k=3, s=2, p=1) conv3 Conv1d(64, 64, k=3, s=2, p=1) conv4 Conv1d(64, 128, k=3, p=1) lstm_cell LSTMCell(128, 128) final_conv Conv1d(128, 1, k=1) Forward: reflect-pad input by 64, STFT-as-conv1d, sqrt(real² + imag²), 4-layer Conv1d feature stack with ReLU, single LSTM step (state across chunks), 1x1 conv + sigmoid -> speech probability. Files added: src/silero_vad.rs ~310 LOC (incl. LSTM cell + downloader) docs/silero_vad_port_notes.md architecture + port plan examples/silero_vad_smoke.rs real-audio discrimination test Plus a new `ureq` direct dep (transport already pulled in via hf-hub). Weights ship via download-on-first-run from the upstream GitHub raw URL into `~/.cache/rtx-csm/silero_vad_16k.safetensors` (1.24 MB). No repo bloat; no .gitignore wrestling. End-to-end smoke (synthetic 50/50 silence/speech WAV at 16 kHz): load (cold): download + parse, < 100 ms after first run VAD sweep: 170 ms over 9.99 s of audio = 0.017x realtime (59x faster) unit test: passes (load weights + run one step) Probability output (per 32 ms chunk): 0-1.5 s: p ~ 0.01-0.07 silence 1.5-5 s: p ~ 1.000 speech (clean ramp at speech onset) 5-10 s: p ~ 0.001 silence Speech-chunk fraction 33% on the 50/50 layout — matches expected. Production angle: dramatically better silence/speech discrimination than the Phase 8.1.3b energy VAD (which only catches obvious silence). Silero V5 catches whisper-quiet speech, breath/lip noise, music vs speech distinction. Drop-in candidate for `--vad-gate` in a future iteration. The ort/protobuf conflict that blocked this for two months is now permanently resolved by NOT using ort. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
osobhandClaude Opus 4.7
|
c92e8f2eef |
rtx-csm: docs/perf_history — add watermarker backend matrix (SilentCipher row)
Captures the Phase 10 work in the consolidated perf doc. New section "Watermarker backend matrix" lists AudioSeal (Meta) and SilentCipher (Sesame's actual) side by side with measured RTF, capacity, conflicts, and a per-use-case recommendation table. Headline: SilentCipher is Sesame's literal production watermarker, now shipping in pure candle with bit-perfect round-trip on real LibriSpeech audio (15/15 codes, confidence 1.0000) and ~10× smaller than AudioSeal (~3M params vs ~30M). The "blow them out of the water" item from the Sesame gap analysis is closed. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
osobhandClaude Opus 4.7
|
385858a3ba |
rtx-csm: Phase 10.5 — SilentCipher in converse_server (Sesame parity shipped)
Adds `--watermark-silentcipher` flag to converse_server. Sesame's actual
production watermarker is now a drop-in option in the conversation
pipeline alongside the existing AudioSeal flags.
Usage:
--watermark-silentcipher hf # download from sony/silentcipher
--watermark-silentcipher /path/to/dir # local checkpoint dir
--watermark-message 0xCAFE # 16-bit message (also drives AudioSeal)
Mutex with `--watermark-generator` / `--watermark-detector` (AudioSeal):
the Generator only carries one watermarker. Both are wrapped with
`ResampledWatermarker(24 kHz <-> 16 kHz)` for the CSM TTS path.
End-to-end production test (Q8 + Kyutai + mock LLM + SilentCipher 0xCAFE):
client TTFA: 7888 ms
total wall: 21860 ms
assistant audio: 12.16 s @ 24 kHz, written to /tmp/converse_silent_response.wav
re-detect: confidence 0.7614, payload 0x0000CAFE (PASS)
The 0.76 confidence (vs 1.00 in the standalone CLI test) is expected —
the assistant audio went through 24->16->24 resample plus stream-
encode-decode, all of which add noise. Still well above the 0.7
threshold we use for `Option<u16> -> Some/None` mapping in the
Watermarker trait impl.
A/B vs AudioSeal on the same /tmp/asr_test.flac (10.43 s @ 24 kHz):
AudioSeal SilentCipher
Embed timing not in CLI 988 ms (0.10x rt)
Detect timing not in CLI 1493 ms (0.14x rt)
Bit accuracy 16/16 bits 15/15 codes
Confidence 1.0000 1.0000
Message 0xCAFE 0xCAFE (decimal 51966)
Both bit-perfect. AudioSeal carries 16 bits, SilentCipher carries up
to ~24 bits per patch (15 base-3 codes). For our use (16-bit job_id
or message hash), either fits.
Production recommendation: ship SilentCipher for literal Sesame
parity AND the structural advantages (smaller model, identical bit
accuracy, confidence-based threshold). AudioSeal stays available for
callers who want the per-sample presence map (which SilentCipher
doesn't provide).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
||
|
osobhandClaude Opus 4.7
|
df89372ad7 |
rtx-csm: Phase 10.4 — SilentCipher detect + Watermarker trait + apply CLI
End-to-end SilentCipher: bit-perfect round-trip on real LibriSpeech
audio. Sesame's actual production watermarker now works in pure
candle 0.9 + Metal.
New components in src/silentcipher.rs:
detect(samples_16k) -> DetectResult
1. RMS-normalize to VCTK baseline (matches embed pre-conditioning)
2. STFT -> magnitude
3. dec_m_0(magnitude) -> (B, message_dim, 1, T) logits
4. argmax along message_dim -> (T,) per-frame predictions
5. Truncate to multiple of message_len
6. Reshape to (n_patches, message_len), per-column mode
7. Find terminator (value 0), rotate so payload follows it
8. Subtract +1 offset -> original codes
encode_bits / decode_bits (Phase 10.4 fix)
Switched from base-4 (2 bits per code) to base-`(message_dim - 1)`.
The 16 kHz model has message_dim=4 = 3 carrier values (1,2,3) +
terminator (0), NOT 4 carrier values. Original base-4 packing
occasionally produced value 3, which Python's
`np.identity(4)[index+1]` would have crashed on. Real capacity:
15 codes x log2(3) ~= 23.78 bits per patch.
SilentCipherWatermark (impl Watermarker)
Wraps a SilentCipherWatermarker with a fixed default_payload so
it satisfies the existing Watermarker trait. Maps confidence ->
DetectionResult.mean_presence and the lower-16-bits of the
decoded payload -> DetectionResult.message (None below confidence
0.7 to suppress false positives).
examples/silentcipher_apply
Mirrors audioseal_apply: --in / --out / --payload / --detect-only.
Loads from sony/silentcipher HF repo, embeds, optionally
resamples back to source rate, optionally re-detects to verify.
Verified end-to-end (LibriSpeech /tmp/asr_test.flac, 10.42 s @ 16 kHz):
Build: 29 ms (3 .ckpt files from HF cache)
Embed: 1213 ms = 0.116x realtime
Detect: 1838 ms = 0.18x realtime
payload: 0x00BC614E (in)
recovered: 0x00BC614E (out)
codes match: 15 / 15
confidence: 1.0000
Clean (un-watermarked) audio: confidence 0.475, codes mostly 0 -
strong signal-vs-noise discrimination at the 0.7 threshold.
This closes the most surprising gap from the Sesame stack analysis:
rtx-csm now has the *literal* Sesame watermarker (not Meta's
AudioSeal) working in pure candle. AudioSeal stays available for
callers that prefer it.
Phase 10.5 (next): wire as a third option in converse_server alongside
AudioSeal, and a 24/16 kHz ResampledWatermarker for the CSM path.
Plus an A/B bench (SilentCipher vs AudioSeal).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
||
|
osobhandClaude Opus 4.7
|
f6bcf0735a |
rtx-csm: Phase 10.3 — SilentCipher embed pipeline end-to-end
End-to-end encode pipeline working: 3 ckpts load from HF, STFT runs,
encoder + carrier-decoder forward, iSTFT reconstructs. Watermarked
audio out preserves length + carries an embedded message.
New components in src/silentcipher.rs (~150 LOC added):
SilentCipherWatermarker bundle of cfg + 3 networks + STFT + device
::from_ckpts(...) load enc_c.ckpt + dec_c.ckpt + dec_m_0.ckpt
pickle files via candle_core::pickle::read_all
::build_message(codes, T) one-hot + tile across time axis to match
n_frames; matches Python letters_encoding
shape semantics
::embed(samples_16k, codes) full encode pipeline:
1. RMS-normalize to VCTK baseline
2. STFT -> magnitude + phase
3. enc_c forward -> 32-channel carrier
4. enc_c.transform_message -> projected msg
5. cat(carrier_enc, mag.repeat(32),
msg_enc.repeat(32)) -> 96 channels
6. dec_c forward + utterance-level
normalization + ensure_negative_message
+ ReLU clamp
7. iSTFT -> watermarked audio
8. de-normalize energy
::encode_bits(payload) pack a u32 into message_len-1 2-bit codes
Smoke test (`examples/silentcipher_smoke`) verified end-to-end:
Build watermarker: 29 ms (loads 3 .ckpt files)
Synthetic sine embed: 187 ms / 1.00 s audio
Real speech embed: 1042 ms / 10.42 s audio = 0.10x realtime
The 0.10x realtime figure is comparable to AudioSeal in Phase 6f.wm
(73 ms per ~6.8 s sentence = ~0.011x realtime, but AudioSeal had
warm-cache benefit). On a fresh cold model, SilentCipher comes in
~10x faster than realtime — order-of-magnitude OK.
SNR vs original: 24.6 dB on the speech sample, target 47 dB per the
released hparams. The watermark is currently more audible than
intended. Likely cause: utterance-level normalization scale factor
needs refinement, OR the ensure_negative_message + ReLU path is
clipping more than the Python path. Will be diagnosed in Phase 10.4
when detection round-trip lands — the real test of correctness is
"can dec_m recover the embedded codes?", not absolute SNR.
Phase 10.4 will:
- Implement detect() to recover the embedded codes via dec_m_0
- Add Watermarker trait impl for SilentCipherWatermarker
- examples/silentcipher_apply CLI mirroring audioseal_apply
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
||
|
osobhandClaude Opus 4.7
|
e7b34abd16 |
rtx-csm: Phase 10.2 — SilentCipher Layer + STFT + scaffolds
Foundation in src/silentcipher.rs (~480 LOC) plus rustfft 6.2 dep.
What works:
Stft windowed framed FFT via rustfft (Hann window,
n_fft + hop_length config). Round-trip unit test
on a 440 Hz sine (16 kHz, 1 s) achieves > 0.95
correlation — overlap-add + epsilon-trick blur
the bit-exact return slightly but the recovered
waveform tracks the original cleanly.
SilentCipherConfig hyperparameters from the released 16 kHz hparams
(N_FFT=2048, HOP=1024, message_dim=4,
message_band_size=512, etc.). Constructor
sixteen_khz() returns the production defaults.
Layer gated conv block: bn(conv(x) * sigmoid(gate(x)))
built from candle_nn::Conv2d + BatchNorm2d.
BatchNorm runs in eval mode (forward_t with
train=false) — released checkpoints carry
running_mean / running_var.
Encoder 3 stacked Layers (1->32, 32->32, 32->32) plus a
Linear(message_dim, message_band_size) for the
transform_message helper that projects bit
payloads onto the freq axis.
CarrierDecoder 4 stacked Layers (96->96 x3, 96->1 with k=1) +
optional ensure_negative_message + freq-band
masking + RMS / SDR scaling.
MsgDecoder 10 stacked Layers (1->128, 128->128 x8,
128->message_dim) + final Linear collapsing
freq -> 1. Slices to message_band_size rows
before processing. Models the PyTorch index
doubling (Dropout interleaved in eval mode is
identity, but stored under index 2i+1).
vb_from_ckpt opens a .ckpt pickle file and exposes a
VarBuilder with the legacy `module.` prefix
stripped, ready for Encoder::new etc.
What doesn't work yet (Phase 10.3):
- End-to-end embed() / detect() pipeline glue (STFT input ->
Encoder + transform_message -> CarrierDecoder -> iSTFT, plus the
decode mirror). Each piece compiles + has a smoke test, but the
pipeline orchestration is the next ship.
- Watermarker trait impl + wiring into Generator.set_watermarker.
- examples/silentcipher_apply (mirror of audioseal_apply).
Tests: 2 new unit tests pass alongside the existing 88. Full lib build
clean on `--features metal`.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
||
|
osobhandClaude Opus 4.7
|
9f5c345b53 |
rtx-csm: Phase 10.1 — SilentCipher inspector + port notes
Foundation for porting Sesame's actual production watermarker (NOT
AudioSeal — the gap analysis identified this as the literal Sesame
parity item). Same iterative-shipping pattern as Phase 8.4 for
Moonshine.
`docs/silentcipher_port_notes.md`:
- Full architecture from SesameAILabs/silentcipher/src/.../model.py
(verified against 95 LOC of source)
- Three small networks of gated 2D convs on STFT:
enc_c 3 layers 1 -> 32 channels
dec_c 4 layers 96 -> 1 channels
dec_m 10 layers 1 -> 128 -> message_dim, plus Linear
- Each Layer = Conv2d * sigmoid(Conv2d) + BatchNorm2d
- Pipeline (encode + decode) walked through step by step
- 10 ordered porting tasks with hour estimates totaling ~1-2 days
- Risks flagged: STFT helper needed, BatchNorm running stats loading,
phase passthrough, message-length differences vs AudioSeal
`examples/silentcipher_inspect`:
- Downloads sony/silentcipher 16 kHz checkpoint from HuggingFace
- Dumps hparams.yaml + tensor shapes per .ckpt file
- Verified output:
N_FFT 2048 HOP 1024 SR 16000
message_dim 4 message_len 16 message_band 512
enc_c 0.17 MB 40 k params
dec_c 2.01 MB 500 k params
dec_m_0 9.54 MB 2.38 M params
Total ~2.92 M params
That's ~10x smaller than AudioSeal's gen+det combined. Port
estimated 1-2 days.
`.ckpt` files are pickle (PyTorch state_dict) — direct loadable via
candle_core::pickle::read_all, same path as audioseal_convert.rs.
No safetensors conversion needed.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
||
|
osobhandClaude Opus 4.7
|
1760309b39 |
rtx-csm: gap analysis vs Sesame Labs full voice stack
Researched what Sesame has actually disclosed publicly (vs. marketed)
and compared systematically against rtx-csm's shipped surface.
Key findings:
1. **rtx-csm has shipped a SUPERSET of Sesame's open release.** CSM-1B
inference + voice cloning + Q8 + production WebSocket server +
three-backend STT pipeline + per-phase observability — Sesame ships
inference code only.
2. **Sesame's deployed Maya is a cascaded STT->LLM->TTS pipeline**,
same architecture pattern as rtx-csm. Their research blog explicitly
states future work is "fully duplex models" — Maya today isn't
duplex either. We're structurally equivalent at the pipeline level
to Kyutai Unmute, Sesame's closest peer.
3. **Marketing latency claims are unverified.** "Sub-200 ms TTFA"
appears in third-party blogs, not in any Sesame paper. Production
benchmarks of similar cascaded stacks show 250-300 ms TTFT
(gpt-realtime, Unmute) — our 280-380 ms TTS-side is competitive.
Z.AI provider TTFT (~1.3 s) is the dominant cost in our 1.96 s
end-to-end.
4. **Critical correction**: Sesame ships SilentCipher (their fork of
Sony's), NOT AudioSeal (which is Meta's). Our Phase 4 AudioSeal
work is functionally equivalent but isn't the *literal* Sesame
watermarker. SilentCipher port is ~1-2 days.
5. **What's gated on Sesame**: CSM-3B / CSM-8B variants (trained but
never released), Maya personality fine-tune dataset, distilled
wearable variant. The Oct 2025 Series B + smart-glasses pivot
suggests they're unlikely to release any of these.
Punch-list of remaining gaps captured in the doc with status (Closed/
Partial/Open/N/A) per capability.
Recommended next chunks (prioritized):
1. SilentCipher port (~1-2 days) — literal Sesame watermarker parity
2. clawsample-csm integration (~5-10 days) — separate plan exists
3. Tier 2.1 VoXtream look-ahead (~5-7 days) — diminishing returns
after Phase 9.2's chunk_frames tuning
4. Tier 3 Frame-Stacked / VADUSA (training-required, ~3-6 weeks)
5. Distilled CSM (speculative, wait for product target)
Defensible framing: "rtx-csm is Sesame's open release + voice cloning
+ production HTTP/WS server + Kyutai-style cascaded duplex. Remaining
gap to internal Maya is (a) SilentCipher watermarker, (b) Sesame's
proprietary fine-tune dataset, (c) larger CSM variants Sesame chose
not to release."
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
||
|
osobhandClaude Opus 4.7
|
bc57ebcde1 |
rtx-csm: Phase 9.2 — drop default stream_chunk_frames 4 -> 2
Quick TTFA win identified by the Phase 9.1 spike's "alternative quick win" recommendation: tune the streaming Mimi chunk size before attempting the full VoXtream port. The default `--stream-chunk-frames` was 4 (320 ms per chunk); dropping to 2 (160 ms) saves ~155 ms on client TTFA with no measurable downside. Bench (Q8 + stream + Moonshine + mock LLM, 3 turns each): chunk_frames=4 llm_to_first_audio 533 ms e2e_first_audio 939 ms chunk_frames=2 llm_to_first_audio 374 ms e2e_first_audio 784 ms ← new default chunk_frames=1 llm_to_first_audio 283 ms e2e_first_audio 664 ms Per-utterance TTS gen comparable across all three (~5.7-5.9 s for the 4-sentence mock LLM reply), so smaller chunks don't add meaningful decode overhead. The trade is just send-loop overhead + slightly more network packets. Production users can drop to 1 for tightest TTFA via `--stream-chunk-frames 1`. The 2 default is the conservative middle ground. This obviates most of the Tier 2.1 (VoXtream look-ahead) urgency: the first-chunk latency is now ~280-380 ms server-side; the remaining bottleneck is Moonshine STT (332 ms) and Z.AI TTFT (1300 ms), not TTS. VoXtream's 102 ms first-packet claim could close the remaining TTS-side gap (374 -> ~100 ms = -270 ms) but the integration cost is high relative to the win. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
osobhandClaude Opus 4.7
|
87fb269a81 |
rtx-csm: Phase 9.1 spike — VoXtream port notes (Tier 2.1 prep)
Phase 9.1 is the foundation step for the Tier 2.1 (VoXtream-style
look-ahead) optimization from the Phase 8 plan. Same pattern as Phase
8.4: don't attempt the full port in one session; ship the architecture
analysis + ecosystem audit + ordered porting plan so a future session
starts cold from concrete data.
Phonemizer ecosystem audited (results in the doc):
Crate Approach Latency Conflicts
espeak-ng 0.1.1 pure Rust eSpeak NG 1-5 ms/word none <- PICK
voirs-g2p 0.1.0-rc.1 neural, candle 0.9.2 10-50 ms/w libc dep
grapheme_to_phoneme 0.1.0 seq2seq RNN ARPAbet 2-8 ms/word none (abandoned 2020)
phonetisaurus-g2p 0.1.1 FST sub-ms none (no pre-trained FST)
espeak-ng is the right pick: zero C linkage default, no conflicts with
candle 0.9 / sentencepiece-sys / hound / symphonia / ebur128, mature.
Per-word latency fits VoXtream's 102 ms first-packet target with room
to spare (5-10 word lookahead = 5-50 ms total phonemizer cost).
Doc captures:
- Why this is worth doing (current ~600 ms TTS first chunk vs 102 ms
claim from arXiv 2509.15969)
- Three-piece integration architecture (phonemizer service, look-ahead
window in Converse, optional Generator-side phoneme hint)
- Three open questions that MUST be resolved before implementing
(mechanism, CSM training compatibility, empirical win on this HW)
- Six ordered porting tasks with hour estimates totaling 5-7 days
- Alternative quick win: drop streaming chunk_frames from 4 to 2 or 1
first; if that closes most of the gap, full VoXtream port may not
be worth the complexity
Recommended order of attack written for the next session: try the
chunk_frames tuning before implementing the phonemizer. The cheapest
move is a 1-line config change.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
||
|
osobhandClaude Opus 4.7
|
893e8e6232 |
rtx-csm: Phase 8.11 — production stack validated against live Z.AI
Updated docs/perf_history.md to reflect the full Phase 8 work:
- New TL;DR has TWO production configs (English-only with --moonshine,
multilingual with default Kyutai). The English path is the new
headline recommendation.
- Captured live Z.AI bench numbers from the new production stack
(Q8 + stream + warmup + Moonshine + thinking-disabled): total_turn
8765 ms vs Phase 6f.q8 era 17145 ms — half the wall-clock latency
end-to-end.
- Phase 8 commit table extended to 8.4 through 8.10.
- Added an STT backend matrix (Kyutai 1B / Whisper-rs / Moonshine)
with measured RTF, build flags, and per-deploy recommendation rows.
Bench command used (single turn, real Z.AI glm-4.5 thinking-disabled,
10.43 s LibriSpeech /tmp/asr_test.flac):
target/release/examples/converse_server \\
--bind 127.0.0.1:18099 \\
--quantized-gguf /tmp/csm_q8.gguf \\
--stream-tts --moonshine \\
--llm-base https://api.z.ai/api/coding/paas/v4 \\
--llm-model glm-4.5 \\
--llm-extra-body '{"thinking":{"type":"disabled"}}'
Server-side timing:
recv_phase 0 ms (Moonshine batch — no parallel STT)
stt_post 332 ms (Moonshine transcribe at EOT)
llm_to_first_audio 1627 ms (Z.AI TTFT ~1.3 s + first TTS chunk)
conv_total 8432 ms
total_turn 8765 ms
Client TTFA: 1959 ms (vs Phase 6f.q8 era ~2 s — comparable; the dominant
remaining latency is the Z.AI provider TTFT, not anything we control).
Z.AI returned a coherent reply: "He eagerly anticipated a hearty stew
with turnips, carrots, potatoes, and savory mutton pieces for dinner."
matching the LibriSpeech ground-truth meaning.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
||
|
osobhandClaude Opus 4.7
|
932a93c2ce |
rtx-csm: Phase 8.10 — Moonshine as third AsrEngine in converse_server
Wires the Phase 8.5-8.9 Moonshine port as a drop-in alternative to
Kyutai STT in the conversation server. New `--moonshine` flag (mutex
with `--whisper` and `--vad`). Pure candle, no external runtime — no
ggml/protobuf conflicts unlike `--whisper`.
Architecture:
- AsrEngine enum extended with Moonshine(MoonshineAsr) variant
- MoonshineAsr bundles { encoder, decoder, tokenizer, cfg, device }
plus a transcribe_24k() that resamples 24->16 kHz, encodes,
greedy-decodes (KV-cached), detokenizes
- Renamed Shared.whisper_mode -> Shared.batch_asr to cover both
Whisper and Moonshine (both batch-only, skip parallel STT)
- Receive loop's match arms now exhaustive over all three variants
- At EOT, transcript construction branches:
Kyutai -> join words from incremental Word/EndWord stream
Moonshine -> transcribe_24k() over accumulated audio
Whisper -> transcribe_24k() (asr feature)
End-to-end verified (mock LLM, Q8 + stream + warmup + Moonshine, real
LibriSpeech 10.43 s):
recv_phase: 0 ms (batch ASR; audio just buffers)
stt_post: 406 ms (Moonshine transcribe at EOT)
llm_to_first_audio: 533 ms
total_turn: 23617 ms
*** Client TTFA: 939 ms *** (sub-second!)
Compared to Kyutai (Phase 8.2 extended warmup baseline):
Kyutai TTFA p50 4915 ms
Moonshine TTFA 939 ms ← -80%
Moonshine produces near-perfect transcript: "He hoped there would be
stew for dinner, turnips and carrots and bruised potatoes, and fat,
mutton pieces to be ladled out in thick, peppered, flour-fat and
sauce." matching the LibriSpeech ground truth.
This is the new production-recommended voice-loop config for
English-only deploys:
converse_server \\
--quantized-gguf <Q8> --stream-tts --moonshine \\
[--vad-gate] # energy VAD still useful for skipping silence
[--llm-extra-body '{"thinking":{"type":"disabled"}}' # for Z.AI]
For multilingual (en+fr) deploys, stick with Kyutai 1B (the default).
Phase 8 is now feature-complete on the optimization tracks the
research surfaced:
- Tier 1 (warmup, energy VAD, ort gate): SHIPPED
- Tier 2.2 (Moonshine candle port): SHIPPED end-to-end (8.4-8.10)
- Tier 2.1 (VoXtream), Tier 3 (Frame-Stacked, VADUSA): deferred,
documented in plan + perf_history.md
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
||
|
osobhandClaude Opus 4.7
|
95015c17e1 |
rtx-csm: Phase 8.9 — Moonshine KV cache + profile binary
KV cache for the decoder turns greedy generation from O(T^2) into O(T)
total work. Per-token decode drops modestly on short transcripts
(7.1 -> 6.0 ms/token at 49 tokens) and compounds on longer ones.
New components in src/moonshine.rs:
RotaryCache::apply_at(x, position, t)
Apply RoPE for a window starting at `position`. Replaces
`apply()` for cached step (which always called positions 0..T).
DecoderSelfAttention::forward_step(xs, cache_k, cache_v, rope, position)
Single-token cached self-attn. Appends new K/V to per-layer cache,
attends across full accumulated history. No causal mask needed
(cache only contains positions <= current).
CrossAttention::precompute_kv(enc) -> (K, V)
One-shot encoder K/V projection for cross-attn. Reused every step.
CrossAttention::forward_step(xs, k, v)
Cached cross-attn. Q computed from new token; K/V from precompute.
DecoderCache { self_k: Vec<Option<Tensor>>, self_v, cross_k, cross_v, position }
Decoder::precompute_cross_kv(enc) -> DecoderCache
Decoder::step(token_id, &mut cache) -> logits (1, vocab)
Decoder::generate_cached(enc, cfg, max_tokens) -> Vec<u32>
Greedy loop using the cached step.
Profile (5 steady-state runs on /tmp/asr_test.flac, 10.42 s LibriSpeech):
warm-up: 344 ms
steady-state mean: 307 ms (p50 305, range 298-319)
realtime factor: 0.0294x
Comparison across all STT in rtx-csm:
Backend RTF Notes
Kyutai STT 1B 1.01x hardware-bound, 3 GB
Whisper-tiny 0.020x breaks CSM (in-process ggml conflict)
Moonshine-tiny 0.0294x pure candle, NO runtime conflict
Moonshine is the only fast STT path that integrates cleanly. ~34x
faster than realtime, ~17x faster than Kyutai 1B, no protobuf or
ggml linkage issues.
New `examples/moonshine_profile` mirrors `stt_profile` and
`whisper_profile` so all three STT backends report comparable numbers.
Phase 8.10 (next): wire as a third AsrEngine variant in converse_server
for English-only deploys. Replace the energy-VAD-gated Kyutai path
when --moonshine flag is set.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
||
|
osobhandClaude Opus 4.7
|
b22ff3544e |
rtx-csm: Phase 8.8 — Moonshine end-to-end transcription works
Full encoder-decoder Moonshine v2 transcribing real audio in pure
candle 0.9 + Metal. No ort, no ggml, no protobuf. The path that
whisper-rs (Phase 7.6) and Silero V5 via ort (Phase 8.1.3) couldn't
deliver due to in-process linkage conflicts.
End-to-end on /tmp/asr_test.flac (LibriSpeech, 10.42 s):
encode: 10 ms
decode: 348 ms (49 tokens, 7.1 ms/token greedy, no KV cache)
realtime factor: 0.068x (~14x faster than realtime)
Output transcript:
"He hoped there would be stew for dinner, turnips and carrots and
bruised potatoes, and fat, mutton pieces to be ladled out in thick,
peppered, flour-fat and sauce."
Ground truth:
"He hoped there would be stew for dinner, turnips and carrots and
bruised potatoes and fat mutton pieces to be ladled out in thick
peppered flour-fattened sauce."
Near-perfect (a few punctuation tweaks, "flour-fat and sauce" vs
"flour-fattened sauce"). WER very low.
Compared to other STT backends in this crate:
Kyutai STT 1B : 1.01x realtime (3 GB, hardware-bound)
Whisper-tiny : 0.020x realtime (in-process ggml -> CSM regression)
**Moonshine-tiny: 0.068x realtime (pure candle, no runtime conflict)**
Components shipped this commit:
- Decoder::generate(encoder_output, cfg, max_tokens) — greedy
autoregressive loop. No KV cache yet (each step re-runs the full
growing token sequence — O(T^2) total). For 49-token transcripts
at <500 ms total, KV cache isn't urgent.
- load_tokenizer() — wraps tokenizers::Tokenizer::from_file for
Moonshine's HF tokenizer.json (BPE, vocab 32_768).
- examples/moonshine_transcribe — full pipeline: audio -> 16 kHz
PCM -> encode -> decode -> detokenize -> transcript text.
Critical bug fixed: SwiGLU gate/up split direction. HF
modeling_moonshine.py says:
hidden, gate = fc1(x).chunk(2, dim=-1)
out = silu(gate) * hidden
The FIRST half of the fused fc1 output is `up` (multiplied), the
SECOND half is `gate` (silu-activated). I had it reversed in Phase
8.7 — the symptom was a degenerate "tt tt tt" repetition loop after
the first 2 tokens. Reversing the split unlocked the working
transcription. Captured in the code comment.
Remaining for Moonshine readiness in production:
Phase 8.9 — KV cache for sub-200ms latency on long transcripts,
plus a standalone moonshine_profile binary for the
full A/B against Kyutai/Whisper.
Phase 8.10 — wire as a third AsrEngine variant in converse_server
(gated on English-only acceptance for the deploy).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
||
|
osobhandClaude Opus 4.7
|
b94edca496 |
rtx-csm: Phase 8.7 — Moonshine decoder transformer (encoder+decoder)
Full encoder-decoder Moonshine v2 working end-to-end on candle 0.9 +
Metal. Loads HF safetensors, runs through every transformer block, and
produces real logits.
Components added to src/moonshine.rs:
CrossAttention MHA with K/V from encoder output (no causal mask)
DecoderSelfAttention MHA with causal mask, partial RoPE on q/k
DecoderMlp SwiGLU: fused fc1 [2304, 288] split gate+up,
silu(gate) * up, fc2 [288, 1152] back to hidden
DecoderLayer Pre-LN self-attn + Pre-LN cross-attn + Pre-LN MLP
Decoder token embed -> 6 layers -> final LN -> tied LM head
load_full() convenience: returns (Encoder, Decoder)
Smoke test verifies end-to-end:
encoder forward : 1 ms (cached after warm-up)
decoder forward : 85 ms (1 token, prefill mode)
logits shape : (1, 1, 32768)
logit max abs : 30.66 (real signal, not zeros)
argmax token_id : 379 (non-trivial prediction; eos=2)
Implementation notes:
- Same (B*H, T, D) 3D matmul pattern as encoder to dodge candle's 4D
Metal matmul shape-mismatch bug.
- LM head tied to decoder.embed_tokens.weight (cached on Decoder for
fast forward; logits = hidden @ embed.T).
- Causal mask is a (T, T) -inf upper-triangular added to scores
before softmax.
- Decoder final LN tensor is `decoder.norm.weight` (NOT
`decoder.layer_norm.weight` — encoder uses the latter naming).
- No KV cache yet: this is prefill mode. Phase 8.8 will add the
streaming-generation loop with cache + tokenizer.
NOT yet verified: numerical parity vs HF Python reference. The token
predicted (id=379) looks plausible for silent-mostly audio, but a
parity check is still needed (Phase 8.9). Architecture appears
correct based on shape + signal sanity.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
||
|
osobhandClaude Opus 4.7
|
a8e729a826 |
rtx-csm: Phase 8.6 — Moonshine encoder transformer block
Full encoder forward path: conv stem -> 6 transformer layers -> final
LayerNorm. Loads HF safetensors, runs end-to-end on Metal.
Components added to src/moonshine.rs:
RotaryCache partial RoPE (32 of 36 head_dim, theta=10000)
EncoderAttention MHA (8 heads, no bias), partial RoPE on q/k
EncoderMlp 288 -> 1152 -> 288 with bias, GELU(erf) activation
EncoderLayer Pre-LN attn + Pre-LN MLP (LayerNorm weight-only)
Encoder stem + 6 layers + final LayerNorm
load_encoder() VarBuilder convenience for the standalone smoke
Smoke test (`examples/moonshine_smoke`) verified end-to-end:
input (1, 1, 160000) -> output (1, 415, 288)
forward: 132 ms (10 s of audio at 0.013x realtime)
max abs: 6.67 (signal preserved, not zeros)
Implementation notes captured in the diff:
- candle Metal 4D batched matmul had shape-mismatch issues for our
(B, H, T, D) pattern. Switched to (B*H, T, D) 3D form which is
unambiguous and avoids the kernel bug.
- LayerNorm is weight-only (no bias tensors in safetensors); we
construct LayerNorm with a zeros bias to satisfy candle's API.
- rotary_dim = floor(head_dim * 0.9 / 2) * 2 = 32 (must be even).
The remaining 4 head_dim channels pass through unchanged via
`narrow + cat` on dim 3.
Numerical parity vs HF Python reference is NOT yet verified — that's
the next bounded chunk (Phase 8.7). Shape + signal correctness are
verified by the smoke test.
Next: decoder transformer block (self-attn + cross-attn + SwiGLU).
~3-4 h of focused work.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
||
|
osobhandClaude Opus 4.7
|
0699cdeb45 |
rtx-csm: Phase 8.5 — Moonshine conv stem (verified end-to-end)
First working piece of the Moonshine v2 candle port. New src/moonshine.rs module with: - MoonshineConfig::tiny() — hyperparameters from HF config.json - ConvStem (Conv1d × 3) — audio stem, raw 16 kHz → 288-d hidden - load_conv_stem() — VarBuilder from HF safetensors Conv layout (verified against HF source): conv1: in=1, out=288, k=127, stride=64, no bias conv2: in=288, out=576, k=7, stride=3, bias conv3: in=576, out=288, k=3, stride=2, bias Activations: tanh after conv1, gelu_erf after conv2 / conv3 Smoke test (`examples/moonshine_smoke`): - Downloads UsefulSensors/moonshine-tiny from HF - Synthetic 10 s @ 16 kHz audio (silence + sine pulse) - input (1, 1, 160000) -> output (1, 415, 288) - Expected T_seq=415 ((160000-127)/64+1 -> 2498 -> 831 -> 415) - Output max abs = 23.17 (real signal, weights loaded correctly) Also extends `examples/moonshine_inspect` to dump conv shapes explicitly (was being truncated by the per-prefix `take(8)` cap). Next ship: encoder transformer block (partial RoPE, GELU MLP) and output layer norm. Tracked in Phase 8 plan; ~2-3 hours of work. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
osobhandClaude Opus 4.7
|
c848abef22 |
rtx-csm: Phase 8.4 spike — Moonshine v2 inspector + port notes
First step on the Tier 2.2 (Moonshine v2 candle port) item from the
Phase 8 plan. Full port is honestly multi-session work (~12-15 hours
of focused implementation across encoder, decoder, generation loop,
tokenizer, weight mapping, smoke test). This commit ships the
foundation so future sessions start from concrete data instead of
arxiv reading.
Two ships:
1. examples/moonshine_inspect — downloads UsefulSensors/moonshine-tiny
from HF, parses safetensors header, dumps all 160 tensors grouped
by prefix with shapes + dtypes. Verified output: 27.1 M params,
108.4 MB safetensors (F32), encoder + decoder layers laid out as
expected.
2. docs/moonshine_port_notes.md — captures every architectural fact
established by the inspector + HF config.json:
- Hyperparameter table (hidden=288, 6+6 layers, vocab=32768,
partial_rotary=0.9, etc.)
- Tensor layout per layer (encoder, decoder)
- Architecture summary (raw waveform input, 3-layer Conv1d stem,
SwiGLU decoder MLP via fused fc1, tied LM head)
- Ordered porting tasks with hour estimates totaling ~12-15 h
- Risks / unknowns (conv strides not in config, tied output head
question, quality-vs-Kyutai concern)
- Recommended order of attack for the next session
The full port itself is deferred. Ship the foundation now so the
remaining work has a clean handoff.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
||
|
osobhandClaude Opus 4.7
|
3cf02f3aee |
rtx-csm: docs/perf_history.md — Phase 6/7/8 consolidated record
Consolidated record of every shipped optimization, every rejected path
with the data behind the rejection, the production-recommended config,
the architectural lessons captured, and the deferred multi-session
work with realistic sizing.
Headline:
- client first_audio_ms p50 = 529 ms (mock LLM, Q8 + stream + extended warmup + VAD gate)
- real Z.AI loop: ~2 s TTFA p50
- boot cost: ~2.7 s (one-time)
Rejected paths captured (so future sessions don't redo the work):
Q4_K_M (2.85x slower than Q8), whisper-rs linkage (2-3x CSM regression),
Silero V5 via ort (protobuf 3.14 vs 3.21 conflict), Mimi codec Q8
(no candle conv-quant path), KV cache reuse (variance is content-
dependent not state-dependent), rayon for single-connection
(overhead exceeds gain on <100µs tasks), codec swaps (require
backbone retrain), custom distillation (no published checkpoint),
Kyutai 4x flush (hardware-bound on M-series).
Architectural lessons:
1. In-process linkage of external ML runtimes is a recurring trap;
default to sidecar-process pattern.
2. Bench thermals dominate single-machine A/B; 90s cooldown often
necessary.
3. First-frame compilation is the dominant cold-start cost — long
warm-ups are essential.
4. Conv-phase variance is content-dependent, not state-dependent.
5. tokio::join! polls cooperatively — spawn separate tasks for real
concurrency between sync compute and async pump.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
||
|
osobhandClaude Opus 4.7
|
64386720a3 |
rtx-csm: Phase 8.2 — extended boot warm-up (~50% TTFA reduction)
Increased the boot warm-up from `max_audio_ms=200` (~3 frames) to `max_audio_ms=2000` (~25 frames) and switched the prompt to a fuller sentence so more Metal kernel paths fire during the throwaway gen. The original 200 ms warm-up only compiled the early fast paths; the first user turn still paid 4-5 s of additional kernel compilation as new branches lit up under longer-context generation. The 2 s warm-up gives the JIT a chance to compile everything. 3-turn bench (mock LLM, Q8 + stream, M-series Metal, 10.43 s LibriSpeech in): Metric Original warm-up Extended warm-up Δ Boot warm-up cost 819 ms 2748 ms +1.9 s client first_audio_ms p50 4915 ms 529 ms -89% llm_to_first_audio 4892 ms 627 ms -87% total_turn 18408 ms 18112 ms wash Sub-second TTFA on every turn. The extra 2 s at boot is paid back on the first user turn — every turn after is pure win. This is the new production-recommended config: --quantized-gguf <Q8> --stream-tts --vad-gate (warm-up always on; no flag toggle). Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
osobhandClaude Opus 4.7
|
f4a6ffeaf5 |
rtx-csm: validate energy-VAD gate on silence-heavy audio
Adds examples/make_silence_test.rs which builds a 10 s WAV that is 50% silence + 50% real speech (1s silence | 4s speech | 5s silence) so we can see the VAD gate work clearly. Bench A/B (mock LLM, Q8+stream, 3 turns each, M-series Metal): Audio No VAD With VAD Δ recv_phase 90% speech (LibriSpeech) 4834 ms 4509 ms -7% 50% silence (synthetic) 6490 ms 2204 ms -66% The structural win scales with silence content as expected. Real-world voice-agent audio (30-50% silence per typical call-center / voice-bot benchmarks) will see ~30-50% recv_phase reduction. The earlier 7% on LibriSpeech wasn't a weak result — it accurately reflected the ~10% silence in that recording. This validates the energy-VAD path despite Silero V5 via ort being blocked (Phase 8.1.3). Production voice loops should default to --vad-gate. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
osobhandClaude Opus 4.7
|
808c9fae54 |
rtx-csm: Phase 8.1.3 — energy-VAD gate (Silero V5 path blocked)
Original plan: gate STT work via Silero V5 VAD using
voice_activity_detector / ort to skip silent chunks. Phase 8.1.2's
ort_conflict_probe initially passed (no Metal regression). But on
first server boot, sentencepiece-sys's protobuf 3.14 collides with
ort's bundled protobuf 3.21 at process startup:
[libprotobuf FATAL ...] This program was compiled against version
3.14.0 of the Protocol Buffer runtime library, which is not
compatible with the installed version (3.21.12) ... in
sentencepiece-sys-0.13.1
Updated the probe binary to also load Kyutai STT (which links
sentencepiece) — now correctly catches the conflict at probe time.
Same risk class as the whisper-rs/ggml issue from Phase 7.6: external
ML runtimes don't always coexist in the same Rust process.
Pivoted to a pure-Rust energy-based VAD: RMS amplitude threshold per
incoming chunk. ~30 LOC, no external runtime, no protobuf risk.
Catches obvious silence (room tone, pauses) — misses quiet speech
(whispering). Acceptable for typical mic-distance voice loops.
Wired into examples/converse_server as `--vad-gate` /
`--vad-gate-threshold 0.01`. The receive loop checks RMS before each
binary chunk's STT call; silence chunks skip step_pcm entirely.
Orthogonal to existing `--vad` (Kyutai semantic-VAD-for-EOT).
Bench (mock LLM, Q8+stream, 10s LibriSpeech in, ~90% speech):
baseline: recv_phase = 4834 ms
--vad-gate: recv_phase = 4509 ms (-7%, matches the ~10% silence
in this audio)
Real-world voice-agent audio is 30-50% silence; expect proportional
savings (~30-50% recv_phase reduction). Larger silence content =
larger VAD win.
Cargo.toml: keeps the optional `vad` feature + voice_activity_detector
dep wired (path remains for a future Silero-via-candle port). The
ort_conflict_probe example continues to require `--features vad` so
the conflict-detection path stays exercised.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
||
|
osobhandClaude Opus 4.7
|
cdcfbcece1 |
rtx-csm: Phase 8.1.2 — ort runtime-conflict gate (PASS)
Adds optional `vad` feature pulling voice_activity_detector v0.2 (Silero V5 via the `ort` ONNX runtime). Gates wiring VAD into converse_server on a regression check: does linking ort into the same binary as candle slow CSM Metal inference the way whisper-rs (ggml) did? examples/ort_conflict_probe.rs times 10 CSM Q8 forwards, loads ort + runs Silero V5 a few times, then times 10 more forwards. Compares. Result on M-series Metal: before ort load: 645.8 ms mean after ort load: 633.2 ms mean ratio: 0.981 (-1.9%, within noise threshold ±5%) PASS — ort coexists cleanly with candle/Metal. The whisper-rs/ggml regression doesn't generalize to all C++ ML runtimes; ort's Metal backend (via WebGPU EP) doesn't appear to fight with candle's. Safe to ship Silero-VAD gating in Phase 8.1.3. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
osobhandClaude Opus 4.7
|
10c0063e46 |
rtx-csm: Phase 7.2 — STT step_pcm in spawn_blocking
Wraps the per-frame Kyutai STT calls in handle_connection's receive loop with tokio::task::spawn_blocking. The async runtime worker is freed during the ~80ms/frame compute so other tasks/connections can run on the same worker pool. Single-connection latency is unchanged (STT is hardware-bound at ~1× realtime). The win is multi-tenant concurrency: with N WebSocket connections, no longer blocks all N on one connection's STT work. Two call sites updated: - carry-over feed (turn-start barge-in audio) - main receive loop (per-binary-chunk step_pcm) Pattern: clone Arc<Shared>, move into spawn_blocking, use tokio::sync::Mutex::blocking_lock() inside the closure (legal there). The blocking_lock requires the Mutex to be reachable via Send + 'static captures — that's why we clone shared rather than borrow. Cannot show the multi-tenant win in the existing single-connection bench harness; bench thermals also confound direct A/B. Functional correctness verified via running bench (transcripts produced, no errors). Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
osobhandClaude Opus 4.7
|
be92c05d05 |
rtx-csm: Phase 7.6 — Whisper STT path + asr-feature regression discovery
Wires --whisper flag in converse_server using the existing whisper-rs asr feature. AsrEngine enum (Kyutai default + Whisper variant gated on "asr" feature) lets the receive loop branch on backend. Whisper path: buffer audio during receive, transcribe full buffer at EOT — batch-only, no VAD, no incremental words. Adds examples/whisper_profile binary measuring Whisper-tiny in isolation against the same audio used for stt_profile. Standalone profile findings (M-series): Kyutai STT 1B : 80.8 ms / 80 ms audio 1.01x realtime Whisper-tiny : 209 ms / 10.43 s audio 0.020x (~50x faster) But the full-stack bench reveals a critical regression: linking whisper-rs's C++ runtime into the same binary as candle/CSM costs 2-3x across ALL CSM inference (recv_phase, tts_per_utterance, total_turn) even when --whisper is NOT used. Build flag matters. Build recv tts/u total --features metal 4196 3113 18707 --features metal,asr (Kyutai) 10019 7803 43366 <- linkage cost --features metal,asr +whisper 0 12921 54028 <- worse Suspected cause: ggml/whisper.cpp's BLAS or Metal context init conflicts with candle's. Production verdict: build WITHOUT asr feature; accept Kyutai's 1x realtime STT cost. The standalone whisper_profile binary still works for batch transcribe measurement. Real Whisper integration would need a sidecar process pattern (whisper running as a separate binary, IPC to converse_server). Documented in the --whisper CLI help. Flag stays as opt-in with explicit warning. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
osobhandClaude Opus 4.7
|
85ce697ffa |
rtx-csm: Phase 7.1 — profile Kyutai STT step_pcm
New examples/stt_profile binary. Loads kyutai/stt-1b-en_fr, feeds PCM in fixed-size chunks, reports per-call latency p50/p95/min/max plus realtime factor. Findings (M-series Metal, 10.43s LibriSpeech in): frame_batch=1 (80ms): mean=80.8ms p50=81.7ms RT=1.01x frame_batch=3 (240ms): mean=308.6ms p50=298ms RT=1.29x Headline: Kyutai STT 1B on Metal saturates at ~1.0x real-time. There is no slack in the existing model on this hardware. Per-call overhead amortizes poorly when batching frames (3 frames takes 3.8x single frame, not 3x). To go faster requires a smaller model (Whisper-tiny via the existing whisper-rs feature) or a Kyutai variant if available. Note: converse_server's measured recv_phase (~4-5s for 10.4s audio) is faster than this profile predicts (~10s). Discrepancy not yet resolved but the optimization conclusion stands: STT model swap is the only lever for the receive phase. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
osobhandClaude Opus 4.7
|
83e80dc721 |
rtx-csm: add deferred clawsample-csm integration plan
Detailed plan for exposing rtx-csm as a managed service on the clawsample platform: crate layout (mirrors clawsample-demucs + clawsample-gen), HTTP routes (/v1/tts, /v1/tts/async, /v1/voice_profile, WS /v1/converse), DB schema, R2 paths, webhook dispatch, ordered TDD task breakdown, open decisions, acceptance criteria, reference commits. Estimated effort: 5-10 days. Trigger to start: a real consumer for the public TTS API, OR standalone converse_server hits a hard ceiling. Until then, integration is product/platform work, not ML work. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
osobhandClaude Opus 4.7
|
ae400a7bd9 |
rtx-csm: Phase 6f.stream-tts — per-chunk TTS streaming with spawned conv task
Adds Converse::run_streaming using Generator::generate_streaming. Chunks are emitted via a per-chunk callback as they're decoded; per-sentence callback fires once for /metrics + transcript bookkeeping. Skips post-process and watermark — both require full-sentence context. Wires --stream-tts and --stream-chunk-frames into examples/converse_server. Combining --stream-tts with --watermark-* is rejected at boot. Critical architectural fix: tokio::join!(conv_fut, pump_fut) polls cooperatively in ONE task, so the sync TTS gen blocks the runtime worker and prevents pump_fut from draining the chunk channel. Switched to tokio::spawn(conv_fut) + pump_fut.await on the original task — now the runtime can schedule pump_fut on a different worker while conv_fut is mid-synthesize. Without the spawn, chunks pile up and arrive in burst at sentence boundaries (defeating the streaming purpose). A/B (warmed FP, mock LLM, 10.43s LibriSpeech in): Phase Non-stream Stream+spawn Δ llm_to_first_audio 4892ms 1942ms -2950ms total_turn 18408ms 19628ms +1220ms (variance) Client-side TTFA 9345ms 6789ms -2556ms ~3s TTFA reduction is the production win. total_turn is roughly unchanged (chunk delivery overhead exists but is small). For voice-loop deployments where time-to-first-audio dominates UX, this is the right default. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
osobhandClaude Opus 4.7
|
2ce6f8ff32 |
rtx-csm: bench harness parses /metrics phase gauges
Pulls the new per-phase server-side gauges (rtx_csm_recv_phase_ms_avg,
rtx_csm_llm_to_first_audio_ms_avg, rtx_csm_conv_total_ms_avg,
rtx_csm_total_turn_ms_avg, plus the older stt/tts/e2e_first ones) and
pretty-prints them in a labeled block right after the client-side
percentile stats. Raw /metrics is still emitted at the bottom for
anyone who wants the original output.
Sample output (mock LLM, 2 turns, FP CSM):
--- per-phase stats (client-side) ---
audio_send (n=2): mean=1ms p50=1ms p95=1ms min=1ms max=1ms
transcript_ms (n=2): mean=4450ms p50=4497ms ...
first_audio_ms (n=2): mean=3908ms p50=3915ms ...
turn_total_ms (n=2): mean=17171ms p50=17176ms ...
--- server-side phase averages (across all turns) ---
recv_phase 4345ms
llm_to_first_audio 3940ms
conv_total 12742ms
total_turn 17127ms
stt_post 0ms
tts_per_utterance 2679ms
e2e_first_audio 8325ms
Client/server numbers align tightly: client transcript_ms ≈ server
recv_phase, client first_audio_ms ≈ server llm_to_first_audio.
Discrepancies above ~5% indicate machine variance or non-realtime
client pacing artifacts.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
||
|
osobhandClaude Opus 4.7
|
abc07ffd7e |
rtx-csm: Phase 6f.warmup — pre-warm Generator at server boot
Issues one throwaway Generator.generate() call after model load + before accepting connections. Pays Metal kernel compilation + first-frame KV cache init up-front so the first real user turn doesn't carry that overhead. A/B (mock LLM, FP CSM, M-series Metal, 10.43s LibriSpeech in): no warm-up sentence[0] tts_gen=5679ms total=24818ms with warm-up sentence[0] tts_gen=4272ms total=18654ms Warm-up cost at boot: 1417ms. Direct saving on first-sentence TTS gen: ~1.4s. The warm-up pays for itself on the first turn and is amortized to zero across the server's lifetime. Failures during warm-up are logged at warn level and the server boots anyway — first turn falls back to the original cold-start behavior. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |