Mirror of --steering-layers for the decoder: comma-separated layer
indices to actually steer (decoder has 4 layers; useful subsets are
[3], [2,3], [1,2]).
Sweep at seed 7, [email protected]:
[0,1,2,3] cos 0.86 WER 0.93 "I'm not that tall ×3" ← repetition
[3] cos 0.58 WER 0.86 "I'll be off and offense..."
[2,3] cos 0.80 WER 1.43 "My daughter, Penny Ryan, and I have…"
[0,1] cos 0.82 WER 1.57 "And I'll check on them..."
[1,2] cos 0.81 WER 1.43 "I'm going to call him an X-Man..."
The repetition is specific to all-layers-at-once steering. Any 2-layer
subset eliminates it while preserving most of the cosine boost. Same
pattern as the backbone's [8,10,12] finding: partial perturbation
lets the unsteered layers act as a stabilizing prior.
[2,3] (decoder last 2) is the new recommended recipe — best cosine
of the no-repetition subsets and the longest fluent transcript.
Documented in docs/perf_history.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Adds set_decoder_steering on Model + Generator and
--decoder-steering-vec / --decoder-steering-scale on examples/generate.
The decoder is already a LlamaModel under the hood, so the existing
LayerSteering hook in csm_fork::Layer::forward applies as-is — only
the public surface needed wiring.
Architectural hypothesis being tested: backbone carries semantic
content (what the model says), depth decoder carries acoustic detail
(how it sounds). Backbone steering shifts character at the cost of
text fidelity (Sprint 2 finding); decoder steering should shift
prosody/timbre without disturbing word content.
Smoke test with random Gaussian decoder vectors (4 layers × 1024
embed_dim, stddev 0.1, scale 0.5):
case cos WER transcript
baseline 0.72 1.0 "No."
backbone 0.83 1.4 "That's for on-beat for bee..."
decoder 0.76 1.0 "So" (premature EOT)
both 0.81 3.0 "I'm going to go to the next one..."
Decoder steering DOES alter output (cosine 0.72 → 0.76, transcript
changes) but random vectors trigger premature EOT — same pattern as
random backbone vectors. The infrastructure works; getting the real
emotion-from-acoustic-codebooks signal needs decoder activation
capture, which the current Model::capture_backbone_activations
doesn't do (it captures the backbone forward only).
Decoder capture is the next-session item. With it we can extract
real per-emotion decoder vectors from RAVDESS and test the
hypothesis properly.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Per-frame CFG scale schedule (arXiv 2509.19668, Zheng & Maleki). Pure
inference-time. Standard CFG uses one fixed scale for the whole
sequence; this lets the scale vary across frames so early frames
(speaker character) get full CFG and later frames (text adherence)
get a lower scale.
What lands:
- src/cfg_schedule.rs: CfgSchedule enum (Constant / Step /
LinearRamp), scale_at(frame_idx), parser for CLI form
`step:E:L:T | linear:S:E:R | const:X`. 6 unit tests.
- src/generator.rs: GenerateOptions::cfg_schedule (takes precedence
over legacy cfg_scale; fixed-f64 path is preserved as
Constant(s) for back-compat). Generation loop reads
schedule.scale_at(frame_idx) and passes per-frame to
generate_frame_cfg.
- examples/generate.rs: --cfg-schedule, --cfg-scale, --enable-cfg
flags. Loading via load_csm_1b_with_cfg when --enable-cfg.
A/B with 6s output on Amini context, prompt about Selective CFG:
case cos WER transcript
no-CFG baseline 0.944 1.50 "Okay, the M.U. worked..." (off)
const:2.0 0.854 0.92 "On the right side." (short)
step:3.0:1.5:12 0.938 1.00 "The officer for the selective
C.F.D. paper recommends" (best)
linear:3.0:1.0:25 0.854 1.08 "On the surface..." (off)
Step schedule produces the transcript closest to the input ("the
selective CFG paper recommends..."). WER stays at 1.0 because
Moonshine doesn't know "CFG" as a word, but qualitatively this is
the only one that's coherently following the prompt. Speaker cosine
stays ≈ baseline (0.94) instead of dropping to 0.85 like the
constant and linear cases.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Adds the apply hook for ActAdd-style activation steering on the Llama
backbone. Inspired by EmoSteer-TTS (arXiv 2508.03543), but adapted: the
paper is flow-matching-specific (DiT layers, 32 CFM steps, per-token
attribution search via mel synthesis), none of which apply to CSM's
autoregressive Llama-over-Mimi-tokens. What's portable is the
underlying difference-in-means construction with residual-stream
addition — the standard ActAdd / contrastive-steering pattern.
What lands:
- src/steering.rs: LayerSteering type, per-layer (1, embed_dim) tensors,
global scale, safetensors load with keys `layer_<i>_steering`. Three
unit tests covering empty/no-op, dimension validation, and apply math.
- src/csm_fork.rs LlamaModel: optional `steering: Option<LayerSteering>`
field, applied after every layer's forward inside the for-loop. Adds
~3 LOC to the hot path; gated by the Option so unsteered generation
has zero cost beyond a None check.
- src/csm_fork.rs Model::set_backbone_steering: installs steering only
on the conditional backbone (cfg_backbone is intentionally left
un-steered so CFG correctly subtracts an unsteered baseline).
- src/generator.rs Generator::set_steering: errors on quantized
backend (only FP supported for now).
- examples/generate.rs: --steering-vec / --steering-scale flags.
- examples/steering_random.rs: smoke helper that writes random Gaussian
vectors so the apply path can be exercised end-to-end before the
real corpus extractor lands. Box-Muller via seeded rand to avoid an
extra rand_distr dep.
Smoke test (16-layer random Gaussian, stddev=0.05, scale=0.5):
- baseline (no steering, same seed/text): 3.04 s @ RMS -19.5 dB
- steered (random vectors): 1.84 s @ RMS -16.2 dB,
EOT triggered earlier
Output clearly differs — pathway is wired correctly. Random vectors
aren't musically meaningful; that's Phase B.
Phase B (next session): corpus extractor that runs forward passes over
emotion-labeled audio (we already have audio_to_manifest emitting
emotion_tag rows), captures per-layer post-residual activations, and
computes the difference-in-means between emotion_X and neutral pools.
Then A/B with quality_eval.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
scripts/pick_context.sh ranks manifest clips by suitability for CSM-1B's
--context-wav conditioning (duration sweet spot 10-13.5s, RMS -25 to
-15 dB, peak ≤ -3 dBFS) and groups by (manifest_stem, speaker_id) since
diarizer labels are per-file.
examples/generate.rs now accepts repeatable --context-wav and
--context-text pairs, zipped into Vec<Segment> for Generator::generate.
Validates equal counts at runtime.
Smoke-tested with two 10-13s spk0 clips from the MIT-2024 manifest.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Trained adapters now embed (rank, alpha, target_modules, crate_version)
as JSON in safetensors __metadata__["rtx_csm_lora"]. apply_lora_adapter
reads it at load time so users no longer have to remember matching
--lora-rank/--lora-alpha/--extended-lora flags from training.
LoraAdapterMetadata::is_extended() heuristic: target_modules contains
any MLP path or output_proj or k_proj. Handles both the canonical
extended() preset and future custom configs that overlap it.
apply_lora_adapter rank/alpha/extended params became Option<_>
(None = use file metadata, Some = override). Both callers updated.
save_lora_adapter_with_metadata is the new path used by both trainers;
the plain save_lora_adapter still exists for the metadata-less case
(per-stage curriculum snapshots).
safetensors dep bumped 0.4 → 0.7 to match candle 0.9's transitive pin
so candle's Tensor: View impl is in scope for serialize_to_file
(candle's own save wrapper hardcodes the metadata arg to None).
Backward compat: pre-12.5 adapters load fine when explicit CLI flags
are passed; auto-detect path is skipped silently.
Verified end-to-end: trained adapter saved with metadata,
`generate --lora <path>` (no other flags) auto-detected
rank=8 alpha=16 extended=true and applied. Older metadata-less adapter
still loaded with explicit flags. 3 new unit tests; lib suite 99/99.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Closes the train→generate loop the lora_train comment has promised
since Phase 3 ("forthcoming --lora flag on generate").
apply_lora_adapter(generator, path, rank, alpha, extended, device)
shared helper in src/training.rs wraps add_lora_to_backbone +
load_lora_adapter + refresh_lora. Both examples/generate and
examples/converse_server now call it instead of inlining their own
versions, and both now take --extended-lora to opt into Phase 12.1
coverage. Classic q+v adapters still load without the flag.
lora_train.rs now prints the exact `--lora <path> --lora-rank N
--lora-alpha N [--extended-lora]` command-line you need to apply the
trained adapter at inference, replacing the (forthcoming) message.
End-to-end verified: a Phase 12.3 curriculum-trained adapter loaded
into generate with identical seed/text produces different audio
(92KB vs 61KB, EOT @ frame 24 vs 16) — confirming the adapter takes
effect through to the sampled output. The 3-utterance smoke adapter
hasn't learned anything meaningful but the wiring is sound.
Phase 12 emotional voice stack now complete end-to-end:
12.1 capacity → 12.2 control tokens → 12.3 curriculum → 12.4 inference.
Lib suite 96/96.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
GenerateOptions gains emotion_hint: Option<String>. After text
normalization, the hint (if Some) is prepended as `<tag> <text>` so the
Llama BPE tokenizer encodes it as ordinary tokens. Plumbed through
generate, generate_streaming, and generate_with_profile (via delegation),
plus a `--emotion-hint` flag on examples/generate and a Shared field
+ CLI flag on examples/converse_server (per-turn ConverseOptions).
GenerateOptions lost Copy because Option<String> isn't Copy; updated
the four callers that depended on it (bench, longform, converse synth
+ stream) to .clone() the opts at the call site. Cheap — the struct
is small and clones are per-turn, not per-frame.
On the un-adapted base this is a no-op cosmetic prefix. The point is to
unlock Phase 12.1-fine-tuned adapters: train with `[whisper] X` paired
with whispered audio, and the adapter learns the tag→prosody mapping at
inference time.
Verified end-to-end: --emotion-hint "[whisper]" --max-audio-ms 3000
produced a valid 24kHz mono WAV through tokenizer → backbone → Mimi
with no panics. Lib suite 96/96 (added 4 apply_emotion_hint unit tests).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
A single \`generate\` invocation now produces a watermarked WAV when
AudioSeal weights are passed via CLI. End-to-end verified on real CSM
speech: mean_presence=1.0000, 16/16 message bits decoded.
- Generator gains \`watermarker: Option<Box<dyn Watermarker>>\` slot;
\`generate_to_wav\` runs \`wm.embed(&pcm)\` after post-process, before
WAV write. Field is Send+Sync so the existing Arc<Mutex<Generator>>
tts_server pattern still works.
- watermark.rs ships ResampledWatermarker<W> adapter for handling rate
mismatches (CSM 24 kHz ↔ AudioSeal 16 kHz). Output length is normalized
to input length so it's a transparent drop-in.
- examples/generate.rs gains --watermark-generator/--watermark-detector/
--watermark-message flags. Loads AudioSeal, wraps in resampler, installs.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
