Closes the LoRA inference path that was previously stubbed. Two new
public APIs in rtx-csm:
1. lora::load_lora_set_from_safetensors(path, config) -> LoraSet
Reads a trained adapter file (produced by training::
save_lora_adapter[_with_metadata]). Pairs the .lora_a / .lora_b
tensors by base-weight prefix into LoraAdapter entries.
2. lora::merge_into_safetensors(base, lora, scale, output)
Reads the base CSM safetensors, folds in the LoRA deltas at the
given scale (typically alpha/rank from training), writes a merged
safetensors. Original dtype preserved (F16 on Metal, BF16 on
CUDA, F32 on CPU). Tensors LoRA doesn't target are passed
through unchanged.
3. Generator::load_csm_1b_from_path(path, device)
Variant of load_csm_1b that takes an explicit weights path
instead of going through the HF cache. Mimi + tokenizer still
resolve via the hub. This is the path consumers use to load a
merged checkpoint.
MergeReport struct restructured to expose merged/skipped/passthrough
counts so callers can verify the adapter actually targeted weights.
The previous typed-error test is replaced with a missing-base-file
test that exercises the real code path.
Used by zeroclaw-channel-voice's `--lora-adapter` flag to bake a
LoRA adapter into a per-process merged checkpoint at boot, with
zero per-inference overhead.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Adds set_decoder_steering on Model + Generator and
--decoder-steering-vec / --decoder-steering-scale on examples/generate.
The decoder is already a LlamaModel under the hood, so the existing
LayerSteering hook in csm_fork::Layer::forward applies as-is — only
the public surface needed wiring.
Architectural hypothesis being tested: backbone carries semantic
content (what the model says), depth decoder carries acoustic detail
(how it sounds). Backbone steering shifts character at the cost of
text fidelity (Sprint 2 finding); decoder steering should shift
prosody/timbre without disturbing word content.
Smoke test with random Gaussian decoder vectors (4 layers × 1024
embed_dim, stddev 0.1, scale 0.5):
case cos WER transcript
baseline 0.72 1.0 "No."
backbone 0.83 1.4 "That's for on-beat for bee..."
decoder 0.76 1.0 "So" (premature EOT)
both 0.81 3.0 "I'm going to go to the next one..."
Decoder steering DOES alter output (cosine 0.72 → 0.76, transcript
changes) but random vectors trigger premature EOT — same pattern as
random backbone vectors. The infrastructure works; getting the real
emotion-from-acoustic-codebooks signal needs decoder activation
capture, which the current Model::capture_backbone_activations
doesn't do (it captures the backbone forward only).
Decoder capture is the next-session item. With it we can extract
real per-emotion decoder vectors from RAVDESS and test the
hypothesis properly.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Per-frame CFG scale schedule (arXiv 2509.19668, Zheng & Maleki). Pure
inference-time. Standard CFG uses one fixed scale for the whole
sequence; this lets the scale vary across frames so early frames
(speaker character) get full CFG and later frames (text adherence)
get a lower scale.
What lands:
- src/cfg_schedule.rs: CfgSchedule enum (Constant / Step /
LinearRamp), scale_at(frame_idx), parser for CLI form
`step:E:L:T | linear:S:E:R | const:X`. 6 unit tests.
- src/generator.rs: GenerateOptions::cfg_schedule (takes precedence
over legacy cfg_scale; fixed-f64 path is preserved as
Constant(s) for back-compat). Generation loop reads
schedule.scale_at(frame_idx) and passes per-frame to
generate_frame_cfg.
- examples/generate.rs: --cfg-schedule, --cfg-scale, --enable-cfg
flags. Loading via load_csm_1b_with_cfg when --enable-cfg.
A/B with 6s output on Amini context, prompt about Selective CFG:
case cos WER transcript
no-CFG baseline 0.944 1.50 "Okay, the M.U. worked..." (off)
const:2.0 0.854 0.92 "On the right side." (short)
step:3.0:1.5:12 0.938 1.00 "The officer for the selective
C.F.D. paper recommends" (best)
linear:3.0:1.0:25 0.854 1.08 "On the surface..." (off)
Step schedule produces the transcript closest to the input ("the
selective CFG paper recommends..."). WER stays at 1.0 because
Moonshine doesn't know "CFG" as a word, but qualitatively this is
the only one that's coherently following the prompt. Speaker cosine
stays ≈ baseline (0.94) instead of dropping to 0.85 like the
constant and linear cases.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Adds the apply hook for ActAdd-style activation steering on the Llama
backbone. Inspired by EmoSteer-TTS (arXiv 2508.03543), but adapted: the
paper is flow-matching-specific (DiT layers, 32 CFM steps, per-token
attribution search via mel synthesis), none of which apply to CSM's
autoregressive Llama-over-Mimi-tokens. What's portable is the
underlying difference-in-means construction with residual-stream
addition — the standard ActAdd / contrastive-steering pattern.
What lands:
- src/steering.rs: LayerSteering type, per-layer (1, embed_dim) tensors,
global scale, safetensors load with keys `layer_<i>_steering`. Three
unit tests covering empty/no-op, dimension validation, and apply math.
- src/csm_fork.rs LlamaModel: optional `steering: Option<LayerSteering>`
field, applied after every layer's forward inside the for-loop. Adds
~3 LOC to the hot path; gated by the Option so unsteered generation
has zero cost beyond a None check.
- src/csm_fork.rs Model::set_backbone_steering: installs steering only
on the conditional backbone (cfg_backbone is intentionally left
un-steered so CFG correctly subtracts an unsteered baseline).
- src/generator.rs Generator::set_steering: errors on quantized
backend (only FP supported for now).
- examples/generate.rs: --steering-vec / --steering-scale flags.
- examples/steering_random.rs: smoke helper that writes random Gaussian
vectors so the apply path can be exercised end-to-end before the
real corpus extractor lands. Box-Muller via seeded rand to avoid an
extra rand_distr dep.
Smoke test (16-layer random Gaussian, stddev=0.05, scale=0.5):
- baseline (no steering, same seed/text): 3.04 s @ RMS -19.5 dB
- steered (random vectors): 1.84 s @ RMS -16.2 dB,
EOT triggered earlier
Output clearly differs — pathway is wired correctly. Random vectors
aren't musically meaningful; that's Phase B.
Phase B (next session): corpus extractor that runs forward passes over
emotion-labeled audio (we already have audio_to_manifest emitting
emotion_tag rows), captures per-layer post-residual activations, and
computes the difference-in-means between emotion_X and neutral pools.
Then A/B with quality_eval.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Closes the personal_voice_training_guide.md §4 stack: capacity (12.1) +
control tokens (12.2) + multi-stage curriculum (this).
TrainingExample gains emotion_tag + stage. Trainer::train applies the
tag via the same apply_emotion_hint helper inference uses (now
pub(crate)) — training and inference must use identical prefix
formatting or the adapter won't transfer.
TrainingDataset::load_from_manifest reads JSONL
`{wav, transcript, emotion_tag?, stage?, speaker?}` rows; wav paths
resolve relative to manifest dir.
CurriculumStage + CurriculumTrainer run N stages sequentially against a
shared VarMap. Per stage: filter by ex.stage label, build a transient
sub-dataset, run Trainer, save snapshot if requested. The "*" stage
name is a global catch-all.
examples/lora_train_emotional.rs wraps the canonical 3-stage recipe:
audiobook (3 ep × lr 1e-4) → podcast (1 ep × lr 3e-5) → va (1 ep ×
lr 1e-5). --extended-lora recommended (FFN is the prosodic-style
carrier per the guide).
Verified end-to-end on Metal: 3-row manifest → all 3 stages execute,
checkpoints + final adapter written, prompt-token lengths varied by
emotion-tag length (9 vs 11 for different tags) confirming the tag
flowed through the training tokenization. Lib suite 96/96.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
GenerateOptions gains emotion_hint: Option<String>. After text
normalization, the hint (if Some) is prepended as `<tag> <text>` so the
Llama BPE tokenizer encodes it as ordinary tokens. Plumbed through
generate, generate_streaming, and generate_with_profile (via delegation),
plus a `--emotion-hint` flag on examples/generate and a Shared field
+ CLI flag on examples/converse_server (per-turn ConverseOptions).
GenerateOptions lost Copy because Option<String> isn't Copy; updated
the four callers that depended on it (bench, longform, converse synth
+ stream) to .clone() the opts at the call site. Cheap — the struct
is small and clones are per-turn, not per-frame.
On the un-adapted base this is a no-op cosmetic prefix. The point is to
unlock Phase 12.1-fine-tuned adapters: train with `[whisper] X` paired
with whispered audio, and the adapter learns the tag→prosody mapping at
inference time.
Verified end-to-end: --emotion-hint "[whisper]" --max-audio-ms 3000
produced a valid 24kHz mono WAV through tokenizer → backbone → Mimi
with no panics. Lib suite 96/96 (added 4 apply_emotion_hint unit tests).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
A single \`generate\` invocation now produces a watermarked WAV when
AudioSeal weights are passed via CLI. End-to-end verified on real CSM
speech: mean_presence=1.0000, 16/16 message bits decoded.
- Generator gains \`watermarker: Option<Box<dyn Watermarker>>\` slot;
\`generate_to_wav\` runs \`wm.embed(&pcm)\` after post-process, before
WAV write. Field is Send+Sync so the existing Arc<Mutex<Generator>>
tts_server pattern still works.
- watermark.rs ships ResampledWatermarker<W> adapter for handling rate
mismatches (CSM 24 kHz ↔ AudioSeal 16 kHz). Output length is normalized
to input length so it's a transparent drop-in.
- examples/generate.rs gains --watermark-generator/--watermark-detector/
--watermark-message flags. Loads AudioSeal, wraps in resampler, installs.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
redclawsystems