rustytorch

502 Commits 1 Branch 0 Tags

3 Commits

Include renames

Author	SHA1	Message	Date
redclawsystems	ae53983c03	style: cargo fmt --all (18 files) Auto-merged by ci-doctor.	2026-05-07 16:30:04 +00:00
osobhandClaude Opus 4.7	af6c246ef6	rtx-csm: layer-subset steering — recovers Sprint 2 fluency EmoSteer-TTS (arXiv 2508.03543) targets only a spaced subset of middle-to-deep DiT layers (1, 6, 11, 16, 21 of 32) rather than every layer. Previously our LayerSteering applied all 16 vectors which — combined with the noisy emotion-label corpus from Phase B — destroyed output fluency at scale 0.5. Adds: - LayerSteering::restrict_to_layers(&[usize]) — clears every vector whose index isn't in the allowlist. Plus active_layers() inspector and a unit test. - examples/generate --steering-layers 8,10,12 — comma-separated CLI flag that runs restrict_to_layers after load. A/B with the existing excited-vs-surprised vectors at scale=0.5 across five layer subsets: case cos WER transcript baseline 0.45 0.92 "It is a very important thing to do." all16 0.37 1.00 "© transcript Emily Beynon" (broken) [8,10,12] 0.63 0.54 "I want to talk about something." ✓ [4,8,12] 0.42 1.00 "Oh, my God." (broken — layer 4 too early) [12-15] 0.48 0.77 "I want to have fun with that." The mid-layer subset is the clear winner — highest speaker cosine, lowest WER, transcript closest to the prompt ("Today I want to talk about something genuinely important..."). Including layer 4 destroys output fluency even at scale 0.5, validating the paper's avoidance of shallow layers. Pure-deep is between mid and broken. This unblocks Phase B's empirical validation: even with the noisy auto-tagged corpus, the extracted vectors produce meaningful steering when applied to the right layers. A real labeled emotion dataset should compound from here. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>	2026-04-29 10:16:09 -07:00
osobhandClaude Opus 4.7	aa274f2210	rtx-csm: Sprint 2 Phase A — activation steering API Adds the apply hook for ActAdd-style activation steering on the Llama backbone. Inspired by EmoSteer-TTS (arXiv 2508.03543), but adapted: the paper is flow-matching-specific (DiT layers, 32 CFM steps, per-token attribution search via mel synthesis), none of which apply to CSM's autoregressive Llama-over-Mimi-tokens. What's portable is the underlying difference-in-means construction with residual-stream addition — the standard ActAdd / contrastive-steering pattern. What lands: - src/steering.rs: LayerSteering type, per-layer (1, embed_dim) tensors, global scale, safetensors load with keys `layer_<i>_steering`. Three unit tests covering empty/no-op, dimension validation, and apply math. - src/csm_fork.rs LlamaModel: optional `steering: Option<LayerSteering>` field, applied after every layer's forward inside the for-loop. Adds ~3 LOC to the hot path; gated by the Option so unsteered generation has zero cost beyond a None check. - src/csm_fork.rs Model::set_backbone_steering: installs steering only on the conditional backbone (cfg_backbone is intentionally left un-steered so CFG correctly subtracts an unsteered baseline). - src/generator.rs Generator::set_steering: errors on quantized backend (only FP supported for now). - examples/generate.rs: --steering-vec / --steering-scale flags. - examples/steering_random.rs: smoke helper that writes random Gaussian vectors so the apply path can be exercised end-to-end before the real corpus extractor lands. Box-Muller via seeded rand to avoid an extra rand_distr dep. Smoke test (16-layer random Gaussian, stddev=0.05, scale=0.5): - baseline (no steering, same seed/text): 3.04 s @ RMS -19.5 dB - steered (random vectors): 1.84 s @ RMS -16.2 dB, EOT triggered earlier Output clearly differs — pathway is wired correctly. Random vectors aren't musically meaningful; that's Phase B. Phase B (next session): corpus extractor that runs forward passes over emotion-labeled audio (we already have audio_to_manifest emitting emotion_tag rows), captures per-layer post-residual activations, and computes the difference-in-means between emotion_X and neutral pools. Then A/B with quality_eval. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>	2026-04-29 08:45:58 -07:00

Author

SHA1

Message

Date

redclawsystems

ae53983c03

style: cargo fmt --all (18 files)

Auto-merged by ci-doctor.

2026-05-07 16:30:04 +00:00

osobhandClaude Opus 4.7

af6c246ef6

rtx-csm: layer-subset steering — recovers Sprint 2 fluency

EmoSteer-TTS (arXiv 2508.03543) targets only a spaced subset of
middle-to-deep DiT layers (1, 6, 11, 16, 21 of 32) rather than every
layer. Previously our LayerSteering applied all 16 vectors which —
combined with the noisy emotion-label corpus from Phase B — destroyed
output fluency at scale 0.5.

Adds:
- LayerSteering::restrict_to_layers(&[usize]) — clears every vector
  whose index isn't in the allowlist. Plus active_layers() inspector
  and a unit test.
- examples/generate --steering-layers 8,10,12 — comma-separated CLI
  flag that runs restrict_to_layers after load.

A/B with the existing excited-vs-surprised vectors at scale=0.5
across five layer subsets:

  case      cos   WER    transcript
  baseline  0.45  0.92   "It is a very important thing to do."
  all16     0.37  1.00   "© transcript Emily Beynon" (broken)
  [8,10,12] 0.63  0.54   "I want to talk about something."  ✓
  [4,8,12]  0.42  1.00   "Oh, my God." (broken — layer 4 too early)
  [12-15]   0.48  0.77   "I want to have fun with that."

The mid-layer subset is the clear winner — highest speaker cosine,
lowest WER, transcript closest to the prompt ("Today I want to talk
about something genuinely important..."). Including layer 4 destroys
output fluency even at scale 0.5, validating the paper's avoidance of
shallow layers. Pure-deep is between mid and broken.

This unblocks Phase B's empirical validation: even with the noisy
auto-tagged corpus, the extracted vectors produce meaningful steering
when applied to the right layers. A real labeled emotion dataset
should compound from here.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

2026-04-29 10:16:09 -07:00

osobhandClaude Opus 4.7

aa274f2210

rtx-csm: Sprint 2 Phase A — activation steering API

Adds the apply hook for ActAdd-style activation steering on the Llama
backbone. Inspired by EmoSteer-TTS (arXiv 2508.03543), but adapted: the
paper is flow-matching-specific (DiT layers, 32 CFM steps, per-token
attribution search via mel synthesis), none of which apply to CSM's
autoregressive Llama-over-Mimi-tokens. What's portable is the
underlying difference-in-means construction with residual-stream
addition — the standard ActAdd / contrastive-steering pattern.

What lands:
- src/steering.rs: LayerSteering type, per-layer (1, embed_dim) tensors,
  global scale, safetensors load with keys `layer_<i>_steering`. Three
  unit tests covering empty/no-op, dimension validation, and apply math.
- src/csm_fork.rs LlamaModel: optional `steering: Option<LayerSteering>`
  field, applied after every layer's forward inside the for-loop. Adds
  ~3 LOC to the hot path; gated by the Option so unsteered generation
  has zero cost beyond a None check.
- src/csm_fork.rs Model::set_backbone_steering: installs steering only
  on the conditional backbone (cfg_backbone is intentionally left
  un-steered so CFG correctly subtracts an unsteered baseline).
- src/generator.rs Generator::set_steering: errors on quantized
  backend (only FP supported for now).
- examples/generate.rs: --steering-vec / --steering-scale flags.
- examples/steering_random.rs: smoke helper that writes random Gaussian
  vectors so the apply path can be exercised end-to-end before the
  real corpus extractor lands. Box-Muller via seeded rand to avoid an
  extra rand_distr dep.

Smoke test (16-layer random Gaussian, stddev=0.05, scale=0.5):
- baseline (no steering, same seed/text): 3.04 s @ RMS -19.5 dB
- steered (random vectors):              1.84 s @ RMS -16.2 dB,
                                          EOT triggered earlier
Output clearly differs — pathway is wired correctly. Random vectors
aren't musically meaningful; that's Phase B.

Phase B (next session): corpus extractor that runs forward passes over
emotion-labeled audio (we already have audio_to_manifest emitting
emotion_tag rows), captures per-layer post-residual activations, and
computes the difference-in-means between emotion_X and neutral pools.
Then A/B with quality_eval.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

2026-04-29 08:45:58 -07:00