rustytorch

63 Commits 1 Branch 0 Tags

2 Commits

Include renames

Author	SHA1	Message	Date
osobhandClaude Opus 4.7	a8e729a826	rtx-csm: Phase 8.6 — Moonshine encoder transformer block Full encoder forward path: conv stem -> 6 transformer layers -> final LayerNorm. Loads HF safetensors, runs end-to-end on Metal. Components added to src/moonshine.rs: RotaryCache partial RoPE (32 of 36 head_dim, theta=10000) EncoderAttention MHA (8 heads, no bias), partial RoPE on q/k EncoderMlp 288 -> 1152 -> 288 with bias, GELU(erf) activation EncoderLayer Pre-LN attn + Pre-LN MLP (LayerNorm weight-only) Encoder stem + 6 layers + final LayerNorm load_encoder() VarBuilder convenience for the standalone smoke Smoke test (`examples/moonshine_smoke`) verified end-to-end: input (1, 1, 160000) -> output (1, 415, 288) forward: 132 ms (10 s of audio at 0.013x realtime) max abs: 6.67 (signal preserved, not zeros) Implementation notes captured in the diff: - candle Metal 4D batched matmul had shape-mismatch issues for our (B, H, T, D) pattern. Switched to (BH, T, D) 3D form which is unambiguous and avoids the kernel bug. - LayerNorm is weight-only (no bias tensors in safetensors); we construct LayerNorm with a zeros bias to satisfy candle's API. - rotary_dim = floor(head_dim 0.9 / 2) * 2 = 32 (must be even). The remaining 4 head_dim channels pass through unchanged via `narrow + cat` on dim 3. Numerical parity vs HF Python reference is NOT yet verified — that's the next bounded chunk (Phase 8.7). Shape + signal correctness are verified by the smoke test. Next: decoder transformer block (self-attn + cross-attn + SwiGLU). ~3-4 h of focused work. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>	2026-04-27 10:39:48 -07:00
osobhandClaude Opus 4.7	0699cdeb45	rtx-csm: Phase 8.5 — Moonshine conv stem (verified end-to-end) First working piece of the Moonshine v2 candle port. New src/moonshine.rs module with: - MoonshineConfig::tiny() — hyperparameters from HF config.json - ConvStem (Conv1d × 3) — audio stem, raw 16 kHz → 288-d hidden - load_conv_stem() — VarBuilder from HF safetensors Conv layout (verified against HF source): conv1: in=1, out=288, k=127, stride=64, no bias conv2: in=288, out=576, k=7, stride=3, bias conv3: in=576, out=288, k=3, stride=2, bias Activations: tanh after conv1, gelu_erf after conv2 / conv3 Smoke test (`examples/moonshine_smoke`): - Downloads UsefulSensors/moonshine-tiny from HF - Synthetic 10 s @ 16 kHz audio (silence + sine pulse) - input (1, 1, 160000) -> output (1, 415, 288) - Expected T_seq=415 ((160000-127)/64+1 -> 2498 -> 831 -> 415) - Output max abs = 23.17 (real signal, weights loaded correctly) Also extends `examples/moonshine_inspect` to dump conv shapes explicitly (was being truncated by the per-prefix `take(8)` cap). Next ship: encoder transformer block (partial RoPE, GELU MLP) and output layer norm. Tracked in Phase 8 plan; ~2-3 hours of work. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>	2026-04-27 10:27:51 -07:00

Author

SHA1

Message

Date

osobhandClaude Opus 4.7

a8e729a826

rtx-csm: Phase 8.6 — Moonshine encoder transformer block

Full encoder forward path: conv stem -> 6 transformer layers -> final
LayerNorm. Loads HF safetensors, runs end-to-end on Metal.

Components added to src/moonshine.rs:

  RotaryCache       partial RoPE (32 of 36 head_dim, theta=10000)
  EncoderAttention  MHA (8 heads, no bias), partial RoPE on q/k
  EncoderMlp        288 -> 1152 -> 288 with bias, GELU(erf) activation
  EncoderLayer      Pre-LN attn + Pre-LN MLP (LayerNorm weight-only)
  Encoder           stem + 6 layers + final LayerNorm
  load_encoder()    VarBuilder convenience for the standalone smoke

Smoke test (`examples/moonshine_smoke`) verified end-to-end:
  input  (1, 1, 160000)  -> output (1, 415, 288)
  forward: 132 ms        (10 s of audio at 0.013x realtime)
  max abs: 6.67          (signal preserved, not zeros)

Implementation notes captured in the diff:
  - candle Metal 4D batched matmul had shape-mismatch issues for our
    (B, H, T, D) pattern. Switched to (B*H, T, D) 3D form which is
    unambiguous and avoids the kernel bug.
  - LayerNorm is weight-only (no bias tensors in safetensors); we
    construct LayerNorm with a zeros bias to satisfy candle's API.
  - rotary_dim = floor(head_dim * 0.9 / 2) * 2 = 32 (must be even).
    The remaining 4 head_dim channels pass through unchanged via
    `narrow + cat` on dim 3.

Numerical parity vs HF Python reference is NOT yet verified — that's
the next bounded chunk (Phase 8.7). Shape + signal correctness are
verified by the smoke test.

Next: decoder transformer block (self-attn + cross-attn + SwiGLU).
~3-4 h of focused work.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

2026-04-27 10:39:48 -07:00

osobhandClaude Opus 4.7

0699cdeb45

rtx-csm: Phase 8.5 — Moonshine conv stem (verified end-to-end)

First working piece of the Moonshine v2 candle port. New
src/moonshine.rs module with:

  - MoonshineConfig::tiny()  — hyperparameters from HF config.json
  - ConvStem (Conv1d × 3)    — audio stem, raw 16 kHz → 288-d hidden
  - load_conv_stem()         — VarBuilder from HF safetensors

Conv layout (verified against HF source):
  conv1: in=1,   out=288, k=127, stride=64, no bias
  conv2: in=288, out=576, k=7,   stride=3,  bias
  conv3: in=576, out=288, k=3,   stride=2,  bias
  Activations: tanh after conv1, gelu_erf after conv2 / conv3

Smoke test (`examples/moonshine_smoke`):
  - Downloads UsefulSensors/moonshine-tiny from HF
  - Synthetic 10 s @ 16 kHz audio (silence + sine pulse)
  - input (1, 1, 160000) -> output (1, 415, 288)
  - Expected T_seq=415 ((160000-127)/64+1 -> 2498 -> 831 -> 415)
  - Output max abs = 23.17 (real signal, weights loaded correctly)

Also extends `examples/moonshine_inspect` to dump conv shapes
explicitly (was being truncated by the per-prefix `take(8)` cap).

Next ship: encoder transformer block (partial RoPE, GELU MLP) and
output layer norm. Tracked in Phase 8 plan; ~2-3 hours of work.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

2026-04-27 10:27:51 -07:00