KV cache for the decoder turns greedy generation from O(T^2) into O(T)
total work. Per-token decode drops modestly on short transcripts
(7.1 -> 6.0 ms/token at 49 tokens) and compounds on longer ones.
New components in src/moonshine.rs:
RotaryCache::apply_at(x, position, t)
Apply RoPE for a window starting at `position`. Replaces
`apply()` for cached step (which always called positions 0..T).
DecoderSelfAttention::forward_step(xs, cache_k, cache_v, rope, position)
Single-token cached self-attn. Appends new K/V to per-layer cache,
attends across full accumulated history. No causal mask needed
(cache only contains positions <= current).
CrossAttention::precompute_kv(enc) -> (K, V)
One-shot encoder K/V projection for cross-attn. Reused every step.
CrossAttention::forward_step(xs, k, v)
Cached cross-attn. Q computed from new token; K/V from precompute.
DecoderCache { self_k: Vec<Option<Tensor>>, self_v, cross_k, cross_v, position }
Decoder::precompute_cross_kv(enc) -> DecoderCache
Decoder::step(token_id, &mut cache) -> logits (1, vocab)
Decoder::generate_cached(enc, cfg, max_tokens) -> Vec<u32>
Greedy loop using the cached step.
Profile (5 steady-state runs on /tmp/asr_test.flac, 10.42 s LibriSpeech):
warm-up: 344 ms
steady-state mean: 307 ms (p50 305, range 298-319)
realtime factor: 0.0294x
Comparison across all STT in rtx-csm:
Backend RTF Notes
Kyutai STT 1B 1.01x hardware-bound, 3 GB
Whisper-tiny 0.020x breaks CSM (in-process ggml conflict)
Moonshine-tiny 0.0294x pure candle, NO runtime conflict
Moonshine is the only fast STT path that integrates cleanly. ~34x
faster than realtime, ~17x faster than Kyutai 1B, no protobuf or
ggml linkage issues.
New `examples/moonshine_profile` mirrors `stt_profile` and
`whisper_profile` so all three STT backends report comparable numbers.
Phase 8.10 (next): wire as a third AsrEngine variant in converse_server
for English-only deploys. Replace the energy-VAD-gated Kyutai path
when --moonshine flag is set.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Full encoder-decoder Moonshine v2 transcribing real audio in pure
candle 0.9 + Metal. No ort, no ggml, no protobuf. The path that
whisper-rs (Phase 7.6) and Silero V5 via ort (Phase 8.1.3) couldn't
deliver due to in-process linkage conflicts.
End-to-end on /tmp/asr_test.flac (LibriSpeech, 10.42 s):
encode: 10 ms
decode: 348 ms (49 tokens, 7.1 ms/token greedy, no KV cache)
realtime factor: 0.068x (~14x faster than realtime)
Output transcript:
"He hoped there would be stew for dinner, turnips and carrots and
bruised potatoes, and fat, mutton pieces to be ladled out in thick,
peppered, flour-fat and sauce."
Ground truth:
"He hoped there would be stew for dinner, turnips and carrots and
bruised potatoes and fat mutton pieces to be ladled out in thick
peppered flour-fattened sauce."
Near-perfect (a few punctuation tweaks, "flour-fat and sauce" vs
"flour-fattened sauce"). WER very low.
Compared to other STT backends in this crate:
Kyutai STT 1B : 1.01x realtime (3 GB, hardware-bound)
Whisper-tiny : 0.020x realtime (in-process ggml -> CSM regression)
**Moonshine-tiny: 0.068x realtime (pure candle, no runtime conflict)**
Components shipped this commit:
- Decoder::generate(encoder_output, cfg, max_tokens) — greedy
autoregressive loop. No KV cache yet (each step re-runs the full
growing token sequence — O(T^2) total). For 49-token transcripts
at <500 ms total, KV cache isn't urgent.
- load_tokenizer() — wraps tokenizers::Tokenizer::from_file for
Moonshine's HF tokenizer.json (BPE, vocab 32_768).
- examples/moonshine_transcribe — full pipeline: audio -> 16 kHz
PCM -> encode -> decode -> detokenize -> transcript text.
Critical bug fixed: SwiGLU gate/up split direction. HF
modeling_moonshine.py says:
hidden, gate = fc1(x).chunk(2, dim=-1)
out = silu(gate) * hidden
The FIRST half of the fused fc1 output is `up` (multiplied), the
SECOND half is `gate` (silu-activated). I had it reversed in Phase
8.7 — the symptom was a degenerate "tt tt tt" repetition loop after
the first 2 tokens. Reversing the split unlocked the working
transcription. Captured in the code comment.
Remaining for Moonshine readiness in production:
Phase 8.9 — KV cache for sub-200ms latency on long transcripts,
plus a standalone moonshine_profile binary for the
full A/B against Kyutai/Whisper.
Phase 8.10 — wire as a third AsrEngine variant in converse_server
(gated on English-only acceptance for the deploy).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Full encoder-decoder Moonshine v2 working end-to-end on candle 0.9 +
Metal. Loads HF safetensors, runs through every transformer block, and
produces real logits.
Components added to src/moonshine.rs:
CrossAttention MHA with K/V from encoder output (no causal mask)
DecoderSelfAttention MHA with causal mask, partial RoPE on q/k
DecoderMlp SwiGLU: fused fc1 [2304, 288] split gate+up,
silu(gate) * up, fc2 [288, 1152] back to hidden
DecoderLayer Pre-LN self-attn + Pre-LN cross-attn + Pre-LN MLP
Decoder token embed -> 6 layers -> final LN -> tied LM head
load_full() convenience: returns (Encoder, Decoder)
Smoke test verifies end-to-end:
encoder forward : 1 ms (cached after warm-up)
decoder forward : 85 ms (1 token, prefill mode)
logits shape : (1, 1, 32768)
logit max abs : 30.66 (real signal, not zeros)
argmax token_id : 379 (non-trivial prediction; eos=2)
Implementation notes:
- Same (B*H, T, D) 3D matmul pattern as encoder to dodge candle's 4D
Metal matmul shape-mismatch bug.
- LM head tied to decoder.embed_tokens.weight (cached on Decoder for
fast forward; logits = hidden @ embed.T).
- Causal mask is a (T, T) -inf upper-triangular added to scores
before softmax.
- Decoder final LN tensor is `decoder.norm.weight` (NOT
`decoder.layer_norm.weight` — encoder uses the latter naming).
- No KV cache yet: this is prefill mode. Phase 8.8 will add the
streaming-generation loop with cache + tokenizer.
NOT yet verified: numerical parity vs HF Python reference. The token
predicted (id=379) looks plausible for silent-mostly audio, but a
parity check is still needed (Phase 8.9). Architecture appears
correct based on shape + signal sanity.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Full encoder forward path: conv stem -> 6 transformer layers -> final
LayerNorm. Loads HF safetensors, runs end-to-end on Metal.
Components added to src/moonshine.rs:
RotaryCache partial RoPE (32 of 36 head_dim, theta=10000)
EncoderAttention MHA (8 heads, no bias), partial RoPE on q/k
EncoderMlp 288 -> 1152 -> 288 with bias, GELU(erf) activation
EncoderLayer Pre-LN attn + Pre-LN MLP (LayerNorm weight-only)
Encoder stem + 6 layers + final LayerNorm
load_encoder() VarBuilder convenience for the standalone smoke
Smoke test (`examples/moonshine_smoke`) verified end-to-end:
input (1, 1, 160000) -> output (1, 415, 288)
forward: 132 ms (10 s of audio at 0.013x realtime)
max abs: 6.67 (signal preserved, not zeros)
Implementation notes captured in the diff:
- candle Metal 4D batched matmul had shape-mismatch issues for our
(B, H, T, D) pattern. Switched to (B*H, T, D) 3D form which is
unambiguous and avoids the kernel bug.
- LayerNorm is weight-only (no bias tensors in safetensors); we
construct LayerNorm with a zeros bias to satisfy candle's API.
- rotary_dim = floor(head_dim * 0.9 / 2) * 2 = 32 (must be even).
The remaining 4 head_dim channels pass through unchanged via
`narrow + cat` on dim 3.
Numerical parity vs HF Python reference is NOT yet verified — that's
the next bounded chunk (Phase 8.7). Shape + signal correctness are
verified by the smoke test.
Next: decoder transformer block (self-attn + cross-attn + SwiGLU).
~3-4 h of focused work.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
redclawsystems