rustyverse/rustytorch
2
0
Fork 0
Code Issues Pull Requests Actions 6 Packages Projects Releases Wiki Activity
413 Commits 1 Branch 0 Tags
1c981566537be1f77693655eb2488e7613eab164
Commit Graph
413 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
osobhandClaude Opus 4.7 3165b9339b rtx-csm: WavLM transformer encoder with gated rel-pos attention 
Phase 5b — full implementation of WavLMEncoderLayer.forward, replacing
the Phase 5a stub. With random weights the encoder is no longer a no-op
(verified: L2(output - input) > 1e-6 in unit test).

- relative_position_bucket helper: T5-style bidirectional bucketing
  matching HF _relative_positions_bucket. 320 buckets, 800 max distance,
  half/half split with linear inner / log-spaced outer.
- WavLmEncoderLayer::compute_position_bias (layer 0 only): builds (T,T)
  bucket index, embedding-looks up rel_attn_embed, permutes to
  (num_heads, T, T) matching HF's compute_bias output.
- WavLmEncoderLayer::gated_position_bias: HF gating math verbatim —
  Linear(head_dim → 8), reshape (..., 2, 4) sum, sigmoid, chunk to
  gate_a/gate_b, compute gate_a * (gate_b * gru_rel_pos_const - 1) + 2,
  broadcast-multiply position_bias.
- WavLmEncoderLayer::attention: multi-head self-attention with the
  gated bias added to scores before softmax. Standard 1/sqrt(d) scale.
- WavLmEncoderLayer::forward_with_bias returns (output, position_bias)
  so Encoder::forward_all_layers can thread bias from layer 0 through
  layers 1-11 (HF's has_relative_position_bias=(i==0) pattern).
- 3 new tests bring wavlm_sv to 12 tests; 75 lib tests total green.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
 2026-04-25 19:48:56 -07:00
osobhandClaude Opus 4.7 90e53b3c0c rtx-csm: WavLM-Base+ SV scaffold for speaker similarity 
Phase 5a — architectural skeleton for the microsoft/wavlm-base-plus-sv
reference, drop-in replacement for the SpectralCentroidSimilarity weak
baseline in speaker_sim.rs.

Modules in src/wavlm_sv.rs (~600 LOC):
- FeatureExtractor: 7-layer Conv1d, 320× downsample, GroupNorm at layer
  0 (num_groups=num_channels=512), GELU activations.
- FeatureProjection: LayerNorm + Linear 512→768.
- PosConv: Conv1d(768, 768, k=128, groups=16, pad=64) + GELU; SamePad
  strips trailing frame for even kernel.
- WavLmEncoderLayer: struct shape complete (Q/K/V/out projections, pre-
  attention LN, FFN intermediate/output, final LN, gru_rel_pos_const +
  gru_rel_pos_linear, optional rel_attn_embed at layer 0). forward() is
  a STUB; Phase 5b implements gated rel-pos attention.
- Encoder: 12 stacked layers, returns Vec<Tensor> of 13 hidden states.
- Tdnn: dilated unfold + Linear(in*kernel, out) — matches HF impl.
- XVectorHead: softmax-weighted layer sum + projector 768→512 + 5 TDNN
  layers (kernels [5,3,3,1,1] dilations [1,2,3,1,1]) + statistics pool
  + 3000→512 embedding projection.
- WavLmSv top-level + zero-mean unit-variance normalize + cosine
  similarity helper for verification scoring.

9 shape-correctness tests; 72 lib tests total green.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
 2026-04-25 19:42:59 -07:00
osobhandClaude Opus 4.7 63979eab45 rtx-csm: Generator inline watermarker + ResampledWatermarker adapter 
A single \`generate\` invocation now produces a watermarked WAV when
AudioSeal weights are passed via CLI. End-to-end verified on real CSM
speech: mean_presence=1.0000, 16/16 message bits decoded.

- Generator gains \`watermarker: Option<Box<dyn Watermarker>>\` slot;
  \`generate_to_wav\` runs \`wm.embed(&pcm)\` after post-process, before
  WAV write. Field is Send+Sync so the existing Arc<Mutex<Generator>>
  tts_server pattern still works.
- watermark.rs ships ResampledWatermarker<W> adapter for handling rate
  mismatches (CSM 24 kHz ↔ AudioSeal 16 kHz). Output length is normalized
  to input length so it's a transparent drop-in.
- examples/generate.rs gains --watermark-generator/--watermark-detector/
  --watermark-message flags. Loads AudioSeal, wraps in resampler, installs.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
 2026-04-25 19:36:17 -07:00
osobhandClaude Opus 4.7 9d28597687 rtx-csm: AudioSeal apply CLI for arbitrary-rate WAVs 
End-to-end watermarker driver that handles any source sample rate by
resampling to AudioSeal's 16 kHz native, embedding, then resampling back.
Tested on 10s of real CSM 24 kHz speech: mean_presence=0.9988 detection,
12/16 message bits round-trip (4-bit erosion from double resample).

- examples/audioseal_apply.rs: --in/--out/--source-rate/--message; loads
  source via audio_io::load_mono_at_rate, calls AudioSealWatermarker
  through the public Watermarker trait, verifies via in-process detect.
- Fix bit-match counter overflow in audioseal_demo.rs and audioseal_apply.rs.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
 2026-04-25 19:32:14 -07:00
osobhandClaude Opus 4.7 938b54a2b0 rtx-csm: AudioSeal watermark — Rust port end-to-end 
SEANet generator + detector matching `facebook/audioseal` reference layout
(weight_norm-merged via pure-Rust pickle reader). Verified on real CSM
speech: mean_presence=0.9943, 16/16 message bits decoded.

- src/audioseal.rs: SeanetEncoder (4-stage strided downsample, 2-layer
  LSTM bottleneck at 512 channels, 128-dim projection), MsgProcessor
  (16-bit message via embedding sum + broadcast-add), SeanetDecoder,
  Generator (encoder+msg+decoder), Detector (encoder + single 320×
  reverse_convolution + 1×1 head). Padding mirrors audiocraft
  _get_extra_padding_for_conv1d exactly.
- src/audioseal_convert.rs: candle_core::pickle reads .pth directly;
  merge_weight_norm computes g*v/‖v‖ over all axes except 0; writes
  flat safetensors keyed identically to what Generator/Detector read.
- examples/audioseal_inspect.rs: dumps tensor keys + shapes.
- examples/audioseal_convert.rs: HF download + convert CLI.
- examples/audioseal_demo.rs: load + embed + detect on real WAV or
  synthetic burst, optionally writes watermarked WAV.
- audio_io.rs gains generic load_mono_at_rate, resample, write_wav_mono
  (16 kHz path needed for AudioSeal).

12 new unit tests + 2 converter tests; 63 lib tests total green.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
 2026-04-25 19:26:13 -07:00
osobhandClaude Opus 4.7 15dd3575d4 Add rtx-csm: Rust-native port of Sesame CSM-1B with LoRA voice cloning 
A new model crate at crates/models/rtx-csm implementing end-to-end
inference, quantization, and fine-tuning for Sesame's Conversational
Speech Model (CSM-1B). Built on candle 0.9 + Kyutai Mimi codec.

Key capabilities:
- Inference (FP F16 on Metal, F32 on CPU, BF16 on CUDA)
- Quantized inference (Q8_0 / Q4_K_M GGUF, ~3x speedup, ~50% memory)
- Streaming Mimi decode with proper StreamTensor state machine
- In-context voice cloning via SpeakerProfile
- Classifier-Free Guidance (Koel-TTS recipe)
- Long-form chunked generation with rolling context
- Audio post-processing (HPF + declick + EBU R128 LUFS)
- Text input normalization (brackets, times, unicode, length caps)
- Frame-level repetition guard (loop-escape)
- Top-k + top-p sampling
- LoRA fine-tuning end-to-end (training + inference, on FP and Q8 bases)
- In-process Whisper ASR via whisper-rs (under --features asr)
- Standalone TTS HTTP server (Axum)
- Bench harness with manifest export + per-prompt WER

Phases delivered: quantization, ASR/WER eval, LoRA voice cloning, HTTP
service. AudioSeal/WavLM/Unmute remain as documented future work.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
 2026-04-25 18:33:57 -07:00
osobhandClaude Opus 4.7 15b62a8f6e rtx-interpret: fix decoder transpose in SAE compute_and_apply_gradients 
The encoder-gradient path through the decoder was transposing the decoder
before the matmul, producing `[batch, d_model] × [d_sae, d_model]` — a
shape mismatch for every batch > 1. The decoder is stored as
`[d_model, d_sae]`, so `recon_grad @ decoder` is already the right shape
(and matches the comment at the call site, which reads
"recon_grad @ decoder @ d_relu").

All 9 existing `sae::tests` still pass. Omni-Cortex's `LatentDictionary`
now trains correctly on batches larger than 1.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
 2026-04-24 13:58:44 -07:00
osobhandClaude Opus 4.6 85b77d49f2 Add audio neural layers and model architectures for ClawSample integration 
New nn layers:
- ConvTranspose1d with stride, padding, output_padding (9 tests)
- LSTM/BiLSTM with multi-layer support and hidden state (10 tests)

Audio source separation:
- Demucs ONNX inference with segmented overlap-add processing
- Native HtDemucs architecture (encoder/decoder with BiLSTM bottleneck)
- StemType enum: vocals, drums, bass, other, piano, guitar

Audio generation:
- Stable Audio Open ONNX inference scaffold
- GenerationParams (prompt, duration, steps, cfg_scale, seed)

ONNX export scripts:
- export_demucs_onnx.py — Demucs v4 to ONNX with segment chunking
- export_stable_audio_onnx.py — Stable Audio Open components
- export_mert_onnx.py — MERT music understanding transformer

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
 2026-04-17 12:27:12 -07:00
osobhandClaude Opus 4.6 5fe8c67b04 Fix rtx-serving-api binary: create main.rs entry point 
The binary path pointed to non-existent src/bin/server.rs.
Created proper src/main.rs with tokio async entry point that:
- Initializes tracing
- Loads config from RTX_HOST/RTX_PORT/RTX_TIMEOUT env vars
- Instantiates ServingServer and calls serve()

Fixed Cargo.toml: path = "src/bin/server.rs" → path = "src/main.rs"

Validated: binary builds, starts, health endpoint returns healthy,
inference endpoint returns mock completions.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
 2026-04-12 20:16:38 -07:00
osobhandClaude Opus 4.6 02d382d5f6 style: apply rustfmt across all crates and demos 
Consistent formatting pass: line wrapping, import sorting, trailing
whitespace removal, let-chain indentation, merged derive attributes,
and unsafe block reformatting.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
 2026-04-12 07:01:58 -07:00
Omar SobhandClaude Sonnet 4.6 bc88a14fa1 docs: add CLAUDE.md with architecture and integration reference 
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
 2026-03-28 04:39:39 -07:00
osobh d52d359f52 Fix 3 compile errors: rtx-metal (Linux cfg), rtx-onnx (ort API), rtx-fusion (edition) 
- rtx-metal: fix MetalError import in sparse/conversion.rs non-macOS stub
- rtx-onnx: update session.rs and tensor_bridge.rs for ort 2.x API changes
- rtx-fusion: fix Cargo.toml package name
- rtx-hub: fix discovery.rs type mismatch
- Full workspace (80+ crates) now compiles clean on Linux
 2026-03-15 17:33:54 -07:00
redclawsystems 4d88dc0584 Initial commit 2026-03-04 00:08:42 +00:00