# SilentCipher candle port notes (Phase 10) Working notes for the candle port of SilentCipher — Sesame's actual production watermarker (not AudioSeal). See `docs/sesame_gap_analysis.md` for the gap analysis that motivated this. ## Why this exists rtx-csm currently watermarks via AudioSeal (Phase 4). AudioSeal is Meta's system; SilentCipher is what Sesame's GitHub fork (`SesameAILabs/silentcipher`) actually uses. Functionally similar (invisible audio watermark + bit-decode), but different architecture and weights. Goal: literal Sesame watermarker parity. SilentCipher is **substantially simpler than AudioSeal**: - AudioSeal: SEANet 1D conv stack on raw audio + LSTM bottleneck + message embedding + decoder, ~14-15 M params each (gen + det) - SilentCipher: 3 small networks of gated 2D convs on STFT, totaling maybe 5-10 M params ## Architecture (verified from `SesameAILabs/silentcipher/src/silentcipher/model.py`) ### Layer (gated conv block, used everywhere) ```python Layer(in, out, k, s, p): conv: Conv2d(in, out, k, s, p, bias=True) gate: Conv2d(in, out, k, s, p, bias=True) bn: BatchNorm2d(out) forward(x): bn(conv(x) * sigmoid(gate(x))) ``` ### Encoder (`enc_c`) 3 stacked Layers (default `enc_n_layers=3`, kernel 3×3 stride 1 pad 1): ``` Layer(1, 32) ── input: (B, 1, n_fft//2+1, T) Layer(32, 32) Layer(32, 32) ── output: (B, 32, n_fft//2+1, T) ``` Plus `Linear(message_dim, message_band_size)` for `transform_message` (broadcasts the bit message across freq bins). ### CarrierDecoder (`dec_c`) 4 stacked Layers (default `dec_c_n_layers=4`, kernel 3×3 stride 1 pad 1 except last is 1×1): ``` Layer(96, 96) ── input: (B, 96, n_fft//2+1, T) [from concatenated carrier+msg+msg_band] Layer(96, 96) Layer(96, 96) Layer(96, 1, k=1) ── output: (B, 1, n_fft//2+1, T) [watermark spectrogram] ``` ### MsgDecoder (`dec_m`) 10 stacked Layers (`dec_m_num_repeat=8` plus first/last, channel_dim=128): ``` Dropout(0) Layer(1, 128) ── input: (B, 1, message_band_size, T) [Dropout(0) + Layer(128, 128)] × 8 Dropout(0) Layer(128, message_dim) Linear(message_band_size, 1) ── output: (B, message_dim, 1, T) reshaped ``` The model has **multiple `dec_m`** instances — one per message channel (`n_messages` from config, typically 1 for our use). Released checkpoints provide `dec_m_0.ckpt`, `dec_m_1.ckpt`, etc. ## Hyperparameters (from config.yaml in checkpoint) Exact values aren't in the repo — they're in `hparams.yaml` shipped with the released weights at `hf.co/sony/silentcipher/{16_khz/97561_iteration, 44_1_khz/73999_iteration}/hparams.yaml`. Values referenced in `server.py` source: - `enc_n_layers = 3` - `dec_c_n_layers = 4` - `dec_m_num_repeat = 8` (so total layers ≈ 10) - `message_dim` — fixed in config (typically 256 for byte-encoding 5 characters of 8 bits = 40 bits per patch) - `message_band_size` — typically a fraction of `n_fft//2+1` (e.g. 256 out of 513 for `n_fft=1024`) - `n_messages` — number of independent watermark channels (typically 1) - `message_len` — number of bytes per "message patch" (5 from the Python demo: `[123, 234, 111, 222, 11]`) STFT params: - 16 kHz model: `N_FFT=1024`, `HOP_LENGTH=??` (from hparams) - 44.1 kHz model: `N_FFT=??` (likely 2048 or 4096 to match SR/HOP ratio) ## Pipeline (encode) 1. Load audio, resample to model SR (16 kHz or 44.1 kHz) 2. Compute STFT: complex → magnitude + phase 3. Encode message bytes as one-hot, replicated across patches 4. `enc_c.transform_message(msg_one_hot)` → padded msg with shape matching mag (B, 1, n_fft//2+1, T) 5. `enc_c(magnitude)` → carrier features (B, 32, n_fft//2+1, T) 6. `enc_c.transform_message` is a separate broadcast pass producing another tensor (B, 1, n_fft//2+1, T) of message info 7. Concatenate (carrier 32, msg 1, msg_band ≤32) along channels → 96 channels (matches `dec_c_conv_dim = 32*3`) 8. `dec_c(...)` → watermark spectrogram (B, 1, n_fft//2+1, T) scaled by SDR 9. `mag_watermarked = magnitude + watermark` (or `relu`/`abs` per config flag) 10. iSTFT(mag_watermarked, phase) → encoded audio ## Pipeline (decode) 1. STFT 2. `dec_m_i(magnitude)` for each message channel → logits per byte position, shape (B, message_dim, 1, T) 3. Argmax along `message_dim`, group T into patches of `message_len` × bytes 4. Per-patch majority vote → recovered byte sequence 5. Confidence = mean softmax probability of argmax tokens ## Weight checkpoints Hosted at `https://huggingface.co/sony/silentcipher`. Two folders: ``` 44_1_khz/73999_iteration/ enc_c.ckpt ~ encoder dec_c.ckpt ~ carrier decoder dec_m_0.ckpt ~ message decoder (one per channel) hparams.yaml ~ config 16_khz/97561_iteration/ enc_c.ckpt dec_c.ckpt dec_m_0.ckpt hparams.yaml ``` `.ckpt` files are PyTorch `state_dict` pickle (same format as the AudioSeal `.pth`). Use `candle_core::pickle::read_all` (already in `audioseal_convert.rs`) to load directly without converting. For our use, the **16 kHz model** is the natural fit: CSM TTS is 24 kHz, we resample to 16 kHz for watermarking (same 24↔16 dance as AudioSeal). The 44.1 kHz model is for high-fidelity music watermarking, irrelevant to our use case. ## Porting tasks (~1-2 days, MUCH simpler than AudioSeal port was) In order of dependency: 1. **`examples/silentcipher_inspect.rs` (~30 min, this commit)** Same pattern as `moonshine_inspect`: download `sony/silentcipher`, dump tensor shapes for the 16 kHz checkpoint. Verifies the layout matches what we read from the source. 2. **`src/silentcipher.rs` skeleton + `Layer` type (~1 h)** - `MoonshineConfig`-style `SilentCipherConfig` from hparams.yaml - `Layer { conv, gate, bn }` impl with the gated activation forward - Verify with a tiny smoke test (random input, shape preserved) 3. **STFT helper (~2-3 h)** - candle has FFT primitives; we need windowed STFT with Hann window - Reference: `src/silentcipher/stft.py` (40 LOC, simple). Forward pads to whole-window multiple, does `torch.stft`, returns magnitude + phase. - Pure-Rust path: implement framing + `rustfft` per frame, or use candle's built-in FFT if available. - Test: round-trip a sine wave, verify error < 1 e-4. 4. **Encoder + CarrierDecoder + MsgDecoder forward (~2-3 h)** - Each is straightforward Sequential of Layers - `enc_c.transform_message` needs the Linear + zero-pad to `n_fft//2+1` - Test: random input, shape sanity through full encode pipeline 5. **Weight loading via `candle_core::pickle::read_all` (~1 h)** - Direct load from `enc_c.ckpt` etc. — no safetensors conversion needed (same approach as `audioseal_convert.rs`) - Map PyTorch `.weight`/`.bias`/`.running_mean`/`.running_var` for BatchNorm to candle's `BatchNorm2d` constructor 6. **`SilentCipherWatermarker` end-to-end (~2-3 h)** - Wraps everything: load config + weights, construct STFT, hold all three networks, expose `embed(samples) -> Vec` and `detect(samples) -> DetectionResult` - Implements the existing `Watermarker` trait so it drops into `Generator::set_watermarker` 7. **`examples/silentcipher_apply` CLI (~30 min)** - Mirror `audioseal_apply` exactly: input WAV → embed → output WAV, plus `--detect-only` for verifying 8. **`examples/silentcipher_demo` (~30 min)** - End-to-end: real LibriSpeech audio → embed `[123,234,111,222,11]` → detect → assert message matches 9. **Bench vs AudioSeal (~30 min)** - Single-WAV benchmark: embed time, detect time, SDR, bit accuracy. Capture in `docs/perf_history.md`. 10. **`--watermark-silentcipher` flag in converse_server (~1 h)** - Mutex with `--watermark-generator`/`--watermark-detector` (AudioSeal). Stripped down: SilentCipher takes a single checkpoint folder. ## Risks / unknowns 1. **STFT in candle**. We may need to add a `rustfft` dep or implement STFT manually. Performance-wise both should be ~ms-scale, fine for our use case. 2. **BatchNorm running stats**. `candle_nn::BatchNorm` exists but we need to verify it loads `running_mean` / `running_var` from pickle correctly. Worst case we manually compute via stored stats (eval mode means `(x - mean) / sqrt(var + eps) * gamma + beta`). 3. **Phase passthrough**. Watermarking only modifies magnitude; phase must be preserved exactly through the iSTFT. Verify there's no accidental phase corruption. 4. **Message length 40 bits**. AudioSeal carries 16 bits; SilentCipher carries 40 bits per "patch" (5 bytes × 8 bits). For our use case (one watermark per utterance), 16 bits is enough — we can either use the lower 16 bits of the SilentCipher message and ignore the rest, or just embed a job_id in the full 40 bits. ## Bench expectations vs AudioSeal AudioSeal numbers from Phase 4f-g and Phase 6f.wm: - Embed: ~30-70 ms per ~1 s audio (real CSM speech) - Detect: ~30-50 ms - SDR: not measured; bit accuracy 16/16 on clean signal, 12/16 on 24↔16 resample - Phase 6f.wm: ~73 ms total cost per utterance in the converse_server (~1% overhead on a ~6.8 s TTS phase) SilentCipher is smaller (5-10 M params vs ~30 M for AudioSeal gen+det combined), so we expect comparable or faster: - Embed: ~20-40 ms per ~1 s audio - Detect: ~10-30 ms - Bit accuracy: stronger (training-time guarantee per the SilentCipher paper) Real numbers will be in the bench comparison after porting. ## Recommended order of attack for the next session 1. This commit: ship port notes + `silentcipher_inspect.rs` 2. Next: STFT helper + smoke test (highest risk; resolve early) 3. Then: model scaffolds + weight loader + standalone embed/detect 4. Last: integration into converse_server + A/B bench Each step is a bounded ship — same pattern as Phase 8.4-8.10 Moonshine. ## Cited sources - Repo: (Sesame's fork) - Original: - Paper: arXiv 2406.03822 (SilentCipher) - Weights: - Architecture verified from `silentcipher/src/silentcipher/model.py` (95 LOC, three classes)