rustytorch/crates/models/rtx-csm/docs/silentcipher_port_notes.md

SilentCipher candle port notes (Phase 10)

Working notes for the candle port of SilentCipher — Sesame's actual production watermarker (not AudioSeal). See docs/sesame_gap_analysis.md for the gap analysis that motivated this.

Why this exists

rtx-csm currently watermarks via AudioSeal (Phase 4). AudioSeal is Meta's system; SilentCipher is what Sesame's GitHub fork (SesameAILabs/silentcipher) actually uses. Functionally similar (invisible audio watermark + bit-decode), but different architecture and weights. Goal: literal Sesame watermarker parity.

SilentCipher is substantially simpler than AudioSeal:

• AudioSeal: SEANet 1D conv stack on raw audio + LSTM bottleneck + message embedding + decoder, ~14-15 M params each (gen + det)
• SilentCipher: 3 small networks of gated 2D convs on STFT, totaling maybe 5-10 M params

Architecture (verified from SesameAILabs/silentcipher/src/silentcipher/model.py)

Layer (gated conv block, used everywhere)

Layer(in, out, k, s, p):
    conv: Conv2d(in, out, k, s, p, bias=True)
    gate: Conv2d(in, out, k, s, p, bias=True)
    bn:   BatchNorm2d(out)
    forward(x): bn(conv(x) * sigmoid(gate(x)))

Encoder (enc_c)

3 stacked Layers (default enc_n_layers=3, kernel 3×3 stride 1 pad 1):

Layer(1,    32) ── input: (B, 1, n_fft//2+1, T)
Layer(32,   32)
Layer(32,   32) ── output: (B, 32, n_fft//2+1, T)

Plus Linear(message_dim, message_band_size) for transform_message (broadcasts the bit message across freq bins).

CarrierDecoder (dec_c)

4 stacked Layers (default dec_c_n_layers=4, kernel 3×3 stride 1 pad 1 except last is 1×1):

Layer(96,  96)  ── input: (B, 96, n_fft//2+1, T)  [from concatenated carrier+msg+msg_band]
Layer(96,  96)
Layer(96,  96)
Layer(96,   1, k=1)  ── output: (B, 1, n_fft//2+1, T)  [watermark spectrogram]

MsgDecoder (dec_m)

10 stacked Layers (dec_m_num_repeat=8 plus first/last, channel_dim=128):

Dropout(0)
Layer(1,   128)  ── input: (B, 1, message_band_size, T)
[Dropout(0) + Layer(128, 128)] × 8
Dropout(0)
Layer(128, message_dim)
Linear(message_band_size, 1)  ── output: (B, message_dim, 1, T) reshaped

The model has multiple dec_m instances — one per message channel (n_messages from config, typically 1 for our use). Released checkpoints provide dec_m_0.ckpt, dec_m_1.ckpt, etc.

Hyperparameters (from config.yaml in checkpoint)

Exact values aren't in the repo — they're in hparams.yaml shipped with the released weights at hf.co/sony/silentcipher/{16_khz/97561_iteration, 44_1_khz/73999_iteration}/hparams.yaml.

Values referenced in server.py source:

• enc_n_layers = 3
• dec_c_n_layers = 4
• dec_m_num_repeat = 8 (so total layers ≈ 10)
• message_dim — fixed in config (typically 256 for byte-encoding 5 characters of 8 bits = 40 bits per patch)
• message_band_size — typically a fraction of n_fft//2+1 (e.g. 256 out of 513 for n_fft=1024)
• n_messages — number of independent watermark channels (typically 1)
• message_len — number of bytes per "message patch" (5 from the Python demo: [123, 234, 111, 222, 11])

STFT params:

• 16 kHz model: N_FFT=1024, HOP_LENGTH=?? (from hparams)
• 44.1 kHz model: N_FFT=?? (likely 2048 or 4096 to match SR/HOP ratio)

Pipeline (encode)

1. Load audio, resample to model SR (16 kHz or 44.1 kHz)
2. Compute STFT: complex → magnitude + phase
3. Encode message bytes as one-hot, replicated across patches
4. enc_c.transform_message(msg_one_hot) → padded msg with shape matching mag (B, 1, n_fft//2+1, T)
5. enc_c(magnitude) → carrier features (B, 32, n_fft//2+1, T)
6. enc_c.transform_message is a separate broadcast pass producing another tensor (B, 1, n_fft//2+1, T) of message info
7. Concatenate (carrier 32, msg 1, msg_band ≤32) along channels → 96 channels (matches dec_c_conv_dim = 32*3)
8. dec_c(...) → watermark spectrogram (B, 1, n_fft//2+1, T) scaled by SDR
9. mag_watermarked = magnitude + watermark (or relu/abs per config flag)
10. iSTFT(mag_watermarked, phase) → encoded audio

Pipeline (decode)

1. STFT
2. dec_m_i(magnitude) for each message channel → logits per byte position, shape (B, message_dim, 1, T)
3. Argmax along message_dim, group T into patches of message_len × bytes
4. Per-patch majority vote → recovered byte sequence
5. Confidence = mean softmax probability of argmax tokens

Weight checkpoints

Hosted at https://huggingface.co/sony/silentcipher. Two folders:

44_1_khz/73999_iteration/
    enc_c.ckpt              ~ encoder
    dec_c.ckpt              ~ carrier decoder
    dec_m_0.ckpt            ~ message decoder (one per channel)
    hparams.yaml            ~ config
16_khz/97561_iteration/
    enc_c.ckpt
    dec_c.ckpt
    dec_m_0.ckpt
    hparams.yaml

.ckpt files are PyTorch state_dict pickle (same format as the AudioSeal .pth). Use candle_core::pickle::read_all (already in audioseal_convert.rs) to load directly without converting.

For our use, the 16 kHz model is the natural fit: CSM TTS is 24 kHz, we resample to 16 kHz for watermarking (same 24↔16 dance as AudioSeal). The 44.1 kHz model is for high-fidelity music watermarking, irrelevant to our use case.

Porting tasks (~1-2 days, MUCH simpler than AudioSeal port was)

In order of dependency:

1. examples/silentcipher_inspect.rs (~30 min, this commit) Same pattern as moonshine_inspect: download sony/silentcipher, dump tensor shapes for the 16 kHz checkpoint. Verifies the layout matches what we read from the source.

2. src/silentcipher.rs skeleton + Layer type (~1 h)

   • MoonshineConfig-style SilentCipherConfig from hparams.yaml
   • Layer { conv, gate, bn } impl with the gated activation forward
   • Verify with a tiny smoke test (random input, shape preserved)

3. STFT helper (~2-3 h)

   • candle has FFT primitives; we need windowed STFT with Hann window
   • Reference: src/silentcipher/stft.py (40 LOC, simple). Forward pads to whole-window multiple, does torch.stft, returns magnitude + phase.
   • Pure-Rust path: implement framing + rustfft per frame, or use candle's built-in FFT if available.
   • Test: round-trip a sine wave, verify error < 1 e-4.

4. Encoder + CarrierDecoder + MsgDecoder forward (~2-3 h)

   • Each is straightforward Sequential of Layers
   • enc_c.transform_message needs the Linear + zero-pad to n_fft//2+1
   • Test: random input, shape sanity through full encode pipeline

5. Weight loading via candle_core::pickle::read_all (~1 h)

   • Direct load from enc_c.ckpt etc. — no safetensors conversion needed (same approach as audioseal_convert.rs)
   • Map PyTorch .weight/.bias/.running_mean/.running_var for BatchNorm to candle's BatchNorm2d constructor

6. SilentCipherWatermarker end-to-end (~2-3 h)

   • Wraps everything: load config + weights, construct STFT, hold all three networks, expose embed(samples) -> Vec<f32> and detect(samples) -> DetectionResult
   • Implements the existing Watermarker trait so it drops into Generator::set_watermarker

7. examples/silentcipher_apply CLI (~30 min)

   • Mirror audioseal_apply exactly: input WAV → embed → output WAV, plus --detect-only for verifying

8. examples/silentcipher_demo (~30 min)

   • End-to-end: real LibriSpeech audio → embed [123,234,111,222,11] → detect → assert message matches

9. Bench vs AudioSeal (~30 min)

   • Single-WAV benchmark: embed time, detect time, SDR, bit accuracy. Capture in docs/perf_history.md.

10. --watermark-silentcipher flag in converse_server (~1 h)

    • Mutex with --watermark-generator/--watermark-detector (AudioSeal). Stripped down: SilentCipher takes a single checkpoint folder.

Risks / unknowns

1. STFT in candle. We may need to add a rustfft dep or implement STFT manually. Performance-wise both should be ~ms-scale, fine for our use case.

2. BatchNorm running stats. candle_nn::BatchNorm exists but we need to verify it loads running_mean / running_var from pickle correctly. Worst case we manually compute via stored stats (eval mode means (x - mean) / sqrt(var + eps) * gamma + beta).

3. Phase passthrough. Watermarking only modifies magnitude; phase must be preserved exactly through the iSTFT. Verify there's no accidental phase corruption.

4. Message length 40 bits. AudioSeal carries 16 bits; SilentCipher carries 40 bits per "patch" (5 bytes × 8 bits). For our use case (one watermark per utterance), 16 bits is enough — we can either use the lower 16 bits of the SilentCipher message and ignore the rest, or just embed a job_id in the full 40 bits.

Bench expectations vs AudioSeal

AudioSeal numbers from Phase 4f-g and Phase 6f.wm:

• Embed: ~30-70 ms per ~1 s audio (real CSM speech)
• Detect: ~30-50 ms
• SDR: not measured; bit accuracy 16/16 on clean signal, 12/16 on 24↔16 resample
• Phase 6f.wm: ~73 ms total cost per utterance in the converse_server (~1% overhead on a ~6.8 s TTS phase)

SilentCipher is smaller (5-10 M params vs ~30 M for AudioSeal gen+det combined), so we expect comparable or faster:

• Embed: ~20-40 ms per ~1 s audio
• Detect: ~10-30 ms
• Bit accuracy: stronger (training-time guarantee per the SilentCipher paper)

Real numbers will be in the bench comparison after porting.

Recommended order of attack for the next session

1. This commit: ship port notes + silentcipher_inspect.rs
2. Next: STFT helper + smoke test (highest risk; resolve early)
3. Then: model scaffolds + weight loader + standalone embed/detect
4. Last: integration into converse_server + A/B bench

Each step is a bounded ship — same pattern as Phase 8.4-8.10 Moonshine.

Cited sources

• Repo: https://github.com/SesameAILabs/silentcipher (Sesame's fork)
• Original: https://github.com/sony/silentcipher
• Paper: arXiv 2406.03822 (SilentCipher)
• Weights: https://huggingface.co/sony/silentcipher
• Architecture verified from silentcipher/src/silentcipher/model.py (95 LOC, three classes)