rtx-csm: Phase 10.1 — SilentCipher inspector + port notes

Foundation for porting Sesame's actual production watermarker (NOT
AudioSeal — the gap analysis identified this as the literal Sesame
parity item). Same iterative-shipping pattern as Phase 8.4 for
Moonshine.

`docs/silentcipher_port_notes.md`:
  - Full architecture from SesameAILabs/silentcipher/src/.../model.py
    (verified against 95 LOC of source)
  - Three small networks of gated 2D convs on STFT:
      enc_c    3 layers   1 -> 32 channels
      dec_c    4 layers   96 -> 1 channels
      dec_m    10 layers  1 -> 128 -> message_dim, plus Linear
  - Each Layer = Conv2d * sigmoid(Conv2d) + BatchNorm2d
  - Pipeline (encode + decode) walked through step by step
  - 10 ordered porting tasks with hour estimates totaling ~1-2 days
  - Risks flagged: STFT helper needed, BatchNorm running stats loading,
    phase passthrough, message-length differences vs AudioSeal

`examples/silentcipher_inspect`:
  - Downloads sony/silentcipher 16 kHz checkpoint from HuggingFace
  - Dumps hparams.yaml + tensor shapes per .ckpt file
  - Verified output:
        N_FFT 2048   HOP 1024   SR 16000
        message_dim 4  message_len 16  message_band 512
        enc_c     0.17 MB    40 k params
        dec_c     2.01 MB   500 k params
        dec_m_0   9.54 MB  2.38 M params
        Total           ~2.92 M params

That's ~10x smaller than AudioSeal's gen+det combined. Port
estimated 1-2 days.

`.ckpt` files are pickle (PyTorch state_dict) — direct loadable via
candle_core::pickle::read_all, same path as audioseal_convert.rs.
No safetensors conversion needed.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

crates/models/rtx-csm/Cargo.toml

@@ -261,3 +261,7 @@ path = "examples/moonshine_transcribe.rs"
 [[example]]
 name = "moonshine_profile"
 path = "examples/moonshine_profile.rs"
+
+[[example]]
+name = "silentcipher_inspect"
+path = "examples/silentcipher_inspect.rs"

crates/models/rtx-csm/docs/silentcipher_port_notes.md

@@ -0,0 +1,263 @@
+# SilentCipher candle port notes (Phase 10)
+
+Working notes for the candle port of SilentCipher — Sesame's actual
+production watermarker (not AudioSeal). See
+`docs/sesame_gap_analysis.md` for the gap analysis that motivated this.
+
+## Why this exists
+
+rtx-csm currently watermarks via AudioSeal (Phase 4). AudioSeal is
+Meta's system; SilentCipher is what Sesame's GitHub fork
+(`SesameAILabs/silentcipher`) actually uses. Functionally similar
+(invisible audio watermark + bit-decode), but different architecture
+and weights. Goal: literal Sesame watermarker parity.
+
+SilentCipher is **substantially simpler than AudioSeal**:
+- AudioSeal: SEANet 1D conv stack on raw audio + LSTM bottleneck +
+  message embedding + decoder, ~14-15 M params each (gen + det)
+- SilentCipher: 3 small networks of gated 2D convs on STFT, totaling
+  maybe 5-10 M params
+
+## Architecture (verified from `SesameAILabs/silentcipher/src/silentcipher/model.py`)
+
+### Layer (gated conv block, used everywhere)
+
+```python
+Layer(in, out, k, s, p):
+    conv: Conv2d(in, out, k, s, p, bias=True)
+    gate: Conv2d(in, out, k, s, p, bias=True)
+    bn:   BatchNorm2d(out)
+    forward(x): bn(conv(x) * sigmoid(gate(x)))
+```
+
+### Encoder (`enc_c`)
+
+3 stacked Layers (default `enc_n_layers=3`, kernel 3×3 stride 1 pad 1):
+```
+Layer(1,    32) ── input: (B, 1, n_fft//2+1, T)
+Layer(32,   32)
+Layer(32,   32) ── output: (B, 32, n_fft//2+1, T)
+```
+Plus `Linear(message_dim, message_band_size)` for `transform_message`
+(broadcasts the bit message across freq bins).
+
+### CarrierDecoder (`dec_c`)
+
+4 stacked Layers (default `dec_c_n_layers=4`, kernel 3×3 stride 1 pad 1
+except last is 1×1):
+```
+Layer(96,  96)  ── input: (B, 96, n_fft//2+1, T)  [from concatenated carrier+msg+msg_band]
+Layer(96,  96)
+Layer(96,  96)
+Layer(96,   1, k=1)  ── output: (B, 1, n_fft//2+1, T)  [watermark spectrogram]
+```
+
+### MsgDecoder (`dec_m`)
+
+10 stacked Layers (`dec_m_num_repeat=8` plus first/last, channel_dim=128):
+```
+Dropout(0)
+Layer(1,   128)  ── input: (B, 1, message_band_size, T)
+[Dropout(0) + Layer(128, 128)] × 8
+Dropout(0)
+Layer(128, message_dim)
+Linear(message_band_size, 1)  ── output: (B, message_dim, 1, T) reshaped
+```
+
+The model has **multiple `dec_m`** instances — one per message channel
+(`n_messages` from config, typically 1 for our use). Released
+checkpoints provide `dec_m_0.ckpt`, `dec_m_1.ckpt`, etc.
+
+## Hyperparameters (from config.yaml in checkpoint)
+
+Exact values aren't in the repo — they're in `hparams.yaml` shipped
+with the released weights at
+`hf.co/sony/silentcipher/{16_khz/97561_iteration, 44_1_khz/73999_iteration}/hparams.yaml`.
+
+Values referenced in `server.py` source:
+- `enc_n_layers = 3`
+- `dec_c_n_layers = 4`
+- `dec_m_num_repeat = 8` (so total layers ≈ 10)
+- `message_dim` — fixed in config (typically 256 for byte-encoding 5
+  characters of 8 bits = 40 bits per patch)
+- `message_band_size` — typically a fraction of `n_fft//2+1` (e.g. 256
+  out of 513 for `n_fft=1024`)
+- `n_messages` — number of independent watermark channels (typically 1)
+- `message_len` — number of bytes per "message patch" (5 from the
+  Python demo: `[123, 234, 111, 222, 11]`)
+
+STFT params:
+- 16 kHz model: `N_FFT=1024`, `HOP_LENGTH=??` (from hparams)
+- 44.1 kHz model: `N_FFT=??` (likely 2048 or 4096 to match SR/HOP ratio)
+
+## Pipeline (encode)
+
+1. Load audio, resample to model SR (16 kHz or 44.1 kHz)
+2. Compute STFT: complex → magnitude + phase
+3. Encode message bytes as one-hot, replicated across patches
+4. `enc_c.transform_message(msg_one_hot)` → padded msg with shape
+   matching mag (B, 1, n_fft//2+1, T)
+5. `enc_c(magnitude)` → carrier features (B, 32, n_fft//2+1, T)
+6. `enc_c.transform_message` is a separate broadcast pass producing
+   another tensor (B, 1, n_fft//2+1, T) of message info
+7. Concatenate (carrier 32, msg 1, msg_band ≤32) along channels → 96
+   channels (matches `dec_c_conv_dim = 32*3`)
+8. `dec_c(...)` → watermark spectrogram (B, 1, n_fft//2+1, T) scaled
+   by SDR
+9. `mag_watermarked = magnitude + watermark` (or `relu`/`abs` per
+   config flag)
+10. iSTFT(mag_watermarked, phase) → encoded audio
+
+## Pipeline (decode)
+
+1. STFT
+2. `dec_m_i(magnitude)` for each message channel → logits per byte
+   position, shape (B, message_dim, 1, T)
+3. Argmax along `message_dim`, group T into patches of `message_len` ×
+   bytes
+4. Per-patch majority vote → recovered byte sequence
+5. Confidence = mean softmax probability of argmax tokens
+
+## Weight checkpoints
+
+Hosted at `https://huggingface.co/sony/silentcipher`. Two folders:
+
+```
+44_1_khz/73999_iteration/
+    enc_c.ckpt              ~ encoder
+    dec_c.ckpt              ~ carrier decoder
+    dec_m_0.ckpt            ~ message decoder (one per channel)
+    hparams.yaml            ~ config
+16_khz/97561_iteration/
+    enc_c.ckpt
+    dec_c.ckpt
+    dec_m_0.ckpt
+    hparams.yaml
+```
+
+`.ckpt` files are PyTorch `state_dict` pickle (same format as the
+AudioSeal `.pth`). Use `candle_core::pickle::read_all` (already in
+`audioseal_convert.rs`) to load directly without converting.
+
+For our use, the **16 kHz model** is the natural fit: CSM TTS is 24 kHz,
+we resample to 16 kHz for watermarking (same 24↔16 dance as AudioSeal).
+The 44.1 kHz model is for high-fidelity music watermarking, irrelevant
+to our use case.
+
+## Porting tasks (~1-2 days, MUCH simpler than AudioSeal port was)
+
+In order of dependency:
+
+1. **`examples/silentcipher_inspect.rs` (~30 min, this commit)**
+   Same pattern as `moonshine_inspect`: download `sony/silentcipher`,
+   dump tensor shapes for the 16 kHz checkpoint. Verifies the layout
+   matches what we read from the source.
+
+2. **`src/silentcipher.rs` skeleton + `Layer` type (~1 h)**
+   - `MoonshineConfig`-style `SilentCipherConfig` from hparams.yaml
+   - `Layer { conv, gate, bn }` impl with the gated activation
+     forward
+   - Verify with a tiny smoke test (random input, shape preserved)
+
+3. **STFT helper (~2-3 h)**
+   - candle has FFT primitives; we need windowed STFT with Hann window
+   - Reference: `src/silentcipher/stft.py` (40 LOC, simple). Forward
+     pads to whole-window multiple, does `torch.stft`, returns
+     magnitude + phase.
+   - Pure-Rust path: implement framing + `rustfft` per frame, or use
+     candle's built-in FFT if available.
+   - Test: round-trip a sine wave, verify error < 1 e-4.
+
+4. **Encoder + CarrierDecoder + MsgDecoder forward (~2-3 h)**
+   - Each is straightforward Sequential of Layers
+   - `enc_c.transform_message` needs the Linear + zero-pad to
+     `n_fft//2+1`
+   - Test: random input, shape sanity through full encode pipeline
+
+5. **Weight loading via `candle_core::pickle::read_all` (~1 h)**
+   - Direct load from `enc_c.ckpt` etc. — no safetensors conversion
+     needed (same approach as `audioseal_convert.rs`)
+   - Map PyTorch `.weight`/`.bias`/`.running_mean`/`.running_var` for
+     BatchNorm to candle's `BatchNorm2d` constructor
+
+6. **`SilentCipherWatermarker` end-to-end (~2-3 h)**
+   - Wraps everything: load config + weights, construct STFT, hold
+     all three networks, expose `embed(samples) -> Vec<f32>` and
+     `detect(samples) -> DetectionResult`
+   - Implements the existing `Watermarker` trait so it drops into
+     `Generator::set_watermarker`
+
+7. **`examples/silentcipher_apply` CLI (~30 min)**
+   - Mirror `audioseal_apply` exactly: input WAV → embed → output WAV,
+     plus `--detect-only` for verifying
+
+8. **`examples/silentcipher_demo` (~30 min)**
+   - End-to-end: real LibriSpeech audio → embed `[123,234,111,222,11]`
+     → detect → assert message matches
+
+9. **Bench vs AudioSeal (~30 min)**
+   - Single-WAV benchmark: embed time, detect time, SDR, bit
+     accuracy. Capture in `docs/perf_history.md`.
+
+10. **`--watermark-silentcipher` flag in converse_server (~1 h)**
+    - Mutex with `--watermark-generator`/`--watermark-detector`
+      (AudioSeal). Stripped down: SilentCipher takes a single
+      checkpoint folder.
+
+## Risks / unknowns
+
+1. **STFT in candle**. We may need to add a `rustfft` dep or implement
+   STFT manually. Performance-wise both should be ~ms-scale, fine for
+   our use case.
+
+2. **BatchNorm running stats**. `candle_nn::BatchNorm` exists but
+   we need to verify it loads `running_mean` / `running_var` from
+   pickle correctly. Worst case we manually compute via stored stats
+   (eval mode means `(x - mean) / sqrt(var + eps) * gamma + beta`).
+
+3. **Phase passthrough**. Watermarking only modifies magnitude; phase
+   must be preserved exactly through the iSTFT. Verify there's no
+   accidental phase corruption.
+
+4. **Message length 40 bits**. AudioSeal carries 16 bits; SilentCipher
+   carries 40 bits per "patch" (5 bytes × 8 bits). For our use case
+   (one watermark per utterance), 16 bits is enough — we can either
+   use the lower 16 bits of the SilentCipher message and ignore the
+   rest, or just embed a job_id in the full 40 bits.
+
+## Bench expectations vs AudioSeal
+
+AudioSeal numbers from Phase 4f-g and Phase 6f.wm:
+- Embed: ~30-70 ms per ~1 s audio (real CSM speech)
+- Detect: ~30-50 ms
+- SDR: not measured; bit accuracy 16/16 on clean signal, 12/16 on 24↔16
+  resample
+- Phase 6f.wm: ~73 ms total cost per utterance in the converse_server
+  (~1% overhead on a ~6.8 s TTS phase)
+
+SilentCipher is smaller (5-10 M params vs ~30 M for AudioSeal gen+det
+combined), so we expect comparable or faster:
+- Embed: ~20-40 ms per ~1 s audio
+- Detect: ~10-30 ms
+- Bit accuracy: stronger (training-time guarantee per the SilentCipher
+  paper)
+
+Real numbers will be in the bench comparison after porting.
+
+## Recommended order of attack for the next session
+
+1. This commit: ship port notes + `silentcipher_inspect.rs`
+2. Next: STFT helper + smoke test (highest risk; resolve early)
+3. Then: model scaffolds + weight loader + standalone embed/detect
+4. Last: integration into converse_server + A/B bench
+
+Each step is a bounded ship — same pattern as Phase 8.4-8.10 Moonshine.
+
+## Cited sources
+
+- Repo: <https://github.com/SesameAILabs/silentcipher> (Sesame's fork)
+- Original: <https://github.com/sony/silentcipher>
+- Paper: arXiv 2406.03822 (SilentCipher)
+- Weights: <https://huggingface.co/sony/silentcipher>
+- Architecture verified from `silentcipher/src/silentcipher/model.py`
+  (95 LOC, three classes)

crates/models/rtx-csm/examples/silentcipher_inspect.rs

@@ -0,0 +1,107 @@
+//! Phase 10.1 — download SilentCipher 16 kHz checkpoint from HuggingFace
+//! (`sony/silentcipher`) and dump every tensor's shape + dtype. Used to
+//! design the candle module shape and verify our port maps the right
+//! weights.
+//!
+//! Architecture (from `SesameAILabs/silentcipher/src/silentcipher/model.py`):
+//!   Three small networks of gated 2D convs on STFT:
+//!     enc_c  : 3 layers, 1 -> 32 channels
+//!     dec_c  : 4 layers, 96 -> 1 channels
+//!     dec_m  : 10 layers, 1 -> 128 -> message_dim, plus Linear
+//!
+//! Each `Layer` is `Conv2d(...) * sigmoid(Conv2d(...))` followed by a
+//! `BatchNorm2d`. The PyTorch state_dict per Layer carries:
+//!     conv.weight, conv.bias
+//!     gate.weight, gate.bias
+//!     bn.weight, bn.bias, bn.running_mean, bn.running_var,
+//!     bn.num_batches_tracked
+//!
+//! Weight files are PyTorch `.ckpt` (pickle), same format as AudioSeal's
+//! `.pth`. We can read directly with `candle_core::pickle::read_all`.
+//!
+//! Usage:
+//! ```bash
+//! cargo run -p rtx-csm --release --example silentcipher_inspect
+//! ```
+
+use anyhow::{Context, Result};
+use hf_hub::api::sync::Api;
+
+const REPO: &str = "sony/silentcipher";
+const CKPT_DIR: &str = "16_khz/97561_iteration";
+const CKPT_FILES: &[&str] = &["enc_c.ckpt", "dec_c.ckpt", "dec_m_0.ckpt"];
+
+fn main() -> Result<()> {
+    let api = Api::new().context("hf_hub init")?;
+    let repo = api.model(REPO.to_string());
+
+    eprintln!("=== SilentCipher 16 kHz checkpoint inspector ===");
+    eprintln!("repo:     {REPO}");
+    eprintln!("ckpt dir: {CKPT_DIR}");
+    eprintln!();
+
+    // Download hparams.yaml first — it contains all the architecture
+    // hyperparameters (n_fft, hop_length, message_dim, etc.).
+    let hparams_path = repo
+        .get(&format!("{CKPT_DIR}/hparams.yaml"))
+        .context("download hparams.yaml")?;
+    let hparams = std::fs::read_to_string(&hparams_path)?;
+    eprintln!("--- hparams.yaml ---");
+    eprintln!("{hparams}");
+
+    // Each ckpt file is a PyTorch state_dict pickle. Use the same
+    // candle_core::pickle::read_all path as audioseal_convert.rs.
+    let dev = candle_core::Device::Cpu;
+    for ckpt_file in CKPT_FILES {
+        let path = repo
+            .get(&format!("{CKPT_DIR}/{ckpt_file}"))
+            .with_context(|| format!("download {ckpt_file}"))?;
+        let size = std::fs::metadata(&path)?.len();
+        println!();
+        println!("=== {ckpt_file} ({:.2} MB) ===", size as f64 / 1e6);
+
+        let tensors = candle_core::pickle::read_all(&path)
+            .with_context(|| format!("pickle read {ckpt_file}"))?;
+        let mut total_params: usize = 0;
+        let mut grouped: std::collections::BTreeMap<String, Vec<(String, Vec<usize>)>> =
+            Default::default();
+        for (name, tensor) in tensors.iter() {
+            let _ = dev; // suppress unused if device path not needed
+            let shape = tensor.dims().to_vec();
+            total_params += shape.iter().product::<usize>();
+            let prefix = name
+                .split('.')
+                .take(2)
+                .collect::<Vec<_>>()
+                .join(".");
+            grouped
+                .entry(prefix)
+                .or_default()
+                .push((name.clone(), shape));
+        }
+        for (prefix, items) in grouped.iter() {
+            println!("  [{prefix}]  ({} tensors)", items.len());
+            for (name, shape) in items.iter().take(8) {
+                println!("    {name:<55} {shape:?}");
+            }
+            if items.len() > 8 {
+                println!("    ... {} more", items.len() - 8);
+            }
+        }
+        println!(
+            "  total params: {total_params} ({:.2} M)",
+            total_params as f64 / 1e6
+        );
+    }
+
+    println!();
+    println!("=== summary ===");
+    println!("Three small networks; combined params likely 5-10 M.");
+    println!("Each layer is gated conv + BatchNorm2d. STFT params");
+    println!("come from hparams.yaml above.");
+    println!();
+    println!("Next: implement Layer/Encoder/CarrierDecoder/MsgDecoder");
+    println!("in src/silentcipher.rs (Phase 10.2).");
+
+    Ok(())
+}