rtx-interpret: fix decoder transpose in SAE compute_and_apply_gradients

The encoder-gradient path through the decoder was transposing the decoder
before the matmul, producing `[batch, d_model] × [d_sae, d_model]` — a
shape mismatch for every batch > 1. The decoder is stored as
`[d_model, d_sae]`, so `recon_grad @ decoder` is already the right shape
(and matches the comment at the call site, which reads
"recon_grad @ decoder @ d_relu").

All 9 existing `sae::tests` still pass. Omni-Cortex's `LatentDictionary`
now trains correctly on batches larger than 1.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
This commit is contained in:
2026-04-24 13:58:44 -07:00
co-authored by Claude Opus 4.7
parent 85b77d49f2
commit 15b62a8f6e
@@ -355,9 +355,13 @@ impl SAETrainer {
// Decoder bias gradient: sum of recon_grad along batch dimension // Decoder bias gradient: sum of recon_grad along batch dimension
let decoder_bias_grad = recon_grad.sum(Some(0))?; let decoder_bias_grad = recon_grad.sum(Some(0))?;
// Encoder gradient (through decoder): recon_grad @ decoder @ d_relu // Encoder gradient (through decoder): recon_grad @ decoder @ d_relu.
let decoder_t = self.sae.decoder().transpose(0, 1)?; // `decoder` has shape [d_model, d_sae] and `recon_grad` has shape
let pre_encoder_grad = recon_grad.matmul(&decoder_t)?; // [batch, d_model], so `recon_grad @ decoder = [batch, d_sae]` —
// no transpose needed. (The previous transpose here was an
// upstream bug: it produced a shape-mismatched matmul for every
// batch size > 1.)
let pre_encoder_grad = recon_grad.matmul(self.sae.decoder())?;
// ReLU gradient: mask where features > 0 // ReLU gradient: mask where features > 0
// Create a mask by computing (features > 0) as 1.0 or 0.0 // Create a mask by computing (features > 0) as 1.0 or 0.0