rtx-interpret: fix decoder transpose in SAE compute_and_apply_gradients
The encoder-gradient path through the decoder was transposing the decoder before the matmul, producing `[batch, d_model] × [d_sae, d_model]` — a shape mismatch for every batch > 1. The decoder is stored as `[d_model, d_sae]`, so `recon_grad @ decoder` is already the right shape (and matches the comment at the call site, which reads "recon_grad @ decoder @ d_relu"). All 9 existing `sae::tests` still pass. Omni-Cortex's `LatentDictionary` now trains correctly on batches larger than 1. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
This commit is contained in:
@@ -355,9 +355,13 @@ impl SAETrainer {
|
|||||||
// Decoder bias gradient: sum of recon_grad along batch dimension
|
// Decoder bias gradient: sum of recon_grad along batch dimension
|
||||||
let decoder_bias_grad = recon_grad.sum(Some(0))?;
|
let decoder_bias_grad = recon_grad.sum(Some(0))?;
|
||||||
|
|
||||||
// Encoder gradient (through decoder): recon_grad @ decoder @ d_relu
|
// Encoder gradient (through decoder): recon_grad @ decoder @ d_relu.
|
||||||
let decoder_t = self.sae.decoder().transpose(0, 1)?;
|
// `decoder` has shape [d_model, d_sae] and `recon_grad` has shape
|
||||||
let pre_encoder_grad = recon_grad.matmul(&decoder_t)?;
|
// [batch, d_model], so `recon_grad @ decoder = [batch, d_sae]` —
|
||||||
|
// no transpose needed. (The previous transpose here was an
|
||||||
|
// upstream bug: it produced a shape-mismatched matmul for every
|
||||||
|
// batch size > 1.)
|
||||||
|
let pre_encoder_grad = recon_grad.matmul(self.sae.decoder())?;
|
||||||
|
|
||||||
// ReLU gradient: mask where features > 0
|
// ReLU gradient: mask where features > 0
|
||||||
// Create a mask by computing (features > 0) as 1.0 or 0.0
|
// Create a mask by computing (features > 0) as 1.0 or 0.0
|
||||||
|
|||||||
Reference in New Issue
Block a user