`normalize_gate: Option<(lo, hi)>` — when set, a decoder column is
rescaled to unit norm only if its norm has left the band; columns
inside are left exactly as the gradient step made them (divisor 1.0).
`None` keeps per-step renormalisation, bit for bit.
Why (omni-cortex D629/D630): per-step rescaling was measured doing
two opposite things on the same 32-unit SAE. With it off, two runs
descended cleanly to floors 3-7x LOWER than with it on — it was
fighting descent. Two other runs (lr 0.01, seeds 7 and 99) diverged
outright without it — it was also the clamp holding an unstable rate
finite, turning a blow-up into a slow oscillation that looked like a
healthy dictionary drifting. The band keeps the second role and drops
the first; D630 measures whether it does both.
The cold-start exemption is applied after the gate, unchanged. Test
pins: in-band columns bit-identical before/after, out-of-band pulled
to unit, gate None == per-step. Fixture norms sit strictly off the
band edge — a hand-scaled 2.0 came out 2.0000002 in f32 and was,
correctly, treated as outside.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
reinitialize_encoder_neuron_cold / SAETrainer::reinitialize_neuron_cold
exempt a freshly reset unit's decoder column from per-step
normalize_decoder for cold_steps training steps, so its small random
init is not blown up to unit norm before it has learned anything —
the mechanism omni-cortex's D605 refutation left as the prime
suspect, now testable (D620 downstream). Runtime-only state, not
carried through checkpoints (documented); cold_steps 0 is exactly
the plain reinit, and existing entry points delegate with 0.
Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01B1feFAQxjbCRHePUdxuNra
SparseAutoencoder::new_seeded draws encoder/decoder weights via
randn_seeded with per-tensor SplitMix64-derived seeds (same
derivation as MambaBlock::new_seeded), so identical (config, seed)
gives bit-exact SAEs — without it, cross-instance loss comparisons
are noise (measured downstream: 0.09 vs 0.65 starts on identical
data).
SAETrainer::reinitialize_neuron couples the encoder-unit weight
reinit with zeroing that unit's optimizer moment rows (encoder row,
bias slot, decoder column), so external generate-and-test callers
can't reset weights while leaving optimizer state stale — previously
only the trainer's internal dead-neuron resampling did both. Note
train_step's update rule is plain SGD today, so the moment reset is
inert until the Adam path is switched on; the coupling is the
contract either way, and a doctored-checkpoint test pins the
row/column semantics.
Also drops a vacuous assert!(true) smoke test that failed clippy's
assertions_on_constants.
Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01B1feFAQxjbCRHePUdxuNra
The encoder-gradient path through the decoder was transposing the decoder
before the matmul, producing `[batch, d_model] × [d_sae, d_model]` — a
shape mismatch for every batch > 1. The decoder is stored as
`[d_model, d_sae]`, so `recon_grad @ decoder` is already the right shape
(and matches the comment at the call site, which reads
"recon_grad @ decoder @ d_relu").
All 9 existing `sae::tests` still pass. Omni-Cortex's `LatentDictionary`
now trains correctly on batches larger than 1.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
osobhandClaude Opus 4.7