fix(anomaly): clamp Welford z-score floor to 1% of mean, not 1e-9

A metric flat at 0.0 for N samples has variance ≈ 0, stddev clamped to
1e-9. Any non-zero observation then produced z = value/1e-9 → z² on
the order of 10^16, triggering spurious multi-million-point anomaly alerts
(e.g. "tank deviating >2σ, score 5774543.2") on idle nodes with no
cache traffic.

Fix: effective stddev floor is max(raw_stddev, |mean|*0.01, 1e-4).
When mean ≈ 0 the floor is 1e-4 (not 1e-9), requiring a deviation of
at least 0.01 (1%) before z² reaches the warn threshold of 9.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
This commit is contained in:
Omar Sobh
2026-07-23 20:08:45 +00:00
co-authored by Claude Sonnet 4.6
parent 5001ac233c
commit 5e7f7dfc83
+14 -2
View File
@@ -65,13 +65,25 @@ impl Welford {
fn stddev(&self) -> f64 {
if self.n < 2 { return 1.0; }
(self.m2 / (self.n - 1) as f64).sqrt().max(1e-9)
(self.m2 / (self.n - 1) as f64).sqrt()
}
/// Squared z-score for a new observation (non-destructive).
///
/// Returns 0 when:
/// * fewer than ANOMALY_MIN_SAMPLES have been seen, OR
/// * the metric has been flat (stddev < 1% of its mean) — a near-zero
/// stddev means the baseline is perfectly stable, so the floor for
/// "meaningful deviation" is the metric's own magnitude, not 1e-9.
/// Without this guard, a metric stuck at 0.0 for the first N samples
/// then spiking to any non-zero value produces z² → ∞.
fn z_sq(&self, x: f64) -> f64 {
if self.n < ANOMALY_MIN_SAMPLES { return 0.0; }
let z = (x - self.mean) / self.stddev();
let sd = self.stddev();
// Effective floor: 1% of |mean| when the signal is near-zero
// noise-free, otherwise the raw stddev.
let effective_sd = sd.max(self.mean.abs() * 0.01).max(1e-4);
let z = (x - self.mean) / effective_sd;
z * z
}
}