fix(anomaly): clamp Welford z-score floor to 1% of mean, not 1e-9
A metric flat at 0.0 for N samples has variance ≈ 0, stddev clamped to 1e-9. Any non-zero observation then produced z = value/1e-9 → z² on the order of 10^16, triggering spurious multi-million-point anomaly alerts (e.g. "tank deviating >2σ, score 5774543.2") on idle nodes with no cache traffic. Fix: effective stddev floor is max(raw_stddev, |mean|*0.01, 1e-4). When mean ≈ 0 the floor is 1e-4 (not 1e-9), requiring a deviation of at least 0.01 (1%) before z² reaches the warn threshold of 9. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
This commit is contained in:
co-authored by
Claude Sonnet 4.6
parent
5001ac233c
commit
5e7f7dfc83
@@ -65,13 +65,25 @@ impl Welford {
|
||||
|
||||
fn stddev(&self) -> f64 {
|
||||
if self.n < 2 { return 1.0; }
|
||||
(self.m2 / (self.n - 1) as f64).sqrt().max(1e-9)
|
||||
(self.m2 / (self.n - 1) as f64).sqrt()
|
||||
}
|
||||
|
||||
/// Squared z-score for a new observation (non-destructive).
|
||||
///
|
||||
/// Returns 0 when:
|
||||
/// * fewer than ANOMALY_MIN_SAMPLES have been seen, OR
|
||||
/// * the metric has been flat (stddev < 1% of its mean) — a near-zero
|
||||
/// stddev means the baseline is perfectly stable, so the floor for
|
||||
/// "meaningful deviation" is the metric's own magnitude, not 1e-9.
|
||||
/// Without this guard, a metric stuck at 0.0 for the first N samples
|
||||
/// then spiking to any non-zero value produces z² → ∞.
|
||||
fn z_sq(&self, x: f64) -> f64 {
|
||||
if self.n < ANOMALY_MIN_SAMPLES { return 0.0; }
|
||||
let z = (x - self.mean) / self.stddev();
|
||||
let sd = self.stddev();
|
||||
// Effective floor: 1% of |mean| when the signal is near-zero
|
||||
// noise-free, otherwise the raw stddev.
|
||||
let effective_sd = sd.max(self.mean.abs() * 0.01).max(1e-4);
|
||||
let z = (x - self.mean) / effective_sd;
|
||||
z * z
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user