Anomaly Detection That Doesn't Cry Wolf

The failure mode of most anomaly detection is not missed incidents. It is alert fatigue. A system that fires fifty times a day for things nobody cares about trains its users to ignore it, and the one alert that mattered gets dismissed along with the noise. Building detection that earns attention is less about exotic algorithms and more about respecting the operator's limited trust.

Static thresholds are a trap

The instinct is to set a threshold: alert if error rate exceeds two percent, if latency exceeds 500 milliseconds, if volume drops below some floor. Static thresholds fail in two directions at once. They fire constantly during normal peaks, and they stay silent during off-peak periods when a real problem hides below the fixed line. Any metric with daily and weekly seasonality, which is almost all of them, defeats a static threshold immediately.

The first upgrade is to model normal behavior instead of asserting it. Seasonal-trend decomposition using LOESS, known as STL, separates a metric into trend, seasonal, and residual components. You detect anomalies on the residual, after seasonality and trend have been removed. A Monday-morning traffic spike no longer looks anomalous because the seasonal component expects it. What remains is the genuinely unexpected.

Ensembles for robustness

No single detector is right for every metric. Isolation forests excel at multivariate outliers by measuring how easily a point can be separated from the rest of the data. Density-based methods catch local anomalies that global methods miss. Simple robust statistics, like a median absolute deviation band, are hard to beat for well-behaved univariate series. We run an ensemble and combine their votes, which makes the system robust to the failure modes of any one method. A point flagged by several independent detectors is far more likely to be a real anomaly than one flagged by a single sensitive detector.

The correlation problem

When something breaks, it rarely breaks in one place. A failing upstream service causes error rates to climb, latency to spike, and downstream volumes to fall, all at once. A naive system sends three separate alerts, and the on-call engineer has to reconstruct that they are one incident. This multiplies the perceived noise and slows diagnosis.

Correlation-aware alerting groups related signals. If several metrics deviate together within a short window and share a dependency in the service graph, they collapse into a single incident with a ranked list of likely root causes. The operator sees one page, not ten, and the page points toward the origin rather than the symptoms. This single change does more for operator trust than any improvement in raw detection accuracy.

Drift is a slower anomaly

Not every problem is a spike. Models decay quietly as the world shifts underneath them. The distribution of inputs drifts away from the training distribution, and predictions degrade without any single dramatic event. Detecting this requires comparing distributions over time rather than watching for point outliers.

Population stability index measures how much a feature's distribution has shifted between a baseline and a current window. The Kolmogorov-Smirnov test detects distributional differences more formally. We run both continuously against the training baseline for every feature and every prediction stream. When drift crosses a threshold, the system does not just alert, it recommends retraining and shows exactly which features moved, so the response is targeted rather than a panicked full retrain.

Adaptive thresholds and feedback

The final ingredient is learning from the operator. Every time an alert is dismissed as a false positive or confirmed as real, that feedback tunes the detection. Bands widen for metrics that are inherently noisy and tighten for metrics that are usually stable. Over weeks, the system converges on a personalized sensitivity that reflects what this particular team actually cares about. Detection that adapts to feedback is the difference between a tool people trust and one they mute.

Measuring the thing that matters

The right metric for an anomaly detection system is not recall in isolation. It is the precision of the alerts that reach a human, because that precision determines whether they keep paying attention. We track median time to detect, the false-alert rate, and the fraction of alerts that operators act on. When those numbers are healthy, the missed-incident rate takes care of itself, because operators trust the system enough to respond when it does fire.

Good monitoring is a trust relationship. Every false alarm is a withdrawal from that account, and every accurate, actionable alert is a deposit. Build the system to stay in credit, and it will be there when it counts.