Why Most Alerts Are Noisy
Alert fatigue is one of the greatest threats to system reliability. When engineers receive hundreds of low-priority or false alerts each week, they inevitably begin to ignore notifications. The root cause is usually simplistic query design: static thresholds placed on non-stationary signals like CPU utilization or raw request count.
To build reliable systems, alerts must measure user impact and utilize mathematical precision.
1. Escaping the Average Trap: Use Percentiles
Never alert on average response time (average(duration)). Averages conceal severe tail-latency spikes that affect your highest-value users.
-- BAD: Masks outliers and customer degradation
SELECT average(duration) FROM Transaction WHERE appName = 'PaymentService'
-- GOOD: Reflects real customer experience at the 95th and 99th percentiles
SELECT percentile(duration, 95, 99) FROM Transaction WHERE appName = 'PaymentService'
2. Detecting Silent Failures: Drop-off Alerting
Some of the worst production outages are "silent failures" where services do not return HTTP 500 errors, but traffic abruptly plummets because an upstream DNS or load balancer broke.
A standard error-rate alert will not trigger because zero errors are being logged. Instead, use a baseline comparison query:
-- Detects unexpected drops in successful transactions compared to last week
SELECT count() FROM Transaction
WHERE appName = 'OrderCheckout'
AND httpResponseCode = '200'
COMPARE WITH 1 WEEK AGO
3. Sliding Windows for Volatile Signals
For high-throughput microservices, single-minute spikes can cause momentary threshold breaches that self-heal before an engineer even opens their laptop. Use sliding windows (SLIDE BY) to smooth out transient jitter while maintaining rapid detection for sustained degradation:
SELECT count() FROM TransactionError
WHERE appName = 'PaymentGateway'
TIMESERIES 5 MINUTES SLIDE BY 1 MINUTE
4. Facet-Based Dynamic Alerting
Instead of creating 50 separate alert conditions for 50 endpoints, use FACET to monitor all routes in a single condition:
SELECT count(*) FROM TransactionError
FACET request.uri
WHERE appName = 'CoreAPI'
New Relic will automatically evaluate each URI independently and open an incident pointing directly to the offending endpoint.
Summary Checklist for Production Alerts
By applying these principles, you can reduce alert volume by 40–60% while simultaneously increasing incident detection speed.