Dynamic Baselines Beat Static Thresholds Every Time

2:00 PM - prod-db-03 disk usage: 87%
Static threshold: 85% exceeded → ALERT
Engineer paged: Investigates, finds normal traffic spike
Result: False positive, wasted 45 minutes

vs.

3:00 AM - prod-db-03 disk usage: 78%
Static threshold: Not exceeded → NO ALERT
Reality: Baseline is 65% at 3 AM, 78% is 4.3σ deviation
Result: Missed real anomaly, incident discovered hours later

January:  Set threshold at 80% CPU (works)
March:    False positives increase, raise to 85%
June:     Still too many alerts, raise to 90%
September: Missing real incidents, lower to 87%
December: Alert fatigue, engineers ignore all CPU alerts
Result: Threshold tuning is a full-time job

Threshold: disk_usage > 85%

2 PM (peak traffic):
  - Disk 87% = normal (lots of temp files from requests)
  - Alert fires = FALSE POSITIVE

3 AM (off-peak):
  - Disk 78% = anomaly (should be 65%, log rotation failed)
  - Alert doesn't fire = MISSED INCIDENT

Threshold: request_rate > 10,000/sec

Normal day: 5,000/sec average, 10,000/sec = real spike
Black Friday: 50,000/sec average, 10,000/sec = impossibly low
Summer: 2,500/sec average, 10,000/sec = never triggers (too high)

Result: Threshold is wrong 80% of the year

# Pseudocode: Time-of-day baseline calculation
def calculate_baseline(metric, host, current_hour):
    # Get last 7 days of data for this hour
    historical_data = get_metric_data(
        metric=metric,
        host=host,
        hour=current_hour,  # e.g., 14 (2 PM)
        last_n_days=7
    )

    # Calculate statistics
    baseline_mean = mean(historical_data)
    baseline_stddev = stddev(historical_data)

    # Define normal range
    normal_min = baseline_mean - (2 * baseline_stddev)
    normal_max = baseline_mean + (2 * baseline_stddev)

    return {
        "mean": baseline_mean,
        "stddev": baseline_stddev,
        "normal_range": [normal_min, normal_max]
    }

# Example output for prod-db-03, disk_usage, 2 PM:
# { "mean": 72.3, "stddev": 2.1, "normal_range": [68.1, 76.5] }

# Pseudocode: Anomaly detection
def detect_anomaly(current_value, baseline):
    deviation = (current_value - baseline.mean) / baseline.stddev

    if deviation > 4.0:
        return "CRITICAL"  # >4σ
    elif deviation > 3.0:
        return "HIGH"      # 3-4σ
    elif deviation > 2.0:
        return "WARNING"   # 2-3σ
    else:
        return "NORMAL"    # <2σ

# Example: disk_usage = 91.4% at 2 PM
# baseline.mean = 72.3%, baseline.stddev = 2.1%
# (91.4 - 72.3) / 2.1 = 9.1σ → "CRITICAL"

Disk usage samples (every 30s):
72%, 73%, 91%, 74%, 72%, 73%
              ↑
          One spike (temporary write, then deleted)

Static threshold: Triggers immediately on 91%

Dynamic baseline with persistence:
  Sample 1: 91% (anomaly detected)
  Sample 2: 74% (back to normal)
  Duration: 30 seconds (< 2 minutes required)
  Action: Ignore (transient spike)

Sustained anomaly:
  Sample 1-6: 91%, 92%, 90%, 91%, 93%, 91%
  Duration: 3 minutes (> 2 minutes required)
  Action: Trigger playbook (real incident)

def detect_persistent_anomaly(metric_history,
                              threshold_sigma=2.0,
                              duration_seconds=120):
    anomaly_start = None

    for sample in metric_history:
        if sample.deviation > threshold_sigma:
            if anomaly_start is None:
                anomaly_start = sample.timestamp
            else:
                duration = sample.timestamp - anomaly_start
                if duration >= duration_seconds:
                    return True  # Persistent anomaly
        else:
            anomaly_start = None  # Reset if back to normal

    return False  # No persistent anomaly

# Known events configuration
known_events:
  - name: weekly_database_backup
    schedule: "0 2 * * 0"  # Cron: Every Sunday 2 AM
    duration: 120  # minutes
    suppress_alerts:
      - disk_usage
      - cpu_usage
      - network_throughput
    hosts:
      - "*.db.*"  # All database servers

def should_suppress_alert(metric, host, timestamp):
    for event in known_events:
        if event.matches(host) and event.is_active(timestamp):
            if metric in event.suppress_alerts:
                return True  # Suppress, known event
    return False  # Don't suppress

Primary anomaly: disk_usage = 91%

Check correlated metrics:
  - inode_usage: 45% (normal, not exhausted)
  - disk_write_iops: 1,200 (normal, not thrashing)
  - disk_read_latency: 3ms (normal, no performance degradation)
  - application_errors: 0 (normal, no failures)
  - tcp_connections: 150 (normal, services healthy)

Interpretation:
  Disk space is high, but performance is normal.
  Likely cause: Legitimate large file write.

Action: Lower confidence (0.75 → requires approval vs. autonomous)

# Multi-metric correlation
correlation_rules:
  disk_usage_critical:
    primary: disk_usage > baseline + 4σ
    correlations:
      - metric: inode_usage
        threshold: > 90%
        weight: 0.3
      - metric: disk_write_iops
        threshold: > baseline + 3σ
        weight: 0.2
      - metric: application_errors
        threshold: > 0
        weight: 0.5

    confidence_boost: 0.2  # If correlations match, increase confidence

CPU usage last 7 days: [40%, 42%, 44%, 46%, 48%, 50%, 65%]

Standard average: 47.9%
Weighted average: 53.6% (Day 7's 65% weighted higher)

Current: 68%
Deviation (standard): (68 - 47.9) / stddev = 4.2σ → CRITICAL
Deviation (weighted): (68 - 53.6) / stddev = 2.8σ → HIGH

Result: Weighted baseline reduces false alarm severity

# Enable weighted baseline
baseline_calculation:
  method: weighted_rolling_average
  weights: [0.5, 0.7, 0.9, 1.1, 1.3, 1.5, 2.0]  # Day 1-7

# Or use exponential decay
baseline_calculation:
  method: exponential_decay
  decay_factor: 0.85  # Recent data 15% more important each day back