The Alert Rules That Actually Get Action (Not Ignored)

Dashboards show trends, but someone has to open them. Alerts do the opposite: they come to you when something needs attention. This guide walks through configuring alerts that catch real problems without drowning your team in noise.

Why Does Proactive Alerting Matter?

The Problem with Dashboard-Only Monitoring

Dashboards need a human staring at them, and busy weeks are exactly when nobody does. By the time someone spots a trend, the context about what changed is gone. Problems fester for days unnoticed.

What Good Alerting Provides

Alerts catch problems when they start, not when they're already critical. They work around the clock even during crunch weeks. And because every alert is timestamped, you get a built-in record of when things went off track.

"The best engineering teams hear about problems in their inbox before the dashboard even looks bad."

What Makes Up an Effective Alert Rule?

Each alert rule in CodePulse has five moving parts:

1. Metric

What you're measuring. The metrics most teams alert on:

• cycle_time_hours - Total PR cycle time
• wait_for_review_hours - Time waiting for first review
• test_failure_rate_percent - CI failure rate
• review_coverage_percent - Percentage of PRs reviewed
• merge_without_approval_rate_percent - Approval bypasses

2. Operator

How to compare the metric value to your threshold:

• > - Greater than (for upper limits)
• < - Less than (for lower limits)
• >= - Greater than or equal
• <= - Less than or equal
• == - Equals (rarely used)

3. Threshold

The number that fires the alert. Base it on:

• Your current baseline (alert on significant deviation)
• Industry benchmarks (alert when below standard)
• Team goals (alert when off-target)

4. Time Period

How often the metric gets checked:

• Daily: Most responsive, good for critical metrics
• Weekly: Smooths out daily variance, good for trends
• Monthly: Big-picture view, good for strategic metrics

5. Severity

How urgent is this alert?

• Critical: Needs same-day attention. Use sparingly.
• Warning: Worth addressing this week, not today.
• Info: Worth knowing. Investigate when you have time.

What Are the Essential Alerts Every Team Should Configure?

Velocity Alerts

Alert: Cycle Time Warning
  Metric: cycle_time_hours
  Operator: >
  Threshold: 48
  Period: weekly
  Severity: warning
  Description: "Weekly average cycle time exceeds 2 days.
    Check for review bottlenecks or blocked PRs."

Alert: Cycle Time Critical
  Metric: cycle_time_hours
  Operator: >
  Threshold: 72
  Period: weekly
  Severity: critical
  Description: "Weekly average cycle time exceeds 3 days.
    Immediate investigation needed - team is significantly blocked."

Review Health Alerts

Alert: Slow First Review
  Metric: wait_for_review_hours
  Operator: >
  Threshold: 8
  Period: daily
  Severity: warning
  Description: "PRs waiting over 8 hours for first review.
    Check reviewer availability and assignment."

Alert: Review Coverage Drop
  Metric: review_coverage_percent
  Operator: <
  Threshold: 95
  Period: weekly
  Severity: warning
  Description: "Review coverage below 95%.
    Ensure all PRs are getting reviewed before merge."

Alert: Approval Bypasses
  Metric: merge_without_approval_rate_percent
  Operator: >
  Threshold: 5
  Period: weekly
  Severity: critical
  Description: "More than 5% of merges bypassing approval.
    Check for branch protection issues or emergency merges."

Quality Alerts

Alert: Test Failure Rate High
  Metric: test_failure_rate_percent
  Operator: >
  Threshold: 15
  Period: weekly
  Severity: warning
  Description: "Test failure rate above 15%.
    Investigate flaky tests or CI issues."

Alert: Test Failure Rate Critical
  Metric: test_failure_rate_percent
  Operator: >
  Threshold: 25
  Period: weekly
  Severity: critical
  Description: "Test failure rate above 25%.
    CI pipeline is unreliable - prioritize fixing."

What Are the Most Common Alert Anti-Patterns?

1. Too Many Alerts (Alert Fatigue)

Problem: So many alerts that people tune them out entirely.

Signs:

• Alerts trigger daily but nobody investigates
• Team asks to disable alerts because "they're just noise"
• Multiple alerts fire for the same root cause

Fix:

• Start with 3-5 alerts, not 20
• Prefer weekly over daily evaluation to cut noise
• Reserve critical for true emergencies; use warning for early signals
• Prune any alert that never leads to action

2. Thresholds Too Tight

Problem: Alerts fire on normal variation, not real problems.

Signs:

• Alert fires, investigation shows nothing wrong
• Tiny swings trigger alerts (cycle time goes from 23h to 25h)
• Thresholds based on ideal state, not actual baseline

Fix:

• Set thresholds from real data, not aspirations
• Build in buffer: if your average is 24h, alert at 36h, not 25h
• Use percentiles to reduce outlier sensitivity

3. Thresholds Too Loose

Problem: Alerts never fire because thresholds are set too high.

Signs:

• Known problems slip through without triggering anything
• Team hears about issues from complaints, not alerts
• Thresholds set at "worst case" levels that never get hit

Fix:

• Tighten thresholds quarterly as your team improves
• Use tiered alerts: warning at moderate deviation, critical at extreme
• Backtest: "Would this rule have caught our last incident?"

"An alert that fires every day and gets ignored is worse than no alert at all. It trains your team that alerts don't matter."

4. No Escalation Path

Problem: Alert fires but nobody knows who should act on it.

Signs:

• Alerts land in a channel everyone ignores
• Nobody feels responsible for investigating
• Same alert fires repeatedly with no resolution

Fix:

• Assign a specific owner for each alert type
• Write action guidance into the alert description
• Review unacknowledged alerts in weekly team meetings

How Should You Configure Alert Notifications?

Slack Integration

Configure Slack delivery in Alerts → Notifications tab:

• Team channel: Good visibility, but gets noisy fast
• Dedicated alerts channel: Keeps alerts organized, though people may mute it
• Direct messages: Best for critical alerts to the on-call owner

Email Options

• Immediate: Each alert sends an email (can be noisy)
• Daily digest: Summary of all alerts once per day
• Critical only: Email only for severity=critical

Quiet Hours

For co-located teams, suppress non-critical alerts outside business hours. Quiet hours stop:

• Weekend notifications for issues that can wait until Monday
• Middle-of-night alerts that no one will act on anyway
• Notification fatigue from off-hours accumulation

How Do You Iterate on Alert Rules Over Time?

Start Conservative

On your first pass, set thresholds looser than you think you need. Run that way for 2-4 weeks, then tighten based on what you learn. Keep iterating until alerts are meaningful without being overwhelming.

Review Dismissed Alerts

When an alert gets dismissed without action, figure out which bucket it falls in:

• False positive? Tighten the threshold or switch metrics
• Real issue the team chose to ignore? Ask why, and whether that's OK
• Threshold wrong for your context? Adjust it

Monthly Alert Health Check

Once a month, run through your alert configuration with these four questions:

1. Which alerts fired, and did anyone act on them?
2. Which alerts never fired? Are those thresholds too loose?
3. Did any incidents slip through that alerts should have caught?
4. Are there new metrics worth alerting on given current priorities?

Connecting Alerts to Action

The difference between a useful alert and a useless one is usually the description field:

For more on specific metrics to alert on, see our guides on Cycle Time Breakdown, Test Failure Rate, and Review Coverage.