Alerts without fatigue: SLOs and error budgets in practice

Alert fatigue follows a familiar pattern. Every incident leaves a new alert behind, nobody retires the old ones, and eventually the on-call shift gets hundreds of notifications a night. Most of them need no action. The team learns to ignore them, and the day the important one arrives, it gets lost in the noise.

Adding more rules does not fix this. What works is changing the criterion: alert when the service is failing its users in a way that matters, and only then. Service level objectives (SLOs) and error budgets provide a concrete method to do it.

SLI, SLO and error budget

Three concepts, in this order:

• SLI (service level indicator): a measure of how users experience the service. The most practical form is a ratio: good events over valid events. For example, requests answered without error and in under 500 ms, over total requests.
• SLO (service level objective): the value that indicator must reach over a time window. For example, 99.9% good requests over 30 days.
• Error budget: what the SLO allows to fail. With a 99.9% SLO, the budget is 0.1% of the events in the window.

The budget changes the conversation. You no longer argue about whether an error is acceptable in the abstract. You ask how much budget is left and how fast it is being spent. While budget remains, the team can deploy and experiment. If it runs out, it makes sense to prioritize reliability over new features until it recovers.

How to choose good SLIs

• Measure as close to the user as possible: at the load balancer, the API gateway or the client, not on the server's disk.
• Start with a few: availability (share of responses without error) and latency (share of responses below a threshold) cover most services.
• Define what counts as a valid event. For example, a 4xx error caused by bad data sent by the client normally should not consume budget.
• Do not set the SLO at 100%. No service meets it, and an impossible target stops guiding decisions.

Alert on symptoms, not causes

A cause is something that may produce a problem: high CPU, a replica down, a disk at 85%. A symptom is the problem as the user experiences it: errors, slowness, operations that do not complete.

Alerting on causes has two flaws. It fires when there is no impact (95% CPU may be normal for a batch job) and it leaves gaps when there is impact without an expected cause (a configuration error does not raise CPU). Alerting on symptoms flips the logic: you alert when users are hurting, whatever the cause.

This does not mean removing cause metrics. They remain essential for diagnosis. The difference is where they go: cause signals feed dashboards and analysis; alerts that wake someone up are reserved for symptoms. The "Monitoring Distributed Systems" chapter of Google's SRE book develops this principle.

Burn rate: the measure that makes the SLO useful

Alerting directly on the SLO has a problem. If you alert when the error rate exceeds 0.1% over the last five minutes, any short spike fires an alert, even if it consumes a tiny fraction of the budget. If you wait until the 30-day SLO is missed, it is too late.

The answer is burn rate: how fast the budget is being spent, compared with the rate that would use it up exactly at the end of the window.

• A burn rate of 1 spends exactly the budget over the 30 days.
• A burn rate of 2 uses it up in 15 days.
• A burn rate of 14.4 uses it up in just over two days (720 hours divided by 14.4 is 50 hours).

Burn rate is the observed error rate divided by the error rate the SLO allows. With a 99.9% SLO, an error rate of 1.44% equals a burn rate of 14.4.

Multiwindow, multi-burn-rate alerts

Google's Site Reliability Workbook, in the chapter "Alerting on SLOs", walks through several ways to alert on SLOs and recommends a combination: several burn rates, each evaluated over two windows.

Why several burn rates

A severe problem burns a lot of budget in a short time and needs an immediate response. A mild, sustained problem burns it slowly, but it also runs the budget out if nobody acts. That is why several levels are defined. For a 30-day SLO, the Workbook proposes this starting point:

Budget consumed	Long window	Short window	Burn rate	Action
2%	1 hour	5 minutes	14.4	Page
5%	6 hours	30 minutes	6	Page
10%	3 days	6 hours	1	Ticket

These values are a reasonable starting point, not a fixed rule. Each organization should tune them to the criticality of the service and its ability to respond.

Why two windows

The long window ensures that a meaningful share of the budget has been consumed, which avoids alerts on brief spikes. But it has a side effect: once the problem is fixed, the long window keeps showing a high rate for quite a while, and the alert is slow to clear.

The short window fixes that. The alert only fires if the burn rate exceeds the threshold in both windows at the same time. So when the problem ends, the short window drops quickly and the alert clears. The Workbook suggests a short window of about one twelfth of the long one, as in the table.

Worked example

This example is illustrative; the numbers do not describe any real service.

Take a payments API with this SLO: 99.9% of valid requests are answered without a server error over a 30-day window. If the API receives, for example, 10 million requests a month, the budget is 10,000 failed requests.

A burn rate alert of 14.4 over one hour means the error rate exceeds 1.44% for that hour. At that volume, about 13,900 requests arrive in an hour, and 1.44% of them is about 200: 2% of the monthly budget. If nobody acted, the full budget would be gone in about 50 hours.

Here is how it looks as Prometheus alerting rules, with the two paging conditions. Metric and label names are examples:

# Example: 99.9% SLO (error budget 0.001) for the payments API
groups:
  - name: slo-payments-api
    rules:
      # 2% of the budget in 1 hour: burn rate 14.4
      - alert: PaymentsFastBudgetBurn
        expr: |
          (
            sum(rate(http_requests_total{job="payments-api",code=~"5.."}[1h]))
            /
            sum(rate(http_requests_total{job="payments-api"}[1h]))
          ) > (14.4 * 0.001)
          and
          (
            sum(rate(http_requests_total{job="payments-api",code=~"5.."}[5m]))
            /
            sum(rate(http_requests_total{job="payments-api"}[5m]))
          ) > (14.4 * 0.001)
        labels:
          severity: page
        annotations:
          summary: "Payments API: fast error budget burn"

      # 5% of the budget in 6 hours: burn rate 6
      - alert: PaymentsSustainedBudgetBurn
        expr: |
          (
            sum(rate(http_requests_total{job="payments-api",code=~"5.."}[6h]))
            /
            sum(rate(http_requests_total{job="payments-api"}[6h]))
          ) > (6 * 0.001)
          and
          (
            sum(rate(http_requests_total{job="payments-api",code=~"5.."}[30m]))
            /
            sum(rate(http_requests_total{job="payments-api"}[30m]))
          ) > (6 * 0.001)
        labels:
          severity: page
        annotations:
          summary: "Payments API: sustained error budget burn"

In production, precompute the ratios with recording rules to avoid repeating expensive queries. Platforms such as Dynatrace or Datadog offer native SLO features and burn rate alerts; the reasoning is the same even if the configuration differs.

Putting it into operation

1. Inventory the current alerts and classify them: symptom, cause or informational. Check how many led to real action in recent months.
2. Define SLIs and SLOs for critical services, together with the business owner. The SLO is an agreement, not just a technical number.
3. Implement burn rate alerts for those services and make them the only alerts that page the shift for those services.
4. Downgrade cause alerts to dashboards, tickets or context inside the incident.
5. Review the budget regularly with product and development, and agree on what happens when it runs out.
6. Tune after every incident: did the alert arrive in time? Was any alert unnecessary? Did the SLI reflect what the customer felt?

Low-traffic services need care. With few requests per hour, a single error can trigger a high burn rate. Common options are grouping related services, generating synthetic traffic or adjusting the windows. The Workbook covers this case in the same chapter.