Y

Monitoring and observability · Advanced

SLOs, error budgets and alerts you can test

4 min read

Why it matters

Alerting on every CPU spike wakes people for nothing; alerting on nothing means customers find outages first. Site reliability and platform roles use service level objectives to decide what matters, and interviewers ask candidates to define one and design the alerts around it.

The concept

  • A service level indicator (SLI) measures user experience: the share of requests that succeed, the share served under 300 ms.

  • A service level objective (SLO) sets a target for it over a window: 99.5% of requests succeed over 30 days.

  • The error budget is what the SLO allows to fail: 0.5% of requests, about 3.6 hours of total outage in 30 days. Spend it on releases and experiments; when it runs out, slow down and fix reliability.

  • Burn rate is how fast you are spending the budget. A burn rate of 1 uses exactly the budget over 30 days; 14.4 uses 2% of the monthly budget in one hour.

  • Multi-window alerts page on fast burns (checked over 1 hour and 5 minutes, so they fire fast and reset fast) and open tickets for slow burns (6 hours and 30 minutes).

Alert on symptoms users feel (errors, latency), not on causes (CPU), and give every alert a runbook.

Worked example: rules for the fare API

For the service from Instrument a service: metrics, logs and request ids:

slo-rules.yml
groups:
  - name: fare-api-slo
    rules:
      # Share of requests that failed (5xx), over several windows
      - record: fare_api:error_ratio:rate5m
        expr: sum(rate(http_requests_total{route="/fare-estimate",status=~"5.."}[5m])) / sum(rate(http_requests_total{route="/fare-estimate"}[5m]))
      - record: fare_api:error_ratio:rate1h
        expr: sum(rate(http_requests_total{route="/fare-estimate",status=~"5.."}[1h])) / sum(rate(http_requests_total{route="/fare-estimate"}[1h]))
      - record: fare_api:error_ratio:rate30m
        expr: sum(rate(http_requests_total{route="/fare-estimate",status=~"5.."}[30m])) / sum(rate(http_requests_total{route="/fare-estimate"}[30m]))
      - record: fare_api:error_ratio:rate6h
        expr: sum(rate(http_requests_total{route="/fare-estimate",status=~"5.."}[6h])) / sum(rate(http_requests_total{route="/fare-estimate"}[6h]))

      # SLO: 99.5% of requests succeed over 30 days, so the error budget is 0.5%.
      # Page when the budget burns 14.4 times too fast (2% of a 30-day budget in 1 hour).
      - alert: FareApiErrorBudgetFastBurn
        expr: fare_api:error_ratio:rate1h > (14.4 * 0.005) and fare_api:error_ratio:rate5m > (14.4 * 0.005)
        for: 2m
        labels:
          severity: page
        annotations:
          summary: "Fare API is burning its error budget fast ({{ $value | humanizePercentage }} errors)"
          runbook: https://example.com/runbooks/fare-api#errors
      # Ticket when it burns 6 times too fast over 6 hours (5% of the budget).
      - alert: FareApiErrorBudgetSlowBurn
        expr: fare_api:error_ratio:rate6h > (6 * 0.005) and fare_api:error_ratio:rate30m > (6 * 0.005)
        for: 15m
        labels:
          severity: ticket
        annotations:
          summary: "Fare API error budget burning steadily"

Alert rules are code, so test them. promtool feeds synthetic time series in and checks which alerts fire:

slo-rules-test.yml
rule_files: [slo-rules.yml]
evaluation_interval: 1m
tests:
  - interval: 1m
    input_series:
      # 100 requests a minute; from minute 60, 10% of them fail
      - series: 'http_requests_total{route="/fare-estimate",status="200"}'
        values: '0+100x59 6000+90x60'
      - series: 'http_requests_total{route="/fare-estimate",status="500"}'
        values: '0x59 0+10x60'
    alert_rule_test:
      - eval_time: 55m
        alertname: FareApiErrorBudgetFastBurn
        exp_alerts: []
      - eval_time: 115m
        alertname: FareApiErrorBudgetFastBurn
        exp_alerts:
          - exp_labels:
              severity: page
            exp_annotations:
              summary: "Fare API is burning its error budget fast (9.181% errors)"
              runbook: https://example.com/runbooks/fare-api#errors
check.sh
promtool check rules slo-rules.yml          # SUCCESS: 6 rules found
promtool test rules slo-rules-test.yml      # SUCCESS

Before failures start (minute 55), no alert. Once 10% of requests fail, the page fires. The summary reports 9.181% rather than 10% because $value is the one-hour ratio, and part of that hour was still healthy: a detail you only discover by testing.

Common mistakes

  • SLOs of 100%. Nothing is perfectly reliable, and a 100% target leaves no room to ship changes.

  • Alerting on causes. High CPU with happy users is not an emergency; an error spike with normal CPU is.

  • Alerts without runbooks. At 3 a.m. nobody remembers what to check.

  • Untested alert rules. A typo in a label silently disables an alert. Test rules in CI.

Interview angle

"Define an SLO for a payments API and design its alerting." Choose SLIs (success ratio, latency at p99 under a threshold), a target with an error budget, burn-rate alerts for paging and tickets, and a policy for what happens when the budget is spent. Mention testing rules and runbooks.

Practice task

  1. Add a latency SLO: 99% of requests under 300 ms, using the histogram buckets.

  2. Write a test where errors are 4% for 8 hours and confirm the slow-burn alert fires (threshold 3%) but not the fast one (threshold 7.2%).

  3. Run the rule tests in CI with the GitHub Actions workflow from the CI/CD shelf.

Further reading