Prompt and context
Design a service that computes SLOs, error budgets, and burn rates for many services and triggers reliability gates and alerts. How do you handle late data, windows, silences, backfills, and exceptions?
This fits SRE, platform, backend, and system-design roles. The Google SRE workbook describes error budgets as a way to balance reliability with delivery speed, while burn-rate alerting focuses on how quickly that budget is consumed. The design must connect definitions, computation, alerts, decisions, and auditability.
What the interviewer is testing
- Distinguish SLI events, the SLO target, compliance period, error budget, and burn rate.
- Explain why fast and slow windows complement each other and how noise is controlled.
- Design deduplication, late data, duplicate samples, backfills, and versioned recomputation.
- Separate alerts from deployment gates, on-call response, silences, and human exceptions.
- Set boundaries for tenant isolation, query cost, historical traceability, and permissions.
- Validate policies with data-quality metrics and replay rather than only drawing a dashboard.
30-second answer framework
“I would split the service into SLO specifications, event ingestion, window aggregation, budget calculation, policy evaluation, and notification audit. Each SLO fixes successful and total events, target, compliance period, and version; burn rate is the observed error rate divided by the allowed error rate. A fast window catches spikes and a slow window confirms persistence, with freshness and sample-size checks. Late or backfilled data creates a versioned recomputation instead of silently rewriting an audited decision.”
Step-by-step deep dive
Step 1: Define the SLI and event contract
An SLO specification names the service, metric type, successful events, total events, target, compliance period, aggregation dimensions, and timezone. Counters need a stable deduplication key and event time. Client cancellations, dependency failures, and platform faults should not be mixed without an explicit policy. Once published, the specification is versioned so historical calculations use the version that was active then.
Step 2: Calculate budget and burn rate
For a 99.9% target, the allowed error ratio over the compliance period is 0.1%. Divide the current window's error rate by that allowed rate to get burn rate: 1 consumes budget at the planned pace, while a value above 1 consumes it faster. Store raw counts and aggregates with numerators, denominators, and time boundaries so results can be recomputed instead of preserving only a percentage.
Step 3: Combine fast and slow windows
A fast window detects a release spike quickly; a slow window confirms that impact persists. A policy records both window lengths, burn-rate thresholds, minimum event volume, and required evaluations. Trigger only when freshness and denominator size are sufficient. Services can use different policies, but templates should state the purpose rather than forcing every alert into one threshold.
Step 4: Handle late, duplicate, and backfilled data
Make ingestion idempotent by event ID or time bucket, and persist an aggregation watermark and correction version. Late events inside the allowed delay trigger recomputation; events outside it mark the data incomplete. A backfill must not silently change an old notification or gate. Record the old value, new value, operator, reason, and affected decision ID.
Step 5: Connect alerts, gates, and exceptions
The alert service notifies and escalates, the deployment controller pauses or requests approval, and the policy service emits evidence-backed state. An exception has scope, expiry, approver, and reason. A silence suppresses notification only; it does not stop budget calculation. Every gate includes the SLO, window, counts, freshness, and rule versions so the decision can be audited.
Step 6: Scale and observe the service
Shard aggregation by service, region, and tenant, limit high-cardinality dimensions, and cache repeated queries. Monitor ingestion lag, loss, duplication, aggregation delay, evaluation time, notification success, and policy errors. Replay historical incidents to validate windows and inject failures to measure end-to-end event-to-alert and event-to-gate latency. Incomplete data should be shown as unknown and escalated, not reported as healthy.
Trade-offs, boundaries, and information gain
The error-budget service turns a reliability target into an auditable engineering decision. More sensitive burn-rate thresholds increase noise and alert burden; longer windows respond more slowly. Separate computed facts, notification actions, and deployment decisions. Permit corrected recomputation while retaining historical versions. The service supplies shared evidence; it does not choose the team's priorities.
Model high-quality answer
“I would start with a versioned SLO specification covering successful and total events, target, compliance period, dimensions, and deduplication. Idempotent ingestion stores numerators, denominators, time boundaries, and watermarks by service and window. Aggregation calculates error rate, allowed error rate, and burn rate. Policy evaluation uses a fast window for spikes and a slow window for persistence, checking minimum volume and freshness.
Notifications, deployment gates, and human exceptions are separate consumers. Silence suppresses notification but leaves computation running; a gate emits evidence with SLO, window, counts, rule, and data versions; exceptions are approved, time-bound, and auditable. Late data creates a versioned recomputation rather than quietly changing a prior decision.
At scale I shard by service, region, and tenant, cap high-cardinality labels, and monitor ingestion lag, duplicates, loss, aggregation delay, and notification success. Replay and fault injection validate the policy. Incomplete data is marked unknown and escalated instead of being counted as zero errors.”
Common mistakes
- Storing only the SLO percentage → sample size and boundaries disappear → retain numerator, denominator, window, and watermark.
- One threshold for every service → risk and traffic differ → version policies by target, volume, and window.
- Treating silence as stopping computation → budget facts are lost → suppress notifications only and keep auditing.
- Overwriting history with late data → release decisions cannot be reconstructed → version recomputations and retain old and new values.
- Treating missing data as healthy → collection failures are hidden → show unknown and alert on data quality.
- Making a gate depend on notification state → notification retries change release results → gates read immutable evaluation evidence.
Follow-up questions and answers
Why use both fast and slow windows?
The fast window reduces detection time for spikes; the slow window filters transient noise and confirms sustained budget consumption. Together they balance response speed and false positives, but thresholds must be validated against event volume.
What happens to an old alert after a backfill changes burn rate?
Keep the old evaluation and notification records, create a recomputation tagged with the new data version, and state whether the current gate changes. Do not delete history; responders need the facts available at the time.
How do you prevent high-cardinality dimensions from overwhelming the system?
Limit allowed labels, sample tenant or route slices by tier, materialize only valuable views, and enforce query budgets and timeouts. A service-level SLO should not require unlimited dimensions.
How do exceptions avoid permanently bypassing gates?
Bind each exception to a service, change, or time range, require an approver and reason, and enforce automatic expiry. Restore the default policy afterward and measure exception coverage as an audit metric.