Prompt and context
The service has a 99.99% availability SLO and a 0.01% monthly error budget. User-visible failures have already exceeded the budget, yet a feature that affects a major renewal is waiting to ship. Explain how you define the indicators, verify that the budget is truly spent, decide which changes pause, and restore a healthy release cadence.
Google SRE treats an error budget as the failure space allowed by an SLO: while it remains, teams may release within reason; after it is spent, changes normally freeze, except urgent security fixes and fixes for the current failure. This is an input to risk decisions, not an unconditional product prohibition.
What the interviewer is testing
A strong answer starts with user-centered SLIs and connects budget consumption to release, rollback, and recovery time. Expect questions about why server metrics can misrepresent user availability, how to separate a short spike from sustained burn, what evidence justifies an exception, and who can lift a freeze.
“Stop all development” ignores security, data integrity, and compliance obligations. “The business is important, so ship” abandons a shared risk language.
Questions to clarify first
User impact and metric boundary
Confirm that SLIs come from real user journeys and include clients, dependencies, and critical workflows. Separate request failures, latency, incorrect data, and unavailability; a single average can hide tail latency or severe impact on one customer class.
Budget window and burn rate
Clarify whether the budget is monthly, quarterly, or rolling, plus current burn rate and uncertainty. The balance should answer “how long can we sustain this?” rather than show only a percentage.
Value and release risk
Assess revenue, compliance, and security value. Can the feature be canaried, rolled back, or limited to selected tenants? Without verifiable value and a recovery path, customer pressure is not risk evidence.
30-second answer
“I first validate user-side SLIs, the SLO window, and burn, checking whether the signal is sustained or an instrumentation error. If the budget is truly spent, I pause non-essential releases and direct capacity and engineering effort to restoring the SLO, fixing causes, and validating monitoring. Security, compliance, or incident-mitigation changes can be exceptions when they are small, reversible, and owned. For the major-customer feature, I would seek tenant isolation or a delayed canary, then use a risk review to decide whether to proceed. Once the agreed budget margin returns, I would reopen releases gradually and review whether the SLO represents user value.”
Step-by-step solution
Step 1: Define SLIs as user outcomes
Define success rate, end-to-end latency, and correctness for critical workflows, using client or edge data where possible. State what counts as a valid request, maintenance windows, and tenant segmentation. If users care about export completion, create a job-completion SLI instead of relying on an API 200.
Step 2: Compute budget and burn
A 99.99% availability target allows a 0.01% failure ratio in the chosen window; minutes depend on the window and counting method. Use one query to calculate remaining budget, burn rate, and uncertainty, and label data delay. Before acting on an anomaly, check duplicate counts, missing telemetry, and dependency attribution.
Step 3: Establish release gates
When the budget is healthy, normal releases still require rollback and monitoring. As it nears exhaustion, raise review level and reduce batch and canary exposure. Once spent, freeze non-essential changes; allow only incident fixes, data-integrity work, severe security, or compliance changes with rollback evidence. Encode the gate in tooling and a responsibility matrix rather than relying on verbal agreement.
Step 4: Evaluate an exception
For every exception, record user benefit, risk, tenants exposed, percentage, rollback condition, and observation window. Isolation to one tenant, shadow traffic, or internal users is useful evidence. Product, the change owner, and SRE approve together; a sales promise alone cannot decide.
Step 5: Restore reliability before chasing perfection
During the freeze, fix root causes, capacity limits, and monitoring gaps, with a recovery SLO, budget margin, and deadline. If the target forces long-term heroics, adjust it during a review, but do not change the window or exclude failures after the fact to make a release possible.
Step 6: Communicate with customers and teams
Explain the affected workflow, time range, mitigations, and next update. Offer a verifiable alternative or schedule for the customer feature; do not promise an unverified recovery time. Use one budget dashboard, incident timeline, and owner list internally to reduce political negotiation between product and engineering.
Step 7: Govern recovery
When the budget reaches the agreed threshold, reopen releases in stages, starting with small batches. Review failed changes, detection delay, rollback duration, and customer impact. If the SLO does not represent user value, propose a data-backed change. Retain freeze, exception, approval, and outcome records for quarterly risk review.
High-quality sample answer
I would not make an absolute decision from “sales is urgent” or “the budget is zero.” I would validate user-side SLIs, the statistical window, data integrity, and burn rate, excluding instrumentation and duplicate-count errors. If the budget is truly spent, I would freeze non-essential releases and invest in recovery, fixes, and monitoring. Security, compliance, data repair, and incident mitigation may request an exception.
If the major-customer feature must move, I would isolate tenants, use a small canary, add automatic rollback, and define an observation window. Product, the change owner, and SRE would approve with benefits, risks, and stop conditions recorded. After the budget recovers, I would lift the freeze gradually and review whether the SLO measures real user outcomes before changing targets or process.
Common mistakes
- Mistake: Freeze every change after budget exhaustion. → Why it fails: Security, data repair, and incident mitigation may be delayed. → Fix: Define bounded exceptions, approvers, and rollback evidence.
- Mistake: Use only average server latency. → Why it fails: It misses client failures, tail latency, and key-tenant experience. → Fix: Define end-to-end SLIs from user journeys.
- Mistake: Change the SLO window or exclude failures to ship. → Why it fails: The budget loses comparability and hides risk. → Fix: Keep the current window and use formal governance for later changes.
- Mistake: Treat one successful canary as proof of full release. → Why it fails: Exposure, dependencies, and tail risk differ. → Fix: Expand gradually with automatic rollback.
Follow-up questions and responses
Follow-up 1: How much budget does 99.99% represent?
The allowed failure ratio in the chosen window is 0.01%. Converting it to minutes requires the window, counting method, and whether the metric is request-based or time-based. The key is to state the window rather than present an unconditional number.
Follow-up 2: Is a sales-promised feature an exception?
Commercial value alone does not grant an exception. Show user, revenue, or compliance impact; provide isolation, canary, rollback, and observation evidence; and obtain joint approval. If exposure cannot be reduced, delay the feature and offer an alternative.
Follow-up 3: What if the budget calculation may be wrong?
Freeze high-risk releases while checking telemetry coverage, duplicate counts, time delay, dependency attribution, and client samples in parallel. Recalculate after correction, but do not expand exposure while the result is uncertain.
Follow-up 4: When should the SLO change?
Change it when stable operation and review data show that the target does not match user value, cost, or capability. Record old target, new target, impact, and approver first. An outage or release pressure is not a reason for an ad-hoc lower target.
Follow-up 5: How do you know the freeze is over?
Define recovery conditions in advance: SLI meets target for an observation window, budget has rebuilt to a threshold, root-cause fixes are verified, and rollback works. Then start with a small batch and continue monitoring instead of instantly returning to full speed.