Question and suitable scenarios
A feature will be enabled in stages and may affect revenue, privacy, reliability, or user habits. Design the pre-launch decision table: what permits expansion, what pauses the rollout, what triggers rollback, and how to distinguish product failure from an observability failure.
GitHub’s pilot guidance asks teams to define success criteria up front and choose deliberately between expanding, holding, and rolling back. Microsoft’s Known Issue Rollback shows a targeted reversal that preserves other changes in the same update. The interview signal is judgment and accountability, not a single conversion number.
What the interviewer is testing
- Separating success, guardrail, diagnostic, and exit criteria.
- Connecting rollback actions to data, code, configuration, and communication.
- Choosing thresholds, windows, and sample sizes from risk.
- Handling delayed metrics, segment harm, and irreversible state changes.
- Pausing when evidence is incomplete instead of expanding by intuition.
Clarifications before answering
- Does the feature change persistent data, billing, or permissions? That determines reversibility.
- How are pilot users selected, and is there an unexposed comparison group? Segmentation affects inference.
- What are metric latency and the minimum detectable change? The window must outlast data arrival.
- Is rollback a flag, configuration restore, reverse migration, or manual compensation? The action changes the threshold.
A 30-second answer framework
I split the rollout into pilot, expansion, and general availability. Each stage has a value metric, unacceptable guardrails, a minimum observation window, and an owner. If data is incomplete, we hold the stage. Rollback criteria name severity, duration, affected segments, and a tested recovery action; “metric down” is not enough. We rehearse flag closure, data compatibility, and user messaging. We expand only after success criteria pass; otherwise we hold, fix, or roll back and update the next launch gate.
Step-by-step deep answer
1. Define decisions instead of one goal
The success metric asks whether users get value, such as task completion. Guardrails ask whether harm is unacceptable, such as errors, refunds, latency, or privacy complaints. Diagnostic metrics locate causes but should not independently authorize expansion.
2. Set stages and observation windows
The pilot must detect severe issues without spreading them to everyone. Each stage gets a minimum window covering daily cycles, asynchronous jobs, and delayed events. When data is incomplete, the state is “wait for evidence,” not default success.
3. Derive risk thresholds
Write each threshold as metric, baseline, deviation, duration, and segment. For example, pause when errors for a high-value segment exceed baseline for two windows; continue collecting for a small conversion change. Thresholds come from risk tolerance and recoverability, not from a desired launch date.
4. Make rollback executable
Prefer reversible flags or configuration. If the feature writes new data, verify that the old path can ignore or read it; otherwise plan migration, compensation, or a write freeze. Every action has an owner, maximum completion time, and verification signal.
5. Handle causality and segments
Compare pilot and control groups and inspect device, region, plan, and cohort interactions. A healthy aggregate with severe segment harm still pauses. Preserve versioned events for replay; do not call correlation causation without evidence.
6. Establish communication and authority
Before launch, name who can pause, who approves expansion, and who communicates with customers. High-risk features need status updates, support language, and an incident record. Amazon’s leadership principles emphasize ownership and evidence-backed challenge, which translates into an explicit escalation path.
7. Learn and update the gate
After rollback, record the trigger, detection delay, action time, affected users, and compensation. If a guardrail was late, improve detection or the window. If recovery could not restore state, raise compatibility requirements for the next pilot.
High-quality sample answer
I model rollout as pilot, expand, hold, and rollback states. Each state has a value metric, unacceptable guardrails, a minimum observation window, and an owner. Expansion requires complete data and passed success criteria; a severe guardrail pauses the rollout and executes a rehearsed action. For persistent writes, I verify old-path compatibility before launch instead of assuming a flag is enough. I inspect segments separately so averages do not hide harm. After rollback I verify errors, data integrity, and customer communication, then update the next gate with whatever the detection or recovery process missed.
Common mistakes
- Watch conversion only → privacy, errors, or valuable-segment harm stays hidden → define guardrails first.
- Say “meaningful drop” → no automation can act → specify baseline, deviation, duration, and segment.
- Treat flag closure as universal rollback → new data may be unreadable → rehearse compatibility.
- Expand with incomplete data → delayed events arrive later → use a wait state and minimum window.
- No pause authority → risk is discovered without an actor → name on-call and escalation owners.
Follow-up questions and responses
Success rises but refunds and complaints rise too. What now?
Make refunds and complaints higher-priority guardrails, pause expansion, and isolate the harmed segment. If it cannot be isolated quickly, roll back while preserving the success signal for diagnosis.
Rollback would lose data users already created. What do you do?
Freeze writes, preserve migration and compensation paths, and use degraded reads or manual handling if safety cannot be proven. Do not make an irreversible switch casually.
The pilot is too small. How do you avoid premature rollback?
Use event-level safety triggers for severe risks, longer windows and larger samples for low-severity metrics, and different evidence gates rather than one threshold for everything.
Who can pause the rollout?
Delegate pause authority by risk: on-call engineers can stop severe harm, product and engineering owners approve expansion, and the responsible owner reviews the decision afterward.
Metrics recover after rollback. Do you relaunch immediately?
No. Confirm root cause, data repair, and monitoring delay. Redefine scope and criteria, then rerun a smaller pilot with the new evidence gate.