Prompt and context
During a production incident, the on-call engineer wants a high-severity response while the product owner sees limited impact. You do not have complete data, but delayed escalation could expand the blast radius. Explain how you move forward, communicate user impact, and improve the severity process afterward.
What the interviewer is testing
- Protecting users under uncertainty before discussing blame or ownership.
- Replacing rank or volume with observable signals and escalation thresholds.
- Turning disagreement into records, actions, and process improvement.
Questions to clarify first
- What impact, users, and critical paths are confirmed right now?
- What do the current severity, on-call authority, and escalation-time rules say?
- Are there delayed metrics, observability gaps, or business signals to verify?
A 30-second answer
I would put known facts, unknowns, and worst-case risk on one timeline, then propose a short time-boxed protective action and escalation threshold. If user or rollback risk is high, I would start the higher-severity response and record it as a reversible protection decision. I would assign a coordinator, set an update cadence, and keep a decision log. After recovery, a blameless review would examine signals, severity rules, and action completion rather than assigning the disagreement to a person.
Step-by-step deep dive
1. Establish shared facts
List alerts, deploys, error rates, user reports, and actions with timestamps. Mark observations separately from hypotheses. Do not treat “it feels severe” as evidence, but do not wait for perfect data before protecting users.
2. Make a temporary risk decision
Write escalation conditions as observable signals: sustained errors on a critical path, an expanding tenant set, uncertain data integrity, or a shrinking rollback window. During disagreement, start at the higher level if needed, set a ten-minute or shorter review point, and make downgrade explicit.
3. Clarify roles and cadence
Assign an incident lead, technical owner, scribe, and communications owner. Route other updates to one channel or ticket and publish internal status on a fixed cadence. External updates should state confirmed impact, mitigation, and the next update time, not guesses about root cause.
4. Offer comparable options to product
Explain escalation cost, the risk of waiting, and the trigger for changing course instead of debating who understands incidents better. Options might include pausing risky releases, read-only mode, or rollback, each with an owner and deadline.
5. Record dissent while acting
The decision log captures evidence, dissent, the chosen action, and the review time. This lets the retrospective test decision quality instead of using the outcome to declare who was right. New risk should have a direct path to re-evaluation.
6. Run a blameless review
GitLab’s incident-review guidance focuses on understanding the system and decision why/how and on preventive actions. Include the timeline, contributing factors, detection gaps, user communication, and action items. “Someone should have been more careful” is not a process improvement.
7. Make improvements verifiable
Give the severity matrix, monitoring, drills, or runbook an owner, due date, and metric. A later drill should reproduce the disagreement and verify that thresholds, permissions, and communication templates reduce delay instead of merely adding pages to a document.
Model answer
I would put confirmed impact, unknowns, and worst-case risk on a timeline and propose a short time-boxed protective action. If a critical path or data integrity is at risk, I would start the higher-severity response, label it reversible, and review it in ten minutes using error rate, user scope, and rollback progress. I would assign coordination, technical, scribe, and communications roles and publish factual updates. After recovery, a blameless review would inspect alerts, severity rules, and communication, then assign owners and verification metrics to each improvement.
Common mistakes
- Waiting for complete data and missing the protection window.
- Using seniority to override a colleague without a reviewable signal.
- Turning the incident channel into root-cause guesses and blame.
- Writing “improve monitoring” without an owner, date, or acceptance metric.
- Treating the eventual outcome as the only evidence of decision quality.
Follow-up questions and responses
What if escalation triggers an expensive cross-team on-call?
Compare that cost with the downside of waiting, and use a short time box with explicit downgrade criteria. A protective escalation is reversible when its review and exit path are visible.
What if the product owner insists on low severity?
Ask them to confirm the impact assumption, then record the risk signal, mitigation, and threshold. Follow the authorized escalation path for safety, data integrity, or compliance risks and notify the owner.
What if monitoring signals conflict?
Mark the data-quality problem, use the conservative user-impact assumption, take a low-risk protective action, and assign people to verify logs, sampled users, and dependencies. Do not hide conflicting evidence.
How do you stop blameless culture from meaning no accountability?
Blamelessness targets learning and system factors, not the absence of ownership. Every action has an owner, due date, and verification; repeated disregard of known risks follows normal governance.
When should a review be public?
Use user impact, contracts, and policy to decide. A public summary should cover impact, timeline, fix, and prevention while removing unnecessary personal data and unverified speculation.
How do you prove the approach worked?
Track escalation delay, false-escalation rate, on-time user updates, action completion, and drill outcomes over multiple incidents or drills. One successful example is not enough evidence.