Prompt and context
You discover that a frequently used incident runbook contains steps for a removed service and could make an outage worse. Explain how you protect the on-call team without interrupting current coverage, validate a replacement, retire or rewrite the runbook, and prove that the improvement was adopted.
What the interviewer is testing
- Reducing unsafe action risk before debating who owns the documentation.
- Turning “the document is stale” into a reproducible, verifiable migration plan.
- Using STAR to explain impact, collaboration, trade-offs, and follow-up metrics.
Questions to clarify first
- Which steps are invalid, and could they delete data, widen traffic, or block recovery?
- Where do on-call engineers actually open the runbook, including cached links and bots?
- Who owns the replacement, what sandbox exists, and how would an emergency rollback work?
A 30-second answer
I would mark the risk at the top of the runbook, notify on-call and the incident owner, and provide a verified temporary path so nobody repeats the invalid step. I would test the replacement in a sandbox, update links, permissions, and rollback guidance with the service owner, then publish and deprecate the old version. Adoption would be measured through link access, drill success, wrong actions, and completed follow-ups, not just a merged documentation change.
Step-by-step deep dive
1. Isolate dangerous steps
Identify commands or decisions with irreversible impact. Add a visible warning, disable automated links, or move the old version to an explicit archive while preserving a temporary path approved by the on-call owner.
2. Reconstruct the real access path
Inspect on-call indexes, search results, bot messages, permissions, and cached bookmarks to find the version engineers actually use. Editing only the source repository leaves copied links circulating.
3. Validate the replacement
Run the new steps in a sandbox or low-risk window and record prerequisites, signals, stop conditions, and rollback. GitLab describes runbooks as initial identification and routine handling, with playbooks or escalation paths for more complex cases; the document boundary should match real responsibility.
4. Rewrite with the owning teams
Ask the service owner, an on-call representative, and security or compliance reviewers to check the change. Separate facts, assumptions, and open checks instead of rewriting critical commands from memory. Each step states scope and the escalation entry point when it fails.
5. Plan publication and retirement
Publish the new version as a candidate and give the old one a deprecation date and replacement link. If the old path can cause serious harm, remove execution permission or make it read-only before following the change process. Keep rollback aligned with deployment permissions and the latest usable version.
6. Test adoption with a drill
Have someone who did not write the runbook complete a drill. Observe whether they find the entry point, recognize prerequisites, and escalate at the stop condition. Record time, wrong steps, and questions instead of substituting author self-testing for real use.
7. Add maintenance triggers
Set an owner, review date, and triggers such as topology, alert-name, or permission changes. Stale documentation may indicate that change review lacks a documentation check; add it to release checklists and incident-review actions.
Model answer
I would confirm the risk of the invalid step, add a warning, notify on-call and the incident owner, and provide a confirmed temporary path. I would inspect the links people actually use, test the replacement in a sandbox with prerequisites, stop conditions, and rollback, and review it with the service owner and on-call representative. Then I would publish the new version, deprecate the old one, and run a drill with someone who did not author it. The acceptance metrics would include finding the entry, wrong actions, escalation timing, and migration completion, with maintenance triggers in the change process.
Common mistakes
- Deleting the old runbook without a safe replacement path.
- Editing the repository while ignoring cached links, bots, and bookmarks.
- Declaring success after the author alone completes a drill.
- Putting every failure procedure in a runbook without playbook or escalation boundaries.
- Counting documentation merges instead of wrong actions and drill outcomes.
Follow-up questions and responses
What if an incident is already active?
Have the incident lead confirm the temporary path and risk, stop propagation of the invalid step, and add the documentation fix to the incident action list. Do not perform an untested large rewrite during the firefight.
What if the service owner denies that the runbook is stale?
Bring the exact step, recent change, and reproducible failure evidence, then propose a small drill. Focus the discussion on user risk and maintenance ownership, not the person.
What if many external teams use the old runbook?
Keep a stable redirect or compatibility note, notify consumers, and set a migration deadline. Restrict high-risk command permissions first, then confirm migration team by team.
When should you delete the old version completely?
Only after the replacement has passed a drill, has an owner, all critical entry points have moved, and rollback is available. Follow stricter retention rules when security or compliance requires them.
What if the replacement fails in the drill?
Stop publication, record failed prerequisites and signals, fix the procedure, and rerun the drill. Failure is evidence that the release gate is not met, not a reason to explain it away.
How would you tell this story with STAR?
Describe the concrete risk and task, then the notifications, validation, collaboration, and publication steps. End with reduced wrong actions, drill pass rate, or migration completion plus the maintenance mechanism you added.