Prompt and context
Design an incident command platform for multiple teams. It must receive alerts, create incidents, assign response roles, coordinate internal and external communication, and retain an auditable timeline after recovery. Explain consistency, permissions, notification storms, and recovery objectives.
This fits system design, SRE, platform, and backend roles. It tests how detection, command, collaboration, and review become a recoverable system under pressure. The platform is not merely chat: it is a control plane with accountable owners, state transitions, evidence, and degradation paths. Clarify normal and peak alert rates, concurrent incidents, participants, channels, and retention.
What interviewers assess
- Separating incident management from monitoring, chat, and ticket systems.
- Defining state, idempotency, a source of truth, and concurrent-edit rules.
- Separating incident manager, tech lead, communications lead, and scribe duties.
- Handling deduplication, correlation, notification storms, and authorization.
- Providing degradation, regional recovery, audit integrity, and retention.
- Validating the design with MTTA, MTTR, alert noise, and communication delay.
A 30-second answer
“I would split the platform into alert intake, incident orchestration, real-time collaboration, notification, and review storage. The intake layer deduplicates by source and time window; orchestration treats the incident as the source of truth and versions state and role changes for audit. The incident manager commands, the tech lead diagnoses, the communications lead updates stakeholders, and the scribe maintains the timeline. Events are persisted before replayable streams push updates. Notifications are prioritized and rate-limited. During a regional failure, the minimum control plane and manual fallback remain available; recovery generates a review from the timeline.”
Step-by-step solution
Step 1: Define boundaries and objectives
Signals may come from metrics, logs, synthetic checks, support escalation, or a human. The platform organizes signals into incidents and response workflow; it does not replace monitoring computation, chat storage, or deployment. Set objectives for incident-creation latency, critical-notification delivery, timeline durability, regional recovery time, and audit retention.
Separate normal load from a large outage: tens of thousands of alerts per minute, hundreds of concurrent incidents, hundreds of participants per incident, and simultaneous SMS, email, push, and webhook delivery. Peaks require backpressure and priority so low-value notifications cannot block command actions.
Step 2: Ingest, deduplicate, and correlate alerts
Preserve source, rule version, timestamp, fingerprint, and raw payload for each signal. Client retries and network replay require idempotency keys; a unique constraint or lease prevents duplicate incidents for one fingerprint. A deduplication window must not hide a new outage forever, so configure it by service, environment, and time.
Correlation may start with service topology, deployment version, region, and shared labels, then allow the incident manager to split or merge. Automatic correlation must record its rule and evidence; it cannot silently change incident scope. Raw alerts are immutable, while aggregation is a derived state.
Step 3: Model state and roles
An incident can use states such as detected, triaged, mitigating, monitoring, resolved, and closed. Define legal transitions, actors, and required evidence. State is not a Boolean; record impact, current hypothesis, next action, and update time together.
Separate response roles from identity. The incident manager owns priority and decisions, the tech lead owns investigation, the communications lead owns internal and external updates, and the scribe maintains the timeline. Role changes append audit events. Leases and versions prevent two responders from silently overwriting a command.
Step 4: Build collaboration around one source of truth
Persist state, roles, action items, and communication summaries to an event log or transactional store before publishing through a bus to WebSocket, SSE, email, and mobile channels. A client may render optimistically, but a version conflict requires a fresh server read. Browser memory is never the final truth.
Store actor, server time, incident version, action type, summary, and related alerts in the timeline. Long chat can live elsewhere, but decisions and recovery evidence belong in a structured timeline. Read models can render the current incident, while replay still reconstructs it.
Step 5: Control notification storms and access
Rate-limit by incident priority, on-call schedule, channel reliability, and acknowledgement. Summarize low-priority alerts; use backoff, escalation, and acknowledgement timeouts for high-priority notifications. One incident should not send hundreds of duplicate messages to one responder. Isolate notification queues from the control plane so a provider outage cannot block state changes.
Authorize observers, responders, incident managers, communications leads, and administrators by tenant, service, and environment. An external status page is a redacted read projection of impact and progress; it must not expose hypotheses, customer data, or credentials. Record reads and exports, and require a reason plus later review for emergency break-glass access.
Step 6: Degrade, recover, and review
If the platform fails, retain a phone bridge, static runbook, or backup incident log. Preserve incident creation, role claiming, and timeline writes first; analytics, search, and historical reports can be unavailable. Regional deployment may use a primary writer with asynchronous replication or sharded writes, but document conflict handling and lease reacquisition during failover.
After recovery, draft a review from detection, confirmation, mitigation, recovery, and closure timestamps, decisions, alert quality, communication, and follow-up work. Do not overwrite the original timeline; revisions append versions. Track MTTA, MTTM, MTTR, duplicate-alert ratio, role-claim latency, notification delivery, and action-item completion.
Information gain and boundaries
The value of an incident command platform comes from structuring ownership, state, and evidence, not from moving every conversation into one page. It cannot guarantee a correct root cause or replace monitoring quality and responder training. Notifications, identity, regional networks, and the platform itself can fail together, so the design needs a minimum control plane and manual fallback.
Model answer
“I would define the boundary first: the platform organizes alerts and response, while monitoring, deployment, and ordinary chat remain separate. The objectives are creation latency, critical notification delivery, timeline durability, regional recovery, and audit retention. Intake preserves raw alerts and uses source fingerprints plus idempotency keys for deduplication. Correlation by service, region, version, and topology is explainable and reversible.
The orchestration layer is a versioned state machine whose state and role changes append to a durable log. The incident manager owns decisions, the tech lead investigates, the communications lead updates stakeholders, and the scribe maintains evidence. Clients receive replayable updates and reread on version conflict; browser cache is not authoritative. Control actions and notifications use separate queues, with priority, on-call, and acknowledgement limits.
Access is isolated by tenant, service, and environment, and the external projection is redacted. During a regional failure, creation, role claiming, and timeline writes stay available while search and reports degrade to read-only or offline. Recovery produces a review from an immutable timeline and computes MTTA, MTTR, alert noise, and action completion. If the platform is unavailable, a phone bridge and static runbook keep command possible.”
Common mistakes
- Treating the platform as chat → key state becomes un-auditable → use a structured state machine and timeline.
- Broadcasting every alert → notification storms block command → deduplicate, prioritize, acknowledge, and rate-limit.
- Keeping roles only in the UI → concurrent responders overwrite each other → use server leases, versions, and audit events.
- Hiding automatic correlation → bad scope cannot be explained → preserve raw alerts and correlation evidence.
- Designing only the healthy region → a platform outage removes command → provide a minimum control plane and manual fallback.
- Overwriting review evidence → decisions and timestamps cannot be verified → keep the raw log immutable and append revisions.
Follow-up questions
What if two incident managers claim control at once?
Use an expiring takeover lease and monotonic version. The server accepts control writes only from the current lease holder. The losing responder rereads and displays the owner; emergency break-glass requires authorization, reason, and audit.
Can a notification-provider outage affect incident state?
It must not make notification delivery a prerequisite for state writes. Persist the incident and pending notification first, then let an isolated worker retry, switch channels, or trigger a phone process while the control plane remains usable.
How do you stop correlation from hiding a new incident?
Correlation creates a candidate relationship and preserves the raw signal. Use multiple evidence dimensions such as time, service, region, and topology. High-impact or low-confidence signals require human confirmation, and rule versions plus splits enter the timeline.
What belongs on an external status page?
Only a privacy-filtered projection of impact, current phase, next update time, and recovery progress. Keep internal hypotheses, customer identifiers, credentials, unconfirmed root causes, and detailed logs in controlled internal views.