Prompt and scope
This system-design question targets platform, cloud-infrastructure, and embedded-systems roles. Devices may be offline for weeks, have limited storage, or pay for cellular bandwidth; a bad image can disconnect an entire fleet. The platform must ensure that only compatible devices install a signed package, that rollout can pause, and that a device can recover or return to its previous version.
Assume one million devices, at most 100,000 downloads per day, and a 20 MiB update. Group devices by model, hardware revision, region, and current version. The answer does not require AWS; cloud products only help make control-plane and data-plane boundaries concrete.
What the interviewer evaluates
- Whether you separate package integrity, publisher authentication, device authorization, and compatibility checks.
- Whether you design a control-plane and device data-plane state machine instead of drawing only a download bucket.
- Whether you calculate bandwidth, concurrency, and abort thresholds and explain pause, retry, rollback, and human escalation.
A strong answer defines “successful rollout” as device verification, installation, reboot, and health confirmation, not CDN download completion.
Clarifications before answering
- Can devices boot from two slots? A/B slots allow writing the inactive slot and falling back after a failed boot; single-slot devices need a more conservative bootloader and field-recovery path.
- Are there safety or regional constraints? Targeting must include model, hardware, region, certificate state, and current version, not only a device label.
- What is the update deadline? The deadline changes batch size, maintenance windows, offline retries, and whether forced installation is acceptable.
- Is failure device-level or cohort-level? A device can retry; a higher failure rate for one model should pause that cohort rather than expand globally.
A 30-second answer
“I would split the platform into a signed-package repository, release control plane, device agent, and telemetry. The release pipeline creates a manifest with model, hardware, version, dependencies, and expiry, signs it in a controlled environment, and makes the device verify the signature, digest, and compatibility before installation. The control plane targets cohorts with canary, fixed-rate, or exponential rollout and stores per-device state with an idempotent job ID. The device downloads chunks resumably into the inactive slot, reboots, and commits only after health checks pass. A rise in failures, download errors, boot rollback, or security alerts pauses the cohort and preserves the old version. Offline devices retry in maintenance windows, and every action is auditable and replayable.”
Step-by-step deep answer
Step 1: Estimate the main bottleneck.
If one million devices download 20 MiB, the total is about 20 TiB. Completing 100,000 devices per day is about 2 TiB/day, or 23.7 MiB/s on average; peaks need headroom for concurrency and retries. Object storage and a CDN distribute packages. The control plane sends manifests and short-lived download authorization instead of making devices poll a large database.
Step 2: Create an authentic package and manifest.
The build pipeline produces immutable bytes, a digest, and a manifest. Signing keys stay in a controlled signing service, and the release records signer version and approval. The device has a trust root and verifies the signature, package digest, target model, minimum bootloader, anti-rollback counter, and expiry. A download URL is a transport mechanism, not the trust boundary.
Step 3: Separate control and data planes.
The control plane creates releases, resolves targets, creates cohorts and device jobs, and issues pause commands. The device data plane downloads chunks from the CDN and reports state to a job endpoint. Each device updates (device_id, job_id) idempotently, so duplicate reports cannot move state incorrectly. States include QUEUED, DOWNLOADING, VERIFIED, INSTALLING, SUCCEEDED, FAILED, ROLLED_BACK, and REJECTED.
Step 4: Design safe installation and rollback.
Write the package to the inactive slot, verify each chunk and the final manifest, then switch the boot slot. The bootloader records attempts and a boot-confirmation deadline. The application confirms only after health checks, critical sensors, and communication recover. Repeated failure selects the old slot and reports the reason. A single-slot device needs a recovery image or field service; it does not get A/B atomicity for free.
Step 5: Control rollout.
Start with internal devices and a small canary, then expand by model and region. Fixed or exponential rates are both valid, but every cohort needs maximum concurrency, a maintenance window, and abort criteria. Measure thresholds per cohort and separate download failure, signature rejection, installation failure, boot rollback, and health-check timeout.
Step 6: Handle offline devices, retries, and audit.
When a device reconnects, it claims an unexpired job. Chunked downloads use resume and exponential backoff; retries reuse the same job and package version. A device past its deadline enters a remediation queue rather than being marked successful. The control plane retains approvals, manifest, target snapshot, state transitions, actor, and pause reason for replay and compliance.
High-quality sample answer
“I would separate the control plane, package repository/CDN, device agent, and telemetry. The release pipeline creates immutable bytes and a manifest containing target model, hardware revision, minimum bootloader, version counter, and digest; a controlled signing service signs it, and the device verifies it with an embedded trust root. The download URL only transports bytes.
With one million devices, a 20 MiB package is about 20 TiB, so the CDN handles distribution while the control plane stores release and per-device job state. Devices download chunks into an inactive slot, verify the digest, reboot, and commit only after health confirmation; failures select the old slot and report a reason. I start with a canary, then expand by model and region. Each cohort has concurrency, failure-rate, and rollback-rate thresholds; exceeding one pauses the rollout instead of spreading a bad image. Offline devices claim the same job in a maintenance window, retries resume, and approvals, states, and versions remain auditable.”
Common mistakes
- Only checking HTTPS → transport security does not authenticate the package or its bytes → verify the manifest signature and package digest on the device.
- Marking success after download → install, boot, and health checks can still fail → make boot confirmation the terminal success condition.
- Launching to the whole fleet → one compatibility defect can affect everyone → use canaries, cohorts, rates, and abort criteria.
- Creating a new job on every retry → state and audit history fragment → reuse an idempotent
(device_id, job_id)state. - Allowing arbitrary downgrade → an attacker can replay a vulnerable image → use signed version counters, anti-rollback policy, and a controlled exception list.
Follow-ups and responses
Follow-up 1: At 2% rollout, one model reaches a 4% boot-rollback rate. What do you do?
Pause that model and package version immediately while continuing observation of successful devices. Slice by hardware revision, bootloader, region, and build to find the compatibility boundary; if needed, deploy a verified old version or recovery command. Do not continue because the fleet-wide average is low.
Follow-up 2: A device loses power at 80% download. How does the next attempt avoid starting over?
Write chunks into the inactive slot and persist package version, chunk digests, confirmed offset, and the overall manifest digest. After reboot, verify local chunks and the manifest, then request a range from the last contiguous trusted chunk. If the package or signature changes, discard the temporary slot instead of combining two versions.
Follow-up 3: How do you stop an attacker replaying a valid old package?
Put a monotonic version counter or security version in the manifest and persist the highest accepted value on the device; the bootloader rejects lower values. Emergency rollback requires a controlled signature, explicit cohort, and time-limited authorization recorded in audit, so an ordinary download endpoint cannot bypass anti-rollback.