Prompt
Design a DNSSEC key-rollover control plane for a platform hosting tens of thousands of signed zones. It coordinates registrar DS records, authoritative DNS DNSKEY/RRSIG records, and tenant approvals without exposing validating resolvers to a broken chain.
Scenario and constraints
KSK and ZSK lifetimes differ, registrar APIs are delayed and retryable, and authoritative nodes span regions. The control plane needs idempotent retries, per-tenant pause, auditable state transitions, and a fail-closed behavior when one step is uncertain.
What this tests
The core is state machines, time invariants, external side-effect orchestration, and observability. Strong answers distinguish DNSKEY prepublication, DS publication, old-key retirement, and signature overlap; generating a key alone is not a rollover.
Reference approach
Persist a per-zone state machine: observe, prepublish, ds-submit, ds-visible, sign-with-both, retire-old, and verify. Record the desired version, operation token, TTL budget, and evidence snapshot at every transition. Publish the new DNSKEY to all authoritative nodes and wait its TTL, then submit DS through the registrar. After the parent DS is visible through several recursive resolvers, sign with both keys for a safety window, and only then remove the old DS and DNSKEY.
Run one zone at a time under a lease. External calls use idempotency keys and exponential backoff. Freeze a zone on SERVFAIL, DS/DNSKEY mismatch, or expired RRSIG; preserve old material and request human approval.
Critical details
The database stores desired state, while DNS queries and registrar readbacks are sources of truth; conflicts must not be overwritten blindly. Timing includes authoritative and recursive TTLs, signature validity, and propagation margin. Keep private keys in controlled KMS storage, and log fingerprints, key tags, versions, and actors rather than private material.
Common traps
Mutating one registrar object concurrently across tenants; checking only authoritative nodes and not the parent DS; deleting the old key immediately after a failed step; treating queue retries as idempotency; and omitting pause or human takeover paths.
Evaluation rubric
Strong answers define boundaries between the control plane, authoritative DNS, registrar, validating resolvers, and audit storage; state at least three safety invariants; and specify entry, exit, and rollback conditions for each state. They also name success rate, SERVFAIL, propagation latency, and stuck-lease metrics.
Follow-up questions
Why is DS publication normally after DNSKEY prepublication?
Prepublication lets authoritative nodes and caches acquire the new DNSKEY first. The parent DS can then point to a key that validators can retrieve, reducing the DS-visible/DNSKEY-missing window.
How do you retry when a registrar call times out but may have succeeded?
Read the registrar's current DS using the zone version and idempotency token, then decide whether to retry. Do not submit a second unknown mutation based only on one timeout.
When may the system automatically remove the old DS?
Only after the new DS remains visible in the target recursive-resolver set, RRSIG validation passes, and TTL and signature windows meet policy. Otherwise the zone stays frozen.