System design interview: how would you design a multi-tenant secrets rotation service?

Question

Design a multi-tenant secrets rotation service. A tenant can configure a database password, third-party API key, or certificate and its rotation period. The service generates a new value, updates the external system, stores a version, gradually notifies workloads, and retires the old value. Cover recovery, concurrency, auditability, authorization, and rollback.

What the interviewer is testing

• Whether you model rotation as a retryable state machine rather than an unrecoverable cron script.
• Whether you handle duplicate messages, concurrent jobs for one secret, and tenant isolation.
• Whether you can stage two versions and roll out gradually instead of causing an immediate fleet-wide failure.
• Whether you can explain retirement, compromise response, audit evidence, and least privilege.

Model answer

Core objects are Secret, immutable Version, RotationPolicy, Job, and Lease. A scheduler emits a job for the next rotation time; a queue is partitioned by secret_id, and a worker acquires a lease with an expiry. A database uniqueness constraint on (secret_id, idempotency_key) makes retries safe.

Use a state machine such as scheduled → generating → external_updated → staged → rolling_out → verified → retired. Persist the external request identifier, version, and next retry time at every step. Create the credential in the external provider first, then store a pending version. Workloads switch gradually through an immutable version reference or refresh mechanism. Promote the version to current only after health checks, error rates, and authorization tests pass.

Keep control metadata separate from encrypted secret payloads. Applications receive short-lived read permission. Every state transition goes to an append-only audit log without secret plaintext. Rollback selects an old version that is still valid; it must not automatically revoke that version before recovery is complete.

Architecture sketch

Scheduler -> Durable Queue -> Rotation Workers
     |              |              |
 Policy DB     Lease/Idempotency  External Provider
     |                             |
 Version Store + KMS        Rollout Controller -> Workloads
     |
 Audit Log / Metrics / Alerts

Workers renew leases; after expiry another worker can take over. Queue messages carry only secret and job identifiers. A worker reads values from a restricted version store, keeping plaintext out of messages, logs, and metric labels.

Critical flow

1. The scheduler creates an idempotent job and applies tenant quotas.
2. A worker acquires the lease and reads the current version and policy; a completed job returns safely.
3. It generates a value and calls the provider with a provider-side idempotency key.
4. It writes a pending version and runs compatibility checks and a small rollout.
5. It observes error rate, authentication success, and health probes; only then promotes current.
6. After all consumers acknowledge, it disables and destroys the old version; failures retry or roll back.

Persist state and external responses at every step. On restart, continue from the last state instead of guessing whether the provider was already changed.

Common pitfalls

• Storing only the next time in cron, leaving no recovery point after restart or duplicate delivery.
• Revoking the old value immediately, ignoring connection pools, caches, and long-lived connections.
• Putting plaintext into queues, logs, tracing spans, or error messages.
• Using one global lock for all tenants, or omitting tenant quotas so one tenant exhausts workers.
• Treating rollback as writing the old value again without checking that it is still valid and active.

Consistency and security trade-offs

Use a strongly consistent database for the state machine and uniqueness constraints. Notifications and rollout can be at-least-once, so consumers must be idempotent. Version reads may be briefly cached, but current-version changes and revocations need an explicit invalidation path. Tenant authorization limits access to its own secrets, jobs, and audit records; workers receive only the provider permissions required for the current step.

Rotation periods should consider key type, exposure risk, provider limits, and recovery windows rather than a fixed timer alone. NIST key-management guidance treats usage period, purpose, protection level, and revocation as one policy decision.

Inject failures when a worker crashes before and after every state transition, when messages duplicate, leases expire, providers time out, a partial rollout fails, and the database fails over. Assert that one job does not create duplicate external credentials, exactly one version becomes current, and the old value is revoked only after the confirmation window. Also test tenant isolation, audit redaction, and alert latency.

• AWS Secrets Manager's AWSPENDING/AWSCURRENT rotation flow: staging labels and completion steps.
• Google Cloud Secret Manager rotation guidance: retries, non-concurrent rotations, gradual rollout, and old-version cleanup.
• NIST SP 800-57 Part 1: key purpose, protection, usage period, and revocation principles.

Follow-up questions

How do you prevent two workers from rotating one secret at once?

Use a database lease or a distributed lock with a fencing token, and write the token into every state update. A recovered worker without the current token cannot overwrite newer state; leases need renewal and a clear expiry.

What if the provider has no idempotent API?

Persist a request fingerprint and external resource identifier in the job, then query the provider before retrying. If the provider cannot be queried, pause the step for human confirmation instead of blindly creating more credentials.

Why not let every application read latest?

latest can push an unverified value to the whole fleet immediately. Immutable versions, staged rollout, and health checks contain the blast radius and preserve a rollback point.

How do you protect the service when rotation jobs pile up?

Set concurrency quotas per tenant and provider, use priorities and exponential backoff, and expose oldest-job age, failure rate, and remaining recovery window. Near-expiry secrets can be prioritized without bypassing authorization or idempotency.

What does the service do after a compromise?

Pause ordinary schedules for affected jobs, create a high-priority emergency rotation, shorten the rollout observation window, and retain forensic logs. Confirm critical consumers switched before revoking the old value, and notify the tenant and security-response process.