System design interview: How do you migrate Kubernetes to KMS v2 and prove Secrets are encrypted?

Prompt and scope

The platform team must protect Secrets in etcd. The cluster has several kube-apiservers, an external KMS plugin, and many historical objects, yet releases cannot stop during migration. Design configuration, rotation, rewrites, verification, monitoring, and rollback.

What the interviewer is testing

• Understanding the KMS v2 envelope-encryption boundary and DEK/KEK relationship.
• Distinguishing encrypted new writes from rewritten historical objects.
• Handling multiple apiservers, KMS outages, rotation, and concurrent updates.
• Proving the result with etcd evidence and API reads rather than a config diff.

Clarifying questions

1. Which Kubernetes and plugin versions and high-availability topology are deployed?
2. Which resources are protected, including custom resources and audit logs?
3. What availability, latency, rotation, and disaster-recovery guarantees does the external KMS provide?
4. What control-plane window, rollback deadline, and compliance evidence format are required?

A 30-second answer

Validate the KMS v2 plugin and permissions in a non-production cluster. Make KMS the first provider, keep the old provider as a read fallback, and roll each kube-apiserver one at a time. New writes become encrypted, but historical objects require no-op updates or storage-version migration. Verify API reads, etcd prefixes, KMS metrics, failure drills, and audit records. Remove the old provider only after every object is rewritten and the rollback window has expired.

Step-by-step design

1. Define the encryption model

Kubernetes uses envelope encryption: a data-encryption key (DEK) protects a resource, while a key-encryption key (KEK) is protected by the external KMS. KMS v2 lowers request overhead with server-side caching and per-apiserver DEK design, but it does not rewrite every old etcd value automatically.

2. Validate plugin and permissions first

In an isolated cluster test the socket, identity, timeouts, restarts, and KMS-unavailable behavior. Confirm the kube-apiserver can decrypt objects written by the old provider and write a new Secret through the new provider. Record latency, errors, cache hits, and KMS call volume.

providers:
  - kms:
      apiVersion: v2
      name: external-kms
      endpoint: unix:///var/run/kms/plugin.sock
  - aescbc:
      keys:
        - name: old-key
          secret: <base64-secret>

3. Roll the high-availability control plane

Put KMS first, retain the old provider for reads, and restart kube-apiservers one at a time. After each change verify API reads and writes, concurrent requests, and audit output. Never restart the whole control plane together. Version the configuration and plugin for rollback.

4. Rewrite historical objects

Changing provider order affects future writes only. Batch no-op updates for Secrets and other protected resources, or use storage-version migration to trigger rewrites; retry conflicts. Shard by namespace, rate-limit work, and record object versions so the API server, etcd, and KMS are not overwhelmed.

5. Collect encryption evidence

Create a new Secret, read its raw bytes from etcd, and verify the KMS v2 encryption prefix; then read it through the API and compare plaintext. Sample old objects and every protected resource type, counting those not yet rewritten. A successful API read alone does not prove the disk value is encrypted.

6. Rotate, fail, and roll back

After KEK rotation keep the old KEK available for decryption, rewrite in batches, and monitor failures. Drill KMS timeouts, socket loss, one-apiserver rollback, and plugin upgrades. If error rates cross a threshold, pause rewrites and restore the old read configuration. Remove the old provider only after the new path and historical coverage are proven.

Model high-quality answer

I would validate the KMS v2 plugin, permissions, latency, and failure behavior in isolation. Production would put KMS v2 first, retain the old provider briefly for reads, and roll kube-apiservers individually. Once new writes are encrypted, I would rewrite objects in bounded resource and namespace batches, recording progress, conflicts, and retries. Evidence would include API reads, etcd prefixes, and sampled historical objects. Only after coverage is complete would I remove the old provider. Rotation and KMS failure need drills, thresholds, and a scoped rollback.

Common mistakes

• Changing config without rewriting old objects → historical plaintext remains → batch updates and completion counts.
• Restarting every apiserver together → control-plane outage → roll one at a time and observe health.
• Looking only at API reads → storage encryption is unproven → inspect raw etcd bytes.
• Removing the old provider immediately → old data cannot decrypt → wait for migration and keep a rollback window.
• Ignoring KMS latency and caching → peak API requests overwhelm KMS → load-test, rate-limit, and monitor calls.

Follow-up questions and responses

Are new Secrets all safe after KMS v2 is configured?

New writes use the first provider, but old objects do not change automatically. Rewrite them and verify with etcd evidence.

Can writes continue while KMS is unavailable?

It depends on plugin caching and configuration; never assume availability. Define timeout, write rejection, alerting, and post-recovery retry behavior, then drill it.

Why is retaining the old provider a risk rather than a permanent solution?

It preserves decryptability during migration and rollback but expands key and configuration scope. Remove it after coverage is proven, retaining rotation and recovery evidence.