System design interview: How do you safely migrate Kubernetes Pod security to enforce?

Prompt and scope

The platform team must move namespaces to the Kubernetes Pod Security Standards restricted level. Workloads come from production services, shared infrastructure, and temporary build jobs. Design discovery, remediation, exceptions, cutover, and rollback.

What the interviewer is testing

• Understanding the different side effects of enforce, audit, and warn.
• Bringing namespaces, workloads, and exceptions into one governance boundary.
• Using observations to drive remediation instead of immediately rejecting everything.
• Considering pinned policy versions, audit trails, and emergency rollback.

Clarifying questions

1. What Kubernetes versions, tenant boundaries, and release tools are in use?
2. Is the target baseline or restricted, and may namespaces use different levels?
3. Which workloads require privileges, hostPath, or host networking?
4. Does rollback mean temporarily lowering one policy or shifting traffic to an old cluster?

A 30-second answer

Start per namespace with audit and warn to collect violations and give submitters actionable feedback; do not turn on global enforce first. Remediate in risk and business order with pinned policy versions. Every exception needs an owner, reason, expiry, and compensating control. After a small pilot, enable enforce namespace by namespace and watch rejection and release-success rates. Emergency rollback lowers only the affected namespace policy while retaining the audit trail.

Step-by-step design

1. Establish an asset and policy baseline

Inventory namespaces, Pod templates, controllers, and release sources. Group production, shared infrastructure, development, and temporary jobs. Choose a target level and version for each group, and record exceptions; an unlabeled namespace is a security gap, not a safe default.

2. Observe before blocking

Enable audit to record violations and the same level in warn for client feedback. Aggregate events by rule, team, and image into a remediation queue. This exposes risk without interrupting current traffic.

metadata:
  labels:
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted

3. Remediate workloads

Remove unnecessary privilege, hostNetwork, hostPID, hostPath, and writable root filesystems. Add non-root, seccomp, and capability restrictions. Put policy checks in CI so a template fails with a specific field before the cluster rejects it.

4. Govern exceptions

Grant exceptions only at a governable namespace or controlled entry point. Record impact, owner, expiry, and compensating controls. An admission exemption is not a permanent allow-list; renewals require review, and system namespaces should not all be unconditionally exempt.

5. Cut over enforce in stages

Enable enforce first for low-risk namespaces. Compare rejection, release failure, restart, and tenant complaint rates before expanding. Pin policy versions; rehearse upgrades in warn and audit so a latest change does not create a surprise rejection wave.

6. Roll back and verify

Release controllers must pause and restore old templates. If a core service is blocked, temporarily lower only that namespace while retaining audit and events. Test new Pods, Deployment templates, rolling upgrades, Jobs, exception expiry, and policy-version upgrades.

Model high-quality answer

I would inventory namespaces and workloads, pin policy versions, then collect violations with audit plus the same-level warn. CI checks security contexts early, while exceptions carry owners and expiry dates. Low-risk namespaces move to enforce first; I expand only after rejection rate, release success, and business errors remain acceptable. Policy upgrades are rehearsed with warn/audit. If a service is blocked, I roll back only its namespace policy, preserve the audit trail, and fix the root cause rather than permanently weakening the cluster.

Common mistakes

• Enabling global enforce immediately → many releases fail together → use audit/warn and stages.
• Treating unlabeled namespaces as safe → policy coverage has blind spots → label and inventory them.
• Creating permanent exemptions → risk remains hidden → require expiry and review.
• Discovering violations only in the cluster → feedback arrives too late → move checks into CI and template review.
• Using latest without rehearsal → a version change causes surprise rejects → pin and pre-observe.

Follow-up questions and responses

Why enable both warn and audit if neither blocks?

warn gives the submitting client immediate feedback; audit records violations for platform measurement. They support different workflows and do not replace each other.

Should an exception be scoped to a Pod or a namespace?

Prefer a governable namespace boundary with a controlled entry point. Per-Pod allowances are easy to copy and bypass; all exceptions still need an owner, expiry, and compensating control.

How do you prove availability did not decline?

Compare rejection, release success, restarts, SLOs, and tenant errors before and after each stage. Replay rolling upgrades and short-lived Jobs as well as long-running services.