Prompt and context
A platform team must inject default security context and observability labels when Pods are created, without maintaining an external mutating webhook. Using Kubernetes MutatingAdmissionPolicy, design the policy, binding, conflict handling, audit, and rollback plan.
Kubernetes v1.36 marks MutatingAdmissionPolicy stable. It uses CEL inside the API server to describe matching and mutations, and can change an incoming object with ApplyConfiguration or JSONPatch. Policy definition and binding are separate. The interview tests whether you can put declarative mutation into an auditable, reversible admission chain rather than merely rewriting a webhook manifest.
What the interviewer evaluates
The interviewer looks for a clear policy-versus-binding boundary; a deliberate choice between ApplyConfiguration and JSONPatch; idempotent mutations with explicit field ownership; handling of failurePolicy, scope, ordering, conflicts, self-protection, upgrades, and rollback; and evidence from audit events and metrics.
Clarifying questions
Target objects and defaults
Ask whether only Pods or also Deployments and Jobs are in scope, which fields are mandatory defaults, which user values may win, and how service accounts, namespaces, and label selectors constrain the change.
Version and runtime boundary
Confirm that the cluster is v1.36, that admissionregistration.k8s.io/v1 is enabled, whether existing webhooks remain in the chain, and whether the policy must be reused across clusters.
Risk and rollback
Identify protected security fields, acceptable failure modes, audit retention, change windows, and the effect of disabling the policy on existing objects and new requests.
30-second answer
“I would define matching and idempotent mutations in a policy, then use a binding to scope namespaces, resources, and parameters. ApplyConfiguration handles simple structured defaults; JSONPatch is reserved for precise array operations. Non-matches skip, while mutation errors follow failurePolicy. I would prevent self-matching, constrain RBAC, and roll out through dry runs, narrow bindings, and audit metrics. Rollback removes the binding and restores a versioned policy; it does not silently rewrite existing objects.”
Step-by-step solution
Step 1: Separate policy and binding
The policy stores rules, variables, match conditions, and mutation expressions. The binding selects resources and namespaces and can provide parameters. One policy can therefore be reused with different bindings for tenants or environments during a staged rollout.
Step 2: Choose a mutation representation
ApplyConfiguration expresses structured defaults close to the object model. JSONPatch handles exact path insertion or removal, but array indices and JSON Pointer escaping must be correct. Do not mix them into an implicit overwrite strategy; every field needs an ownership and precedence rule.
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingAdmissionPolicy
metadata:
name: pod-default-observability
spec:
matchConstraints:
resourceRules:
- apiGroups: [""]
apiVersions: ["v1"]
operations: ["CREATE"]
resources: ["pods"]
mutations:
- applyConfiguration:
expression: >-
Object{metadata: Object{labels: {"observability.example.com/enabled": "true"}}}
---
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingAdmissionPolicyBinding
metadata:
name: pod-default-observability-binding
spec:
policyName: pod-default-observability
matchResources:
namespaceSelector:
matchLabels:
platform.example.com/enabled: "true"Step 3: Make mutations idempotent and non-destructive
Write defaults only when fields are absent, preserving explicit user values. For lists, define keyed merge or whole-list replacement; retries, repeated admission, and updates must not append duplicates. Check whether evaluating the mutated object can trigger the same rule again, creating a self-amplifying loop.
Step 4: Limit matching and protect the policy
Narrow scope with apiGroups, resources, operations, namespaceSelector, and objectSelector. A MutatingAdmissionPolicy cannot match itself or its binding, preventing a policy from changing its own configuration into an unrecoverable state. Use explicit allowlists for high-risk fields so user-provided CEL parameters cannot expand write authority.
Step 5: Define errors and ordering
Distinguish a non-match, an empty expression result, a mutation error, and a temporary API-server failure. Choose Fail or Ignore through failurePolicy and pair it with alerts. Do not rely on hidden ordering among mutators; separate field ownership or centralize a decision when policies would write the same field.
Step 6: Migrate and version webhooks
Compare the old webhook result with the new policy in a shadow or audit-only phase, then bind the policy to a small namespace set. Record policy version, request UID, an original-object summary, and mutation reason. Remove the webhook gradually after conflicts are understood, keeping versioned manifests for comparison and rollback.
Step 7: Roll back and observe
For an emergency rollback, pause or delete the binding so new requests stop changing, then restore the previous policy version. Existing objects are not reverse-mutated by removing a binding; cleanup needs a separate, approved controller or batch process. Monitor admission latency, rejection rate, mutation count, expression errors, and hit rate by namespace.
Model answer
I would treat the policy as a versioned rule and the binding as the rollout and permission boundary. Scope it to Pod CREATE in selected namespaces and apply idempotent defaults only to missing labels and security fields; use tested JSONPatch for exact array paths. Define failurePolicy, field allowlists, RBAC, and self-protection before rollout. Compare old and new results, bind a small namespace set, and expand using audit, latency, and error metrics. Rollback removes the binding and restores the previous policy; existing objects do not automatically roll back, so cleanup is a separate audited process.
Common mistakes
- Mistake: Replacing the whole object. → Why it fails: Explicit user configuration is erased and ownership conflicts appear. → Fix: Write only missing fields and define list-merge rules.
- Mistake: Assuming deleting a binding restores old objects. → Why it fails: Admission affects requests; it does not provide reverse mutation. → Fix: Use a separate audited cleanup process and state the existing-object behavior.
- Mistake: Depending on a fixed order among policies. → Why it fails: Admission ordering changes can change the result. → Fix: Separate field ownership or centralize the decision.
- Mistake: Replacing the production webhook immediately. → Why it fails: Expression differences can surface at peak load. → Fix: Shadow first, roll out by namespace, and retain versioned manifests.
Follow-ups and responses
How do you choose between ApplyConfiguration and JSONPatch?
Use ApplyConfiguration for structured defaults and clearer intent. Use JSONPatch for precise insertion, deletion, or escaped paths. Test repeat execution and array conflicts with either form.
Should failurePolicy always be Fail?
Security baselines and compliance fields often favor Fail. Non-critical observability labels may use Ignore with explicit alerting and compensation. Tie the decision to impact, availability goals, and audit evidence.
How do you test that a policy will not corrupt objects?
Build a matrix with server dry runs, fixed input snapshots, repeated admission, namespace labels, pre-set user fields, empty lists, and expression errors, then compare the old webhook and new policy results.
Why not write a controller instead?
Admission blocks or changes a request before persistence, which suits defaults and entrance constraints. Controllers reconcile asynchronously and repair existing objects. They can complement each other, but a controller does not replace an entrance security policy.