Prompt and context
Enterprise customers fear that changing an automation policy could affect many resources incorrectly. Decide whether the SaaS should preview potential allows, denials, and affected objects before a change is submitted.
This is a product trade-off question, not a requirement to use OPA, Terraform, or a particular workflow engine. Focus on preview-versus-execution differences, user trust, permissions, and gradual rollout.
What the interviewer evaluates
User problem
Can you distinguish “who will be affected?” from “the final execution is guaranteed to match,” and identify high-risk policies and roles?
Accuracy boundary
Can you state the snapshot, external facts, and evaluation time used by a preview without presenting stale simulation as a guarantee?
Product scope
Can you select an initial policy subset, object scale, diff view, approval, and rollback path instead of promising perfect simulation for every rule?
Value validation
Can you validate value with unintended-impact rate, undo rate, preview adoption, execution divergence, and support tickets?
Clarifying questions to ask
- Which policies can cause large-scale or irreversible impact?
- Do customers need an object list, summary, diff, or cost estimate?
- Must the preview use live external state, or is a snapshot acceptable?
- Do changes require multiple approvers, rollback, or audit retention?
- Could results expose sensitive resource names or cross-tenant information?
- What latency and object-count limits are acceptable?
30-second answer framework
“I would first validate whether high-risk customers delay or mis-submit changes because impact is invisible. The first release would cover explainable, enumerable rules and return additions, removals, allows, denials, and unknowns from a timestamped snapshot. The preview would explicitly not guarantee execution; submission would re-evaluate and show drift. It would reuse execution permissions, be auditable, and run asynchronously for large scopes. Preview adoption, execution divergence, undo rate, and incidents would determine expansion.”
Step-by-step deep dive
Step 1: Validate the problem and segments
Interview administrators, auditors, and operators about losses from unintended policy effects, approval delays, or difficult rollback. Segment by irreversibility, object count, and compliance requirements.
Step 2: Define the preview contract
Specify policy version, evaluation time, state snapshot, external-fact version, and output types. At minimum distinguish changed, unchanged, not applicable, and unknown objects with reasons.
Step 3: Select the first scope
Prioritize rules with explicit logic, enumerable objects, and explainable outcomes. Defer rules that depend on live randomness, manual actions, or unobservable external systems, returning an explicit unknown state.
Step 4: Design interaction and guardrails
Show summaries, object samples, downloadable lists, sensitive-field masking, and drift warnings. High-risk changes require reconfirmation, two-person approval, or staged execution; preview and submission use the same authorization check.
Step 5: Handle races and privacy
Re-evaluate current state at submission and compare preview with execution; pause or ask for confirmation when drift exceeds a threshold. Isolate results by tenant, minimize display, and retain audit records.
Step 6: Roll out and measure
Start with internal users and low-risk customers. Record preview latency, adoption, execution divergence, undo actions, unintended impact, and support tickets. If previews often disagree with execution, fix snapshots or narrow the promise before expanding.
Strong sample answer
“I would not position dry run as a global safety guarantee. I would start with high-risk, enumerable actions such as deletion or bulk authorization, and verify that customers need an impact list to reduce mistakes. Preview input would bind policy version, tenant, evaluation time, and state snapshot; output would classify added, removed, unchanged, and unknown objects with rule explanations.
The result would show an expiry and use the same execution engine at submission. State drift would pause the change or request confirmation. Permissions would match real execution, sensitive resources would be summarized, and large results would be asynchronous. I would pilot with low-risk customers, measuring execution divergence, undo rate, unintended impact, and tickets before adding more rules.”
Common mistakes
- Calling a simulation an execution guarantee.
- Ignoring state races between preview and submission.
- Promising every policy, object scale, and external dependency in v1.
- Showing one total without reasons, samples, or unknown states.
- Giving preview broader permissions and exposing cross-tenant resources.
- Omitting policy version, state timestamp, and audit evidence.
- Measuring clicks instead of execution divergence and unintended impact.
- Expanding while previews disagree instead of fixing the boundary.
Follow-up questions and responses
Follow-up 1: What if preview and execution disagree?
Re-evaluate at submission and show the drift; pause above a threshold. Record policy, state, time, and decision identifier for both evaluations, then classify the cause.
Follow-up 2: Why not copy production data for simulation?
Copying adds privacy, cost, and freshness problems. Prefer isolated snapshots or redacted projections, retaining only needed fields and the snapshot time.
Follow-up 3: Which customers should get it first?
Choose customers with clear object boundaries, mature approvals, rollback capability, and feedback capacity. Confirm audit, retention, and isolation needs with regulated customers first.
Follow-up 4: What if the preview is too large?
Show grouped summaries and representative samples first, then offer an asynchronous list and completion notice. Set limits and cost warnings so preview queries do not starve execution.
Follow-up 5: How do you prove it deserves long-term maintenance?
Compare enabled customers with a control group on mistakes, undos, incidents, approval time, and tickets, combining preview adoption with execution divergence to assess accuracy and savings.