Prompt and context
A search team has completed a new ranking algorithm, but the launch method is undecided. Engineering wants a feature flag, growth wants an A/B test, and operations is concerned about incidents. Explain how you choose the mechanism, define success, control exposure, and roll back a bad result. Assume the algorithm is already deployed; the question is about who sees it, how to compare it, and when to expand.
What the interviewer is testing
- Whether you separate the goals: a flag controls exposure, a progressive rollout controls risk, and an A/B test answers a causal comparison.
- Whether you define user, business, and system guardrails before discussing traffic percentages.
- Whether you explain stable bucketing, contamination, rollback authority, and safe defaults.
- Whether you include flag ownership, lifecycle, and cleanup instead of treating launch as a button click.
Clarifying questions before answering
- Are we learning value or only shipping safely? Unknown value calls for an experiment; known value with high risk calls for a progressive rollout.
- What is the experiment unit? Search ranking normally needs stable user or account bucketing; request-level randomness makes one user switch experiences.
- What is the primary metric and which guardrails are non-negotiable? More clicks cannot justify worse latency, complaints, conversion, or errors.
- Are there cross-device, regional, or dependency constraints? Inconsistent exposure contaminates conclusions; narrow the scope or fix allocation first.
30-second answer framework
“I first clarify whether the goal is learning, risk control, or both. To compare ranking value, I use a user-level stable A/B split; when value is known but failure risk is high, I start with a small progressive rollout behind a kill switch. Before launch I define a north-star metric and guardrails such as latency and errors, and log the variant ID, exposure, and outcome. I expand only after pre-set thresholds are met, pause and roll back on failure, and remove the experiment branch and flag after the decision.”
Step-by-step deep answer
1. Map each mechanism to the question
A feature flag is a runtime control plane: code can be deployed without exposing it to everyone. A progressive rollout is an exposure policy: move from internal accounts to a small customer cohort and then wider traffic. An A/B test is a measurement design: compare variants in comparable groups. They can be layered, but one does not replace the others.
2. Choose the unit and stable bucketing
Google Cloud’s example uses userID for sticky bucketing and descriptive variant IDs for analysis. For ranking, hash a user or account into baseline or experimental so the same person does not switch during the test. Request-level randomization contaminates caching, learning behavior, and returning-user experience.
3. Define metrics and stop rules
The north-star metric should represent the user task, such as a satisfied click after a useful search or task completion. Guardrails cover P95 latency, zero-result rate, error rate, complaints, and commercial harm. Write expansion, pause, and rollback thresholds before seeing results so a short-term click lift cannot hide long-term or system damage.
4. Use progressive rollout for irreversible risk
Microsoft describes separating deployment from exposure and moving from team accounts to selected customers and then broader users. Internal users validate correctness first; region or cohort expansion follows. Check metrics and logs at every stage. The kill path must be independent of the new algorithm so the rollback control does not depend on the failing code path.
5. Prevent contamination and audit variants
Log the user, variant, timestamp, version, exposure event, and outcome event. Google Cloud recommends descriptive variant names instead of a bare boolean. If a user is split differently across devices or groups interfere, mark the contamination and stop interpreting results. Analysis should trace each outcome back to the flag rules active at the time.
6. End the experiment and govern the flag
Optimizely separates experiments from targeted delivery: experiments answer which option is better, then the winning option is rolled out with a flag. Atlassian and Microsoft both emphasize removing flags after full rollout to avoid branch, coordination, and maintenance debt. Register an owner, expiry date, default, and deletion task when the flag is created.
High-quality sample answer
I would first ask whether the team needs learning or risk control. Ranking value is unknown, so I would start with a user-level stable A/B experiment, wrapped in a small progressive exposure gate that can be switched off when a guardrail is breached. The primary metric would be satisfied clicks after a useful search; guardrails would include P95 latency, zero-result rate, errors, and complaints.
I would log the variant ID, exposure, and outcome, never randomize per request, and avoid looking at clicks alone. I would start with internal accounts and a small customer cohort, verify logging, cache behavior, and cross-device allocation, then expand at explicit thresholds. Once the decision is made, I would make the winner the default and delete the experiment branch, owner record, and flag. If guardrails worsen, I would pause, restore baseline, and investigate. This covers learning, safety, and lifecycle ownership.
Common mistakes
- Mistake → Calling a feature flag an A/B test. Why it fails: a flag controls exposure but does not automatically create a comparable measurement. Fix: specify groups, metrics, exposure logging, and analysis.
- Mistake → Randomizing every request. Why it fails: one user flickers between experiences and pollutes cache and behavior. Fix: choose a stable unit and use sticky bucketing.
- Mistake → Defining only a growth metric. Why it fails: clicks can rise while latency, errors, or complaints worsen. Fix: pair the north-star metric with system and experience guardrails.
- Mistake → Keeping the flag after full rollout. Why it fails: branches and rules accumulate, making future behavior hard to reason about. Fix: create an owner, expiry date, and removal task with the flag.
Follow-up questions and responses
The experiment increases clicks but also latency. Which result wins?
Check the pre-set guardrail threshold first. If it is breached, pause expansion and restore baseline; then segment user value to decide whether to optimize the algorithm or keep the test only for latency-tolerant contexts. A click lift does not override system degradation.
The same user receives different variants on web and mobile. What do you do?
Define the experiment unit. If cross-device consistency matters, use an account-level key and one allocation service. If it cannot be unified, limit the conclusion to one surface instead of treating cross-device interaction as an experiment effect.
When is an A/B test not worth doing?
Skip it when correctness or compliance dominates, the sample cannot detect the target difference, or an external constraint already determines the launch. A small progressive rollout with guardrails is cheaper when the decision uncertainty is low.
What should happen if the flag service is unavailable?
Define a safe default for each variant, usually the verified baseline, and keep the last usable configuration locally when appropriate. Record evaluation failures so a flag-service outage does not become a core business outage.