Prompt and setting
The visible result is measured only among users with an observed outcome. The interview asks whether users who churned, failed telemetry, or were filtered out differ systematically, and how to restore a valid decision population.
What the interviewer tests
- Recognizing survivorship bias as conditioning on selected survivors.
- Reconstructing the assignment denominator before reading outcome rates.
- Separating missingness, exposure, and treatment effects.
Clarifying questions before answering
- Is the primary estimand intent-to-treat over all assigned users or a per-protocol effect?
- When and why can users leave, disable telemetry, or become ineligible?
- Are assignment counts, exposure counts, and outcome counts available by arm and segment?
- Was the retention filter defined before launch, or introduced after seeing results?
30-second answer framework
I would freeze the decision, rebuild the funnel from assignment to exposure to outcome, and compare attrition by arm. The primary analysis should keep all randomized users in the denominator, treating missing outcomes explicitly. I would inspect sample-ratio mismatch, telemetry loss, and pre-treatment segments, then report sensitivity bounds for plausible missing outcomes. A survivor-only uplift can be descriptive, but it cannot be the launch effect without stronger assumptions.
Step-by-step deep dive
1. Define the population
Name the unit, assignment time, eligibility rule, and measurement window. Preserve every assigned user in the intent-to-treat population; do not silently replace it with users who opened the feature or remained active.
2. Rebuild the funnel
For each arm, count assigned users, exposed users, active users, observed outcomes, and missing outcomes. Add rates and absolute counts. A higher conversion rate among survivors can coexist with worse total conversion if treatment causes more early exits.
3. Diagnose the selection mechanism
Check sample-ratio mismatch, rollout rules, client versions, geography, device, latency, and event delivery. Compare pre-treatment behavior for retained and missing users. If missingness depends on treatment or outcome risk, complete-case analysis is biased.
4. Choose the primary estimate
Use intent-to-treat for the launch decision, with missing outcomes handled by a pre-specified rule or bounds. Report per-protocol or survivor-only results as secondary and label their assumptions. If randomization or telemetry is broken, repair the pipeline or rerun instead of narrating a winner.
5. Communicate uncertainty
Show arm counts, attrition differences, intervals, and sensitivity scenarios. Segment only on pre-treatment variables and pre-specify the analysis. Escalate when the decision changes across plausible missing-outcome assumptions.
High-quality sample answer
“I would pause the launch and reconstruct the assignment-to-outcome funnel for both arms. My primary estimate keeps every randomized user, while survivor-only conversion is secondary. I would test sample-ratio mismatch, telemetry loss, and arm-specific attrition, then compare pre-treatment risk across observed and missing users. I would publish intent-to-treat intervals plus sensitivity bounds; if the conclusion depends on untestable missing outcomes or a broken randomization path, I would fix the experiment or rerun it.”
Common mistakes
- Use only active users → treatment-related churn disappears → retain all assigned users for intent-to-treat.
- Treat missing events as random → telemetry loss can depend on arm or outcome → diagnose missingness and report sensitivity.
- Call exposure the denominator automatically → exposure may be post-treatment → state the estimand before filtering.
- Hide attrition in a percentage → unequal counts can reverse the decision → show absolute counts and arm differences.
Follow-up questions and responses
Can missing outcomes simply be imputed?
Only with a stated model and sensitivity analysis. Multiple imputation may be useful under missing-at-random assumptions, but it does not prove those assumptions; worst-case or tipping-point bounds can reveal decision fragility.
When is a per-protocol result useful?
It can describe effect among users who followed a defined protocol, but adherence is often post-treatment. Treat it as secondary and explain the assumptions or use a causal method designed for treatment adherence.
How do you distinguish survivorship bias from sample-ratio mismatch?
Sample-ratio mismatch concerns assignment counts deviating from the planned split. Survivorship bias concerns conditioning on a selected subset after assignment. They can coexist, so audit both the randomization denominator and downstream retention.