Prompt and scope
A product team launches a new checkout flow in only some regions. Before launch, conversion is 10% in treated regions and 8% in control regions; afterward it is 14% and 9%. How would you evaluate the change with difference-in-differences?
This is a causal-inference prompt for data science, product analytics, and experimentation-platform roles. The four percentages are interview assumptions, not observed business results. Separate descriptive movement from causal identification and discuss assignment, time windows, parallel trends, and staggered rollout risk.
What the interviewer evaluates
A strong answer writes the estimand before checking assumptions; it does not jump to “conversion rose by three points.” The interviewer also looks for selection bias, concurrent shocks, spillovers, repeated observations, clustered standard errors, and the risk that a naive two-way fixed-effects model mixes effects under staggered adoption.
Clarifying questions
- How are regions assigned to treatment, and did the business choose expected high-growth regions?
- Are there several pre-launch periods for a trend check?
- Did pricing, marketing, seasonality, or traffic mix change at the same time?
- Can users cross regions or migrate, creating spillover into controls?
- Was the launch simultaneous or staggered by date?
- Is the unit a user, order, or region, and at what level should errors be clustered?
30-second answer
“First I would write the change as difference-in-differences: the treated change minus the control change. The point estimate here is three percentage points. It has a causal interpretation only if pre-treatment trends are comparable and there are no differential concurrent shocks or material spillovers. I would plot multiple pre-periods, control justified covariates, and cluster errors at the assignment level. For a staggered rollout, I would use an estimator designed for staggered treatment and compare robustness rather than rely on naive two-way fixed effects.”
Step-by-step deep dive
Step 1: Compute the descriptive estimate
Treatment regions move from 10% to 14%, a four-point increase. Controls move from 8% to 9%, a one-point increase. The DiD is (14%-10%) - (9%-8%) = 3 percentage points. That is a relative change, not yet evidence that the product caused it.
Step 2: Check identification assumptions
The key assumption is parallel trends: without the change, the outcome gap would have evolved similarly. Plot several pre-launch means and gaps to see whether groups were already diverging. Check for region-specific campaigns, logistics changes, macro shocks, and migration. Parallel trends cannot be established by one before-and-after comparison; combine multiple periods with a business explanation.
Step 3: Choose estimation and inference
With simultaneous treatment and one adoption date, use region and time fixed effects and cluster errors by region. With few treated regions, asymptotic standard errors may be unreliable, so add randomization inference or a small-cluster method. For staggered adoption, estimate cohort-specific dynamics first. NBER’s staggered-adoption guidance warns that simple two-way fixed effects can combine effects with problematic weights when effects vary by cohort or time; compare estimators suited to staggered treatment.
Step 4: Run counterfactual and robustness checks
Try placebo launch dates, placebo regions, and alternate windows. Add justified covariates and compare estimates; inspect negative outcomes and retention for implausible changes. If spillover is material, use non-adjacent controls or a higher-level treatment. Report the point estimate, interval, sample size, cluster level, and missing-data handling, not only three percentage points.
High-quality sample answer
“The arithmetic is four points of growth in treated regions minus one point in controls, so the DiD point estimate is three percentage points. That is only a relative change. It becomes a causal checkout-flow effect if, absent the launch, the groups would have followed comparable trends, no region-specific shock coincided with launch, and spillover is limited.
I would use several pre-launch periods to plot conversion and the gap, verify how treatment was assigned, and investigate marketing, price, traffic, and holiday changes. For a simultaneous launch, I would use region and time fixed effects with region-clustered errors. With few regions, I would add randomization or small-cluster inference.
For a staggered launch, I would estimate dynamic cohort effects and compare an estimator designed for staggered treatment instead of relying on naive two-way fixed effects. I would run placebo dates, placebo regions, alternate windows, and negative-outcome checks, then report intervals and assumptions. If pre-trends are clearly non-parallel or spillover cannot be controlled, I would downgrade the result to an association and recommend a randomized or better-isolated rollout.”
Common mistakes
- Calling three points causal → only a relative change was computed → state identification assumptions and uncertainty.
- Checking one pre-period point → parallel trends cannot be assessed → use multiple periods and an event-study plot.
- Ignoring assignment bias → high-growth regions may have been selected → analyze assignment and justify covariates.
- Applying naive TWFE to staggered rollout → cohort effects can contaminate one another → use staggered-treatment estimators and sensitivity checks.
- Clustering by user → treatment was assigned by region → cluster at the assignment level or use robust inference.
- Ignoring spillover → controls may receive the treatment indirectly → use non-adjacent controls or a higher-level rollout.
- Reporting only a point estimate → precision and sample boundaries are hidden → report intervals, counts, missingness, and windows.
Follow-up questions
Follow-up 1: Pre-trends already diverged. Can you still use DiD?
Look for a mechanism and a more comparable control. A justified covariate or narrower window can help, but regression should not cosmetically flatten an unexplained divergence. If parallel trends is not credible, use randomization or label the result associative.
Follow-up 2: There are only three treated regions. How do you infer uncertainty?
Do not rely on large-cluster asymptotics. Consider randomization inference, wild bootstrap, or a synthetic-control design suited to few clusters, and report region-level uncertainty transparently.
Follow-up 3: Users move between regions. What changes?
Define exposure, such as first stable region or account home region. Exclude clear movers for sensitivity analysis and measure migration. Strong spillover breaks the region-level counterfactual.
Follow-up 4: The result is significant but revenue is flat. How do you explain it?
Decompose conversion into order value, refunds, retention, and acquisition cost, and inspect negative outcomes. DiD identifies the target metric’s change; it does not automatically establish total business value.