Prompt and scope
Treat this as a product execution and metrics case. Revenue per user rises while a retention measure falls after a launch. Do not assume causality, equal customer value, or that either metric is the company goal. The answer should define the decision population, diagnosis window, guardrails, and action rule before recommending a rollout.
What the interviewer is testing
The interviewer wants diagnosis before advocacy. Interview Pilot describes PM execution interviews as tests of success metrics, prioritization, and metric-change diagnosis, and warns against choosing vanity metrics or jumping to fixes. A strong answer distinguishes observation from causal evidence, segments the effect, states which long-term outcome matters, and makes rollback criteria explicit. A weak answer says “retention is more important” without defining the business model or checking who is paying.
Questions to clarify before answering
- What does revenue mean: gross bookings, recognized revenue, margin, or revenue per active user? Each changes the trade-off.
- Which retention cohort and horizon moved: day 7, month 1, renewal, or logo retention? A short-term and contract-renewal signal are not interchangeable.
- Which users saw the launch, and was exposure randomized? Segment, plan, geography, tenure, and device can reveal composition changes.
- What is the decision deadline and rollback cost? A reversible flag supports a different threshold from a permanent pricing migration.
- What guardrails are non-negotiable: refunds, complaints, support load, latency, or regulatory limits?
A 30-second answer framework
“I would first verify the metric definitions, exposure, cohorts, and time window. Then I would test whether the revenue gain and retention loss are causal and concentrated in a segment. I would define the company objective and guardrails, estimate long-term value rather than optimize a single snapshot, and keep the launch behind a reversible control. My decision rule would specify when to continue, narrow the segment, change the experience, or roll back.”
Step-by-step diagnosis and decision
1. Validate the movement
Check instrumentation, exposure assignment, denominators, refunds, seasonality, and cohort maturity. Compare treatment with control when possible. Revenue per user can rise simply because low-value users churned; that is a composition change, not necessarily better monetization.
2. Segment the trade-off
Break results down by plan, tenure, acquisition source, geography, device, and feature exposure. Compare revenue, retention, gross margin, support contacts, refunds, and usage quality for each segment. A premium segment with stable renewal and a free segment with rapid churn may justify a targeted change rather than a global rollback.
3. Choose primary and guardrail metrics
Name the durable outcome the business is optimizing, such as contribution margin over a renewal horizon. Keep retention, refund rate, complaint rate, and service quality as guardrails when they represent unacceptable harm. Statsig distinguishes primary metrics from secondary metrics that detect unintended side effects; Amplitude documents success metrics and guardrail thresholds. Do not add so many guardrails that every decision becomes impossible.
4. Apply a reversible decision rule
Keep a control group or feature flag while evidence matures. Continue when the primary outcome clears its minimum effect and no guardrail breaches its threshold. Narrow the rollout when harm is segment-specific. Change pricing, onboarding, or value communication when diagnosis identifies a mechanism. Roll back when a critical guardrail breaches a pre-agreed limit or long-term value is clearly negative. Record the rule before looking at the next slice of data.
High-quality sample answer
“I would not choose revenue or retention by instinct. First I would verify that revenue per user uses the same eligible population as retention and that the launch exposure is randomized. I would inspect cohorts by plan and tenure, then compare renewal intent, refunds, complaints, and contribution margin. Suppose a premium cohort has higher margin and unchanged renewal, while new free users see a sharp day-30 retention decline after a paywall change. I would keep the premium treatment, narrow the paywall experiment for new users, and use renewal plus refund rate as guardrails. The launch stays behind a flag with a control group. I would continue only if the margin target clears its minimum effect and no guardrail crosses its threshold; otherwise I would revert the paywall and test a less disruptive value explanation. The key is an explicit, segment-aware rule rather than defending the metric that moved first.”
Common mistakes
- Declare the winner from one aggregate chart → composition and cohort maturity can reverse the conclusion → validate denominators and segment first.
- Treat retention as universally superior → business models and horizons differ → define the durable objective and customer value.
- Call every secondary metric a co-equal goal → decision rules become paralyzed → limit guardrails to material, actionable harms.
- Stop an experiment because of early noise → repeated peeking increases false decisions → predefine duration, thresholds, and emergency-only checks.
- Roll back globally when one segment is harmed → discards value from unaffected users → localize exposure and test a mechanism-specific change.
- Ignore margin and refunds → revenue can grow while economics worsen → include contribution and post-purchase costs.
Follow-up questions and answers
What if revenue rises and retention falls in every segment?
Recheck attribution and horizon, then estimate expected lifetime value with explicit uncertainty. If the durable value is negative or a critical guardrail is breached, pause or roll back while testing a less extractive design. Do not hide the loss inside an aggregate forecast.
What if retention is noisy and the interview time is short?
State a proxy only when it has a validated relationship to the long-term outcome. Use a staged answer: instrument now, monitor a leading guardrail, and reserve the final rollout decision until the cohort matures. Label the uncertainty instead of inventing a precise threshold.
How would you communicate the decision to leadership?
Show the objective, segment table, confidence and decision rule on one page. Say what was learned, what remains uncertain, which users are affected, and what you will do if a guardrail moves. Leadership can then debate the objective explicitly rather than argue over a single chart.