Evaluate a Classifier on Imbalanced Data

Question and when to use it

A fraud classifier is evaluated on 100,000 transactions. Of these, 1,000 are fraudulent, so the positive rate is 1%. The model reports 99% accuracy. A review team can investigate at most 1,000 alerts per day, and a missed fraud, a false block, and an alert review have different costs. Decide whether the model is useful, choose a production threshold, and describe the offline evaluation and production monitoring plan.

This question fits data science, machine learning engineering, and risk roles. The task is not to name a metric that is supposedly “good for imbalanced data.” A useful answer turns the confusion matrix, ranking quality, business costs, processing capacity, and probability quality into one operational decision. The 1% positive rate, 100,000 validation examples, and daily capacity of 1,000 are interview assumptions, not general industry constants.

What the interviewer is evaluating

The first signal is whether the candidate catches the accuracy trap. Predicting every transaction as legitimate already produces 99% accuracy and zero recall. Merely replacing accuracy with F1 is incomplete: F1 gives precision and recall equal weight by default and says nothing about true negatives, review capacity, or unequal error costs.

The second signal is whether the candidate separates ranking from the business decision. Metrics such as AUROC and average precision summarize ranking across thresholds; the production system must still choose an operating point. A strong answer reports precision, recall, false-positive rate, alert volume, and cost from the same confusion matrix, then explains why the selected threshold fits capacity.

The third signal is leakage-free experimentation. Model fitting, probability calibration, threshold selection, and final evaluation must not repeatedly consume the same labels. A strong answer selects the threshold on cross-validation or a separate validation set, leaves the final test set untouched until the decision is frozen, and preserves a production-like class ratio in validation and test data.

The fourth signal is whether the candidate can detect failure after launch. The class base rate, population mix, score distribution, and label delay can all change the selected operating point's results. The monitoring plan should use time slices and important segments rather than treating one offline score as permanent.

Questions to clarify before answering

• What does the positive class mean, and which error is more expensive? Missing fraud usually causes a loss, while a false positive can add both review cost and user friction. Screening for disease or filtering spam would produce different error meanings and objectives.
• Does the model output a ranking score or a trustworthy probability? Taking the highest-scoring 1,000 cases under fixed capacity mainly requires good ranking. Computing expected monetary loss requires a calibrated score that can be interpreted as a probability.
• Is review capacity a hard ceiling or an average budget? A hard ceiling points to top-k, precision@k, and recall@k. Elastic capacity permits a direct comparison of the incremental benefit and cost of different thresholds.
• How long does it take to receive labels? Chargebacks may take weeks to mature. Same-day precision is then unavailable, so the team needs unlabeled leading indicators and outcome metrics recomputed on a mature-label window.
• Does production have the same positive rate as validation? Precision changes with the base rate. If validation data was oversampled or undersampled, the operating point must be evaluated again on data that retains the production ratio.
• Do error costs vary by transaction value or user segment? Missing a high-value fraud is not equivalent to missing a low-value one. A single global threshold may lose to a segmented policy or a per-example expected-cost decision.

30-second answer framework

“A 99% accuracy result does not establish that the model works. Predicting every case as legitimate also reaches 99% while missing all fraud. On a separate validation set with the production base rate, I would build the TP, FP, FN, and TN counts for candidate thresholds, then report precision, recall, false-positive rate, and alert volume. If the team can review only 1,000 alerts, I would evaluate precision@k and recall@k for the top 1,000; if costs are measurable, I would minimize the combined expected cost of misses, false positives, and reviews. AUROC and precision-recall or average precision compare rankings but do not select the operating point. I would tune thresholds and calibration only on data outside model fitting, use the final test set once, and monitor base rate, segments, capacity, and mature-label outcomes in production.”

Step-by-step solution

Step 1: Establish the no-skill baseline and full confusion matrix

Predicting every case as legitimate yields 99,000 true negatives and 1,000 false negatives: 99% accuracy and zero recall. This baseline shows that 99% accuracy only reflects class counts and does not prove that the model finds any fraud.

Suppose two candidate thresholds produce these results on the same validation set:

Lowering the threshold to 0.50 catches 150 additional frauds but creates 1,350 additional false positives and 1,500 additional reviews. The 0.80 threshold has higher accuracy and F1, while 0.50 has higher recall. These differences do not crown one metric; they expose the business trade-off that must be resolved.

Step 2: Evaluate ranking, the operating point, and probability quality separately

AUROC can be interpreted as the probability that a random positive ranks above a random negative, so it is useful for comparing ranking across thresholds. Under severe imbalance, a large number of true negatives can make a small false-positive rate look harmless even when the absolute number of false alerts is large. The precision-recall curve and average precision therefore add a view of how many issued alerts are correct and how much of the positive class is recovered. PR metrics still depend on prevalence, so they should not be compared without the data distribution.

The business decision lands on one threshold or a top-k cutoff. With capacity fixed at 1,000, the 0.80 threshold happens to produce 1,000 validation alerts, for precision@1000 of 70% and recall@1000 of 70%. Production volume and score distributions change. A more reliable capacity policy ranks cases each day, takes the top 1,000, and continuously measures both metrics instead of assuming that 0.80 will always produce the same count.

Calibration matters when a score is used as a probability. A calibrated 0.8, for example, should mean that roughly 80% of comparable predictions are positive. Reliability diagrams and scores such as Brier score can check calibration, and the calibrator should be fit on independent data or out-of-fold predictions. A fixed top-k policy mainly requires stable ranking and does not require every score to be perfectly calibrated. Monetary expected-cost decisions do require credible probabilities.

Step 3: Turn threshold selection into a cost rule

Let C_FN be the average cost of missing one fraud, C_FP the user-friction cost of one false positive excluding manual review, and C_R the cost of reviewing one alert. Validation cost at a threshold is FN × C_FN + FP × C_FP + alert count × C_R.

Moving from 0.80 to 0.50 avoids 150 misses but adds 1,350 false positives and 1,500 reviews. Under this scenario, lowering the threshold is worthwhile only when 150 × C_FN > 1,350 × C_FP + 1,500 × C_R, which reduces to C_FN > 9 × C_FP + 10 × C_R. This is a decision boundary derived from the example confusion matrices, not a universal constant.

If 1,000 reviews is an inviolable ceiling, the 0.50 threshold cannot go straight to production even when its expected value is higher. Options include reviewing only the top 1,000, automating a high-confidence and low-risk subset, or sending different values and segments to separate queues. Each option requires a fresh capacity, user-harm, and reversibility analysis.

Step 4: Account for base-rate changes

Precision depends on the positive rate as well as model behavior. Given true-positive rate TPR, false-positive rate FPR, and prevalence p, precision is TPR × p ÷ [TPR × p + FPR × (1 - p)].

At the 0.80 threshold, the example TPR is 70% and FPR is 0.303%. At 1% prevalence, the formula gives about 70.0% precision. If TPR and FPR remain unchanged but prevalence falls to 0.5%, precision falls to about 53.7%. Offline test data must therefore retain a production-like base rate, and production monitoring must track prevalence and segment mix. Resampling can help model training, but its artificial class ratio must not become the evaluation distribution.

Step 5: Design leakage-free evaluation and production monitoring

Split training, validation, and final test data by time, adding user, device, or event grouping when needed to prevent related examples from crossing partitions. After fitting the model, use cross-validation or separate validation data for hyperparameters, calibration, and threshold selection. Evaluate the final test set once after the decision is frozen. If a threshold-tuning tool uses cross-validation internally, verify that it is not tuning on the same complete data used to fit that exact model.

The production dashboard needs at least two layers. The immediate layer tracks score distributions, alert count, the top-k cutoff, review-queue age, important segments, and system errors. The mature-label layer recomputes precision@k, recall@k, FN cost, FP cost, calibration error, and deviation from offline expectations. Threshold changes should reuse the same cost and capacity rule and be replayed on an out-of-time set; a one-day fluctuation should not trigger a moving target.

Example of a strong answer

“I would not accept the 99% accuracy result first. Only 1% of the validation set is fraud, so predicting every transaction as legitimate also gives 99% accuracy and zero recall. I would clarify the positive-class definition, the three costs, whether 1,000 daily reviews is a hard limit, and the label delay.

At the 0.80 threshold, the given results are 700 TP, 300 FP, 300 FN, and 98,700 TN. Precision and recall are both 70%, and the alert count is exactly 1,000. The 0.50 threshold raises recall to 85% but creates 2,500 alerts, which exceeds capacity. It catches 150 additional frauds while adding 1,350 false positives and 1,500 reviews. I would use C_FN > 9 × C_FP + 10 × C_R to test the incremental value of lowering the threshold. With a hard capacity limit, I would instead take the top 1,000 scores and report precision@1000 and recall@1000.

I would use both AUROC and precision-recall or average precision during model comparison, then select the operating point with the confusion matrix and costs. If scores drive monetary decisions, I would calibrate them on data outside model fitting. If the system only needs top-k, I would focus on ranking stability. Hyperparameters, calibration, and thresholds stay within validation or cross-validation, while a production-ratio final test set is evaluated once.

In production, I would monitor alert capacity and mature-label metrics together. Even if TPR and FPR stay fixed, lowering the positive rate from 1% to 0.5% reduces the example precision from about 70.0% to 53.7%. That is why the plan also tracks base rate, score distribution, calibration drift, and performance by time and key segment.”

Common mistakes

• Comparing only accuracy → An all-negative classifier already reaches 99%, so the metric does not prove that positives were found → Start with the no-skill baseline, then report the confusion matrix, precision, recall, and absolute alert count.
• Automatically choosing F1 for imbalance → F1 implies equal precision-recall weight and ignores capacity and true negatives → Choose F-beta, a cost function, or a fixed-capacity metric from actual error costs.
• Claiming PR is always better than ROC → They answer different questions, and PR depends on prevalence → Use AUROC for ranking, PR or AP for positive-alert quality, and select the operating point separately.
• Using the default 0.5 threshold → A statistical default is not a business optimum → Tune the threshold on data outside model fitting against cost and capacity.
• Tuning the threshold on training data → The threshold adapts to training error and produces optimistic evaluation → Use out-of-fold predictions or a separate validation set, preserving a final test set.
• Reporting precision on a resampled test set → The artificial class ratio changes precision and alert volume → Evaluate on a production-like base rate and keep resampling in the training stage.
• Treating a raw score as probability → An uncalibrated 0.8 need not mean an 80% event rate → Validate and calibrate probabilities for expected-cost decisions; state the score semantics when only ranking.
• Monitoring only AUROC after launch → Stable global ranking can hide capacity overflow, segment failure, or a base-rate shift → Monitor the operating point, queue, costs, and mature-label outcomes as well.

Follow-up questions and responses

Follow-up 1: What if review capacity falls from 1,000 cases to 500?

Recompute precision@500, recall@500, and expected net value for the first 500 cases rather than scaling the old threshold. If high-value transactions offer greater benefit, rank by per-example expected loss rather than fraud probability alone. Check whether that ranking imposes disproportionate false blocks on any user segment.

Follow-up 2: How would you monitor the model when labels take 60 days to mature?

Separate immediate proxies from delayed outcomes. Immediately track missing inputs, score distribution, alert volume, reviewer decisions, and queue age. Build fixed maturity windows by event date and recompute precision, recall, and cost for the same cohort after 60 days. Proxies can trigger investigation but must not be presented as final label metrics.

Follow-up 3: Can the same threshold be used in a new market with a 0.5% fraud rate?

The old market's 70% precision is insufficient evidence. Even if the 70% TPR and 0.303% FPR transfer, the lower base rate reduces precision to about 53.7%; actual TPR and FPR may also shift with the population. Re-estimate the confusion matrix, calibration, and capacity on out-of-time samples from the new market, then apply the same cost rule.

Follow-up 4: AUROC improved, but precision among the top 1,000 fell. Which model wins?

If production can act on only the top 1,000, local ranking quality directly controls value, so precision@1000, recall@1000, and cost constraints take priority. AUROC spans all thresholds, and its gain may occur in a region the business never uses. Confidence intervals or repeated time slices should confirm that the top-k difference is not sampling noise.

Follow-up 5: Should transactions with different values share one threshold?

A single threshold is simple but treats a one-unit error and a ten-thousand-unit error as equal. With calibrated probabilities and estimated costs, use per-transaction expected loss or value bands with separate thresholds, then validate total capacity, segment fairness, interpretability, and rollback. A segment with too few examples should not tune its own threshold because it will chase noise.