Data Science Interview: How Do You Assess Probability Calibration?

Prompt and context

A fraud model outputs a risk probability from 0 to 1 for every transaction. The business wants to use 0.8 as a manual-review threshold. Explain how to assess calibration, separate ranking from probability quality, choose a calibration method, and handle class imbalance, drift, and review-capacity limits.

PracHub's public 2026 machine-learning interview question explicitly asks about model calibration, reliability diagrams, ECE, Brier score, and threshold selection. The scikit-learn documentation provides the implementation boundaries for calibration, binned reliability diagrams, and independent calibration data. This question is not tied to a specific company.

What interviewers assess

An average answer says “check accuracy or AUC.” A strong answer defines the probability: among cases predicted near 0.8, the long-run positive rate should be near 0.8. It then separates reliability, discrimination, base rate, and the business threshold, explaining why a high-AUC model can still be overconfident.

Interviewers also check for leakage-free splits, a representative calibration set, sufficient observations per bin, and a post-calibration cost check. Calibration does not make the model smarter; it makes the score interpretable on a probability scale.

Clarifying questions

• What decision uses the probability? Ranking may need AUC and top-k; capacity allocation or expected-loss decisions need calibration.
• When do labels mature? Fraud confirmation may be delayed, so the evaluation window must wait for mature labels.
• Does the online positive rate match training? Sampling, weighting, and time changes alter the prior and require target-distribution evaluation or prior correction.
• Must every segment have trustworthy probabilities? High-risk systems should inspect region, channel, and product slices because aggregate calibration can hide local errors.
• What are review capacity and error costs? The threshold is a business decision, not an automatic output of a calibration curve.

30-second answer

“I first confirm whether probabilities drive review decisions and wait for mature labels. Calibration means that cases predicted at 0.8 have an observed positive rate near 0.8. On data independent from training and representative of production, I draw a reliability diagram and report bin counts, ECE, and log loss; Brier is a composite score and cannot prove calibration alone. I choose the threshold from cost and review capacity, check that ranking quality remains stable, inspect segments, and monitor prior and calibration drift over time. For miscalibration I start with sigmoid scaling; with enough data and a nonlinear shape I try isotonic, and I never fit a calibrator on training predictions.”

Step-by-step answer

Step 1: Separate calibration from ranking

Binary calibration requires that samples predicted near p have an observed positive frequency near p. AUC depends only on ordering, and a monotonic probability transformation can preserve AUC, so AUC cannot replace a calibration check.

Step 2: Plot reliability and report uncertainty

Bin predicted probabilities, then compute the mean prediction and observed positive rate in each bin. Plot the mean prediction on the x-axis and the positive rate on the y-axis; perfect calibration is near the diagonal. Show each bin's count or a histogram so a tiny high-score bin is not overinterpreted.

from sklearn.calibration import CalibrationDisplay
from sklearn.metrics import log_loss

display = CalibrationDisplay.from_predictions(
    y_true=y_cal,
    y_prob=p_cal,
    n_bins=10,
    strategy="quantile",
)
loss = log_loss(y_cal, p_cal)

Bin boundaries are not objective truth: equal-width bins may be mostly empty under severe imbalance, while quantile bins change the probability range per bin. With few observations, show confidence intervals, merge sparse bins, or use a reproducible optimized binning method instead of trusting one line.

Step 3: Understand ECE, Brier, and log-loss boundaries

ECE usually computes a weighted absolute gap between mean prediction and observed frequency per bin. It is easy to communicate but depends on the bin count and rule. Brier score is squared probability error, while log loss penalizes confident wrong predictions; both mix reliability with discrimination and outcome uncertainty.

Therefore “lower Brier means better calibration” is not a valid conclusion. Combine the diagram, bin counts, the reliability component of a decomposition, and log loss on the same mature time window. Research on reliability diagrams also shows that naive binning is sensitive to implementation choices, so report the method and uncertainty.

Step 4: Calibrate on a leakage-free split

Training predictions were seen by the base model, so fitting a calibrator on them produces an optimistic mapping. The safe sequence is: fit the base model; fit the calibrator on out-of-fold or held-out probabilities; evaluate once on data that participated in neither fit.

train: fit base model
calibration: fit calibrator on out-of-fold or held-out probabilities
test: evaluate calibration, ranking, and business threshold once

Use time-based splits for delayed-label or time-series cases; do not shuffle future information into the past. The calibration set must match production's positive rate, feature distribution, and label-maturity window, or it measures the training sample rather than the probability the business will use.

Step 5: Choose calibration and the operating threshold

Sigmoid or Platt scaling fits one-dimensional logistic regression on raw scores and is stable with limited data when the error is roughly S-shaped. Isotonic regression is more flexible but can overfit when the calibration set is small. Temperature scaling is common for multiclass logits and still needs independent validation.

Calibration determines the probability scale; the threshold remains a business decision. If reviewers can process only 2,000 cases per day, choose a threshold from capacity, false-positive cost, false-negative cost, and delay rather than assuming 0.8 is meaningful. Validate threshold changes with representative replay and online guardrails.

Step 6: Handle imbalance and production drift

Oversampling, class weights, and focal loss can change the probability meaning of a score. If training has a 10% positive rate but production has 2%, ranking may remain stable while probabilities become miscalibrated. Re-evaluate on the target prior, apply a prior correction when justified, or recalibrate.

After launch, compute rolling reliability, ECE, log loss, and positive rate, with slices by channel, region, and customer tier. Seasonality, policy changes, and label delay cause drift. Trigger recalibration when error exceeds a business tolerance, a critical segment drifts, or the prior moves beyond a threshold, using fresh data with mature labels.

High-quality sample answer

“I first ask whether these probabilities drive review, when labels mature, and whether the online positive rate matches the training sample. Calibration means that transactions predicted near 0.8 have a long-run fraud rate near 0.8. The evaluation set must be independent, time-aware, label-mature, and representative of production.

“I draw a reliability diagram with bin counts and confidence intervals, then report ECE, log loss, and Brier. Brier is not proof of calibration because it also includes discrimination. AUC answers ranking, not whether a probability is trustworthy. I use sigmoid for a small, roughly S-shaped error and try isotonic with enough data for a nonlinear mapping, never fitting the calibrator on training predictions.

“Finally I set the threshold from review capacity and error costs, inspect segment calibration, and monitor mature-label reliability and log loss over time. A high-AUC model that overestimates risk still cannot treat 0.8 as real risk.”

Common mistakes

• Symptom → Using accuracy or AUC to prove trustworthy probabilities → Why it fails → Neither compares predicted values with observed frequencies → Fix → Use a reliability diagram, bin counts, and calibration metrics.
• Symptom → Fitting a calibrator on training predictions → Why it fails → The mapping is overfit and miscalibrates new samples → Fix → Use out-of-fold predictions or an independent calibration set.
• Symptom → Reporting one ECE number → Why it fails → ECE depends on binning and sparse observations → Fix → Report the bin rule, diagram, counts, and uncertainty.
• Symptom → Treating 0.8 as a threshold because AUC is high → Why it fails → Good ordering does not imply a correct probability scale → Fix → Choose thresholds from capacity and error cost.
• Symptom → Ignoring sampling and base-rate shifts → Why it fails → Calibration targets the training population rather than production → Fix → Evaluate on the target distribution and monitor prior drift.

Follow-ups and responses

What if AUC is high but the reliability diagram is S-shaped?

Keep the base model's ranking, fit sigmoid or isotonic on independent data, and re-evaluate calibration, AUC, and business cost. Monotonic calibration normally preserves ordering, but the split and implementation still need testing.

What if positive labels are confirmed only after 30 days?

Define a label-maturity window and evaluate or fit the calibrator only on records at least 30 days old. Recent records can monitor prediction volume and delay, but cannot be treated as negatives for reliability.

What if regions have very different calibration curves?

Check sample size, label definitions, and base rates first. Then choose a global calibrator, segment calibrators, or segment thresholds. If a segment lacks data, use a stable global mapping with interval monitoring instead of fitting a noisy independent curve.