1. Prompt
You have a point predictor f(x), but the product needs an interval such as [lower, upper] with a target coverage of 1 - alpha. Design a split conformal prediction workflow and explain how the training set, calibration set, and test point are used.
2. Constraints and clarifications
- Start with regression and absolute-residual scores; classification sets, online calibration, and conditional coverage are extensions.
- Fit the model first and freeze it; do not reuse the calibration set to fit the point predictor.
- Coverage is a marginal guarantee over an exchangeable batch, not a promise that every subgroup or point has the same rate.
- Distinguish i.i.d.-like data, time series, and covariate shift when discussing validity.
3. Core approach
Split the data into training and calibration sets. Fit the point predictor on training data, then compute calibration nonconformity scores s_i = |y_i - f(x_i)|. For target coverage 1 - alpha, take a finite-sample conformal quantile q; for a new x, return [f(x) - q, f(x) + q].
Under exchangeability, the construction gives a marginal coverage guarantee: the new label falls inside the interval with probability at least approximately 1 - alpha, with a conservative order statistic for finite calibration sets. The point model need not be correct, but its residuals and noise determine interval width.
4. Reference implementation
def fit_split_conformal(train, calibration, alpha):
model = fit_point_predictor(train.x, train.y)
scores = sorted(
abs(y - model.predict(x))
for x, y in zip(calibration.x, calibration.y)
)
# Use the finite-sample conformal quantile, not a naive percentile.
rank = ceil((len(scores) + 1) * (1 - alpha))
q = scores[min(rank, len(scores)) - 1]
def predict_interval(x):
center = model.predict(x)
return center - q, center + q
return predict_interval5. Correctness and evaluation
Evaluate coverage and mean interval width on an independent test set, then slice by time, region, or user group. Coverage below target can come from non-exchangeable data, a small calibration set, distribution shift, or an implementation bug; an aggregate rate can hide systematic failure for one subgroup.
Width depends on the score. Absolute residuals produce symmetric intervals, while quantile regression or local-scale scores can produce asymmetric or adaptive intervals. A narrower interval is not automatically better; compare it jointly with coverage and business loss.
6. Follow-ups and traps
- “Distribution-free” does not mean an unconditional guarantee for every distribution; the classical guarantee relies on exchangeability between calibration and new samples.
- Temporal dependence can invalidate naive split conformal. Use rolling calibration, weighted methods, or explicitly narrow the guarantee.
- Under covariate shift, density-ratio weighting can help, but wrong weights affect coverage; do not claim the original guarantee is unchanged.
- Marginal coverage is not conditional coverage. Small groups or extreme inputs can still receive very wide or very narrow intervals.
7. Further reading
Compare split, cross-conformal, online, and weighted conformal methods: they trade data reuse, compute cost, shift adaptation, and theoretical guarantees. A production system should monitor coverage, interval width, residual drift, calibration freshness, and rejection or human-review rates.
8. Interview scoring points
Can separate training and calibration
The candidate should fit the point predictor only on training data, compute scores on held-out calibration data, and use a finite-sample quantile.
Can state coverage assumptions
They should distinguish exchangeability, marginal coverage, and conditional coverage, without presenting the result as a deterministic guarantee for each point.
Can handle drift and temporal dependence
They should propose rolling, weighted, or online calibration and explain the extra assumptions and changed guarantees.
Can balance interval width
They should report coverage and width together, monitor subgroup failures, and connect statistical metrics to business loss.