Data Science Interview: How Do You Validate a Time-Series Forecasting Model Without Leakage?

Question and Applicable Scenarios

You have two years of daily ride demand for 200 cities. The production job retrains every Monday and forecasts the next 7 days for each city. Features include past demand, day of week, holidays, weather forecasts, and planned price promotions. The team proposes a random train-validation split and wants to select the model with the lowest aggregate MAPE.

Design leakage-free offline validation. Cover splitting, feature availability, baselines, metrics, aggregation across cities, model selection, and launch gates. Two years, 200 cities, 7 days, and weekly retraining are interview assumptions, not industry benchmarks.

This question applies to data science, machine learning, and forecasting roles. Its core is reproducing what was genuinely known at each historical forecast origin, so the category is data.

What Interviewers Evaluate

First, can the candidate translate the production forecasting process into an evaluation protocol? A strong answer fixes forecast origins, horizon, retraining cadence, and training-window policy before discussing models. “Split chronologically” alone is incomplete.

Second, can the candidate find leakage outside the split? Future information can enter through full-data scaling, rolling windows that cross the origin, revised data, realized weather, promotions not yet approved, or target encodings fitted on all dates.

Third, do the metrics correspond to decisions? One aggregate can hide distant-horizon failure, weak low-volume cities, persistent bias, or miscalibrated intervals. A strong answer retains results by horizon, city segment, and temporal fold, with simple baselines beside the model.

Fourth, can the candidate separate tuning from final estimation? Rolling validation selects models and thresholds. A final contiguous period that never influenced selection estimates the chosen pipeline's performance. Repeatedly inspecting that holdout while changing the model turns it into another validation set.

Questions to Clarify Before Answering

• What is the actual forecast origin? A Monday 06:00 run must freeze data at that time. Daily rolling forecasts create different origins and retraining costs.
• Is the 7-day path produced directly or recursively one day at a time? The strategy changes feature generation and requires results for horizons 1 through 7 separately.
• Which future covariates are known at the origin? Calendars are normally known; a weather forecast is available, but realized weather is not; only approved and published promotions qualify.
• When are labels complete? If ride counts arrive two days late, training needs an equivalent gap or point-in-time snapshots.
• Are overforecasting and underforecasting equally costly? Capacity planning can have asymmetric shortage and idle-capacity costs that MAE alone cannot express.
• Are cities equally important? An equal-city macro average measures coverage; volume weighting measures aggregate impact. Neither replaces the other.
• How much history does production use? A fixed window can adapt to structural change, while stable annual seasonality may require a longer history.

30-Second Answer Framework

“I would work backward from production and run a rolling-origin backtest. At each historical Monday, I would use only data available and complete at that time, fit with the production window, forecast the next 7 days, and roll forward. Every transform, lag feature, and tuning decision is fitted inside the training fold; realized future weather cannot replace its forecast. I would compare against a seasonal-naive baseline and report MAE, bias, and business loss by horizons 1 through 7, city, and temporal fold; a percentage metric needs an explicit zero policy. After selection, I would evaluate once on an untouched contiguous holdout and launch only if critical cities, peak weeks, and interval coverage pass predefined gates.”

Step-by-Step Deep Dive

Step 1: Treat one production run as one backtest fold.

For forecast origin t, training may contain only records available at t with complete labels. The test interval is t+1 through t+7. Move the origin forward by 7 days to simulate weekly retraining. Skip early origins that do not contain enough history for the seasonal patterns the pipeline requires.

An expanding window uses all history through t, which improves data use but retains old regimes. A fixed window adapts faster but may discard annual seasonality. The backtest must reproduce the intended production policy; choosing a policy after observing the final holdout contaminates that holdout.

Step 2: Define a feature-availability contract.

For each feature, record event time, system availability time, revision policy, and missing-value behavior. Reconstruct the snapshot available at each origin:

• a one-day lag comes from t or earlier, and a 7-day rolling mean cannot cross t;
• scaling, imputation, encoding, feature selection, and target transformations fit only on that fold's training data;
• use the weather forecast published by t, not later realized weather;
• future promotion features include only plans confirmed by t;
• if labels arrive two days late, end training at t-2 or apply an equivalent gap.

Executable checks should assert available_at <= t for every training cell, regenerate features after deleting all raw records after the origin, and replay several origins from saved snapshots. A historical forecast that changes when unavailable future data is removed reveals leakage or an irreproducible dependency.

Step 3: Establish baselines that are hard to game.

At minimum, compare a seasonal-naive forecast that uses the same weekday from the previous week. Add a prior-year baseline if annual seasonality is stable enough. A complex model earns its cost only when it wins consistently on identical origins, available features, and scored rows. An implausibly large gain should trigger a time-alignment and leakage audit before celebration.

Step 4: Match metrics to failure costs.

Use MAE for typical absolute error, RMSE to expose large misses, and mean signed error to reveal persistent over- or underforecasting. MAPE is undefined at zero and unstable near zero, so it cannot be the only metric. For cross-scale comparison, MASE can scale error by a seasonal-naive error computed from the training fold. WAPE can support aggregate planning, but high-volume cities dominate it.

For quantile forecasts, score each quantile with pinball loss and compare empirical coverage with the intended level, such as P90. Coverage must be paired with interval width: an extremely wide interval can cover well while being useless for a capacity decision.

Step 5: Preserve the structure of error before aggregating.

Keep origin, horizon, city, actual, and forecast on every scored row, then report:

1. horizons 1 through 7 separately, so near-term accuracy cannot hide distant-horizon collapse;
2. both an equal-city macro average and a volume-weighted result;
3. the distribution across temporal folds, not only its mean;
4. peak periods, holidays, low-volume cities, new cities, and unusual-weather windows separately;
5. runs of same-direction bias within a city.

Step 6: Select inside rolling folds and lock the final test.

Use development rolling folds for models, windows, and hyperparameters. Record experiments when comparison count is large so repeated trials do not silently optimize noise in a few folds. Freeze the complete pipeline, then evaluate once on the final contiguous 8 to 12 weeks. That duration is another interview assumption and should change with seasonality and sample needs.

Predefine launch gates: aggregate business loss beats seasonal naive; critical cities do not exceed a regression threshold; horizon 7 remains acceptable; bias stays within capacity tolerance; and interval coverage and width pass. If the average wins while a critical gate fails, do not launch globally—canary only the cities that pass or keep the baseline.

Step 7: Extend the protocol into production monitoring.

Log model version, data snapshot, forecast origin, horizon predictions, and feature versions. When labels mature, backfill the same metrics and compare them with backtest distributions. Monitor data availability, missingness, signed bias, error by horizon, and delta from the baseline. After a regime change, reassess the training-window policy instead of assuming frequent retraining fixes the protocol.

High-Quality Sample Answer

“I would reproduce one production run at a series of historical origins. If the job retrains Monday at 06:00 and emits seven days at once, each fold sees only data available with complete labels at that time, trains with the production window, and scores the following seven days. If labels lag by two days, training ends two days before the origin. Weather uses the forecast version available then, never the realized observation.

All transforms live inside the fold. Scalers, imputers, encoders, and feature selection fit only on training rows, while lags and rolling windows must end no later than the origin. I would also delete post-origin raw data and regenerate features to verify that the historical prediction does not change.

The first comparator is last week's same-weekday seasonal-naive forecast. I retain results by city, origin, and horizon, then report MAE, RMSE, signed bias, and business cost. Cities get both equal weighting and volume weighting. MAPE fails on zeros, so I would not use it alone. For quantiles, I would add pinball loss, coverage by horizon, and interval width.

Rolling folds select the pipeline; a final contiguous period is used once. Launch gates are fixed before that test: beat the baseline overall, avoid unacceptable critical-city regressions, pass horizon 7 and peak weeks, and meet bias and interval-calibration requirements. In production, I log origins and data versions so matured labels can reproduce the same slices. That makes the offline gain comparable with the system we will actually run.”

Common Mistakes

• Randomly shuffling dates → Training sees mechanisms and feature statistics from after the test date → Use rolling origins that reproduce production runs.
• Generating features and scaling on the full dataset → Validation information has entered training → Fit every transform per fold and replay features by availability time.
• Replacing a weather forecast with realized weather → Offline evaluation has information production lacked → Store and use the forecast vintage available at each origin.
• Reporting one aggregate MAPE → Zeros, small cities, and distant horizons are mishandled or hidden → Combine absolute error, bias, business loss, and segmented results.
• Omitting a seasonal-naive baseline → Model complexity has no credible incremental benchmark → Score the baseline on identical folds and rows.
• Changing the model after viewing the final test → The holdout now participates in selection → Freeze once; after failure, wait for a new future holdout.
• Launching globally because the average wins → High-volume cities can hide critical subgroup regressions → Predefine subgroup, peak, and horizon gates and canary by city.

Follow-Up Questions and Responses

How would you evaluate a city that has never appeared in training?

Ordinary rolling folds place the same cities in train and test, so they cannot estimate cold start. Add a city-held-out evaluation: remove a group of cities completely while training a shared model, then use only static attributes or short history genuinely available at launch. Report zero-history and short-history cases separately against regional and global naive baselines.

A promotion affects 14 days while the forecast horizon is 7 days. Do you need a gap?

The gap follows information availability and overlapping labels, not the horizon mechanically. If a training label or aggregate consumes outcomes from 14 days after the origin, truncate it or leave sufficient separation. If the promotion plan was confirmed before the origin and the feature contains only that plan, its 14-day duration does not itself create leakage. Draw a timeline for each field.

What if the new model improves only high-volume cities?

Show the volume-weighted benefit beside the equal-city regression and convert business requirements into gates. Launching the new model only for high-volume cities while retaining a baseline or hierarchical model elsewhere may be valid. A single weighted average does not establish that every city benefits.

Online error is much worse than the backtest. What do you inspect first?

Start with reproducibility: can the exact model, origin, feature snapshot, and exogenous-variable versions rebuild the prediction? Then separate data delay or definition changes, training-serving transformation differences, regime shifts that also hurt the baseline, and model-specific drift. Locate the protocol mismatch before choosing retraining, a shorter window, rollback, or a new validation design.