Data Leakage: The Model That Only Works Before Deployment

What problem does this solve?

A utility builds a pump-failure model on two years of sensor history. Cross-validated ROC-AUC comes out at 0.98. The team ships it. Over the following quarter it predicts almost nothing useful, and the failures it does flag are ones the maintenance planners had already scheduled.

Two things went wrong. The rows were shuffled randomly before splitting, so readings from ten minutes after a failure sat in the training set while readings from ten minutes before sat in the test set — the model was effectively interpolating within a single event it had already seen.

Second, one feature was `work_order_raised`, populated by the maintenance system. It is present in the historical table and it is a consequence of the failure, not a predictor of it. At prediction time it is always empty. The model had learned to read the answer.

How the solution works

Ask one question of every feature: would this value exist, with this value, at the moment the prediction has to be made? If the answer is no, or if it is populated by a process that happens after the event, it leaks.

Split by time for anything with a temporal ordering. Training on the past and testing on the future is the only split that reflects how the model will be used.

Split by group where records share an entity. All readings from one pump, all images of one patient, all transactions from one account must fall entirely on one side of the split.

Fit every transformation inside the fold. Scalers, imputers, encoders and resamplers must learn their parameters from training data only, or statistics from the test set flow backwards into training.

1. 1
   Information enters the feature set A column derived from the outcome, or computed using the whole dataset, is included among the inputs.
2. 2
   The model finds it Optimisation exploits the strongest available signal, and a leaked feature is by construction the strongest.
3. 3
   Validation confirms it The validation split carries the same contamination, so the score rises rather than falling.
4. 4
   Nobody questions the number A high score is reported as success. This is the point at which leakage should have been suspected.
5. 5
   Production removes it At inference the leaked field is empty or the future is genuinely unknown, and performance collapses to the model's real ability.

Reference architecture

Leakage enters at four points. Each is prevented by a different discipline, and a pipeline that guards only one of them is not protected.

Deployment options: The strongest structural defence is a feature store with explicit point-in-time semantics, so a feature's value is always retrieved as of the prediction timestamp rather than as of today.

Key capabilities

Integrations

Leakage is prevented in the data platform more than in the model code, because the guarantees have to hold across every retrain.

Industry use cases

Implementation approach

1. 1
   List and interrogate features For each, establish whether it exists at prediction time and what process populates it.
2. 2
   Choose the split deliberately Time-based for temporal problems, grouped for repeated entities, both where both apply.
3. 3
   Move everything into the pipeline No fitting of any transformation outside a fold, including scaling, imputation and resampling.
4. 4
   Screen for suspicious features Score each feature alone; anything near-perfect on its own is a leakage candidate.
5. 5
   Re-baseline the expectation Report the honest score to stakeholders before the previous number becomes the target.

Security & deployment

Removing a leaked feature can change which data the model needs, and occasionally reduces the sensitivity of the training set — a useful side effect worth recording. Keep the feature availability audit as a governance artefact: when a regulator or an internal auditor asks why a model performs differently in production than in validation, that document is the answer.

A worked example

The same pump dataset evaluated three ways, showing how much of the score was leakage.

1. Random split, all features. ROC-AUC 0.98. Rows from the same failure event appear on both sides, and `work_order_raised` is included.
2. Random split, leaked column removed. ROC-AUC 0.91. Removing the target-derived feature costs seven points.
3. Time-based split, leaked column removed. ROC-AUC 0.74. Training on the first eighteen months and testing on the last six removes the temporal advantage.
4. Time-based split, grouped by pump. ROC-AUC 0.71. No pump appears in both training and test.

The honest figure is 0.71, not 0.98. Twenty-seven points of apparent performance were leakage. A model at 0.71 may still be commercially worthwhile — but only the 0.71 version would have survived deployment, and only that number should have gone to the steering committee.

In code

Two habits prevent most leakage: put every transformation inside a Pipeline, and use a splitter that respects time and groups.

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.impute import SimpleImputer
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import TimeSeriesSplit, GroupKFold, cross_val_score

# WRONG - the scaler sees the whole dataset, including the test fold.
# X_scaled = StandardScaler().fit_transform(X)
# scores = cross_val_score(model, X_scaled, y, cv=5)

# RIGHT - every transformation is refitted inside each training fold.
pipe = Pipeline([
    ("impute", SimpleImputer(strategy="median")),
    ("scale", StandardScaler()),
    ("clf", GradientBoostingClassifier(random_state=0)),
])

# Temporal data: never shuffle. Train on the past, test on the future.
ts = TimeSeriesSplit(n_splits=5)
print("time-aware:", cross_val_score(pipe, X, y, cv=ts, scoring="roc_auc").mean().round(3))

# Repeated entities: keep every pump entirely on one side of the split.
gk = GroupKFold(n_splits=5)
print("grouped:", cross_val_score(pipe, X, y, cv=gk, groups=pump_id, scoring="roc_auc").mean().round(3))

# Cheap leakage screen: a single feature that alone predicts almost perfectly.
from sklearn.tree import DecisionTreeClassifier
for col in X.columns:
    auc = cross_val_score(DecisionTreeClassifier(max_depth=1),
                          X[[col]], y, cv=ts, scoring="roc_auc").mean()
    if auc > 0.90:
        print(f"SUSPECT: {col} alone reaches AUC {auc:.3f}")

Two honest cross-validation scores and a list of individually suspicious columns. Any single feature that reaches AUC above about 0.90 on its own is almost always derived from the target rather than predictive of it.

Diagnostic checks

• Score each feature individually. A single column reaching AUC above 0.90 alone is almost always derived from the target.
• Compare random-split against time-based-split performance. A large gap is temporal leakage.
• Check whether any entity identifier appears in both training and test sets.
• Confirm every scaler, imputer and encoder is fitted inside the fold rather than on the full dataset.
• Ask when each feature is physically written. Anything populated by a downstream process after the event leaks by definition.
• Verify that feature selection and hyperparameter search did not see the final test set.

Limitations & prerequisites

• Not all leakage is detectable statistically; some requires domain knowledge of how a field is populated.
• Time-based splitting reduces the effective training set and usually lowers the reported score, which can be politically difficult after a high number has circulated.
• Grouped splitting can leave too few groups for a stable estimate on small datasets.
• Point-in-time correctness is expensive to retrofit onto a warehouse that was not designed for it.

The four leakage types

They present identically as an inflated score. The diagnostic and the fix differ.

A pipeline can be guarded against one type and wide open to another; check all four rather than assuming a Pipeline object covers it.

Key takeaways

• Leakage means training on information that will not exist at prediction time.
• It inflates the validation score because the validation set carries the same contamination.
• There are four distinct types and a Pipeline object only guards against one of them.
• Split by time when data is ordered and by group when entities repeat.
• An unexpectedly high score is a reason to investigate, not to ship.

FAQ

Related topics & solutions

• Class Imbalance: When 95% Accuracy Means the Model Found Nothing
• SMOTE: Inventing Minority Examples Without Copying Them
• Duplicate Data: When One Patient Becomes Four Records
• AI Products Portfolio
• Random Undersampling: Throwing Away Data on Purpose
• Selection Bias: When Your Data Only Contains the People Who Showed Up
• ADASYN: Generating Where the Model Is Actually Struggling
• Random Oversampling: Duplication as a Deliberate Choice