Random Oversampling: Duplication as a Deliberate Choice

What problem does this solve?

A motor insurer holds 84,000 settled claims. Special investigations confirmed 290 as fraudulent — roughly one in every 290 claims. A gradient-boosted baseline reports high accuracy and refers almost nothing, because predicting 'legitimate' for every claim is the cheapest way to minimise the loss.

The team has the features they would expect: time between policy inception and first notification, whether the same repair garage appears across multiple claims, mismatch between reported impact point and the invoiced panel work, number of occupants claiming injury, and whether the assessor's photographs were taken at the garage rather than the scene.

None of that helps while the training signal is dominated by 83,710 legitimate claims. The 290 fraud records contribute so little to the aggregate loss that the model can safely disregard them, and the fields above are never exercised.

How the solution works

Random oversampling changes the arithmetic rather than the data. Duplicating the 290 confirmed cases until they represent, say, one in four of the training fold means the loss function now pays a real penalty for missing them.

Because it duplicates rather than interpolates, it is safe on the categorical fields this dataset is full of — garage identifier, policy type, region, impact point. No synthetic method can average two garage codes into something meaningful.

The cost is duplicate exposure. Those 290 claims appear many times each, so a flexible model can fit them individually rather than learning the pattern they share. Depth limits, early stopping and honest cross-validation are what keep that in check.

The second cost is calibration. After oversampling, the model's predicted probabilities reflect the resampled ratio, not the real one in 290. Any downstream process that treats the score as a probability of fraud will be wrong unless the scores are recalibrated.

1. 1
   Split first Create the training and validation folds before any resampling, so no duplicated row appears on both sides.
2. 2
   Choose a target ratio Decide the minority proportion for the training fold. Full balance is a default, not a recommendation.
3. 3
   Sample with replacement Draw minority rows at random, with replacement, until the target count is reached. Some rows will be drawn many times, others once.
4. 4
   Train on the enlarged fold The classifier now sees the minority class often enough for it to affect the gradient.
5. 5
   Recalibrate and threshold Map the model's scores back to real-world probabilities, then choose the operating threshold against the cost of a referral versus a missed fraud.

Reference architecture

Four layers decide whether duplication helps or simply produces a memorised model.

Deployment options: Claims data is personal data and usually policyholder-identifiable. Resampling produces additional copies of real records rather than synthetic ones, so the training set inherits the source classification without dilution.

Key capabilities

Integrations

Oversampling lives in the training pipeline, but its consequences surface in the claims workflow, where scores become referrals and referrals become workload.

Industry use cases

Implementation approach

1. 1
   Count the confirmed set Establish how many adjudicated fraud cases exist and how they were confirmed. Below roughly a hundred distinct cases, sampling is not the binding constraint.
2. 2
   Compute the duplication factor Derive it from the target ratio and inspect it. A factor above about fifty warrants a milder target.
3. 3
   Constrain the model Cap depth, set a minimum leaf size and enable early stopping before increasing the ratio.
4. 4
   Recalibrate Fit isotonic or Platt calibration on untouched data so scores can be read as probabilities.
5. 5
   Set the threshold on capacity Choose the operating point from the number of referrals investigators can actually process.

Security & deployment

Random oversampling multiplies real policyholder records inside the training set. Unlike synthetic methods there is no anonymising step at all, so the resampled dataset carries exactly the same personal-data classification and access controls as the source. Keep the resampled fold inside the training environment, never export it for benchmarking, and delete it with the training artefacts rather than retaining it as a convenience dataset.

A worked example

Working the arithmetic on one training fold of the claims dataset — 67,200 legitimate and 232 confirmed fraud after an 80/20 split.

1. Target one in four. With p = 0.25 and N_maj = 67,200, the required fraud count is 0.25 * 67,200 / 0.75 = 22,400.
2. Duplication factor. 22,400 drawn from 232 distinct claims means each real claim appears about 97 times on average.
3. What the model sees. A single staged-collision claim with an unusual garage code now contributes roughly 97 times its original weight to the gradient.
4. Effect on scores. The model's average predicted probability rises toward 0.25, though the true base rate remains 0.0035.

Ninety-seven copies of one claim is the whole technique in a single number. It explains both why recall improves — the class is finally visible — and why a deep unconstrained model will overfit those 232 specific claims rather than the behaviour they represent. A milder target such as p = 0.10 gives a duplication factor near 30 and is usually the better starting point.

In code

Two things matter in the code: the sampler sits inside the fold, and the raw scores are recalibrated before anyone treats them as probabilities.

import numpy as np
from sklearn.model_selection import StratifiedKFold, train_test_split
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.calibration import CalibratedClassifierCV
from sklearn.metrics import average_precision_score, brier_score_loss
from imblearn.over_sampling import RandomOverSampler
from imblearn.pipeline import Pipeline

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=11,
)

# sampling_strategy is the minority:majority ratio AFTER resampling.
# 0.33 -> one minority row for every three majority rows.
pipe = Pipeline([
    ("ros", RandomOverSampler(sampling_strategy=0.33, random_state=11)),
    ("clf", HistGradientBoostingClassifier(
        max_depth=4, early_stopping=True, random_state=11)),
])
pipe.fit(X_train, y_train)
raw = pipe.predict_proba(X_test)[:, 1]

# Oversampling shifts the score distribution away from the true base rate.
# Calibrate on data the sampler never touched.
calibrated = CalibratedClassifierCV(pipe, method="isotonic", cv=5)
calibrated.fit(X_train, y_train)
cal = calibrated.predict_proba(X_test)[:, 1]

print("base rate:      ", round(y_test.mean(), 5))
print("mean raw score: ", round(raw.mean(), 4))
print("mean cal score: ", round(cal.mean(), 4))
print("PR-AUC raw:     ", round(average_precision_score(y_test, raw), 3))
print("Brier raw:      ", round(brier_score_loss(y_test, raw), 5))
print("Brier calibrated:", round(brier_score_loss(y_test, cal), 5))

# Referral capacity is finite: pick the threshold from the budget, not from 0.5.
capacity = 400
threshold = np.sort(cal)[-capacity]
print("threshold for", capacity, "referrals:", round(threshold, 4))

The mean raw score sits far above the true base rate while the calibrated mean sits close to it, and the Brier score improves after calibration even when PR-AUC is unchanged. Ranking quality and probability quality are separate properties, and oversampling damages only the second.

Diagnostic checks

• Compute the duplication factor explicitly. A single claim appearing hundreds of times is a memorisation risk, not a modelling strategy.
• Compare the mean predicted score against the true base rate. A large gap confirms the calibration distortion oversampling introduces.
• Check the Brier score before and after calibration; PR-AUC alone will not reveal the problem.
• Verify no duplicated row appears in both the training and validation folds.
• Inspect the highest-importance features. If a garage identifier or claim reference dominates, the model has learned specific claims rather than behaviour.
• Re-run with several random seeds. Large variance indicates the result depends on which few claims were drawn most often.

Limitations & prerequisites

• It adds no information; the model still learns from the same 290 distinct claims regardless of how often they are copied.
• Predicted probabilities no longer reflect the real base rate without recalibration.
• Training time grows with the resampled fold, which matters when the majority class is large.
• Flexible models fit duplicated rows individually, producing validation scores that will not survive contact with new claims.

Random oversampling among the alternatives

The choice is driven by feature types and minority size more than by sophistication.

For a categorical-heavy claims table with a few hundred confirmed cases, random oversampling with a constrained model and explicit recalibration is a defensible first choice.

Key takeaways

• Random oversampling copies minority rows; it changes exposure, never information content.
• The duplication factor follows from the target ratio and should be computed before training, not inferred afterwards.
• It is the safe choice when features are categorical, because nothing is interpolated.
• Predicted probabilities shift away from the true base rate and must be recalibrated.
• Constrain model capacity, or a few hundred claims will be memorised rather than generalised.

FAQ

Related topics & solutions

• Class Imbalance: When 95% Accuracy Means the Model Found Nothing
• SMOTE: Inventing Minority Examples Without Copying Them
• ADASYN: Generating Where the Model Is Actually Struggling
• Random Undersampling: Throwing Away Data on Purpose
• AI Products Portfolio
• Missing Values: Why the Gap Itself Carries Information
• Selection Bias: When Your Data Only Contains the People Who Showed Up
• Outliers: Telling a Broken Sensor From a Broken Chiller