Class imbalance is when one class holds far more training examples than another — 950 legitimate transactions against 50 fraudulent ones, for instance. A model trained on that data can predict the majority class every single time and still report 95% accuracy, while catching none of the fraud. The imbalance is not itself an error; it reflects reality, because rare events are rare. The error is evaluating the model with a metric that cannot see the failure, and training it with a loss function that has no reason to care.
The fix is rarely one technique. It is a decision about which errors the business can absorb, expressed through the metric, the sampling strategy and the decision threshold together.
What problem does this solve?
A fraud team ships a model reporting 95% accuracy. In the first month it flags nothing. The distribution was 950 legitimate to 50 fraudulent, and predicting 'legitimate' every time scores exactly 95%. The number was never measuring detection.
The same shape appears wherever the interesting event is rare: a defect on a production line running at 99.2% yield, a security event in a week of clean CCTV, a disease present in 3% of screened patients. In each case the majority class is large enough to dominate the loss function, so the cheapest way for the model to reduce error is to ignore the minority entirely.
How the solution works
Change the metric first, before touching the data. Precision, recall, F1 and PR-AUC all remain informative when one class is rare; accuracy does not. This step costs nothing and often reveals that the model was never as good as reported.
Then decide which error is more expensive. A missed fraud case and a falsely flagged customer are not equivalent costs, and the decision threshold — not the model — is where that trade-off is set.
Only then consider resampling. Oversampling the minority, undersampling the majority, or synthesising new minority examples with SMOTE or ADASYN each change the training distribution in different ways, and each carries its own failure mode.
- 1Measure the distribution Count examples per class, per split. An imbalance that exists in training but not in test, or vice versa, is a different and more serious problem.
- 2Establish a baseline Score a model that always predicts the majority class. Any real model must beat this on recall and PR-AUC, not on accuracy.
- 3Choose the operating metric Pick the metric that matches the decision being automated, and agree it with the business owner before tuning begins.
- 4Adjust training Apply class weights, resampling or a cost-sensitive loss. Class weighting is the least invasive and should be tried first.
- 5Tune the threshold Sweep the decision threshold across the validation set and select the point that matches the agreed cost of each error type.
Reference architecture
Imbalance is handled at four independent points, and confusing them is the most common source of wasted effort.
| Layer | What it contains |
|---|---|
| Data layer | Class distribution, label quality, and whether the rare class is rare in reality or only rare in this sample. |
| Training layer | Class weights, resampling and cost-sensitive loss functions — all applied to the training split alone. |
| Evaluation layer | Precision, recall, F1, PR-AUC and the confusion matrix, computed on an untouched test set with the original distribution. |
| Decision layer | The threshold that converts a score into an action, set against the business cost of each error type. |
Deployment options: On-premise, edge or private cloud according to data residency. Fraud, health and government screening data frequently cannot leave the entity's boundary, which constrains where training and monitoring can run.
Key capabilities
Distribution and baseline audit
A documented class distribution per split and a majority-class baseline that any model must beat.
availableMetric selection workshop
An agreed operating metric and error cost signed off by the business owner, not chosen by the data team alone.
availableResampling pipeline design
Resampling applied inside cross-validation folds so the reported score is not inflated by leakage.
custom developmentThreshold and cost modelling
A threshold chosen from a measured precision-recall curve against the real cost of a miss and a false alarm.
custom developmentIntegrations
Imbalance handling is part of the training pipeline, not a separate product. It has to survive retraining, which means the sampling strategy belongs in versioned code rather than in a notebook someone ran once.
| System | Integration point & data exchanged | Direction |
|---|---|---|
| Feature store or data warehouse | Class distribution is monitored over time; a shifting base rate silently invalidates a tuned threshold. | bi-directional |
| Training pipeline | Resampling runs inside the cross-validation fold, never before the split. → Data Leakage: The Model That Only Works Before Deployment | bi-directional |
| Model monitoring | Recall and precision are tracked per class in production, not just aggregate accuracy. | bi-directional |
Industry use cases
Banking and payments
Fraud is a fraction of a percent of transactions; recall and the cost of a false decline drive the threshold.
Healthcare screening
A missed positive is far more costly than a recalled patient, so the threshold is deliberately set toward high recall.
Industrial quality inspection
On a line running at high yield, defect examples accumulate slowly, and the training set has to be built deliberately rather than sampled.
Security and CCTV analytics
Genuine incidents are rare against hours of ordinary footage, and a model tuned on accuracy will report an empty week as a success.
UAE & GCC considerations
For UAE and GCC deployments, the constraint is usually data residency rather than modelling. Fraud, patient and government screening data often cannot leave the entity's environment, which rules out several managed AutoML services and pushes training on-premise or into an approved regional private cloud. Confirm the residency position, the approved deployment zone, Arabic and English operating requirements, and who owns the retraining schedule before selecting a platform.
Implementation approach
- 1Agree the cost Establish with the business owner what a missed positive and a false alarm each cost. Without this, threshold tuning has no objective.
- 2Fix the metric Replace accuracy in every report and dashboard. Leaving it visible guarantees someone will quote it.
- 3Try class weighting Start with class_weight='balanced' or an equivalent. It is one parameter and often closes most of the gap.
- 4Resample if needed Add SMOTE, ADASYN or undersampling inside the cross-validation fold, and compare against the weighted baseline honestly.
- 5Tune and monitor Select the threshold from the precision-recall curve, then monitor the base rate and per-class recall after deployment.
Security & deployment
Training data for fraud, health and security models is among the most sensitive an organisation holds. Keep it inside the approved boundary, use least-privilege access to the training environment, and treat synthetic minority examples as derived personal data rather than as anonymous — SMOTE interpolates between real records and can carry identifying structure.
A worked example
Take 1,000 transactions: 950 legitimate, 50 fraudulent. A model that always answers 'legitimate' produces the following.
- Accuracy. 950 correct out of 1,000 = 95.0%. This is the number that reaches the steering committee.
- Recall on fraud. 0 of the 50 fraud cases detected = 0.0%. This is the number that matters.
- Precision on fraud. Undefined — the model never predicts fraud, so there are no positive predictions to be correct about.
- F1 on fraud. 0.0, because it is the harmonic mean of precision and recall and recall is zero.
Two metrics on the same predictions: 95% and 0%. Only one of them describes whether the system does its job. This gap is the accuracy paradox, and it is why the metric decision comes before the modelling decision.
In code
The critical detail is the split ordering. Resampling before splitting copies or synthesises minority examples that then appear on both sides of the split, which inflates the test score and hides the problem instead of fixing it.
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report, average_precision_score
from imblearn.over_sampling import SMOTE
# 1,000 rows, 5% minority - roughly the fraud shape described above.
X, y = make_classification(
n_samples=1000, n_features=10, n_informative=4,
weights=[0.95, 0.05], random_state=42,
)
# Split FIRST. Everything that follows touches training data only.
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.25, stratify=y, random_state=42,
)
# Baseline: no resampling, but tell the loss function the classes differ.
baseline = LogisticRegression(max_iter=1000, class_weight="balanced")
baseline.fit(X_train, y_train)
# SMOTE applied to the TRAINING SPLIT ONLY - never before the split.
X_res, y_res = SMOTE(random_state=42).fit_resample(X_train, y_train)
resampled = LogisticRegression(max_iter=1000)
resampled.fit(X_res, y_res)
for name, model in [("class_weight", baseline), ("smote", resampled)]:
proba = model.predict_proba(X_test)[:, 1]
print(name, "PR-AUC:", round(average_precision_score(y_test, proba), 3))
print(classification_report(y_test, model.predict(X_test), digits=3))Both models report far lower accuracy than a majority-class predictor and far higher recall on the minority class. Compare them on PR-AUC rather than accuracy; on a 5% minority the ranking between the two approaches often changes with the random seed, which is itself a useful signal that the difference is not yet meaningful.
Diagnostic checks
- Compute the majority-class baseline. If the model's accuracy is close to it, the model may have learned nothing.
- Read the confusion matrix, not the summary score. A column of zeros for the minority class is unambiguous.
- Check whether resampling happened before or after the train/test split. Before is leakage and the test score is invalid.
- Compare PR-AUC against ROC-AUC. ROC-AUC can look strong on heavily imbalanced data while precision at the operating threshold is unusable.
- Verify the class distribution in production matches the distribution the threshold was tuned on.
When to use it
- The minority class is genuinely rare and genuinely important — fraud, defects, disease, intrusions.
- The cost of a missed positive clearly exceeds the cost of a false alarm.
- You have enough minority examples to represent the class; resampling amplifies what is there, it does not create new information.
- The base rate is stable enough that a tuned threshold will still be valid next quarter.
When not to use it
- The imbalance is mild. Below roughly a 1:4 ratio, class weighting and a sensible threshold are usually sufficient and resampling adds risk without benefit.
- The minority class is rare because of a labelling failure rather than reality — fix the labels first, because resampling will faithfully amplify the error.
- Only a handful of minority examples exist. Synthetic oversampling will interpolate between too few points and produce examples that do not resemble the real class.
- The deployment decision does not depend on the rare class at all, in which case the imbalance may simply not matter.
Limitations & prerequisites
- Resampling changes the training distribution but not the information content; it cannot manufacture signal that the features do not carry.
- SMOTE interpolates between existing minority points and can create implausible examples in categorical or high-dimensional feature spaces.
- Undersampling discards majority examples and with them potentially useful information about the decision boundary.
- A threshold tuned on one base rate becomes wrong when the base rate shifts, which it usually does.
Choosing a response to imbalance
These options are not ranked. The right one depends on how rare the minority is, how many examples exist, and what the errors cost.
| Approach | Best suited to | Main risk |
|---|---|---|
| Class weighting | First attempt in almost every case | Can destabilise training at extreme ratios |
| Random oversampling | Small datasets, simple models | Duplicates examples and encourages overfitting |
| Random undersampling | Very large majority class | Discards information about the boundary |
| SMOTE | Continuous features, moderate minority count | Implausible synthetic points in categorical space |
| ADASYN | Minority examples near the decision boundary | Amplifies noise and mislabelled points |
| Threshold tuning | Every deployment, always | Invalid once the base rate shifts |
In practice the sequence is: fix the metric, weight the classes, tune the threshold, and only then reach for synthetic resampling.
Key takeaways
- Accuracy on an imbalanced dataset measures the base rate, not the model.
- Change the metric before changing the data — it is free and often decisive.
- Class weighting is the first intervention; synthetic resampling is the third.
- Apply any resampling after the train/test split, inside the fold, or the score is invalid.
- The decision threshold is where the business cost of each error is actually expressed.
FAQ
There is no fixed threshold. Around 1:4 most models cope with class weighting alone; by 1:100 accuracy is meaningless and resampling or cost-sensitive training is usually needed. The ratio matters less than whether the minority class has enough examples to be learnable.
No. Start with class weighting, which is simpler and cannot leak. Reach for SMOTE when weighting is insufficient and the features are continuous enough for interpolation between minority points to be meaningful.
The model is predicting the majority class for every input. Accuracy rewards this because the majority dominates the count. Read the confusion matrix and the per-class recall instead.
Only alongside per-class metrics, and preferably not in reporting at all. Once accuracy appears on a dashboard, someone will quote it in isolation.
Only if the additional data contains more minority examples. Collecting ten times as many majority records makes the ratio worse, not better.
Inside the training fold, after the split. Resampling the full dataset before splitting places synthetic copies of training points into the test set, which produces an inflated score that will not survive production.
Plot the precision-recall curve on validation data, attach a cost to each error type, and select the point that minimises expected cost. The default of 0.5 is a convention, not a recommendation.
Is your model reporting accuracy that nobody trusts?
Send the class distribution, the current metric and one example of a miss that mattered. We will identify whether the problem is sampling, threshold, features or labels before anyone retrains anything.
Request an AI model assessmentSources & evidence
- imbalanced-learn user guide — Official documentation for resampling methods including SMOTE and ADASYN.
- scikit-learn: class imbalance and metrics — Official reference for precision, recall, F1 and average precision.
- Chawla et al., SMOTE (JAIR 2002) — The original synthetic minority oversampling paper.
- NIST AI Risk Management Framework — Governance context for measurement and validity of AI systems.
Vendor and product names are trademarks of their respective owners; references are for technical context and do not imply partnership, certification or endorsement unless stated on the vendor's official pages.