Random Undersampling: Throwing Away Data on Purpose

What problem does this solve?

A managed SOC ingests roughly 41 million events across a rolling ninety days. After correlation, 2.6 million become alerts. Of those, 1,180 were confirmed as genuine incidents by tier-two analysts. The ratio at alert level is close to one in 2,200.

Training a classifier to rank alerts for triage runs into two separate walls. The first is computational: fitting on 2.6 million rows for every experiment makes iteration slow enough that the team stops experimenting. The second is statistical: the benign class dominates the loss so completely that the model converges on ranking everything as low priority.

The instinct is to sample the benign alerts down. The danger is that benign is not one thing. It contains nightly backup traffic, scheduled patch deployments, authorised vulnerability scans, service-account activity, VPN reconnections and a long tail of application noise. A uniform random draw takes 99% of everything, including 99% of the only forty examples of an authorised scan — and the model then flags the next authorised scan as an incident.

How the solution works

Stratify the draw. Sample within each benign subtype rather than across the pooled majority class, so rare-but-legitimate behaviours retain proportionally more of their examples than high-volume noise does.

Where no subtype labels exist, cluster the majority class first and sample within clusters. This approximates stratification using structure the data already contains.

Consider ensembles instead of a single draw. EasyEnsemble trains several models, each on a different undersampled subset of the majority class, and combines them — so across the ensemble very little majority data goes entirely unused.

Distinguish informed undersampling from random. Tomek links remove majority points that sit directly against a minority point, cleaning the boundary rather than thinning the class. NearMiss selects majority points by proximity to the minority, which sharpens the boundary but is notably sensitive to noise.

1. 1
   Split before sampling Build the folds first. Undersampling the full dataset before splitting produces an evaluation set that no longer reflects live alert volumes.
2. 2
   Define the majority strata Use the existing alert taxonomy, the detection rule identifier, or a clustering pass where no taxonomy exists.
3. 3
   Allocate per stratum Decide how many rows to retain from each subtype, keeping a floor for low-volume behaviours rather than a flat percentage.
4. 4
   Draw without replacement Sample within each stratum. No row is duplicated; discarded rows are simply absent from this fold.
5. 5
   Evaluate on the original distribution Validation and test sets keep the real one-in-2,200 ratio, so precision at the operating threshold reflects the workload analysts will actually see.

Reference architecture

Undersampling touches four concerns, and only the first is about class ratio.

Deployment options: SOC telemetry frequently contains user identifiers, host names and internal addressing. Undersampling reduces volume but does not de-identify anything, so the retained fold carries the same handling requirements as the full log store.

Key capabilities

Integrations

The model sits between the correlation layer and the analyst queue, so its integration surface is the alerting pipeline and the case management system rather than a data science platform.

Industry use cases

Implementation approach

1. 1
   Inventory the benign classes List the distinct legitimate behaviours in the majority class and their volumes before choosing any sampling rate.
2. 2
   Set floors, then allocate Guarantee a minimum per subtype first, and distribute what remains by volume.
3. 3
   Prefer an ensemble Where compute allows, train several models on different subsets rather than discarding data once for all time.
4. 4
   Evaluate operationally Report incidents caught at a fixed review budget alongside PR-AUC.
5. 5
   Monitor per subtype Track false positives by benign behaviour after deployment; a subtype that was thinned too far shows up here first.

Security & deployment

A sampled training fold drawn from SOC telemetry still contains host names, user identifiers and internal network structure, and is therefore an attractive target in its own right. Keep it inside the same enclave as the log store, apply the same access controls, and set an explicit deletion date — derived training sets are commonly overlooked by retention policies written for source logs. Record which subtypes were thinned, because that record is the first thing to consult when an unexplained false-positive pattern appears months later.

A worked example

One training fold: 944 confirmed incidents and 2.08 million benign alerts, targeting a one-in-four minority proportion.

1. Required majority count. With p = 0.25, retain 944 * 0.75 / 0.25 = 2,832 benign alerts out of 2.08 million.
2. Uniform draw retention rate. 2,832 / 2,080,000 is a retention rate of about 0.14%, applied identically to every benign behaviour.
3. Effect on a high-volume subtype. Nightly backup traffic contributes 610,000 alerts and retains roughly 830. Ample.
4. Effect on a low-volume subtype. Authorised vulnerability scans contribute 40 alerts across the window and retain 0.056 — in practice, zero.
5. Stratified alternative. Apply a floor of 25 rows per subtype first, then distribute the remainder proportionally across the high-volume strata.

The uniform draw eliminates an entire legitimate behaviour from the training fold. The model has then never seen an authorised scan labelled benign, and the first one it meets in production is ranked as a probable incident. The floor costs 25 rows out of 2,832 and prevents exactly that class of false positive.

In code

The comparison worth running is uniform against stratified, measured on subtype coverage rather than on the aggregate score.

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import average_precision_score, precision_recall_curve
from imblearn.under_sampling import RandomUnderSampler, TomekLinks
from imblearn.ensemble import EasyEnsembleClassifier

X_train, X_test, y_train, y_test, sub_train, sub_test = train_test_split(
    X, y, alert_subtype, test_size=0.25, stratify=y, random_state=5,
)

def stratified_undersample(X, y, subtype, target_majority, floor=25, seed=5):
    """Keep a minimum per benign subtype, then distribute the remainder by volume."""
    rng = np.random.default_rng(seed)
    maj = np.where(y == 0)[0]
    groups = pd.Series(subtype).iloc[maj].groupby(lambda i: subtype[maj[i]]).groups
    keep = []
    remaining = target_majority - floor * len(groups)
    total = len(maj)
    for name, idx in groups.items():
        idx = np.asarray(maj)[list(idx)] if False else np.asarray(list(idx))
        share = int(round(remaining * (len(idx) / total))) if remaining > 0 else 0
        n = min(len(idx), floor + max(share, 0))
        keep.extend(rng.choice(idx, size=n, replace=False))
    keep = np.concatenate([np.asarray(keep, dtype=int), np.where(y == 1)[0]])
    return X[keep], y[keep]

n_min = int((y_train == 1).sum())
target_maj = int(n_min * 0.75 / 0.25)

# 1. Uniform random undersampling
Xu, yu = RandomUnderSampler(sampling_strategy=0.33, random_state=5).fit_resample(X_train, y_train)

# 2. Stratified undersampling with a per-subtype floor
Xs, ys = stratified_undersample(X_train, y_train, sub_train, target_maj)

# 3. Ensemble over multiple undersampled subsets - little majority data wasted
ens = EasyEnsembleClassifier(n_estimators=10, random_state=5).fit(X_train, y_train)

for name, (Xf, yf) in [("uniform", (Xu, yu)), ("stratified", (Xs, ys))]:
    clf = RandomForestClassifier(n_estimators=300, random_state=5).fit(Xf, yf)
    p = clf.predict_proba(X_test)[:, 1]
    print(name, "PR-AUC:", round(average_precision_score(y_test, p), 3))

print("easyensemble PR-AUC:",
      round(average_precision_score(y_test, ens.predict_proba(X_test)[:, 1]), 3))

# Operational view: at a fixed analyst budget, how many incidents are caught?
prec, rec, thr = precision_recall_curve(y_test, ens.predict_proba(X_test)[:, 1])
budget = 500
cut = np.sort(ens.predict_proba(X_test)[:, 1])[-budget]
flagged = ens.predict_proba(X_test)[:, 1] >= cut
print(f"at {budget} alerts reviewed: caught {int(y_test[flagged].sum())} of {int(y_test.sum())}")

Three PR-AUC figures and one operational number. The operational line is the one to take to the SOC manager: at a fixed review budget, how many genuine incidents the ranking surfaces. Aggregate PR-AUC often differs little between uniform and stratified, while subtype-level false positives differ substantially.

Diagnostic checks

• Count retained rows per benign subtype. Any subtype reduced to zero is a false-positive source waiting to activate.
• Compare false-positive rates per subtype between uniform and stratified sampling; the aggregate score will hide the difference.
• Confirm the validation and test sets retain the original ratio rather than the undersampled one.
• Re-run with several seeds. If PR-AUC swings noticeably, too much majority data was discarded for a stable estimate.
• Check whether an entity — host, user, service account — appears entirely on one side of the split, since SOC records repeat heavily per entity.
• Compare against class weighting on the full dataset. If weighting matches the undersampled result, the discarded data was buying nothing.

Limitations & prerequisites

• Discarded rows are gone from that fold entirely; unlike duplication, the loss cannot be undone by tuning.
• Uniform draws are blind to majority-class structure and silently delete rare legitimate behaviours.
• Aggregate metrics conceal the damage, which surfaces as subtype-specific false positives after deployment.
• NearMiss and other proximity-based selectors are sensitive to noise and can sharpen a boundary around mislabelled points.

Undersampling variants for alert triage

The choice is between speed, boundary quality and how much information you are prepared to lose.

For SOC triage, stratified undersampling with per-subtype floors is the pragmatic default, and EasyEnsemble is the upgrade when the compute budget allows it.

Key takeaways

• Undersampling is the only resampling method that permanently discards information.
• Benign is not one class; a uniform draw deletes low-volume legitimate behaviours entirely.
• Stratify by subtype with a per-stratum floor, or cluster first where no taxonomy exists.
• EasyEnsemble recovers most of the discarded data by combining models over several subsets.
• Evaluate on the original ratio and report incidents caught at a fixed analyst review budget.

FAQ

Related topics & solutions

• Class Imbalance: When 95% Accuracy Means the Model Found Nothing
• Random Oversampling: Duplication as a Deliberate Choice
• SMOTE: Inventing Minority Examples Without Copying Them
• Outliers: Telling a Broken Sensor From a Broken Chiller
• Duplicate Data: When One Patient Becomes Four Records
• SAIF – Cybersecurity
• ADASYN: Generating Where the Model Is Actually Struggling
• Data Leakage: The Model That Only Works Before Deployment