An outlier is an observation far from the rest of the distribution. In machine learning the useful question is never whether a point is unusual, but why. A cold-chain dataset contains two kinds of extreme reading that look identical in a histogram: a probe glitch that reported minus forty for one sample, and a genuine excursion where a reefer unit failed and the load warmed for six hours. Automatic removal deletes both, and the second is the one the business needed.
Outlier handling is a classification problem before it is a cleaning step. Decide what each extreme value represents, then decide what to do with it.
What problem does this solve?
A distributor moves temperature-controlled pharmaceutical and fresh-food consignments across the Gulf. Each reefer container carries probes logging at five-minute intervals: supply air, return air, ambient and door state. Twelve months of history covers roughly 4,300 journeys.
The dataset contains extremes of several distinct kinds. Some probes report a single sample at minus forty or plus ninety, physically impossible values that appear for one reading and vanish. Some show a slow upward drift over several hours after a compressor stops. Some jump five degrees for twenty minutes every time a door opens at a delivery point, which is expected. And a handful of journeys show ambient temperature exceeding fifty degrees while supply air stays in range, which is the unit working exactly as designed on an August afternoon.
The analytics team applied a three-sigma filter and dropped every reading beyond it. Model performance improved on the validation set. It also deleted 61% of the confirmed excursion events, because a genuine refrigeration failure is by definition an extreme temperature reading. The cleaning step removed the target.
How the solution works
Separate impossible from unusual. A value outside the sensor's physical measurement range is an instrument artefact and can be removed on a domain rule, with no statistics involved. A value inside the range but far from normal is a candidate incident and must not be removed on statistical grounds alone.
Prefer robust statistics over the mean and standard deviation. Both are pulled by the very points you are trying to identify, so a genuine excursion inflates the standard deviation and then hides inside the widened threshold.
Distinguish univariate from multivariate. Ambient at fifty-two degrees is unremarkable in July, and supply air at nine degrees is unremarkable for chilled produce, but the two together with the door closed is a failing unit. Neither column alone flags it.
Use isolation-based or density-based detectors for the multivariate case, then route what they surface to a human or a domain rule rather than to a delete statement.
- 1Apply physical bounds first Reject readings outside the probe's rated measurement range. This is a specification check, not a statistical one, and it removes true artefacts safely.
- 2Profile with robust measures Use median and interquartile range rather than mean and standard deviation, since the latter are distorted by the points under investigation.
- 3Look across columns Evaluate supply air, return air, ambient and door state together. Most genuine excursions are visible only as a combination.
- 4Score rather than label Isolation Forest and Local Outlier Factor produce a continuous anomaly score. Keep the score; do not collapse it to a binary and discard the row.
- 5Decide by cause Artefact: remove. Expected event such as a door opening: keep and encode. Genuine excursion: keep, label, and treat as the positive class.
Reference architecture
Four layers, and only the third involves an algorithm.
| Layer | What it contains |
|---|---|
| Specification layer | Probe measurement range, accuracy tolerance and calibration date. Values outside the rated range are artefacts by definition. |
| Process layer | Door events, defrost cycles, compressor state and loading stops - the expected reasons a reading moves. |
| Detection layer | Robust univariate profiling plus Isolation Forest or Local Outlier Factor over the combined signal. |
| Decision layer | A rule that maps each flagged point to remove, keep and encode, or escalate as a candidate incident. |
Deployment options: Logger data usually arrives in batches when a container regains connectivity, so detection runs on ingest rather than in real time, and the same logic must work on backfilled history.
Key capabilities
Physical-bounds validation
Instrument artefacts separated from real readings using probe specifications rather than statistics.
availableRobust univariate profiling
Thresholds built from median and IQR so a genuine excursion cannot widen the threshold that should catch it.
availableMultivariate excursion detection
Isolation Forest and LOF over combined channels, surfacing failures no single column reveals.
custom developmentExcursion adjudication workflow
A ranked review queue with domain rules, so flagged points become labelled incidents instead of deleted rows.
custom developmentIntegrations
Outlier handling for cold chain sits between the telemetry platform and the quality system, because the output is a labelled excursion record rather than a cleaned table.
| System | Integration point & data exchanged | Direction |
|---|---|---|
| Telemetry and logger platform | Probe calibration status and measurement range read alongside values so bounds checks use current specifications. → Missing Values: Why the Gap Itself Carries Information | bi-directional |
| Quality management system | Confirmed excursions written back as labelled events, which is how the positive class grows. → Label Noise: The Accuracy Ceiling Nobody Put There Deliberately | bi-directional |
| Transport management | Door events, stops and route context joined so expected movements are not mistaken for failures. → Octopus WMS | bi-directional |
Industry use cases
Pharmaceutical distribution
Excursion evidence is a regulatory record, so removing extreme readings is not merely a modelling choice.
Fresh and frozen food logistics
Shelf-life models depend on cumulative time above setpoint, which is precisely the extreme tail.
Reefer fleet maintenance
Slow drift patterns before failure are weak outliers that a hard threshold never surfaces.
Last-mile chilled delivery
Frequent door openings produce expected extremes that must be encoded rather than cleaned away.
UAE & GCC considerations
Gulf ambient conditions change what counts as unusual. A reefer working correctly in August will show ambient readings that would be extreme anywhere else, along with longer compressor duty cycles and more frequent defrost activity, all of which shift the distribution of every derived feature. A threshold fitted on winter data will flag ordinary summer operation as anomalous. Fit thresholds per season, or include ambient as a conditioning variable, and confirm the probe's rated range covers the temperatures actually reached inside a container standing on a yard at midday. Where consignments are pharmaceutical, excursion records are regulated evidence and the retention rules for them are usually stricter than for general telemetry.
Implementation approach
- 1Collect the specifications Obtain the rated measurement range and accuracy for each probe type before writing any rule.
- 2Bound, then profile Remove out-of-range artefacts first, then profile what remains with median and IQR.
- 3Join the process context Attach door state, compressor state and stop events so expected extremes can be recognised.
- 4Score multivariately Run Isolation Forest or LOF over the combined channels and keep the score as a feature.
- 5Adjudicate, do not delete Route high scores to review, and record the outcome as a label rather than dropping the row.
Security & deployment
Excursion records for pharmaceutical consignments are regulated evidence and frequently discoverable, so deletion of extreme readings should be logged and reversible rather than performed in place. Keep the raw ingest immutable and apply cleaning as a derived view. Journey and container identifiers can reveal customer shipping patterns, so access to the raw telemetry should be scoped in the same way as the consignment records themselves.
A worked example
One journey, Jebel Ali to Riyadh, with four extreme readings that a single filter would treat identically.
- Reading A - minus 41.2 C for one sample. The probe is rated to minus 30. The value is outside the instrument's physical range, so it is an artefact and can be removed on a bounds rule.
- Reading B - 8.4 C rising to 14.1 C over five hours. Inside range, sustained, correlated with the compressor-off flag. This is a genuine excursion and is the event the programme exists to detect.
- Reading C - 11.9 C for eighteen minutes at a delivery stop. Inside range, coincides with door state open, returns to setpoint afterwards. Expected behaviour; keep it and encode the door state as a feature.
- Reading D - ambient 53.6 C, supply air 4.9 C. Ambient is extreme for the column but the unit is holding setpoint. Univariate detection flags it; multivariate context shows nothing is wrong.
- Three-sigma filter applied to supply air. Mean 5.1, sd 1.9, threshold 10.8. It removes A and B, keeps C, and never examines D.
The filter removed one artefact and one genuine failure, kept an expected door event, and missed the case that needed multivariate context entirely. Its accuracy on the task it was given was one in four.
In code
The ordering matters: physical bounds first, robust univariate profiling second, multivariate scoring third, and no automatic deletion at any stage.
import numpy as np
import pandas as pd
from sklearn.ensemble import IsolationForest
from sklearn.neighbors import LocalOutlierFactor
from sklearn.preprocessing import RobustScaler
PROBE_MIN, PROBE_MAX = -30.0, 60.0 # from the datasheet, not from the data
# 1. Physical bounds - a specification check, safe to act on automatically.
artefact = (df["supply_air"] < PROBE_MIN) | (df["supply_air"] > PROBE_MAX)
print("instrument artefacts:", int(artefact.sum()))
clean = df.loc[~artefact].copy()
# 2. Robust univariate profiling. Mean and sd are moved by the very points
# under investigation; median and IQR are not.
q1, q3 = clean["supply_air"].quantile([0.25, 0.75])
iqr = q3 - q1
lo, hi = q1 - 1.5 * iqr, q3 + 1.5 * iqr
clean["iqr_flag"] = (clean["supply_air"] < lo) | (clean["supply_air"] > hi)
z = (clean["supply_air"] - clean["supply_air"].mean()) / clean["supply_air"].std()
print("iqr flags:", int(clean['iqr_flag'].sum()), "| 3-sigma flags:", int((z.abs() > 3).sum()))
# 3. Multivariate: the combination is what identifies a failing unit.
cols = ["supply_air", "return_air", "ambient", "door_open", "compressor_on"]
X = RobustScaler().fit_transform(clean[cols])
iso = IsolationForest(n_estimators=300, contamination="auto", random_state=4)
clean["iso_score"] = -iso.fit(X).score_samples(X) # higher = more anomalous
lof = LocalOutlierFactor(n_neighbors=35, novelty=False)
lof.fit_predict(X)
clean["lof_score"] = -lof.negative_outlier_factor_
# 4. Domain rules turn a score into a decision. Nothing is deleted here.
clean["expected_door_event"] = clean["door_open"] & (clean["supply_air"] < 15)
clean["candidate_excursion"] = (
(clean["iso_score"] > clean["iso_score"].quantile(0.99))
& ~clean["expected_door_event"]
& ~clean["compressor_on"]
)
print(clean.groupby("candidate_excursion")[["supply_air", "ambient"]].median())
# Review queue for a human, ordered by how anomalous the combination is.
review = clean.nlargest(50, "iso_score")[cols + ["iso_score", "journey_id"]]A count of instrument artefacts, a comparison showing IQR and three-sigma disagreeing on how many points are extreme, median profiles for candidate excursions against normal operation, and a ranked review queue. The three-sigma count is typically the lower of the two on skewed temperature data, because the extremes have already widened the standard deviation.
Diagnostic checks
- Count how many confirmed excursions your cleaning step removes. If the answer is not zero, the cleaning is deleting the target.
- Compare IQR flags against three-sigma flags. Large disagreement indicates a skewed distribution where the z-score is unreliable.
- Check whether flagged points cluster at door-open or defrost events; if so they are expected behaviour, not anomalies.
- Plot ambient against supply air. Points extreme in one axis but normal in combination are univariate false positives.
- Verify every removed value lies outside the probe's rated range. Anything inside the range was removed on statistics, not on evidence.
- Re-fit thresholds by season and compare. A large shift means a single annual threshold is wrong for most of the year.
When to use it
- Extreme values have identifiable physical causes that can be checked against a specification.
- Process context such as door state or compressor state is available to explain expected movements.
- The rare event is the modelling target, so extremes must be preserved and labelled.
- Multiple correlated channels are recorded, which makes multivariate detection possible.
When not to use it
- The extreme values are the phenomenon being modelled and no cleaning of any kind is appropriate.
- Only one channel is available with no process context, where multivariate methods have nothing to combine.
- The dataset is small enough that every extreme can be inspected individually, which is better than any detector.
- Readings are already validated upstream by the logger firmware and duplicated validation adds only false positives.
Limitations & prerequisites
- Isolation Forest and LOF surface unusual combinations, not causes; a flagged point still requires domain interpretation.
- Contamination and neighbourhood parameters materially change what is flagged and have no universally correct value.
- Density-based methods degrade when the normal operating regime itself changes seasonally.
- No detector distinguishes a rare-but-correct reading from a fault; only the specification and process context can.
Detection methods against the cold-chain problem
The methods answer different questions, and only the first is safe to automate.
| Method | Answers | Weakness here |
|---|---|---|
| Physical bounds | Is this reading possible at all? | Cannot see a genuine excursion in range |
| Z-score | How many sd from the mean? | Distorted by the extremes it should find |
| IQR rule | Is it far from the quartiles? | Univariate; blind to combinations |
| Robust scaling | Comparable features for a model | A transform, not a detector |
| Isolation Forest | Is this combination easy to isolate? | Parameter-sensitive; no cause given |
| Local Outlier Factor | Is local density unusual here? | Struggles with seasonal regime change |
| Domain rules | Does the process explain this? | Needs process data to exist |
Bounds remove artefacts, robust statistics profile the normal regime, multivariate detectors nominate candidates, and domain rules make the decision.
Key takeaways
- Ask why a value is extreme before deciding what to do with it.
- Physical bounds are safe to automate; statistical thresholds are not.
- Mean and standard deviation are distorted by the very points you are hunting - use median and IQR.
- Most genuine cold-chain failures are multivariate and invisible in any single column.
- Score and adjudicate rather than delete; a removed excursion is a removed label.
FAQ
No. Remove values that are physically impossible for the instrument. Anything within the sensor's range is a candidate incident and should be investigated rather than deleted, particularly when rare events are the modelling target.
It uses the mean and standard deviation, both of which are pulled by the extreme points. A genuine excursion widens the standard deviation and can then fall inside the threshold meant to catch it.
A univariate outlier is extreme in one column. A multivariate outlier is unremarkable in every column individually but implausible as a combination - supply air in range while ambient is extreme and the compressor is off.
Isolation Forest scales better to large datasets and handles higher dimensionality. LOF is stronger when normal behaviour has varying density, though it is more sensitive to the neighbourhood size.
Join the door-state channel and encode it as a feature. An expected extreme with a known cause should be represented, not removed, so the model learns the difference.
No. It rescales features using the median and IQR so extreme values distort the scale less. The points remain in the data.
Prefer to keep the continuous score and choose an operating threshold from review capacity, rather than committing to a contamination rate that asserts what proportion of the data is anomalous.
Start with the confirmed excursion records the quality system already holds. Measuring recall against those is more informative than any unsupervised score.
Temperature history you cannot fully trust?
Send the logger export, the sampling interval and the excursion definition you report against. We will separate instrument artefacts from genuine excursions and show which extremes your current cleaning removes.
Book a cold-chain monitoring reviewSources & evidence
- scikit-learn: novelty and outlier detection — Official reference for Isolation Forest, Local Outlier Factor and related estimators.
- scikit-learn: RobustScaler — Official reference for median and IQR based scaling.
- Liu et al., Isolation Forest (ICDM 2008) — The original isolation-based anomaly detection paper.
- Breunig et al., LOF: Identifying Density-Based Local Outliers — The original Local Outlier Factor paper.
Vendor and product names are trademarks of their respective owners; references are for technical context and do not imply partnership, certification or endorsement unless stated on the vendor's official pages.