Missing Values: Why the Gap Itself Carries Information

What problem does this solve?

A facilities team runs analytics over a mixed-use tower: zone temperature, relative humidity, CO2, occupancy counts and sub-metered power at fifteen-minute intervals across 340 points. Around 9% of expected readings are absent over a twelve-month window.

The gaps have several distinct causes. Some wireless sensors on the upper floors miss readings during high wind because of antenna movement. Several CO2 sensors drop out during their scheduled auto-calibration. Two chiller power meters lose long blocks during network switch maintenance. And a handful of zone temperature sensors stop reporting precisely when the plant room runs hottest, because the transmitter exceeds its own operating range.

The team's first pass filled every gap with the column mean. Energy consumption forecasting accuracy improved slightly on paper. In production the model failed to anticipate exactly the peak-load afternoons it was built for — because the readings that mattered most were the ones that had been absent, and they had all been replaced with an average summer afternoon.

How the solution works

Classify the mechanism before choosing a method. MCAR means the probability of absence is unrelated to anything — a random packet loss. MAR means absence is explained by other observed variables — dropouts correlate with recorded wind speed. MNAR means absence depends on the unobserved value itself — the sensor fails because the temperature is high.

For MCAR, most reasonable imputations are defensible. For MAR, use methods that condition on the other columns, such as iterative or KNN imputation. For MNAR, no imputation is correct on its own; the fact of absence must be carried as a feature.

For time series, respect the axis. Temperature at 14:00 is far better estimated from 13:45 and 14:15 than from the annual mean, so short gaps take interpolation and long gaps are better left absent than invented.

Add missingness indicators. A binary column marking that a value was absent lets the model learn the pattern directly, which is the only honest way to handle MNAR without pretending to know the missing number.

1. 1
   Quantify per point Compute missingness rates per sensor, per hour of day and per month rather than one dataset-level percentage.
2. 2
   Test for MCAR Compare the distribution of other variables between rows where the target column is present and rows where it is absent. A material difference rules out MCAR.
3. 3
   Look for MAR structure Model the missingness indicator itself as a classification target. If other columns predict absence well, the mechanism is MAR and conditional imputation is appropriate.
4. 4
   Suspect MNAR at the range limits Check whether the last recorded values before a gap cluster near a sensor's operating limit. That is the signature of a device failing under the condition being measured.
5. 5
   Impute inside the fold Fit the imputer on training data only. Fitting on the whole dataset lets test-set statistics flow backwards into training.

Reference architecture

Missing data touches four layers, and only one of them is the imputation call.

Deployment options: Occupancy counts and zone-level data can become personal data when they identify individual working patterns, particularly in small tenancies, so retention and aggregation levels should be agreed before the analytics platform is built.

Key capabilities

Integrations

In a building estate, missing data is a maintenance fact before it is a modelling problem, so the integration surface includes the BMS and the CMMS as well as the analytics platform.

Industry use cases

Implementation approach

1. 1
   Profile before filling Produce missingness rates per point, per hour and per month. A single overall percentage conceals every pattern that matters.
2. 2
   Classify the mechanism Test whether other columns predict absence, and inspect the values immediately preceding gaps for range-limit clustering.
3. 3
   Match the method Time interpolation for short gaps, conditional imputation for MAR, and indicators wherever MNAR is suspected.
4. 4
   Cap the gap length Set a maximum interpolation span. Beyond it, leave the value absent rather than manufacturing a plausible line.
5. 5
   Monitor the rate Track dropout per sensor after deployment and route sustained increases to maintenance.

Security & deployment

Occupancy counts and zone-level environmental data can identify individual working patterns in small tenancies, and imputation can make that worse by producing a continuous inferred record where the real data was sparse. Agree aggregation levels and retention with the building owner before modelling, and keep imputed values flagged so an inferred occupancy figure is never mistaken for a measurement in a tenant-facing report.

A worked example

One zone temperature sensor over a single July week at fifteen-minute resolution, with a 90-minute gap on the hottest afternoon.

1. The surrounding readings. The last value before the gap is 38.4 C at 14:15; the first after is 37.9 C at 15:45. The annual column mean is 24.1 C.
2. Global mean imputation. Six consecutive intervals are filled with 24.1 C, inserting a 14-degree trough into the hottest afternoon of the year.
3. What the model learns. Peak cooling load appears to coincide with a sharp temperature drop, which is the opposite of the physical relationship.
4. Linear interpolation. The gap fills smoothly from 38.4 to 37.9, preserving the shape but assuming nothing unusual happened in between.
5. Interpolation plus indicator. The same interpolated values, with a flag marking the interval as imputed and the reason as transmitter over-range.

The last row is the honest treatment. Interpolation keeps the temporal shape plausible, and the indicator lets the model learn that this sensor stops reporting under extreme heat — which is a genuine predictor of peak load, and precisely the pattern global mean imputation destroyed.

In code

Three things worth doing in order: test whether missingness is random, compare methods honestly inside folds, and keep the indicator.

import numpy as np
import pandas as pd
from sklearn.experimental import enable_iterative_imputer  # noqa: F401
from sklearn.impute import SimpleImputer, KNNImputer, IterativeImputer, MissingIndicator
from sklearn.ensemble import HistGradientBoostingRegressor, RandomForestClassifier
from sklearn.pipeline import Pipeline, FeatureUnion
from sklearn.model_selection import TimeSeriesSplit, cross_val_score

# 1. Is the missingness random? Predict the gap itself.
# High AUC here means other columns explain absence -> MAR, not MCAR.
is_missing = df["zone_temp"].isna().astype(int)
predictors = df.drop(columns=["zone_temp"]).fillna(-999)
auc = cross_val_score(
    RandomForestClassifier(n_estimators=200, random_state=0),
    predictors, is_missing, cv=TimeSeriesSplit(n_splits=4), scoring="roc_auc",
).mean()
print("missingness predictability (AUC):", round(auc, 3))

# 2. Where do gaps start? Values near the sensor limit suggest MNAR.
gap_start = df["zone_temp"].notna() & df["zone_temp"].shift(-1).isna()
print("last value before gap - describe:")
print(df.loc[gap_start, "zone_temp"].describe())

# 3. Compare strategies inside time-ordered folds.
# The indicator is kept alongside the imputed values, never instead of them.
with_flag = FeatureUnion([
    ("imputed", IterativeImputer(max_iter=10, random_state=0)),
    ("was_missing", MissingIndicator(features="all")),
])

strategies = {
    "mean": SimpleImputer(strategy="mean"),
    "knn": KNNImputer(n_neighbors=5),
    "iterative": IterativeImputer(max_iter=10, random_state=0),
    "iterative+indicator": with_flag,
}
cv = TimeSeriesSplit(n_splits=5)
for name, imputer in strategies.items():
    pipe = Pipeline([("impute", imputer),
                     ("model", HistGradientBoostingRegressor(random_state=0))])
    s = cross_val_score(pipe, X, y_energy, cv=cv, scoring="neg_mean_absolute_error")
    print(f"{name:20s} MAE {-s.mean():.3f} +/- {s.std():.3f}")

# Short gaps only: interpolation is a time-series operation, not a column statistic.
df["zone_temp_interp"] = df["zone_temp"].interpolate(method="time", limit=4)

An AUC for predicting absence, a summary of the values immediately preceding gaps, and mean absolute error per strategy. A high AUC rules out MCAR. Values clustered near the top of the sensor range before gaps indicate MNAR, and in that case the indicator column usually improves the model more than any change of imputer.

Diagnostic checks

• Compute missingness per sensor, per hour of day and per month; a dataset-level percentage hides seasonal and diurnal structure.
• Train a classifier to predict the missingness indicator. High AUC means the mechanism is not MCAR.
• Inspect the last recorded value before each gap. Clustering near a device's operating limit is the MNAR signature.
• Compare mean imputation against interpolation on the same time-ordered folds; a large difference means the temporal axis carries real signal.
• Check whether gaps align with calibration schedules or network maintenance windows, which are explainable and should be excluded rather than filled.
• Verify the imputer is fitted inside the fold; a global fit leaks test statistics into training.

Limitations & prerequisites

• No imputation adds information; it produces a plausible value, not a correct one.
• MNAR cannot be corrected from the observed data alone, which is why the indicator matters more than the method.
• Iterative imputation is expensive on wide sensor estates and its uncertainty is not reflected in a single filled value.
• Interpolation assumes smooth behaviour between endpoints, which is exactly what a fault event violates.

Methods against gap type

Selection follows the mechanism and the gap length, not convenience.

For building sensor data the practical default is time interpolation with a capped span, always paired with an indicator column.

Key takeaways

• Why a value is missing matters more than how many are missing.
• MCAR, MAR and MNAR require different treatments and only the first is forgiving.
• Global mean imputation destroys temporal and seasonal structure in sensor data.
• A missingness indicator preserves MNAR information that no imputer can recover.
• Fit imputers inside time-ordered folds, and monitor dropout rates as a maintenance signal.

FAQ

Related topics & solutions

• Outliers: Telling a Broken Sensor From a Broken Chiller
• Data Leakage: The Model That Only Works Before Deployment
• Duplicate Data: When One Patient Becomes Four Records
• Facility Management
• Random Oversampling: Duplication as a Deliberate Choice
• Class Imbalance: When 95% Accuracy Means the Model Found Nothing
• ADASYN: Generating Where the Model Is Actually Struggling
• Selection Bias: When Your Data Only Contains the People Who Showed Up