Duplicate Data: When One Patient Becomes Four Records

What problem does this solve?

A hospital group runs a readmission-risk model across four facilities. The master patient index holds 612,000 records. Clinical audit believes those represent roughly 540,000 distinct people, so around 12% of records are duplicates of someone already present.

The duplicates arise the way they always do. Emirates ID is captured at some registration desks and skipped at others when a patient arrives unconscious. Arabic names transliterate several ways, so Mohammed, Mohamed and Muhammad all appear. Dates of birth default to the first of January when only a year is known. Two facilities merged their systems in 2023 and matched on name and date of birth alone. A patient who married and changed surname exists twice by design.

The model reported an AUC of 0.89 in validation and 0.71 in the first quarter after deployment. Nobody had checked whether the same person appeared on both sides of the split. Around 12% of the test set was people the model had already seen, and those were the easiest predictions in it.

How the solution works

Separate the three duplicate types, because they need different treatment. An exact duplicate is a byte-identical row, usually an ingestion fault, and can be removed by hashing. A near duplicate is the same entity with field-level variation and needs similarity matching. A repeated encounter is legitimately several rows for one person and must not be removed at all.

Resolve entities before splitting. Blocking narrows the comparison space, similarity scoring compares candidates within a block, and a clustering step assigns a stable enterprise identifier to each resolved person.

Split by that identifier, not by row. Group-aware splitting guarantees every record belonging to one patient falls entirely on one side, which is what makes the reported score honest.

Keep the merge reversible. A wrongly merged pair of patients is a clinical safety event, not a data-quality one, so every merge decision needs an audit trail and an unmerge path.

1. 1
   Hash for exact duplicates Normalise then hash the row. Identical hashes are ingestion artefacts and are safe to collapse.
2. 2
   Block to reduce comparisons Compare only records sharing a coarse key such as birth year plus a name phonetic code. Comparing 612,000 records pairwise is 187 billion comparisons; blocking makes the problem tractable.
3. 3
   Score similarity per field Use edit distance on names, exact match on national identifier, and date proximity on birth date, then combine into one score.
4. 4
   Cluster into entities Group records above threshold into clusters and assign a stable enterprise identifier per cluster.
5. 5
   Split by entity Use the enterprise identifier as the grouping key so no patient appears in both training and test.

Reference architecture

Four layers. The identity layer is the one that decides whether the evaluation layer can be trusted.

Deployment options: Patient identifiers are among the most sensitive fields a hospital holds. Matching runs inside the facility's approved environment, and comparison logs contain identifying data in their own right.

Key capabilities

Integrations

Entity resolution touches the registration workflow and the master patient index far more than it touches the model, because the durable fix is upstream.

Industry use cases

Implementation approach

1. 1
   Measure before building Estimate the duplicate rate on a manually reviewed sample so the pipeline has a target to hit.
2. 2
   Fix capture where possible Mandatory identifier capture at registration prevents more duplicates than any matching algorithm resolves.
3. 3
   Design blocking deliberately Choose blocking keys that survive Arabic transliteration variation, and measure how many true matches each key would miss.
4. 4
   Set thresholds by consequence In a clinical setting a false merge is worse than a missed one, so the threshold should be conservative and borderline pairs should go to a human.
5. 5
   Split by entity everywhere Apply the enterprise identifier as the grouping key in every split, including cross-validation folds.

Security & deployment

Entity resolution requires comparing identifying fields in clear text, so the matching process handles the most sensitive data the organisation holds and must run inside the approved clinical environment. Comparison logs, candidate-pair files and review queues all contain patient identifiers and inherit the same classification as the records themselves. A false merge exposes one patient's clinical history to another and is a patient-safety incident: every merge needs an audit trail, an authorising user and a tested unmerge path before automation is enabled. Nothing in this article requires or exposes real patient data - all examples are illustrative.

A worked example

Four records that entity resolution should recognise as one person, and one that it must not.

1. Record 1. MOHAMMED AL SAYED, DOB 1971-03-14, national ID present, emergency admission.
2. Record 2. Mohamed Alsayed, DOB 1971-03-14, national ID absent, outpatient clinic. Name distance is small and the date matches exactly.
3. Record 3. M. AL-SAYED, DOB 1971-01-01, national ID present and identical to record 1. The date is a default placeholder; the identifier is decisive.
4. Record 4. Mohammed Al Sayed, DOB 1971-03-14, transferred from another facility, different medical record number. Same person, different source system.
5. Record 5 - the trap. Mohammed Al Sayed, DOB 1971-03-14, different national ID, different address. A common name and a shared birth date. This is a different person and merging it would be a patient-safety incident.

Records 1 to 4 resolve to one enterprise identifier and carry three legitimate encounters between them. Record 5 stays separate because a present-and-different national identifier outweighs a strong name and date match. That asymmetry - identifiers can veto, names cannot confirm alone - is the core rule of clinical entity resolution.

In code

The pipeline is hash, block, score, cluster, then split by the resolved identifier. The last step is the one that actually protects the evaluation.

import hashlib
import pandas as pd
from itertools import combinations
from difflib import SequenceMatcher
from sklearn.model_selection import GroupShuffleSplit

# 1. Exact duplicates: normalise, then hash. Ingestion artefacts only.
def row_hash(r):
    key = "|".join(str(r[c]).strip().upper() for c in ["name", "dob", "national_id"])
    return hashlib.sha256(key.encode()).hexdigest()

df["row_hash"] = df.apply(row_hash, axis=1)
exact = df["row_hash"].duplicated().sum()
print("exact duplicates:", int(exact))
df = df.drop_duplicates(subset="row_hash").copy()

# 2. Blocking: never compare every pair. n(n-1)/2 is not tractable at scale.
def phonetic(name):
    s = "".join(ch for ch in str(name).upper() if ch.isalpha())
    return s[:4]

df["block"] = df["dob"].str[:4] + "_" + df["name"].map(phonetic)
print("blocks:", df["block"].nunique(), "| largest:", df["block"].value_counts().max())

# 3. Similarity scoring inside each block.
def score(a, b):
    name = SequenceMatcher(None, str(a["name"]).upper(), str(b["name"]).upper()).ratio()
    dob = 1.0 if a["dob"] == b["dob"] else 0.0
    ida, idb = a["national_id"], b["national_id"]
    if pd.notna(ida) and pd.notna(idb):
        # A present-and-different identifier vetoes the match outright.
        return 1.0 if ida == idb else 0.0
    return 0.6 * name + 0.4 * dob

pairs = []
for _, grp in df.groupby("block"):
    for i, j in combinations(grp.index, 2):
        sc = score(df.loc[i], df.loc[j])
        if sc >= 0.90:
            pairs.append((i, j, sc))
print("candidate matches:", len(pairs))

# 4. Cluster matched pairs into entities (union-find).
parent = {i: i for i in df.index}
def find(x):
    while parent[x] != x:
        parent[x] = parent[parent[x]]; x = parent[x]
    return x
for i, j, _ in pairs:
    ri, rj = find(i), find(j)
    if ri != rj: parent[ri] = rj
df["enterprise_id"] = [find(i) for i in df.index]
print("records:", len(df), "-> distinct patients:", df["enterprise_id"].nunique())

# 5. THE STEP THAT MATTERS: split by patient, never by row.
gss = GroupShuffleSplit(n_splits=1, test_size=0.2, random_state=8)
train_idx, test_idx = next(gss.split(df, groups=df["enterprise_id"]))
overlap = set(df.iloc[train_idx]["enterprise_id"]) & set(df.iloc[test_idx]["enterprise_id"])
print("patients in both splits:", len(overlap))   # must be 0

A count of exact duplicates, the blocking statistics, the number of candidate matches, the reduction from records to distinct patients, and a final overlap count that must be zero. If that last number is not zero the evaluation is invalid regardless of every other result.

Diagnostic checks

• Count distinct enterprise identifiers appearing in both training and test. Anything above zero invalidates the evaluation.
• Compare validation and production performance. A large drop with no distribution change is a classic duplicate-leakage signature.
• Sample the highest-similarity non-merged pairs and the lowest-similarity merged pairs; both reveal a badly set threshold.
• Check duplicate rate per facility and per registration route. A single desk or a single import usually accounts for most of it.
• Look for placeholder birth dates such as the first of January. They cluster duplicates and defeat date-based blocking.
• Verify every merge is reversible and logged before any automated merging is enabled.

Limitations & prerequisites

• Blocking trades recall for tractability; any key will miss some true matches and the trade should be measured rather than assumed.
• String similarity is weak on short names and on transliteration variants that differ early in the string.
• Thresholds encode a value judgement about false merges versus missed merges and cannot be set from the data alone.
• Resolution is never final: new records arrive continuously and identifiers change with marriage, correction and reissue.

Duplicate types and their treatment

Only the first is safe to remove automatically. The third must not be removed at all.

Removing repeated encounters as duplicates is the most common error in the list, and it silently deletes the longitudinal history most clinical models depend on.

Key takeaways

• Most hospital duplicates are near duplicates, not exact copies, so hashing alone finds very few of them.
• A duplicated patient across the split turns an evaluation into a memory test.
• Repeated encounters are legitimate records and must not be deduplicated away.
• A present-and-different national identifier should veto an otherwise strong name match.
• Split by resolved entity identifier, and confirm the overlap between splits is exactly zero.

FAQ

Related topics & solutions

• Data Leakage: The Model That Only Works Before Deployment
• Missing Values: Why the Gap Itself Carries Information
• Outliers: Telling a Broken Sensor From a Broken Chiller
• Label Noise: The Accuracy Ceiling Nobody Put There Deliberately
• Selection Bias: When Your Data Only Contains the People Who Showed Up
• Document Management & Correspondence System
• SMOTE: Inventing Minority Examples Without Copying Them
• ADASYN: Generating Where the Model Is Actually Struggling