1. Question and context
This question tests entity resolution or record linkage for data engineering. A public interview prompt asks candidates to reconcile noisy person records from several sources and probes normalization, conflict resolution, data quality, precision, and recall. It fits data platform, master-data, and customer-data integration roles.
2. What the interviewer is testing
- Whether you define “same person” and business error costs before choosing signals.
- Whether you reduce all-pairs work with candidate generation and explain blocking recall risk.
- Whether you use automatic merge, human review, and no-match bands to control false merges.
- Whether every master-field choice retains source, evidence, and version history.
- Whether you evaluate precision and recall on labeled pairs instead of reporting accuracy alone.
3. Questions to clarify first
- Which fields are stable identifiers? Are email and phone verified, and can names be transliterated or aliased?
- Which is more expensive: a false merge or a missed merge? Is the result used for marketing, payment, compliance, or support?
- Is this a historical backfill or a daily incremental stream? Do sources emit update and deletion events?
- When fields conflict, which source is trusted? Must reviewers see every original value and its provenance?
4. A 30-second answer
I would define the matching unit, error costs, and auditable output first. I would normalize fields deterministically, use email, phone, or composite keys to generate candidate blocks, and avoid all-pairs comparison. For each candidate I would score positive and negative evidence, with separate auto-merge, human-review, and no-match thresholds. I would build the master record using source trust, verification, and freshness while retaining every source value. Finally, I would calibrate on labeled pairs, report precision, recall, and review volume, and monitor incremental drift and false merges.
5. Step-by-step solution
Step 1: Define and normalize records
Keep source, source_id, arrival time, and raw fields for every input. Normalize names with Unicode, case, whitespace, and punctuation rules; trim email addresses and apply an explicit case policy; convert phone numbers to a canonical country-code form. Normalized values must be reproducible without overwriting raw values. Missing should mean unknown, not evidence that two empty strings match.
Step 2: Generate candidates and score matches
Index normalized email, phone, or name-plus-postal-code keys, and run multiple blocking passes for different missing-data patterns. Then compare edit distance, shared fields, verification state, and negative evidence; two verified, different phone numbers should reduce confidence. Scoring within blocks makes the work close to candidate count instead of O(n²), but blocking misses must be sampled and measured.
Step 3: Set thresholds and merge safely
Split scores into auto-merge, human review, and no-match bands. Calibrate thresholds from labeled positive and negative pairs plus business error costs. Log the rule and score for every automatic merge; show conflicting fields and evidence to reviewers; retain rejection reasons. If using union-find or connected components, prevent chains of weak edges from merging distinct people.
Step 4: Build the master and monitor increments
Choose each master field by verified source, business precedence, and freshness, while storing provenance, prior values, and effective time. Compare new records only with relevant blocks and support replay after source or rule changes. Sample reviews regularly and monitor precision, recall, review rate, component size, and drift by source, language, region, and time window. Pause automatic publication and trace affected keys when false merges rise.
6. A strong sample answer
I would model each input as raw plus normalized values, with source, source key, verification state, and event time. I would normalize names, emails, and phones using explainable rules, then generate candidates through multiple blocks such as email, phone, and name plus region. Candidate scores would combine shared fields, edit distance, verification, and conflict penalties, producing auto-merge, human-review, and no-match bands calibrated with labeled pairs and business costs.
I would not reduce a merge to one surviving row. Each master field would retain its chosen source, rule version, effective time, and discarded values; only strong evidence would create a connected component, while weak edges go to review. Offline evaluation would report precision, recall, F1, review rate, and false-merge examples. Incremental runs would monitor source and time-window drift, so a new source can be added safely and a bad rule can be traced, split, and replayed.
7. Common mistakes
- Treating exact name equality as identity despite aliases, transliteration, shared contacts, and homonyms.
- Comparing every pair without describing candidate generation or measuring blocking recall.
- Treating a fuzzy score as truth without a review band or conflict penalties.
- Keeping only one master row and discarding source values, rule versions, and evidence.
- Reporting accuracy alone instead of precision, recall, and business-weighted errors.
- Allowing one weak edge to create a huge connected component and an irreversible over-merge.
8. Follow-up questions
Follow-up 1: Why not train a classifier directly?
With enough labels, a model can learn match scores, but it still needs explainable features, threshold calibration, human review, and versioned replay. Rules, models, and reviewers should all emit evidence for new sources and compliance questions.
Follow-up 2: How do you trade precision against recall?
Translate false-merge and missed-merge costs into a comparable objective, then inspect threshold curves on a validation set. Payment or compliance usually protects precision first; deduplication may accept higher recall, while both retain review and sampling.
Follow-up 3: How do you detect over-merging?
Monitor component size, low-score edge share, cross-source conflicts, and manual split rate; expand evidence graphs for unusually large components. Pause auto-merge, split affected entities by rule version, and replay incremental results.