What to do when 'duplicate' rows have different values in other fields
Two rows share the same email but have different phone numbers or job titles — which one is right? A decision framework for conflicting duplicate data.
Before you work through this by hand: you can check your own CSV free and see which of these problems your file actually has — duplicate rows, near-duplicates, broken quoting, encoding damage. It runs in your browser tab, nothing is uploaded, and there's no signup or email.
The problem exact/fuzzy matching doesn't solve
Our other guides cover finding duplicate and near-duplicate rows, but finding them is only half the problem. Once you've identified that two rows represent the same person or record, you often hit a harder question: they don't agree on everything. Same email, but one row has an old job title and the other a new one; same customer ID, but a different phone number in each. Simply keeping "the first row" or "the last row" can silently keep the wrong value.
A few common resolution strategies
- Most recent wins: if rows have a timestamp or "last updated" field, keep the value from the newer row for each conflicting field.
- Most complete wins: prefer the row with fewer blank fields overall, on the theory that a more complete record was probably entered more carefully.
- Manual review for conflicts, automatic merge for agreements: auto-merge fields where both rows agree, and flag only the genuinely conflicting fields for a human to resolve rather than reviewing every duplicate pair in full.
Implementing "most recent wins" with pandas
import pandas as pd; df = pd.read_csv('file.csv'); df = df.sort_values('last_updated').drop_duplicates(subset=['email'], keep='last') — sorting by the timestamp first means keep='last' retains the most recently updated row for each duplicate group.
Don't silently discard the losing data
Whichever strategy you use, consider keeping a log of what was discarded (the old phone number, the superseded job title) rather than deleting it outright — if the "winning" value turns out to be wrong, you'll want to know what the alternative was. This is the same backup-first principle from our backup and versioning guide, applied to individual field values instead of whole files.
See CSV Repair & Deduper (offline browser tool) — $19 one-time