How to find duplicate customer IDs in a CSV before a CRM import
A review-first workflow for repeated customer or record IDs, including blank IDs, conflicting fields, and safe export without silently merging legitimate history.
Before you work through this by hand: you can check your own CSV free and see which of these problems your file actually has — duplicate rows, near-duplicates, broken quoting, encoding damage. It runs in your browser tab, nothing is uploaded, and there's no signup or email.
Confirm what the ID represents
A customer ID, account number, or external record ID can be a strong identity key, but only within the system that issued it. Before removing anything, confirm whether the export contains one row per customer, one row per order, or a mixture. A repeated customer ID may be a duplicate contact, or it may be valid history with multiple orders or interactions.
Find exact repeated IDs first
Keep an untouched source copy. In Excel or Google Sheets, use conditional formatting or a pivot table on the ID column, then filter the repeated groups. In Python, inspect the groups without changing the source:
import pandas as pd
df = pd.read_csv("export.csv", dtype="string")
ids = df["customer_id"].str.strip()
repeated = df[ids.notna() & ids.ne("") & ids.duplicated(keep=False)]
print(repeated.sort_values("customer_id"))
Normalize only the comparison key. Preserve the original ID text and row number so each proposed change can be traced back to the export.
Handle blanks and conflicts separately
Do not group every blank ID together: a missing identifier is an exception queue, not proof that the rows are duplicates. If repeated IDs have different names, phone numbers, statuses, or timestamps, flag the group for review. Use a documented rule such as newest update or most complete record only when the source system supports that rule; never silently keep the first row just because it appears first.
Verify the cleaned export
Record the source row count, repeated groups found, groups approved, and rows retained. Export to a new filename, reopen it with a CSV parser, and check the header, ID uniqueness where expected, blank-ID count, and a sample of retained records. If the file mixes contacts and transactions, deduplicate at the correct record level instead of collapsing legitimate history.
You can check your CSV free before deciding what needs review. CSV Repair & Deduper runs locally in your browser, flags exact and near-duplicate candidates, keeps original values visible, and lets you review and export a new file without uploading it. See also our CRM import guide and conflicting-data guide.
See CSV Repair & Deduper (offline browser tool) — $19 one-time