How to merge multiple CSV exports into one file without creating duplicates
A step-by-step way to combine several CSV exports (e.g. contact lists from different tools) into one clean file, and where duplicates usually sneak in.
Before you work through this by hand: you can check your own CSV free and see which of these problems your file actually has — duplicate rows, near-duplicates, broken quoting, encoding damage. It runs in your browser tab, nothing is uploaded, and there's no signup or email.
1. Make sure every file has matching columns first
Before combining anything, check that each source file uses the same column names and order. If one export calls a column "Email" and another calls it "email_address", merging will either fail or create a mess of extra columns. Rename headers to match before combining.
2. Combine the files
Spreadsheet method: copy the data rows (not the header) from each additional file and paste them below the first file's data, so you end up with one header row followed by all rows from every source.
Python method: import pandas as pd; import glob; combined = pd.concat([pd.read_csv(f) for f in glob.glob('*.csv')]); combined.to_csv('combined.csv', index=False)
3. Where duplicates come from after merging
Once files are combined, the same contact or record often appears more than once because it existed in two of the source files — sometimes with identical values (an exact duplicate), and sometimes with small differences like a different capitalization, an extra space, or a slightly different spelling (a near-duplicate). See the duplicate-removal guide for the exact-duplicate methods; near-duplicates need a fuzzy-match/similarity-threshold approach, which is the harder part to get right by hand.
4. Sanity-check row counts
Before and after merging, note the row count of each source file and the combined file. If the combined total is noticeably lower than the sum of the sources, something other than intentional dedup (like a header row being mistaken for data, or a parsing error) may have dropped rows. If you merged by concatenating raw files rather than with a real CSV parser, also check our guide to duplicate header rows.
See CSV Repair & Deduper (offline browser tool) — $19 one-time