Fixing 'UnicodeDecodeError' and other CSV encoding errors in Python
Why pandas or the csv module raises UnicodeDecodeError on some CSV files, and the encodings to try instead of guessing.
Before you work through this by hand: you can check your own CSV free and see which of these problems your file actually has — duplicate rows, near-duplicates, broken quoting, encoding damage. It runs in your browser tab, nothing is uploaded, and there's no signup or email.
Why this happens
A CSV file is just bytes; the encoding tells your program how to turn those bytes into text. If a file was saved as Windows-1252 (common from older Excel exports on Windows) and you read it assuming UTF-8, you'll hit an error like:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x92 in position ...
Quick fixes to try, in order
pd.read_csv('file.csv', encoding='utf-8-sig')— handles a UTF-8 byte-order mark (BOM) that plainutf-8doesn't strip automatically.pd.read_csv('file.csv', encoding='cp1252')— the common fallback for older Windows/Excel exports.pd.read_csv('file.csv', encoding='latin1')— rarely raises an error (it maps every byte to a character), but can silently produce wrong characters, so only use it as a last resort and spot-check the output.
Detecting it automatically instead of guessing
You can install charset-normalizer or chardet and let it inspect the file's byte patterns to guess the encoding, but for one-off files it's often faster to just try the three options above in order.
If you'd rather not deal with encodings at all
This is exactly the kind of detail CSV Repair & Deduper handles automatically (BOM detection plus a Windows-1252 fallback) so you don't have to identify the encoding yourself before cleaning the file.
See also: our broader guide to fixing a broken CSV file, which covers delimiter and row-shape issues alongside encoding.
See CSV Repair & Deduper (offline browser tool) — $19 one-time