UnicodeDecodeError When Reading a CSV in Python? Fix the Encoding

Your CSV file opens perfectly in Excel, yet Python greets you with UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 123 the moment you try to read it. The file is not corrupt. It is simply saved in a different text encoding than the one Python assumed. This guide explains encodings in plain language, shows you how to identify the right one, and gives you the reliable fixes for the open() function, the csv module, and pandas.

Quick Answer

The error means your CSV uses an encoding other than UTF-8, commonly Windows-1252 or Latin-1 from Excel exports. Fix it by naming the encoding when you open the file: open("data.csv", encoding="cp1252") or, for Excel files with a byte order mark, encoding="utf-8-sig". In pandas, pass the same value: pd.read_csv("data.csv", encoding="cp1252").

What an Encoding Is, in Plain Words

A text file stores numbers, not letters. An encoding is the lookup table that turns those numbers, called bytes, into characters. UTF-8 is the modern standard, but Windows Excel in many regions saves CSV files in a local encoding such as Windows-1252, where accented characters like é use single bytes that are invalid in UTF-8. When Python opens a file without being told otherwise, it guesses based on your system, and on most modern setups the csv workflow assumes UTF-8. The first byte that does not fit the UTF-8 pattern triggers the UnicodeDecodeError, and the message even tells you the byte value and its position in the file.

Two details from the error message help you identify the culprit. A byte like 0xe9 is é in Latin-1 and Windows-1252. Bytes 0x93 and 0x94 are the curly opening and closing quotes that Excel loves and UTF-8 never uses in that form. Seeing those values is a strong hint that the file is Windows-1252.

Fix 1: Set the Encoding in open()

1. Change your file opening line to include the encoding: open("data.csv", newline="", encoding="cp1252"). The newline="" part is the csv module’s own recommendation and prevents blank-row problems.
2. Run the script again. If it works, read a few rows and print them, checking that accented names and currency symbols display correctly, not just that the error is gone.
3. If cp1252 fails too, try encoding="latin-1". Latin-1 accepts every byte value, so it never raises this error, though very rarely it can display a wrong character.
4. If the file came from modern Excel on a Mac or from a web download and starts with odd characters in the first cell, try encoding="utf-8-sig", which is UTF-8 with handling for the invisible byte order mark many programs prepend.

Fix 2: The Same Setting in pandas

pandas takes the encoding as a parameter: pd.read_csv("data.csv", encoding="cp1252"). If you are unsure which encoding applies and the file is small, encoding_errors="replace" will substitute a placeholder for bad bytes so you can inspect the damage, but treat that as a diagnostic step, not a final fix, because replaced characters are lost data.

How to Detect the Encoding Instead of Guessing

Guessing between two or three candidates is fine for one file. For recurring files, detect it. On the command line, the file utility on Linux and macOS often reports the charset. In Python, the third-party charset-normalizer package (a modern successor to chardet) can inspect the raw bytes: open the file in binary mode, read a chunk, and ask the library for its best guess, then use that encoding in your real read. Opening in binary mode with "rb" never raises a decode error, because no decoding happens until you ask for text. The official Python Unicode HOWTO explains the byte-to-text model behind all of this.

Fix It at the Source: Save the CSV as UTF-8

If you control the Excel file, the cleanest long-term fix is to save it properly once. In Excel, use Save As and choose CSV UTF-8 (Comma delimited) instead of the plain CSV option. Every downstream tool, not just Python, will then read it without special settings. When a system sends you files automatically, asking the sender for UTF-8 output saves repeated pain. If you are new to reading files generally, our guide to running Python files from the terminal helps you iterate on these tests quickly.

Common Mistakes to Avoid

• Using errors=”ignore” as the final fix. It deletes the unreadable bytes silently, mangling names and values. Your file loads, but the data is quietly wrong.
• Assuming the error means the file is corrupt. It almost never does. An encoding mismatch looks dramatic but leaves the underlying data untouched.
• Changing the encoding but not checking the output. Latin-1 will open almost anything, including files it interprets slightly wrong. Print rows with special characters and eyeball them.
• Forgetting the encoding when writing output. If you read with cp1252 and write results for other tools, specify encoding="utf-8" when writing so the next program has an easier time.

Special Case: Only One Row Breaks the File

Occasionally a file is 99 percent clean UTF-8 with a single bad byte hiding in one record — a name pasted from an old system, a stray smart quote in one description. The position number in the error is your map: it counts bytes from the start of the file to the first undecodable byte. To inspect that spot, open the file in binary mode, read up to a little past the position, and decode just that neighbourhood with errors="replace" so you can see the surrounding text and the offending character rendered as a placeholder. Very often you will recognise a familiar shape — a single é or an apostrophe stored in Windows-1252 inside an otherwise UTF-8 export.

For a one-off file, the pragmatic answer is to read the whole file with the encoding that matches the stray bytes, usually cp1252, which also reads pure UTF-8’s plain ASCII parts identically. For files that arrive regularly, fix the producing system if you can, and otherwise make your reader tolerant deliberately: read with UTF-8 first inside a try block, and on UnicodeDecodeError retry with cp1252, logging that the fallback fired so the data problem stays visible rather than silently absorbed.

Conclusion

A UnicodeDecodeError while reading a CSV is a translation problem: the file speaks Windows-1252 or Latin-1 while Python listened in UTF-8. Tell Python the right encoding in open() or read_csv(), verify the special characters look right, and save future exports as CSV UTF-8 whenever you control the source. The error byte and position in the message will even point you to the right answer once you know how to read them.

Frequently Asked Questions

Which encoding should I try first for an Excel CSV?

For files saved by Excel on Windows, try cp1252 first. For files that came from modern Excel, Google Sheets, or web systems and fail only at the very first character, try utf-8-sig. Keep latin-1 as the fallback that always opens, then verify the characters by eye.

Why does the error mention a position number?

The position counts bytes from the start of the file up to the first byte Python could not decode. It is useful because it tells you the problem is not at the start: everything before that byte decoded fine, so the file is mostly UTF-8 with a few stray bytes, or entirely another encoding that happens to share the early characters.

Can I make Python always open files as UTF-8?

Python 3.15 is moving toward UTF-8 as the default, and you can enable UTF-8 mode today with the environment variable PYTHONUTF8=1 or the -X utf8 flag. That helps only when files really are UTF-8, though; it does nothing for a genuine Windows-1252 file, which still needs an explicit encoding.

What is the difference between latin-1 and cp1252?

They agree on most characters, but cp1252 uses the byte range that latin-1 leaves as control codes for printable characters such as curly quotes, the euro sign, and bullet points. For real-world Excel files, cp1252 is the more accurate choice; latin-1 can display those punctuation bytes as invisible or wrong characters.

Why does the same file read fine on my colleague’s computer?

Because the default encoding Python uses when you do not specify one depends on the operating system and regional settings. A Windows machine set to a Western European locale may default to cp1252 and open the file without complaint, while your machine defaults to UTF-8 and fails. Specifying the encoding explicitly makes the code behave identically everywhere.