Mojibake

Mojibake is garbled text produced when bytes written in one character encoding are decoded with a different one. The bytes are intact; the reader's map from bytes to characters is wrong. The word is Japanese — 文字化け, "character transformation" — from a time when Shift-JIS, EUC-JP and ISO-2022-JP files were routinely opened with the wrong one.

The Western classic: UTF-8 read as Windows-1252

UTF-8 writes every character outside ASCII as two to four bytes. Windows-1252 maps each single byte to its own character. Read one as the other and every accented letter or curly quote multiplies:

Character UTF-8 bytes Read as Windows-1252
é C3 A9 é
ü C3 BC ü
’ E2 80 99 ’
— E2 80 94 —
€ E2 82 AC €

Spotting it is easy once you know the tell: a capital à or a †pair where a single character should be.

The other direction: �

Decode a Windows-1252 or Latin-1 file as UTF-8 and the single byte E9 (é) is not a valid UTF-8 sequence. The decoder replaces it with U+FFFD, �. Technically not mojibake — no wrong character was chosen — but it is the same mistake, and it is worse: the original byte is gone.

Repairing it

Symptom Recoverable? How
é, ’ Yes s.encode("cp1252").decode("utf-8")
é Yes Reverse twice, or ftfy.fix_text(s)
� No Go back to the original file and re-decode

The real fix is upstream: write files as UTF-8 and declare it — a <meta charset="utf-8"> in HTML, charset=utf-8 in the HTTP Content-Type, or a byte order mark where the reader (Excel) ignores everything else.

In HTML reports

A browser picks an encoding from, in order, a BOM, the HTTP header, a <meta charset> in the first 1024 bytes, and finally a guess. A generated report opened straight from disk has no header, so a missing meta tag means the browser is guessing — and on many systems it guesses Windows-1252. Every ’ in an HTML page traces back to that chain.

Try it

Comma is free — unlimited reports, unlimited commenters, unlimited revision history.

Fix ’ and é in an HTML page →

Related