Mojibake
Mojibake is garbled text produced when bytes written in one character encoding are decoded with a different one. The bytes are intact; the reader's map from bytes to characters is wrong. The word is Japanese — 文字化け, "character transformation" — from a time when Shift-JIS, EUC-JP and ISO-2022-JP files were routinely opened with the wrong one.
The Western classic: UTF-8 read as Windows-1252
UTF-8 writes every character outside ASCII as two to four bytes. Windows-1252 maps each single byte to its own character. Read one as the other and every accented letter or curly quote multiplies:
| Character | UTF-8 bytes | Read as Windows-1252 |
|---|---|---|
é |
C3 A9 |
é |
ü |
C3 BC |
ü |
’ |
E2 80 99 |
’ |
— |
E2 80 94 |
— |
€ |
E2 82 AC |
€ |
Spotting it is easy once you know the tell: a capital à or a â€
pair where a single character should be.
The other direction: �
Decode a Windows-1252 or Latin-1 file as UTF-8 and the single byte E9
(é) is not a valid UTF-8 sequence. The decoder replaces it with U+FFFD,
�. Technically not mojibake — no wrong character was chosen — but it is
the same mistake, and it is worse: the original byte is gone.
Repairing it
| Symptom | Recoverable? | How |
|---|---|---|
é, ’ |
Yes | s.encode("cp1252").decode("utf-8") |
é |
Yes | Reverse twice, or ftfy.fix_text(s) |
� |
No | Go back to the original file and re-decode |
The real fix is upstream: write files as UTF-8 and declare it — a
<meta charset="utf-8"> in HTML, charset=utf-8 in the HTTP
Content-Type, or a byte order mark where
the reader (Excel) ignores everything else.
In HTML reports
A browser picks an encoding from, in order, a BOM, the HTTP header, a
<meta charset> in the first 1024 bytes, and finally a guess. A
generated report opened straight from disk has no header, so a missing
meta tag means the browser is guessing — and on many systems it guesses
Windows-1252. Every ’ in an HTML page
traces back to that chain.
Try it
Comma is free — unlimited reports, unlimited commenters, unlimited revision history.
Fix ’ and é in an HTML page →