What is a byte order mark?

A byte order mark (BOM) is the Unicode character U+FEFF written at the very start of a text file to signal its encoding. In UTF-8 it is the three bytes EF BB BF; in UTF-16 it is FF FE (little-endian) or FE FF (big-endian), where it also tells the reader which byte order the file uses.

Should a UTF-8 file have a BOM?

Usually not. UTF-8 has no byte order to signal, and the Unicode standard neither requires nor recommends a BOM for it. The exception is a CSV meant to be double-clicked into Excel, which reads a CSV without a BOM in the legacy system code page and turns accented characters into mojibake.

What breaks when a file has a BOM?

Anything that expects the first bytes to be meaningful: a shell script's #! line stops working, JavaScript's JSON.parse throws on the leading character, PHP sends it as output before headers, and CSV readers that don't strip it put an invisible character in the first column's name.

How do I remove a BOM?

Most editors show the encoding in the status bar and can save as UTF-8 without BOM. In Python, read with encoding utf-8-sig, which strips it if present. On the command line, sed -i '1s/^\xEF\xBB\xBF//' file removes it with GNU sed.

Byte order mark (BOM)

A byte order mark is the Unicode character U+FEFF placed at the very start of a text file to signal its encoding. It is invisible in every editor that understands it, and a stray garbage character — ï»¿ — in every program that doesn't.

Encoding	BOM bytes
UTF-8	`EF BB BF`
UTF-16LE	`FF FE`
UTF-16BE	`FE FF`

In UTF-16 the BOM does real work: it says which byte comes first. UTF-8 has no byte order, so there it is only an encoding signature — and the Unicode standard neither requires nor recommends it.

Where you want one: CSV for Excel

Double-click a UTF-8 CSV without a BOM on Windows and Excel decodes it in the legacy system code page. Zoë Müller becomes ZoÃ« MÃ¼ller. Add the three BOM bytes and the same file opens correctly — which is why JSON to CSV writes one by default, and why a CSV that opens wrong in Excel is so often fixed by re-saving as "CSV UTF-8".

Where it breaks things

Shell scripts. The kernel looks for #! in the first two bytes. A BOM in front means the shebang is ignored.
JSON. RFC 8259 says senders must not add one. JavaScript's JSON.parse throws on a leading ; many other parsers tolerate it.
PHP and templates. The BOM is output before <?php, so headers have already been sent.
CSV readers. Python's csv with encoding="utf-8" keeps it, so the first header is "id" and row["id"] raises KeyError. Use utf-8-sig.

In HTML

Browsers check for a BOM before anything else, and the HTML standard lets it override both the Content-Type charset and <meta charset>. A report whose <meta> says one encoding and whose bytes start with a BOM for another will render by the BOM. If a published report shows Ã© where é should be, check the first three bytes before the meta tag.

Adding and removing it

Remove: "Save with encoding → UTF-8" (not "UTF-8 with BOM") in VS Code; sed -i '1s/^\xEF\xBB\xBF//' file with GNU sed; utf-8-sig when reading in Python.
Add: prepend to the string before writing the file, or choose "CSV UTF-8" in Excel's Save As.
Check: head -c3 file | xxd prints efbb bf when one is there.

Try it

Comma is free — unlimited reports, unlimited commenters, unlimited revision history.

Convert JSON to an Excel-ready CSV →