Skip to content
AZ Tools

Why your text turns into strange characters

There is no such thing as plain text on disk. A file holds bytes, and an encoding is the agreement that says which byte or group of bytes stands for which character. A plain text file carries no record of that agreement, so every program that opens one is making an assumption. When the assumption is wrong you get mojibake, and the particular shape of the mess is a reliable clue to what happened.

The garbage has signatures worth recognising

The most familiar one is an accented letter that turns into two. In UTF-8 the letter é is stored as the two bytes C3 and A9, and a program reading it one byte at a time in a Western European code page prints those as à and ©. Every UTF-8 file read as a legacy single-byte encoding turns each non-English character into two or three characters this way, and the run of them is what makes Korean or Japanese text read as a wall of Latin debris.

Three other marks say something different. A file that opens with  is UTF-8 with a byte order mark being read as something else. A black diamond with a question mark is the replacement character, which a decoder writes when the bytes could not be valid in the encoding it was told to use. A plain question mark where a letter should be usually means the text was converted into an encoding with no room for that letter.

Two of those signatures mean the text is already gone

é is recoverable, because the original bytes are still in the file and only the interpretation is wrong. Reading the same bytes with the right encoding brings the text back exactly.

The replacement character and the bare question mark are different: they are what the file now contains. The original bytes were discarded at the moment of conversion, and no later step can work out whether that diamond used to be é or ü. This is why the first rule of repairing an encoding problem is to stop saving over the file — every save through the wrong encoding converts a reversible problem into an unrecoverable one.

Double encoding, and why it repeats

If a program reads UTF-8 as a single-byte encoding and then saves the result as UTF-8, the mistake becomes permanent-looking: the file now genuinely contains the characters à and ©, each stored as two bytes of its own. Do it twice and one letter becomes four characters, then eight.

It is still reversible, because the process is exactly invertible if you know how many rounds happened: encode back to the single-byte code page, then decode as UTF-8, once per round. The reason this pattern shows up so often is that it usually happens inside a pipeline — a form, a database column, an export — where the same conversion runs on data that has already been through it.

The legacy encodings you still meet

Western text usually lands in Windows-1252, which is what most people mean when they say Latin-1. The difference matters: Windows-1252 fills the range that Latin-1 leaves empty with curly quotes, the em dash and the ellipsis, which is why a smart quote so often surfaces as “. Text from older CJK systems arrives in Shift_JIS, EUC-KR, Big5 or GB18030, and Cyrillic in Windows-1251 or KOI8-R.

Spreadsheets are the usual source in an office. Exporting a CSV on Windows still tends to write the system code page rather than UTF-8, so a file that looks perfect on the machine that made it turns to mojibake everywhere else — and re-importing it on that same machine hides the problem from the person who created it.

Detection is guessing, and it is worse on short text

UTF-8 is self-validating: its multi-byte sequences follow a strict pattern, so a file that decodes cleanly as UTF-8 almost certainly is UTF-8. That is the one confident answer a detector can give.

Every single-byte encoding, by contrast, is valid for every possible byte, so nothing rules any of them out. A detector picks between them by looking at which letters and letter pairs are common in real languages, which works on a paragraph and fails on a name or a column heading. A tool that reports Windows-1252 with low confidence for a twenty-byte string is being honest, not unhelpful.

Where a file is supposed to declare itself

Formats that carry a declaration are much easier to get right. HTML has a meta charset, XML has its declaration, and anything served over HTTP can state a charset in the Content-Type header — which wins over what the document says about itself, so a correct page served with a wrong header still breaks.

A byte order mark at the start of a UTF-8 file is an alternative kind of declaration, and it cuts both ways. It removes the guesswork for editors that understand it, and it breaks tools that do not: a JSON parser that reports an unexpected character at position zero, a shell script whose first line is no longer a shebang, or a CSV whose first column name has an invisible character glued to it.

How to actually fix a broken file

Work from the bytes, not the screen. Look at the file in hex to see whether the accented characters are stored as one byte or two, since that alone separates a legacy file from a UTF-8 file being misread. Then try decoding with a candidate encoding and check that real words come back — the test is whether the text is right, not whether it changed.

Once it is readable, save a copy as UTF-8 and keep the original untouched until you are sure. And fix it at the boundary rather than in the middle: decode when data enters, keep one encoding everywhere inside, declare it on the way out. Most repeat offenders are pipelines that convert somewhere in the middle, which is exactly how a single mistake gets applied twice.

Related tools