Unicode Normalization and Invisible Characters
Text that looks the same is not always the same. Unicode lets one visible character be built in more than one way, allows characters that render as nothing at all, and includes pairs that are indistinguishable on screen but different underneath. This guide explains where those differences come from and what to do about them before they turn into a failed login, a duplicate record or a spoofed domain.
Code points, bytes, and what "length" means
A string has at least three plausible lengths, and they disagree. "héllo" might be 5 characters to a reader, 5 or 6 code points depending on how the é is built, and 6 or 7 bytes in UTF-8. Emoji make the gap wider: a family emoji is a single picture assembled from several people joined by invisible connectors, and most languages report its length as somewhere between 2 and 11.
Pick the unit that matches the question. Byte length is what a database column limit and an HTTP header measure. Code-point count is what most string APIs give you. What a person means by "characters" is the grapheme cluster count — the thing that moves the cursor one step — and getting a truncation to land on that boundary is the difference between a clean cut and a broken emoji.
The same text, two encodings: NFC and NFD
"é" can be a single code point (U+00E9) or two: a plain "e" followed by a combining acute accent (U+0301). Both render identically, but they are different sequences, so a naive equality check says they are different strings, a database unique index accepts both, and a search for one misses the other. Normalization picks a canonical form: NFC composes into the single code point, NFD decomposes into base plus mark.
This is not exotic. Text typed on one platform and pasted from another routinely disagrees — macOS has historically stored filenames decomposed while most of the web sends composed — and Korean Hangul has the same split between precomposed syllables and separate jamo. NFC is the right default for storage and transmission; the important part is picking one and applying it at the edge, not sprinkling it wherever a comparison happens to fail.
NFKC and NFKD: when "almost the same" should count
The K forms fold compatibility differences as well as canonical ones. Under NFKC, the fi ligature becomes "fi", fullwidth Ａ becomes "A", the superscript ² becomes "2", and the non-breaking space becomes an ordinary one. That is exactly what you want when comparing identifiers, usernames or search terms, where two spellings should not be able to claim the same slot.
It is exactly what you do not want for display or storage of arbitrary text, because the transformation is lossy and irreversible: mathematical styled letters collapse into plain ASCII, and formatting the author chose disappears. Use the K forms to derive a comparison key, and keep the original for showing back to the user.
The characters you cannot see
Plenty of code points render as nothing. A byte order mark at the start of a file breaks the first key of a JSON document or the first header of a CSV. Zero-width spaces and joiners survive a copy-paste out of a document and quietly break an exact match. Non-breaking spaces look like spaces but do not split on whitespace, and soft hyphens appear only when a word wraps.
Some are worse than untidy. Bidirectional override characters can reorder how source code displays without changing what the compiler reads — the "Trojan Source" trick, where a comment on screen is executable code underneath. And homoglyphs — Cyrillic а, Greek ο, fullwidth letters — let a domain or username look exactly like a familiar one. If a string crosses a trust boundary, inspect its code points rather than trusting your eyes.
Rules that keep text safe
Normalize at the boundary, once. Convert incoming text to NFC as it enters the system, and store it that way; then internal comparisons are comparing like with like. For anything used as an identity — usernames, email local parts, slugs, lookup keys — derive a separate comparison key with NFKC plus case folding, and enforce uniqueness on that key rather than on the display value.
Then decide, explicitly, what you allow. Strip the BOM, reject or strip zero-width and bidi control characters in fields that have no business containing them, and treat a mixed-script identifier as suspicious rather than clever. None of this is expensive, and all of it is far cheaper than debugging why one particular user cannot log in with a password they are typing correctly.
- NFC for storage and transport; NFKC + case folding for comparison keys.
- Normalize once at the edge, not at every comparison.
- Strip the BOM before parsing JSON, CSV or config files.
- Inspect code points when a string looks right but behaves wrong.