Here's what Daniel wrote in this week. He says we've spent a few episodes on multilingual TTS and speech to text, on the road bumps you hit when you code switch or try to get accurate output across languages. And underneath all of that, he says, is one thing nobody thinks about: what character set you're actually working in. He wants to talk about that layer. The modern standard is UTF-8, the predecessor is ASCII, and he's seen firsthand the kind of crazy-inducing failure modes that show up when people collaborate on documents without paying attention to the system.
He's got a story, doesn't he.
He's got a story. A few years back he was involved in a major technical operation preparing for an event in New York City, several hundred attendees. Distributed team, Israel, the UK, New York, all working out of a Google Sheets attendee list full of names from Norway, Spain, France, accents and diacritics everywhere. He eventually standardized the backend on Airtable, but by then he was left with the hopeless task of trying to sanitize a CRM export that some program had standardized on UTF-8 and, in the process, mangled the names of a lot of people who were about to get an email.
Hopeless is the right word for it. That's the part people don't understand.
So that's the shape of today. Why the character set is such a fundamental aspect of computing, and prior even to the keyboard layout variants most of us use. How things get complicated when Unicode runs into different world languages. And what you actually need to know if you're working with international data and international collaborators. Where do we start with that?
Start with what the thing is, because "character set" is one of those phrases everybody's heard and almost nobody can define. A character set is the mapping between the abstract symbols of written language and the bytes a computer stores. That's it. It's a lookup table. And it sits below everything else you touch. Below the keyboard layout, below the font, below the file format. Everything above it assumes it's correct.
Which is exactly why nobody notices it until it's wrong.
Right, and there are three levels here that people constantly collapse into one, and the whole episode falls out of keeping them separate. First, the code point. That's an abstract number assigned to a symbol. U plus zero zero E nine is é. Just a number, no bytes yet. Second, the encoding. That's how that number becomes actual bytes on disk. UTF-8, UTF-16, Latin-1, Windows-1252, all different answers to the same question. Third, the grapheme. That's what a human being looks at and calls one character.
And those three are not the same thing.
They are not the same thing, and that's where the bodies are buried. Here's the thesis for the whole episode: when the encoding layer is wrong, nothing above it can be trusted. Not the spreadsheet. Not the CRM export. Not the phonemizer in your text to speech system.
So the arc is mechanism first, then the failure pattern, then what it means for anyone actually shipping multilingual anything.
That's the arc.
Then start with the mechanism. What ASCII actually is, and how UTF-8 grew out of it.
ASCII is 1963. It defines one hundred and twenty-eight code points, zero through one hundred and twenty-seven, covering the English letters, the digits, punctuation, and a set of control characters that were originally for teletypes. It's a seven-bit encoding. That's the ancestor. And its ghost haunts every legacy system you will ever touch.
One hundred and twenty-eight characters. That's the entire world, as far as 1963 was concerned.
And it's worth being fair to it. For English, ASCII is complete. Every letter, every mark you need. The problem is that "English" is not "the world," and the people who designed the systems on top of ASCII spent about thirty years pretending otherwise.
So how does UTF-8 relate to it? Because that's the part I think people get fuzzy on.
UTF-8 is a strict superset of ASCII. The first one hundred and twenty-eight code points encode to identical single bytes. Byte for byte, an ASCII file is already a valid UTF-8 file. You don't convert anything. You don't migrate anything. You just declare it UTF-8 and it works.
And that's why it won.
That's exactly why it won. It's the only design that could be adopted without breaking thirty years of existing files. Every other candidate required a flag day. UTF-8 required nothing. You could roll it out incrementally, file by file, and the ASCII files kept working the whole time.
Which is a lesson that generalizes way past encoding.
It's the single most important lesson in standards, honestly. The format that wins is almost never the most elegant one. It's the one that doesn't require anybody to rewrite anything.
So ASCII is one byte per character, always. UTF-8 is not.
UTF-8 is variable width. One byte for anything in the ASCII range, two to four bytes for everything else. And that's the root of the first failure class, which is the byte length trap. Byte length is not character count. If you have code that slices a string by bytes, it will split a multi-byte character in half and produce garbage.
Give me the concrete version.
The concrete version is a form field with a character limit that counts bytes instead of characters. Somebody types a name with an accent in it, the field says you're over the limit, and they haven't typed anything unusual. Or the reverse. A truncation routine that cuts at byte two hundred and leaves half a character at the end of a name, and now the last glyph in the email is a black diamond with a question mark in it.
And this is the point where somebody in the meeting says "just strip the accents."
Somebody always says that. Hold that thought, because that's a whole separate failure pattern and it's the one Daniel ran into.
Then let's get the other structural facts on the table, because there are a few and they don't blur together. UTF-16.
UTF-16 is what Windows, Java, and JavaScript use internally. It has big-endian and little-endian variants, which means the same file can be read two different ways depending on which order you assume the bytes go in. That's why UTF-16 needs a byte order mark. The mark tells the reader which endianness it's looking at.
And UTF-8 doesn't need one.
UTF-8 does not need one, because there's no endianness to get wrong. It's a byte stream, not a sequence of two-byte units. And a byte order mark in a UTF-8 file is not standard. It's not forbidden exactly, but it's abnormal, and the standards people will tell you not to do it. One practitioner put it more bluntly than I would and called it a Microsoft hack, because endianness has no impact on UTF-8 at all.
Noted, and we'll come back to the byte order mark, because that's the crux of the Excel problem and it's a trap that looks like a fix.
It absolutely is. Now the grapheme problem, which is the one that actually matters for anybody doing internationalized work. Take é. That can be stored as one code point, U plus zero zero E nine. Or it can be stored as two: a plain letter e, followed by a combining acute accent, U plus zero three zero one. Visually identical. Two completely different byte sequences.
So two files can look the same on screen and not be equal.
Not equal, not sort the same, not match in a search. And here's the line I'd put on a poster. Graphemes are almost always the closest thing to what people mean when they say "character." The human meaning lives at the grapheme level. The bytes live at the code point level. And any code that operates on bytes or code points while the meaning lives at the grapheme level is going to mangle things.
Which is why the advice is don't split or concatenate strings when you're doing internationalized work.
Don't do it. Do not pluck out the first grapheme of a name to make an avatar initial. Do not truncate a display name to twelve characters by counting bytes. It will look like nonsense, and the person whose name it is will see it.
There's a related thing here, normalization.
Normalization is the formal name for the problem I just described. The same visible character can be stored two ways, precomposed or decomposed, and Unicode defines forms that say which one you're using. NFC is precomposed. NFD is decomposed. And the historical split is that macOS enforced NFD, while most of the rest of the world used NFC.
And that produced real bugs.
It produced a nasty class of bug. A file named café, written on Linux in NFC, could appear to be deleted on macOS, because macOS was looking for the NFD form and the file it found didn't match. Git had to add special workaround code to handle it. And it's worse than the simple version, because modern macOS doesn't enforce NFD at the operating system level anymore, but Finder and a lot of Apple applications still use it, and it's Apple's own modified version. So you can type a filename identically and have the find command fail to locate it.
That's a very specific kind of madness.
Apple's filesystem had the same problem. APFS was initially unusable with most non-English languages until Apple added normalization-insensitive volumes, and that didn't land until macOS ten point thirteen.
So a filesystem that would lose your files if your language used accents.
Not lose them. Fail to find them, which from the user's chair is the same experience.
Let's do the astral plane, because that one has a great historical punchline.
Characters outside the Basic Multilingual Plane need four bytes in UTF-8. That's emoji, rare CJK characters, historic scripts. And MySQL famously destroyed those characters outright until 2010, when utf8mb4 arrived in version five point five point three.
The timing there is not a coincidence.
The timing is not a coincidence. Emoji started spreading outside Japan around 2008. Two years later, the database that couldn't store them got fixed. I'd like to say that's a coincidence, but that's not really likely.
So the lesson is that "UTF-8 support" is often a lie.
"UTF-8 support" is often a lie. A lot of systems say UTF-8 and mean "UTF-8 up to three bytes." They silently truncate anything above that. You don't get an error. You get a replacement character, or you get nothing, and the emoji or the rare character just isn't there anymore. That's the misconception to kill: supporting UTF-8 is not the same as supporting Unicode.
That's the theory. Now let's talk about what actually happens when this meets a spreadsheet and a team in three time zones.
This is where Daniel's story becomes the whole episode. Excel on Windows, opening a UTF-8 CSV that has no byte order mark, will not read it as UTF-8. It will read it as a legacy eight-bit codepage. Usually Windows-1252. And what you get is mojibake. José becomes J, o, s, A with a tilde, copyright sign. Björk becomes B, j, A with a tilde, paragraph sign, r, k.
That's the signature.
That's the double-encoding signature. It's unmistakable once you've seen it. The bytes were correct. The interpretation was wrong. And because the interpretation was wrong, the program rendered each byte as if it were a separate character in a different alphabet, and now you have two characters where there should be one.
So Daniel's attendee list, with Norwegian and Spanish and French names.
Every accented name in that list goes through that transformation the moment somebody opens the file the wrong way. And here's the thing about a distributed team. The person in New York opens it and sees mojibake. The person in Israel opens it and sees it fine. The person in the UK opens it and sees something else. And they're all looking at the same file.
So the meeting becomes an argument about whose screen is right.
And nobody knows the word mojibake, so nobody can even name what's happening.
Now, the fix everybody reaches for is the byte order mark.
The fix everybody reaches for is the byte order mark, and it's a trap. Adding a UTF-8 BOM does make Excel detect the encoding correctly. That part works. But the standard doesn't permit a BOM in UTF-8, and it can cause problems on other systems. It's a workaround that trades one class of failure for another.
And there's official guidance on this.
The UK Government's Open Standards Board looked at exactly this and recommended UTF-8 without a byte order mark. And their guidance for Windows Excel users is to use Import Text rather than double-clicking the file. That's the whole fix. Don't open the CSV. Import it, and tell the importer what encoding it is.
Which is a much less satisfying answer than a magic byte sequence.
It's a much less satisfying answer, and that's why nobody does it.
But the byte order mark isn't even the worst outcome. The silent re-save is.
The silent re-save is the real killer, and this is the part that explains why Daniel called his task hopeless. You open the CSV, it's already mangled, you fix a couple of names by hand, and then you press Ctrl+S. And Excel saves it in some arbitrary codepage, and now the mangling is written to disk. You end up with a document where the name column reads one, question mark question mark question mark question mark, two, Kevin Dub question mark i s.
Kevin Dub?is.
Kevin Dub?is. That's a real example from a practitioner describing exactly this. And the data is destroyed on save, often with no warning at all. The original bytes are gone.
Gone as in recoverable-with-effort, or gone as in gone?
Gone as in not reliably recoverable. And this is the part people don't believe until it happens to them. The transformation is lossy. Multiple distinct inputs can map to the same question mark. So when you see a question mark in the output, you cannot know what character was there. Was it an accent? A different letter entirely? A character from a script you don't recognize? There's no way to invert it, because the information isn't in the file anymore. It was thrown away at save time.
So the "sanitize the CRM export" job wasn't hard. It was impossible.
It was impossible, and it was impossible in a way that's invisible from the outside. Daniel's description of it as hopeless is exactly right. You can guess. You can look up the person on LinkedIn and see how they actually spell their name. But you cannot recover it from the file, because the file no longer contains it.
And there's a locale wrinkle on top of all this.
There's a locale wrinkle. Excel uses locale-dependent field separators. In some countries, a CSV is semicolon-separated, not comma-separated. So the same file opens differently depending on the user's locale, and a comma-separated file opened on a machine expecting semicolons comes in as one giant column.
So you have two independent failure pattern stacked on the same file.
Two independent failure pattern, and they interact. You can fix the encoding and still have the columns wrong, or fix the columns and still have the names mangled.
Now, Daniel specifically wanted the keyboard layout connection, because he says the character set is prior even to that.
He's right, and it's worth stating cleanly. A keyboard layout is just a mapping from physical keys to code points. QWERTY, AZERTY, Dvorak, Colemak, they're all answering the same question: when I press this key, which code point comes out? The layout determines which code points you can type easily. The encoding determines how those code points get stored. Two completely different layers.
So you can have a perfect AZERTY layout and still produce mojibake.
You can have a perfect AZERTY layout and produce mojibake all day long, because the layout did its job and the encoding didn't. And the reverse is true too. A US keyboard can type any Unicode character you want, through Alt codes, compose keys, or an input method editor. The layout is a convenience layer. It's not a constraint on the character set.
Which means the character set really is the more fundamental thing.
It's the more fundamental thing, and it's the one you can't fix by buying a different keyboard.
Alright. Connect this to the speech stuff, because that's where Daniel started and I don't want to leave it hanging.
This is the payoff for the last few episodes. Every multilingual text to speech system and every code-switching system assumes the text arrives correctly encoded. That's the assumption underneath everything. Garbage in the character set means garbage in the phonemes, which means garbage out of the speaker.
And there's current work that names this directly.
There's a paper from September this year, out of the NCMMSC conference, that diagnoses how low-resource multilingual TTS fails. And the failure list is exactly what you'd predict. Numbers, dates, named entities, long sentences, code-switched expressions, punctuation-related structures. And they introduce something called a Text Risk Score, which flags risky inputs before synthesis even runs.
So they're scoring the text before they try to speak it.
They're scoring the text before they try to speak it, which is the speech-side mirror of the spreadsheet problem. The structure of the input text determines whether the output is intelligible. And encoding is part of that structure. If your é arrived as two characters, the phonemizer is looking at a letter e and a stray accent mark, and it has to guess.
And code-switching specifically.
Code-switching is an active frontier. F5-TTS advertises seamless code-switching capability. There's Indonesian-English work, Hindi and Indian-English accent work, and the thing they all have in common is per-word language identification. You have to know which language each token belongs to before you can pronounce it. And that presupposes clean, correctly-encoded text, because if the token is mangled you can't even tell which language it was.
The datasets are scaling up too.
SwitchLingua is four hundred and twenty thousand code-switched text samples across twelve languages, with over eighty hours of audio. CS-FLEURS covers a hundred and thirteen language pairs across fifty-two languages. The field is building the infrastructure to handle exactly the multilingual reality Daniel's attendee list represented.
So the attendee list was a preview of a research problem.
The attendee list was a preview of a research problem, and the researchers are solving it at the dataset level while the office is still solving it by hand in a spreadsheet.
Give me the practical guidance, then. What does somebody actually do.
Never assume UTF-8, but default to it. Detect, don't guess. If the first bytes of a file are F F F E or F E F F, that's UTF-16, not UTF-8, and you should stop and look at it.
Prefer UTF-8 without a byte order mark for interchange.
Prefer UTF-8 without a byte order mark. Use the mark only as a targeted workaround for Excel on Windows, and know what you're trading away when you do it.
Never round-trip international data through CSV in Excel.
Save as dot x l s x. Use Import Text for CSVs. That one habit would have saved Daniel's attendee list entirely.
Normalize explicitly.
Normalize explicitly, usually to NFC, at your system boundaries. And be aware that macOS and Finder may hand you NFD whether you asked for it or not.
Don't sanitize names by stripping non-ASCII.
Don't sanitize names by stripping non-ASCII. That's not sanitization, that's data destruction. If you need to transliterate, do it deliberately, do it reversibly, and keep the original alongside it. Never throw away the source.
Treat encoding as a schema decision.
Treat encoding as a schema decision, not an afterthought. Daniel's Airtable standardization worked because it fixed the encoding contract at the platform level. That's the same principle as choosing utf8mb4 in a database. You decide once, at the foundation, and everything above it inherits the decision.
Test with real international data.
Test with Norwegian ø and å, Spanish ñ, French é and ç. And crucially, test with four-byte characters. Emoji, CJK. Those are the ones that expose the three-byte truncation bugs, because a system that lies about supporting UTF-8 will pass every European test you throw at it and fail the moment somebody puts a flag emoji in a name field.
The three-byte lie.
It's the most common form of the problem and the hardest to catch, because it works perfectly until it doesn't.
The word you keep using is encoding. I want to push on it, because I don't think it's the word most people would use for this. Everyone I know would call it a character set. You keep saying encoding.
Hilbert: Character set is the wrong word for what went wrong.
Go on.
Hilbert: A character set is what the characters are. What bit you in the spreadsheet is how they were written down. Different thing. And the reason I know that is I spent about four months doing data migration for an insurance company. Mid-sized outfit, and they'd been running the whole book of business on a system that went in in the eighties and had never been replaced. My job was to move the policyholder records across. Names, addresses, beneficiary details. All of it.
The encoding was the problem.
Hilbert: Nobody knew what the encoding was. It was proprietary, it predated anybody caring, and there was no documentation. So I reverse-engineered it by hand. Sat there with a printed code table I found in a filing cabinet, matching byte values against it, one at a time, for weeks.
A printed code table.
Hilbert: Paper. Folded in thirds. Somebody had written their lunch order on the back of it. I kept it.
You kept it.
Hilbert: It's in a box. But here's the thing I came out to say. The problem isn't only that systems re-encode without telling you. It's that the people who built the original system had no idea they were making an encoding decision at all. They took whatever the vendor gave them. It was invisible to them. And it stayed invisible for thirty years, until it wasn't.
Then it was your problem.
Hilbert: Then it was my problem. There was one policyholder. Surname had a character in it that mapped to three different byte values depending on which decade the record had been entered. Three. The system had been migrated twice before me and nobody had noticed either time.
The same name, three different spellings, all in the same database.
Hilbert: All in the same database, all technically correct for the decade they were entered. I don't fully remember what we did with him. I think we picked one and moved on. Anyway. I have to go, the place on Emek Refaim closes at six and I need to collect something.
Hilbert, that code table is a perfect artifact of the problem. An encoding decision nobody knew they were making, written down on paper, kept in a box.
The three-byte surname is the whole episode in one record. Same name, three answers, and nobody in the building who could tell you which one was right.
Here's the thing I keep circling. If encoding is a schema decision, and the guidance is clear, and the failure pattern are this well documented, why do we still treat it as an afterthought? The same mojibake is happening in offices right now that was happening fifteen years ago.
It gets more critical, not less, as the speech systems scale. Every advance in code-switching TTS assumes clean text going in. The character set isn't a legacy concern. It's the substrate.
Which means the next time you open a CSV and see José, you'll know exactly what happened. And you'll know it might already be too late to fix.
Thanks as always to our producer, Hilbert Flumingtop.
This has been My Weird Prompts, the human-AI collaboration podcast. If you want to send us something, email us at show at my weird prompts dot com. We'll be back soon.
