Vishnu Haridas has a nice post on UTF-8 and how it works. He’s impressed with the brilliance of its design. And why not? It provides code points for virtually every written language in the world while maintaining compatibility with ASCII. Among other things, “compatibility” means that any ASCII character is represented the same way in UTF-8 and that every file containing only ASCII characters is a valid UTF-8 file and that every UTF-8 file containing only ASCII characters is a valid ASCII file.
It almost seems like magic and Haridas is right to be impressed. He doesn’t mention the back story and if he’s unfamiliar with it, he’d be even more impressed. As I’ve written before, UTF-8 was designed over dinner on a paper placemat by the legendary Ken Thompson and Rob Pike. But wait. There’s more. After dinner Pike and Thompson returned to Bell Labs and while Thompson implemented the packing and unpacking code, Pike dug into the Plan 9 graphics library and began changing it to use UTF-8. By the end of the night they were done and by the end of the next day, Plan 9 was running UTF-8 and only UTF-8.
These days it seems we can’t even get a vacation request accomplished that quickly but Thompson and Pike designed, implemented, and ported into Plan 9 a world changing character encoding that revolutionized digital typography in the space of a couple of days.
Another part of the back story is that Pike and Thompson didn’t just decide to invent UTF-8 on a whim. They were motivated by what they considered the inadequate, committee designed proposals floating around and, in particular, one from IBM that they were asked to review. You can read the whole story as told by Rob Pike if you want to know all the details and get a peek at Thompson’s original code.