A month ago, we discussed an article about just how difficult text rendering is, and today we get to take a look at the other side of the coin – text editing.
Alexis Beingessner’s Text Rendering Hates You, published exactly a month ago today, hits very close to my heart.
Back in 2017, I was building a rich text editor in the browser. Unsatisfied with existing libraries that used ContentEditable, I thought to myself “hey, I’ll just reimplement text selection myself! How difficult could it possibly be?” I was young. Naive. I estimated it would take two weeks. In reality, attempting to solve this problem would consume several years of my life, and even landed me a full time job for a year implementing text editing for a new operating system.
Best line in the article.
I think unicode has just gotten out of hand. Way too much work goes into emojis and there’s too much chaos. The bugs described in the article are the result of trying to abuse unicode to become a kind of markup language with things like color modifiers. I think expanding unicode this way was a huge mistake, one that we may never recover from.
I don’t have a problem with emojis per say, but it should have been implemented as a separate markup language on top of unicode so as to not pollute unicode itself. Different clients don’t always show the same emojis for the same code, so things are pretty broken today. Instead of adding tens of thousands of character combinations to the unicode standard for the fun of it, a simple/consistent/portable alternative could be to use a markup language with embeddable GIFs or better yet scalable vector images so we don’t have to keep adding stupid things into unicode. This way you could have a picture of an old man with green hands and a purple hat without having to petition for more garbage in unicode. This would not only ensure both the sender and receiver are always looking at the same thing regardless of vendor, but it’s also future-proof and doesn’t require updates just to see the latest emojis.
So your alternative to address the comparatively small bit of complexity emojis introduce is to replace it with something that is vastly more complex at every level, but is also less portable and would be even more difficult to handle?
Drumhellar,
It doesn’t sound like you read the article. Turning unicode into a markup language for emojis makes unicode more complex and buggy. Separating unicode from emojis would simplify both. Emojis would not only be more flexible but they would be atomic if they were not constrained to unicode, which simplifies everything.
So, using html tag style here purely for illustrative purposes here, but it doesn’t have to be HTML:
Which would appear as
Edit: No joke, the comment I posted contained the emoji, but apparently it’s not supported…what irony.
Having a dedicated markup language outside of unicode would be much more future-proof. Adding new properties would not be an issue, such as adding full color gamut emojis rather than just skin tones.
https://emojipedia.org/modifiers/
If someone wants to support emojis in their application, then that’s fine, but by making it part of unicode it makes everything that inputs or outputs text potentially more complex and buggy even in contexts where there’s no good reason to support emojis in the first place. That’s the reason they should not be part of unicode.
Those types of modifiers exist throughout Unicode, and have existed long before emojis. The implementation of a skin-tone modifier for certain emojis is something that already exists in various places in within Unicode, and in an identical form. It is merely the application of something like Combining Diacritical Marks on yet another section of Unicode. This application to emojis is really just the first time it’s common to English speakers. Other languages using Latin character sets have to deal with this exact situation, but also with the added complexity that many equivalent diacritical + character combinations already have specific characters to represent them.
For example, c and c are the same characters, but the first is represented in the byte stream as U+0237 U+0063, and the second is U+00E7. (Note that it may not display properly in your browser – it displays properly in charmap.exe, but not in the text box, and I haven’t seen how it is rendered once the comment is finished. Notepad renders it correctly) In the first example, the cedilla ( , ) is the second character of the two, in the exact same way skin tone modifiers trail the corresponding emoji. The cedilla isn’t something that can be applied to all characters, though. If you place the combining cedilla character after a Z, it won’t combine correctly, in the same way a skin-tone modifier won’t modify a car emoji, because it isn’t meant to.
Drumhellar,
Yes, unicode must be able to represent letters and their marks, but emojis take it to a ridiculous level. There’s a smiley face, a smiley face with big eyes, a smiley face winking, a smiley face with one tear, a smiley face with two tears, a smiley face sideways, a slightly less smiley face, an upsidedown smiley face, a smiley face with red cheeks, a smiley face with a halo, a smiley face with two heart eyes, a smiley face with three hearts, a smiley face blowing a heart, a smiley face with stars, a smiley face with puckered lips, a smiley face with puckered lips and closed eyes, a smiley face licking lips, a smiley face with tongue out, and the list just keeps going with different combinations of hair styles, hats, jewelry, pet faces, etc. It’s just getting ridiculous and it’s quite evident to me that people want emojis for literally everything – and they should be able to have them because real life has an infinite number of permutations. The thing is, in hindsight it was a mistake to add these to unicode because it does not handle arbitrary permutations particularly well. A dedicate format for emojis would not only be much more future-proof, but it would allow more precision and variety than unicode could ever make possible.
Anyways, like many other technologies developed in the past, it’s very difficult to replace or fix now that it’s so pervasive. It would have been better to develop a superior emoji standard the first time round, but now we’re stuck with them in unicode for the long hull; like the ipv4/ipv6 debacle, haha.
Sure, Unicode had modifiers. What it didn’t have was anything to do with color (it was all monochrome, where any color information came from unrelated markup). Emojis added the need to support color, multiple colors in the same glyph, and color overrides (and contrast concerns, and trouble for color blind users). That’s a major set complication to add, just to have some gimmicky trash ruining something that previously (despite Klingon) was a relatively professional and well respected spec.
Well, first of, I think that to have some more symbols, and that what emojis are, are a nice addition to fonts. What is insane is people wanting to mix R2L(right to left) and L2R things on same input for no other reason than “I want.”. What about T2B? Vertical movement? Let the user set by char count (what usually means column) or rounded column (this means, column +- char length at it). Why not keep the text orientation of the system and be done with it? May the user want, there are already options on the system to change the order. Color tags in emojis fall in the same principle, if your input method does not need it, mask them out.
The whole thing is one more case of over-engineering. For 99.99 % of the cases, his “considerations” don’t matter or steps on things he should not. It all remember me of the problems we face on translation: step out of verbs (even on them things are not easy), and you will face problems that can not be solved in general case for all cases we have.
And he haven’t even gotten to gamma-correction. Because a line at offset 0.5 is not two gray pixels with 0.5 blackness, because due to gamma, two pixels of 0.5 looks much brighter than one pixel of 0.0, so you need something like two pixels of 0.15 next two each other to get the visual darkness of a black line moved half a pixel width.
Now add in semi-transparent text-colors (like macOS now default to), and things get outright evil.
There is nothing about text-rendering that doesn’t hate you.
WYSIWYG/Rich text isn’t “text editing”. It’s closer to “text processing”, in the way word processors work.