https://lord.io/blog/2019/text-editing-hates-you-too/ Folks who are given to arguing that complexity is spiraling out of control, what say you to this?

Because of Unicode, it's essentially impossible to implement a text editor as part of a single person dev project. Which is sad, because I'd like to build a radically different kind of text editor as part of a projectional editor for my programming language.

Sounds to me like structural editing is the way forward — even if you’re not editing a programming language, but just Unicode text. And we already have it for text input on mobile platforms, looking at the examples with Chinese characters and emojis. I’m using an iPhone right now, and as far as I know there’s no way for me to input a thumbs up emoji followed by a skin tone modifier separately using the built-in keyboard. If you relate that to structural editing like I just did, that would be a syntax error. The onscreen keyboard in iOS prevents me from inputting that incorrect state. Let’s not forget that the “easy” way of text editing from the past based on ASCII was based on Americans building something that didn’t even consider any other languages, alphabets, scripts, and cultures. That was all bolted on afterwards with code pages until we got to Unicode. Now we have a technology that sort of works for almost every person on the planet who can read and write (and Klingons!) — and that turned out to be a little more complex problem to solve. But I take that over any simple solution that only works by excluding a majority of people based on their language or culture.

Because Curv is a hobby research project (no corporate sponsorship, no paid staff), I restricted it to ASCII only. There was a poll of users about whether we should support non-ASCII characters in identitifiers and operators, and the vote was NO: people generally do not know how to type non-ASCII characterts on a keyboard, and this change would make it too difficult to read/modify/reuse other people's source code.

Nah, I don’t have any problem with researchers or hobbyists cutting corners to focus on what's important to learn something or test a hypothesis. I was more coming from a perspective where you intend to build a product that you hope will have some significant success. We finally start talking about bias in tech and I think that's important for tech in general, and also for the future of coding. I guess the larger point here is that you shouldn’t be in a position where Unicode support, accessibility, etc. are creating such huge efforts for you. It should just be a foundation you can build on, regardless of what underlying language, environment, and platform you build on. These are areas where we need good solutions so that people like you have a chance to work on important new ideas and still build on an infrastructure that doesn’t exclude people. That we don’t live in a world like this yet, isn’t your fault and not your responsibility. But it would be great if we had more people taking on these responsibilities. And in the discussion of exploding complexity it's easy to categorize Unicode as a bloated mess designed by committee. And maybe it is and could be done much better. It's just dangerous to compare it with a simpler past, ignoring the non-technical aspects of the design shortcomings of those simple solutions and fall exactly into the trap that makes many people outside our industry look at us as privileged white nerd boys that dream of changing a world that happens to exclude most of them.

@Stefan with respect to your last comment

I should also point out that the linked article primarily makes a point about the complex interactions of text editing and Unicode just happens to play a part in this.

I didn't mean for Unicode to be the focus of the discussion. Rather, I had in mind the complexities of text editing, and whether those complexities are the sort that can be reduced (and thus are part of the "spiralling") or not, and why, and how.

Yeah, I guess my comment on this was drowned out by my social justice rant. I can see why this works so well on Twitter. ;-)

Sounds to me like structural editing is the way forward — even if you’re not editing a programming language, but just Unicode text. And we already have it for text input on mobile platforms, looking at the examples with Chinese characters and emojis.

I’m using an iPhone right now, and as far as I know there’s no way for me to input a thumbs up emoji followed by a skin tone modifier separately using the built-in keyboard. If you relate that to structural editing like I just did, that would be a syntax error. The onscreen keyboard in iOS prevents me from inputting that incorrect state.

To add a bit more to this: the complexities of implementing this also could benefit from a more declarative approach. If you read the article, the author is basically describing the rules of how the interaction should behave in the text — that should be more or less directly translated into code, perhaps as rules constraining the movement of the caret/cursor and which keyboard interactions are possible.

To make things simple, I support ASCII only. I don't support NaN, because it ruins the semantics of equality (a==b is no longer an equivalence relation). In Curv, 0/0 raises an exception instead. I only support a single numeric type, which is a float. I'm willing to jettison "traditional" features & semantics that add a lot of complexity, if they don't directly support my research goals.

I thought you couldn't do equality with float values, even ignoring NaN.

0.3 - 0.2 != 0.1

Many algorithms, and more importantly, many end users, implicitly assume that equality is an equivalence relation: (1) a==a, (2) a==b implies b==a, (3) a==b and b==c implies a==c. End users can assume this without knowing what an equivalence relation is. Without NaN, IEEE floating point equality is an equivalence relation, even if a lot of other semantics are screwed up.

I wrote custom text input/editing/rendering for http://strlen.com/treesheets/ and custom font rendering for http://strlen.com/lobster/ and I am pretty sure I am violating a ton of rules in these 2 articles 😞

I'm less interested in general text editing than I am in structured program editing. One of my lines of thought: Users should not be responsible for laying out a program by manually inserting tabs, spaces and newlines. The editor should perform automatic syntax-directed layout. The layout should be automatically recomputed when the width of the output window changes. So it is the system that chooses where line breaks occur, not the user. The position of line breaks changes when you resize the window. How do you design a syntax for a programming language that is compatible with automatic layout?

That said, Unicode has gotten to a point that no-one should attempt to handle that by themselves anymore 😞

Another line of thought: a projectional code editor that displays your code either using a text based syntax, or using a visual/graphical syntax. You can just flip back and forth, editing the program first using the text representation, then the graphical representation, and the automatic layout algorithm makes either form look good. The text based syntax is useful for copy and paste of text to/from the internet, and for people who like typing text and don't mind memorizing syntax. The graphical syntax has the benefits that are associated with visual programming languages, eg node+wire languages. The question is, how do you design an abstract syntax for a programming language that maps onto either of these presentations, and how does the automatic layout work in the two cases?

I think part of the problem is thinking that there are two distinct representations, one for text, one visual/graphical. There are only graphical representations! If you have an old command line terminal in text mode you just happen to have a relatively simple graphical representation that fits same-size glyphs into a fixed grid. So we can abstract that away and think in rows and columns instead of pixels. And that's where we accidentally forget about the translation from logical character position to pixel position because it's completely transparent (and it was even done in hardware back then). Then you start using proportionally spaced fonts or introduce emojis or start playing with different typefaces and suddenly your glyphs have different sizes and don’t line up that neatly anymore and it gets more complicated. Still just a layout algorithm taking little rectangles and arranging them within larger rectangles. Once you can see the nested rectangles, you can arbitrarily play with what gets nested into what and how multiple rectangles on the same level are laid out. And then you're pretty much at the level of arbitrary 2D view hierarchies — what most user interfaces are made out of these days. iOS offers a UI element called collection view, which comes with a flow layout. People use it mostly to arrange photos or graphical elements in rows or columns, filling them up, and then breaking into the next row/column. That is almost the same algorithm that calculates glyph positions in a paragraph of text and could be generalized to use the same algorithm. For historical reasons though (and because the requirements for text rendering Apple has were always ridiculous) it's not today.

@Stefan That seems like a good starting point.

Text is pretty intrinsically hard. All the same, with most system complexity I've run across, you usually have a much simpler system available if you give up a few features. For example back in the day, I remember having an XML parser for about 10% of the grammar which handled all the XML that we cared to throw at it.

I'd like to build a radically different kind of text editor as part of a projectional editor for my programming language.

@Doug Moen can you describe what you're looking for in more detail? Have you written this up somewhere? One way would be to customize one of the existing text editors.

Very interesting topic. I agree with @Stefan about “there are only graphical representations”! I hit text editing complexity quite hard while implementing this user interface prototype for iOS: https://www.achrafkassioui.com/backspace-mirror/ Backspace Mirror attempts to reduce the stress of deleting text when holding backspace. Since the deletion eventually accelerates, I wanted to show at which point the pace of the cursor would change. The hardest part was to programmatically highlight the text before the cursor using web APIs. Traversing the DOM tree that represents styled text was incredibly complicated. I had to decouple the text (one layer) from the highlights (another layer) to get a working solution. Interestingly enough, Ken Kocienda writes about this exact same problem in his book Creative Selection: https://twitter.com/achrafkassioui/status/1192935988097699842?s=21 He calls caret insertion the hardest programming problem he ever faced.