I'm not sure if this is going to be discovery or thinking together, but oh well. I've had the idea in the back of my mind for a long time that UI sucks. It's bad for us as programmers when we have to build an app across 5+ form factors (more when you consider the possibility of AR, VR, or voice assistant applications). It's bad for the consumer when we develop a bad UI. It's bad for the consumer when we build a good UI, but it's not talored to how individuals use it. For example I use about 5% of the potential functionality of my IDE, but it's extremely tedious ranging to impossible to customize the UI for myself. Yes, keyboard bindings exist but for the most part let's ignore that. Most casual users won't learn Keybindings. And, that ignores that I'd like to comstomize the application state I can view. OK, the problem established, I'd like to think about a solution: stop writing UI. Instead, write api descriptors so an agent can design a UI for you, knowing the capabilities of your application. Early versions of an agent will require massive amounts of hints, but that's probably still easier than writing UI. Having an agent (ai maybe?) would also allow input from individual users to the system with hints as to how they want to view the app. "show the buttons in this order. " "show the status of this in that box." etc. Graphql is the closest approximation of such an api descriptor I can think of. On to the questions: Has this been considered before? What prior art is in this space? Are we near technical feasibility of such a project?

This sounds exactly like a description of what HTML and CSS are. HTML is a description of a document without much regard for how a particular client will render it. CSS is a bunch of optional hints to the client about how to render it. Browser clients are designed for all sorts of different device types, and when a new device type comes along there's often a web browser for it that adapts existing HTML to work passably well with the new type of device. (I'm always shocked when I load my wildly complex website, loaded with SVG animations, on, say, a PlayStation, and it works 99% perfectly.)

It does. But html and css are far more concrete than I'm thinking. Instead of the presented api being in the form of user interactable elements, present a machine api (think graphql ish) and let a program design a bridge from your api to UI. Maybe a better example is in order. Will see if I can make one when I'm at my desktop.

I would encourage you not to think of HTML in terms of "user interactable elements" but rather data types. Most of the data types are well named enough (input types: text, date, color, datetime, email) some less well named (input types: checkbox <should be boolean>, radio <should just be hints on select>) output types: p, section, h1. Because it's old and has accumulated age there are warts to it, but I think you'll find it still fits the bill and has the benefit of already working even when used well

I'm not necessarily opposed to thinking about HTML in that way. I just consider it slightly more constraining than what I'd like (which may or may not be possible). This is all indirectly inspired by Smalltalk and the ability in older systems to edit the UI of your own operating system immediately. That would be challenging in modern apps, if only because of the resulting merge conflicts 🙂. Here's a very rough sketch of what I'm thinking of. The hope would be that this kind of api declaration could also span to entirely different UI types, because the programmer in fact makes no statement about any UI type at all. But then it starts to sound like SciFi where "Magic AI does everything"