Lately I’ve been working on and thinking about gui layout. I’m building a gui builder, and I’m trying to figure out what the interface should be between parent and child components to allow for maximum flexibility of layouts while still having things mostly just work when they get combined. I’ve been taking inspiration from the existing gui toolkits (html/css, QT, GTK, WinForms), and from more fanciful gui designs (the iron man interfaces, rainmeter skins). I’m trying to understand what tradeoffs are being made in these systems in a holistic way. Like, what layouts are expressible. I would like to generalize away from are the heavy focus on rectilinearity. Things I’ve been imagining are force-based layout algorithms, and some sort of path-based deformation/matching between child and parent, but they’re just thoughts for now. Has anybody come across any literature on these sorts of topics? The only thing I’ve come across is constraint-solvers.

Here's a question that has usefully guided my thinking on the subject: Have you ever seen a force-directed GUI layout that looked good? I haven't, so what I've pivoted toward is "make the tools really good, so it's easy to design a nice layout" rather than "make the layout happen automagically". Not sure if you'd find this framing helpful, but it might help you narrow what sort of research is relevant. To me, a force-directed layout is appropriate for something like data-viz, where you can be a little loosey-goosey about where things end up. In a GUI, you generally care a lot more about getting a result that feels good in a way that's hard to encode algorithmically.

Nice links! Will definitely take a look at those. I guess the thing that’s leading my thinking in a more automatic direction is I’m picturing an ecosystem and sort of appstore of components. I want somebody to be able to develop a parent component without knowing what type of components it’s going to contain, and somebody else to develop a child component without knowing what type of container it will be put inside (or what siblings it will have), and for that to just work. Or I want to figure out why that’s wrong.

Follow-up question — when you say "path-based deformation/matching", what do you mean by path? Like, the conceptual path to the element in the hierarchy (like CSS selectors) or some sort of visual path in the layout (like aligning things to splines)?

Along the non-rectilinear idea, a parent telling the child, I can give you this shape to display yourself in (maybe a rectangle, maybe something else), and the child either rendering according to that, or saying, I’d really like to be this shape, and some sort of middle ground being achieved. Path in the visual path/SVG sense.

can you do away with the notion of parent-child? perhaps you could think in terms of clustering things, scrolling, zooming, connecting, etc. All components could live at the same level instead of being nested in one another

The trick with constraint layout systems is that they're often too complicated to reason about reliably. You don't know who is in charge: the container or the content. Simultaneously, they are often too rigid. You discover you want two things to lineup cross-hierarchy, and your kind of out of luck. What to do then? Give feedback. Explain why a thing ends up the way it does. Then make it easy to give examples of what you want, which may be realized in more than one way. Browsers have been getting better with explaining CSS, but not without eliminating rage inducing headaches. For instance, it might be hard to select a border in order to determine where it is coming from.

The Iron Man interfaces are wonderfully entertaining pieces of graphic art. There are just a few people in hollywood who know how to make those things. They are mixture of 3D and 2D animations, always using glowing lines and very few colors so it looks terrific. But as a user interface it would be horrible because all the buttons look alike, and the font sizes are microscopic. So although they are sexy from a distance, if you had one in front of you, it would be confusing, hard to use. A lot of music software is enraptured with sub-6pt fonts, and microscopic icons. So i would be wary of chasing Hollywood's ideas. Remember Star Trek next generation, with their buttonless screens that would be sure to create carpal tunnel syndrome in record time? A lot of the futuristic stuff is unergonomic, and possibly harmful to the human body, and presumes a magical artificial intelligence that just graduated from CalArts and spent 1000 hours making a screen that appears in the movie for 2 minutes. A 100 million motion picture is a time compression device, where 50,000 man hours go into making something that lasts 2, so a 25,000 : 1 ratio there. By all means see what you can come up with, but don't hold yourself to unattainable graphic standards that are the work of genius graphic designers who are at the top of a very specialized field (i.e. fake UI's for futuristic films). Hollywood has a long history of faking the future; Irwin Allen was a famous TV producer who loved banks of flashing lights and nutty rubber monsters. I think you are better off studying from the great masters of the Renaissance, who used proportions to draw everything, and also the great book "Designing for people" by Henry Dreyfuss, the greatest industrial designer in the history of the world. Everyone who does any design work should know that man's output because he was so great. He didn't do computer interfaces, but he had a systematic way of attacking every problem that resulted in an optimal final design. It is his methodology that is still useful today.

I come to realize again and again that great design comes from building something over and over again, exploring different options and putting in what you learned from the last time you built it. Great designers build lots of prototypes to figure out what works and what doesn’t. In software engineering we often see design more as planning, as that one time we think a little about architecture and then we just have to implement it. Sometimes that works and it good enough. But if you really want to build something great, you’ll likely have to build it several times.

I have an open source library for plotting widgets on the screen according to inequalities https://github.com/samsquire/inequality-solver

can you do away with the notion of parent-child?

This is the kind of outside-the-box thinking I was looking for (pun-intended). I’m not sure. I think containers are inherent, but maybe a model where a container is just a set of constraints over some children with no graphical presence itself would be possible. This deserves more thought.

You don’t know who is in charge: the container or the content.

This is very much the issue I’m running into. I keep thinking of these fairly naive systems where its either parent-driven or child-driven, or possibly there is a point in the tree where it switches, but I don’t know how to rectify them into something consistent and intuitive. I guess that’s why constraint-layout is a thing. @Edward de Jong / Beads Project Great points. I think there’s an element of me being suckered into thinking these interfaces would be more useful than they really would be (and I will definitely be reading Designing For People). Part of my goal, though, is to just avoid baking in too many assumptions about what makes for a useful/ergonomic/good UI. Which is perhaps itself naive.

great design comes from building something over and over again

Absolutely. Part of my goal with this system is for things to get forked and redesigned and reshared by end users over and over again. I think even a great designer can’t anticipate an individual’s idiosyncracies as well as the individual can tweak and reshape something to fit their own brain.

The Flutter layout model is very simple and expressive - . The proposed model for extensible layouts on the web uses almost exactly the same api. Many immediate-mode gui libraries don't have a concept of parent-child relations. Instead widget functions take a rect that tells them where to draw themselves, and there are a bunch of convenience functions for splitting rects in different ways. Eg page 31 of https://ourmachinery.com/docs/writing-tools-faster/index.html

Tangent: slide 31:

No need for "layout managers" -- instead we split rects directly in code

I understand the sentiment but dismissing automatic layout seems wrong. In a database analogy, would be like doing away with SQL and performing all queries programmatically with lower level API calls and for loops. Makes me think there must be a different way to layout graphics that is to CSS as Datalog is to SQL 🙂

Speaking of movie UIs...

@Emmanuel Oga I think of it more as not having one layout manager. The widget functions just want a rect. You can write the rect out by hand, or write some simple layout calculation, or import a layout manager library if you want, or use multiple different layout algorithms in different places. But the widget libraries don't care.

I spent some time looking into flutter. They have this RenderObject base class that is almost exclusively used through a RenderBox subclass, but in their examples they also have a RenderSector subclass, which does polar coordinate widgets. This was my strawman non-rectilinear example that I was thinking about. So I think their RenderObject interface is probably more or less what I was looking for in terms of layout generality.

Flutter to my understanding is a unique beast among graphical layers. It renders the entire UI into bitmap form, and sends the bitmaps to the underlying OS. So It is effectively its own virtual machine that interfaces with the world by emitting bitmaps. Only an entity with Google's money would attempt such a huge task. My own Beads project maps my language concepts into the lower layers, which for web apps is JS. Flutter is its own universe, and of course is rather tied to the Dart language.

@Jared Windover Nope, I can't find it either. I remember it was part of a collection of proposed extensions aiming to un-hardcode some of the lower levels of the web.

@Jared Windover you mean "Layout API"? https://houdini.glitch.me/layout

Thanks @Niko Autio! I had found houdini, but I hadn’t seen that layout page.