Is anyone aware of a good meta-analysis on why node and link based programming has only found limited success? lookin' to learn more

I have the littlest stub of something like this in my Visual Programming Codex: https://github.com/ivanreese/visual-programming-codex/blob/master/criticisms.md

I have a theory about this, but haven't seen it explored formally anywhere. Mostly, I think node-and-wire programming hasn't tried to be generally successful and is happy with being used in limited contexts. Consider python, a 'generally successful' programming language. It can be used to program embedded systems, write web servers, script game engines, and script 3d modeling tools like blender. It's widely used for scientific computing and data processing. There's some things you might not want to do with python, but there's nothing you can't do with python. Compare that to the most successful node-and-wire systems. They're always tied up in some complex runtime. I can't copy Unreal Blueprints into Max MSP the way I could copy python from a web server to a Blender script. It feels like node-and-wire systems are in the same place scripting languages were in the 90s. Every application has its own bespoke node-and-wire system. None of them are very high performance, and none of them work together. Often, there's no library ecosystem for node-and-write tools, and even if there were an ecosystem, every platform would end up re-implementing the same libraries. Presumably, expert users also fall out of these tools. At some point, I assume the best users of Unreal Blueprints just start learning C++, so you end up with only the mediocre users and the users with a principled attachment to the tool sticking around. Which is to say, node-and-wire tools need to reach all the way down the stack (there can't be a moment where you need to leave the node and wire system to use the more powerful tool 'underneath') and they need to reach across the stack - a node-and-write tool that only works for one very particular use case will never have the general success that a tool like C# has.

^ i think that criticism is actually captured in the snippets Ivan pulled out from No Silver Bullet, haven't been any examples of abstraction in the visual world that is as powerful as objects in the OO world

@Ivan Reese I'd be curious to hear more about those 20 weeds-y things if you ever write them down. We're taking an approach pretty similar to what you described and always looking to learn more.

oh nevermind sorry it's not in No Silver Bullet, but i think the argument still has weight, it's hard to capture procedure + state together in a visual abstraction that can be re-used

why node and link based programming has only found limited success?

My guess is that node and link programming expressivity is limited. I mean it is extremely expressive for some programming artifacts (say, audio processing, event flow, etc.) but very poor for some others (math expressions first come to mind, but I need to find mor examples). So I guess a language that is exclusively based on nodes and links won't be able to be general enough for major success.

plus won’t it still need labels and text? To be powerful it would need primitives, combinations, abstraction and a way of generalising over patterns. not obvious to me how to do that with just nodes. Sounds like it would require learning another alphabet

That is why I was really impress by the expressivity and easy of use of PureData and Max/MSP for lots of programming task and feel very frustrated when I wanted to do some simple message/string manipulation. I then fall back to JavaScript and the combination of the two was very good.

I wish there was visual documentation/annotation instead. Perhaps might help with documentation and debugging. I know I often need to draw diagrams to design/understand what my code does

That means that I don't fully agree with @Ivan Reese point "Don't make it easy to drop down to text" 😉 Or, I can agree if there is another visual "semantic" for the cases where "node and link" are poorly expressive.

I suspect visual programming will eventually be adapted for a very mundane reason - textual programming languages are running out of space. A rust function signature might look like

fn longest<'a, 'b>(x: &'a str, y: &'b str) -> &str

and you can easily imagine a language wanting to say

public static async fn longest<'a, 'b>(x: &'a str, y: &'b str) -> &str

If its not happening already, textual languages will eventually have to turn down features because they just make the function signature too long.

But I fully agree with this one: "Focus on expert users, ensuring they can fly through the tool like experts can fly through Photoshop or Modo" 💯 Power users are very important. In fact one way to achieve that is to consider yourself (as a designer of a visual language) as the first user with all the tools you like (good shortcuts, etc.).

@Ivan Reese nice stuff in that repo! Defo hadn’t heard of Dynamic land

Spatial thinking is a skill that has to be obtained, vs linear thinking which is typically innate https://www.nap.edu/read/11019/chapter/6

@Chris G Yes, but some problems don't fit well with linear representations, mainly problems that are inherently trees or graphs. For example: audio processing, infrastructure architecture, relations in a database...

I wasn't suggesting that problems be fit into a particular mode

Most node and wire tools confine themselves to a domain— they are designed to work on one kind of data, and the primitives are chosen by the system designers to be the ones they think you need for that domain. If you need new/different primitives to solve your problem, it's usually difficult or impossible to create/express them. Successful programming languages have primitives which are agnostic about the domain/problem you are solving, and furthermore are chosen to elegantly span the space of computable functions, which means you never have to leave the system to fully express your solution. Disagree that the first successful node-and-wire programming language will be targeted at high-power developers. See: the exact reasons that Dark had to scale back the other week.

I would distinguish node+wire style visual programming from visual programming in the broader sense. In the broad sense, it's definitely possible to create a powerful, wide spectrum visual programming language that scales as well as text only languages. I would start with the principle that code has a hierarchical structure (like a syntax tree). Each node in the tree has potentially multiple different visualizations available, depending on its type, and you can flip between different visual syntaxes. You can create different visual DSLs for different domains (eg, music sequencing languages use different visual conventions than 3D graphics languages). And you can mix multiple DSLs in the same program: these are just different node types.

Here’s my cognitive-science-hobbyist theory of what’s the problem with node-link languages: Node-link makes effective use of the link image schema (https://en.m.wikipedia.org/wiki/Image_schema). Such schemas are universal patterns that structure our thinking on the lowest levels. These patterns also provide the infrastructure for spoken and written language as well, so they sit both under visual and text-based languages. (If “sitting under” makes sense to you in that previous sentence, that’s because it’s an example for a spatial metaphor build on top of image schemas.) Node-link languages work well, because they hook into that particular link pattern which we all intuitively understand. That’s where their power comes from. Unfortunately, one pattern isn’t enough to model everything. However, it’s great when it’s almost all you need in the domain you are modeling for. Some node-link languages add a second image schema by allowing containment. The container schema is also why files and folders and directory structures are so pervasive. That’s a little more expressive, but two patterns are still not enough to model all the complex things general programming languages can. Where node-link languages break down is: they usually offer no solution to the problem of anchoring things to locations. It usually doesn’t matter where you put nodes on an often infinite canvas. Location has no meaning, like it does in a map, or a scatter plot, where each dot is precisely tied to a location. While placing things arbitrarily on a canvas sounds positive like flexibility and freedom, it creates cognitive effort because we will quickly feel overwhelmed placing large number of objects in a space without any supporting structure. So we invent that structure on the fly, trying something like ordering nodes by data flow, which works well enough for when you’re just modeling how data flows through a number of filters and transformers. But often we end up with conflicting structures. That’s when we start talking about how it all looks like spaghettis. For some reason media apps are good at taking advantage of image schemas and metaphorical structuring, which I believe is all happy accidents simply because these patterns are how we think and when designing UI we gravitate towards them even if we don’t know why. Think waveform visualizations in audio recording software or MIDI notes. X-axis is often bound to time giving position meaning — the same reason why charts and diagrams are often easier to grasp: image schemas giving rise to powerful spatial metaphors. That media apps seem to discover these schemas is less of a coincidence and connected to how we understand media as events spread across time (yep, guess what’s the basis for that…). In general programming we think much more abstract and have many completely incompatible orderings at the same time, e.g. data flow with concurrency, static type hierarchies, hierarchical stack frames, sequential memory — it’s easy to visualize each one of them at a time, but it’s essentially just different views on the same system. I firmly believe that we can deliberately take advantage of this particular knowledge in cognitive science in UI design and ultimately for designing better “visual” languages. If you see any research that’s combining these fields, please send it my way. And if you want to learn more about the cognitive science parts feel free to contact me. I’m happy to provide plenty of pointers to resources I can recommend.

my sketchy thoughts: • node and wire diagrams tend to prioritize a bunch of things but very rarely have clear points of view on visual hierarchy (in the graphic design sense). so it’s hard to see which 1-4 things are of primary interest when looking at an expansive diagram • they don’t tend to have super-robust notions of focusing in on a particular eg function or object. i think there’s a lot of available blue ocean space for tools that let you filter down to a compact representation of what you’re working on and its immediate dependencies • visual display of data structures: node and wire diagrams tend to do a lot for visualizing where data comes from but not a lot for what your data is shaped like or what operations are available for that data shape. this is particularly weird to me because data formats, whether json or haskell, tend to be pretty visual in ascii-land—containers for stricter or maps represented by {}, or strings with quotes, or even the idea of ordering fluent OOP interfaces where something.operation.secondOperation(arg1, arg2) usually means “take a thing AND THEN do operation AND THEN do second operation and stick these arguments INSIDE second operation” • finally you should check out self if you haven’t and

I'll second @Stefan, "Where node-link languages break down is: they usually offer no solution to the problem of anchoring things to locations." The difference between a delight and a tangle. For this reason, I favor near-automatic layout.

hah, I was writing about just that yesterday:

In both blocks and nodes editors, the interaction loop boils down to summoning a node and then connecting, or placing, it into context. This ends up being a very verb-centric model where you get the

+

node/block and then wire it up. This is like painting off to the side, moving it to the correct location, and then blending it into the rest of the painting. While there are times where that flow is useful, it’s much less direct than just painting in place and requires tedious actions inserted in the middle. Going from two steps to three for every action in a system adds up quickly and makes even the most basic operations feel heavy in these editors. Rather than operating in context, you’re always working off to the side and then rewiring things to make them work.

One fresh take from Loglo https://loglo.app/ is sticking to a grid favoring arguments coming from the left and maybe above but with the freedom to reference whatever cell. I say this as one who gets on a concatenative language kick every few years. Where they represent one extreme (single line of nodes), a regular node-link system represents the other: anything goes.

Re: William

For this reason, I favor near-automatic layout.

One of my weed-dweller ideas is that automatic layout is useful, but not in the arrangement of nodes and wires themselves, which (I argue) must be laid out by hand with great care – and tooling needs to be excellent, in order to make this process a joy (like gardening) rather than a chore (like cleaning up). (The cases where automatic layout is useful... that's much deeper in the weeds. At some point, I'll dredge that patch and plop whatever trinkets and bike tires and fish guts come out on my blog)

If your visual representation is interactive and bi-directional, moving things around could change their meaning/properties based on where you place them. Think interactive scatter plot where moving a data point changes its values. How useful that particular use case is, is a separate debate. But in general I like to think much more about interactions than about visualizations. You might have caught me before putting the “visual” in “visual language” in quotes. For reasons. Aren’t all the good examples for visual representations we admire getting most of their usefulness out of their interactions? Wires that snap into place on a connector. MIDI notes that change their pitch when moving them up and down. Scrubbing through an audio or video file. I’d have lots more to say about how that also hooks into image schemas and metaphors and embodied cognition, and it’ll tell you why I’m all about leaving mice and keyboards behind (just as optional accessories, not killing them entirely) and fully embracing both touch and freehand gestures. But I’ll save that for another thread.

I like Stefan's comment about making interaction the focus. As for layout, I don't want a visualization that "must be laid out by hand with great care" [Ivan], because that's a big waste of my time. In the Go language community, they have this culture where you don't format your code by personal preference, there is only one true layout, enforced by the gofmt program. I want to use code visualizations where there is automatic layout, and there is only one way to lay out any given program. Different layout means different meaning. I don't want to "burn" interaction gestures on manual layout. There's only so much convenient interaction syntax available, and it must be used wisely. If I move a graphical object from one position to another, it shouldn't be because the object looks prettier in the new position, the gesture must change the meaning of the program (like moving a MIDI note up and down in Stefan's example).

Extremely obligatory: https://en.m.wikipedia.org/wiki/Deutsch_limit

Who said anything about looking prettier? Overlooked in all this talk about whether and how positioning has meaning: manual layout is meaningful to the author, for the sake of their own thought process, that the environment doesn't need to care about. Go formats your code, but it doesn't force you to use automatically generated variable/function names. You choose the names, and it's worth spending the effort to choose great names, because that's how you structure your thinking about the problem. In other words, the formatting of text code and the layout of visual code are not directly comparable.

(except insofar as a lot of formatting of text code is an impoverished attempt to garner some of the benefits of visual layout)

Go doesn't force you to use automatically generated variable/function names. . You choose the names, and it's worth spending the effort to choose great names...

Inventing names for trivial things is a pain. One of the killer features of node+wire programming is that you aren't forced to choose names for local variables. Choosing names for important high level concepts, or using 2D layout as a way to organize important high level, is beneficial. But don't make me manually lay out the nodes for 'x + y + z'.

@Doug Moen this brings up something I'm curious about, which is how to talk about node and wire code. Text is cool because it can always be verbal or written. Code isn't quite natural langauge, so something like

function add(a, b)

doesn't necessarily have a canonical spoken representation, but it feels more obvious than if it were three unlabeled boxes.

Nodes and wires programming doesn't necessarily imply that you have to position and link things with tedious mouse manipulations. The merit of nodes and wires is mainly their expressivity for some programming artifacts. But one can imagine some HCI that make it easier to manipulate than clicking somewhere, dragging in the canvas, dropping, etc. In fact, like most people coming from text languages I was frustrated to have to always use the mouse with Max/MSP, and as someone that once tasted the power of vim, I was really frustrated. So I experimented that in my "zed" editor where (nearly) every action can be done "easily" with the keyboard: in my #C0120A3L30R video of last week I only use the mouse to move in the canvas, all (the few) editing/navigating actions are done with the keyboard. For example hitting "a" then "s" adds a script node just below the current node, then hitting "i" connects the node input to the output of the previous, etc. I agree this kind of manipulations is reserved to power users, but as Ivan previously said it is important to make things work fine for power users and it shows that if the mouse is seen as a complicated way to edit/navigate one can also imagine some easier way to do things.

on the mouse/keyboard sub-subject: I wish more apps took the desktop-video-game approach of prioritizing inputs for simultaneous keyboard-and-mouse use—eg wasd to move and mouse to select

I like the idea and yet never thought to try it. I definitely will! Thanks @Garth Goldwater 🙂

Re: Doug

[very good comment that I enjoyed and agreed with, ending with...] But don't make me manually lay out the nodes for 'x + y + z'.

Agreed! No good node editor would make you do that. (There is a paucity of good node editors.) That said,

Vec3.add(x, Vec3.add(y, z))

is bad too — and it's common in languages that don't have operator overloading. But operator overloading isn't a total win either, given precedence is typically fixed (good luck using

^

for the geometric wedge product). So in the end, you still often have to manually lay out your equations, just using extra verbosity instead of extra cabling.

But don't make me manually lay out the nodes for 'x + y + z'.

> Agreed! No good node editor would make you do that.

Show me an example of one that doesn't make me do that. The problem with node+wire is that once you add a node to the canvas, it is stuck in that position until you move it. So what happens after you have written a substantial amount of code, and need to revise it? Suppose I want to insert 'x + y + z' into the middle of a computation, but it won't fit, there is no room on the canvas to put those nodes where I want. So either I manually rearrange all the nodes to make room for the new code, or I put the new code off to one side where there is space, and drag input and output wires from the place where the computation needs to be, off to where I put the nodes, and now the code looks like a plate of spaghetti. So that's why I want automatic layout. If I insert nodes, the code automatically reflows to make room. If I delete nodes, the code automatically reflows to fill in the gap.

@Doug Moen Though few editors do this, "moving stuff out of the way to make room for new construction" is the kind of "auto-tidying" that a good editor should do. I don't think that's in conflict with allowing the user to spatially organize, as long as it doesn't outright prevent the user from moving stuff around on their own.

moving stuff out of the way

3d modelling tools have solved this problem without automatic layout. They give you rich tools to manipulate objects in space, and the space itself, by leveraging the properties of the structure of objects and space. Want to insert something in the middle of a dense mesh? Grab a vertex and spread out new vertices along every inbound edge, turning the original vertex into a face — in 1 keypress + click. Or, slice a plane through the mesh, and push both halves of the mesh apart. Or, band-select all the faces pointing toward the camera, and pull them away from all the faces looking away from the camera. At all times, you have full control over exactly where every vertex is, and when editing in bulk, exactly how each vertex is moving — you would absolutely not want to tell the program "just move my vertices out of the way _somehow_". There are, also, automatic tools you can use. But the manual modelling tools are so good, you rarely want to use the automatic tools. (This gets more weedsy when talking about animation, UV mapping, mesh sculpting, etc. — those areas have manual+automatic hybrids. But they're also less about putting things in places and more about setting relationships, which is relevant to us, but not relevant to this question.)

Are we agreeing that the ideal is not node+wire?

A rich symbolic algebra editor could (and should) just be another node. (And, ideally, you should be able to crack it open and play with the nodes inside it.) This thread is broadly about node-and-wire programming, implicitly vs text editing, so that's where I've been focusing my thoughts. But do count me in with team "node-and-wire isn't the best that visual programming has to offer".

Show me an example of one that doesn't make me do that.

"There is a paucity of good node editors." So, you'll have to use your imagination :( (There are a bunch of node-and-wire tools that let you put JavaScript or C++ or whatever inside a node. I view that as cheating. There are some node editors, though their names escape me, that do have dedicated equation nodes. None of them are especially rich.)

Notebooks feel like the first avenue where node-and-wire will go beyond PoC applications — the scripts to replace are small / the number of pieces to display and enable are limited (and are already on screen, to understand) — and notebooks are already a half-visual / hybrid context, going full-visual is a win for the audience (vs a meh for developers anywhere else) Something like https://github.com/IBM/node-red-dsx-workflow but cell-by-cell within a notebook

@Max Krieger thanks for bringing up the Deutsch limit. I don't know about others, but when a whole page of text is in front of me, I certainly cannot parse out 50 items at once. It may be dense, but it is no more clear than zooming out on a diagram (or a mechanical assembly). As for the quote, "Well, this is all fine and well, but the problem with visual programming languages is that you can’t have more than 50 visual primitives on the screen at the same time. How are you going to write an operating system?" Is he trolling us? I mean I have handy here a project of... ~20,000 LoC. I can't see all of that at once.* A dozen people are working on it. I'm sure there are horrors to be found that a decent visualization would reveal — both at the level of basic formatting, style, structuring and at the deeper level of knotted up semantics, conditionals, wrapping something up only to unwrap it elsewhere. The usual suspects.

We shouldn't discuss the Deutsch limit without also discussing APL/J/K. In the communities that use these languages, dense code is highly valued. Because this allows you to put a large program on the screen in its entirety, and read it without scrolling. APL was invented in the late 1950's, and there has been 60 years of work on achieving ever higher code densities. So now you can put the logic from 10,000 lines of C++ code in two pages and read it without scrolling. The claim is that once you surmount the steep learning curve, you become more productive. You can write a page of C code in a few keystrokes. There is less use of libraries: the attitude is that if the name of a library function is more keystrokes than inline coding the same logic using language primitives, then you should use language primitives instead.

I recently tweeted a reflection on the Deutsch Limit: https://twitter.com/spiralganglion/status/1272613075292532736

APL also has a great sense of using positioning and punning to deliver more information @Doug Moen . glad to see it brought up in combination. also, some of the symbols are IMO just plain visual genius (eg: tally)

I just read this thread and wonder if node and link would work better if you added some useless details to give a name to an "area." It feels like my world religions class where they introduced sacred vs profane and how a "flat" profane world would lack meaning and be unattractive at best. Suggestions: islands with fixed sizes and names, generated Risk-like country maps, or even just thumbtacks with gifs/pics/docs attached to certain nodes in a meta-view programmers are forced to move around.

When I'm programming, I usually think about code in terms of both control flow and data flow. Most dataflow systems seem to only have one type of wire. Are there systems with different wires to represent different flows through the graph (eg blue wire for data flow, red wire for control flow)?

Max/MSP and Pure Data have one kind of wire for messages (discrete data), and another for audio signals ("continuous" data — not truly continuous, but discrete samples of a continuous function at a sufficiently high sampling rate). Many visual programming tools will assign a type to their ports, and whenever a wire is connected to one such port, it can only be connected to other ports with a compatible type. In my project Hest, wires are used both to convey data (and there may be different kinds of wire, with different appearance and properties and modes of interaction based on the type of data being conveyed), and to exist as "edges" in the sense of 2d vector / 3d graphics, so in that way they're both an aspect of the code and data literal.