As a follow-up to the above, there's this nice blog post from @tonsky on diagrams as code: Where Should Visual Programming Go?

Level 3: Diagrams are code

This is what the endgame should be IMO. Some things are better represented as text. Some are best understood visually. We should mix and match what works best on a case-by-case basis. Don’t try to visualize simple code. Don’t try to write code where a diagram is better.

Hear, hear!

“Level 3”, when it’s not code & diagram as universal parallel representations but domain-specific visual syntaxes (as @tonsky says he wants) has been explored a number of times. Here is the relevant section from https://arxiv.org/pdf/2303.06777, with some hopefully helpful citations.

Yeah this is basically notebook programming. There are two levels to what you want to diagram too. 1. the program specification (e.g. the source or configuration) or 2. The runtime state. Text source code is just a narrow view of 1. I think that can be enhanced with diagrams across for both 1 and 2. https://observablehq.com/plot/ is incredibly flexible for quite a huge range of visualizations (grammar of graphics). I use it for schematic like diagrams, but obviously it also does more recognisable mathy charts too.

@Tom Larkworthy This isn’t “basically notebook programming” if that’s taken in the sense of notebook programming today, right? Notebooks today can’t use graphics to define part of a program, only to visualize data. Here’s another screenshot from my paper I linked above.

notebooks are definitely capable of reading external data and making dynamic, executable decisions based on it. The also have a ton of affordances for representing data visually, and for implementing UIs inline. So together all those things seem like level 3 to me, but some assembly is required. The browser itself has local storage and the env has internet access so I think persistence is a red herring if you consider the system as a whole.

What @tonsky is arguing for in level 3, and what @Joshua Horowitz seems to be referring to, is the desire for such tools that don't require "some assembly". The amount of assembly needed to go from notebook to programming-with-diagrams is about the same amount of assembly to go from smalltalk to programming-with-diagrams or the web platform to programming-with-diagrams. So while notebooks certainly are nice environments to work in, and while it would be nice if notebooks offered p-w-d out of the box, I don't think it's true that "this is basically notebook programming", unless it's also true that this is basically smalltalk or basically javascript.

I appreciate the reminder to go reread @Joshua Horowitz's paper. Very useful framework and I'd completely forgotten about it.

@Tom Larkworthy: Notebooks are great at displaying runtime state. You can also use them to render UIs, with which you can modify transient state. But the persistent code that makes up a notebook can only be edited through the notebook’s code editor. If I want to edit a state chart graphically and have that be part of the code my notebook runs, I can’t do that with Observable or Jupyter or whatever without very awkward workflows (like copy-pasting text from an Observable output cell into a code cell). Interactions with graphical things rendered in a notebook don’t survive a browser refresh. So they do not pull off the @tonsky level-3 thing.

observable views are a UI + a data-channel and they can be 2-way bound to other UI components, composed hierarchically, or even bound to persistence [local view, shared view]. Diagrammatic editing is a little hard, and this conversation has made me try to see if Plot can be made editable (they do expose their grammar-of-graphics scales and an interactive pointer mark which seems relevant)

@Tom Larkworthy The only part of your comment which is relevant to what I’m talking about is the part about local/shared persistence. I appreciate the effort you’ve put into getting around Observable’s lack of support for this stuff! But those approaches are limited and awkward: • If we’re talking about using UI to edit your notebook’s code, local persistence is a non-starter… the point of notebooks is to share code. • The shared view has no authentication. So if you used it to author part of the notebook’s code, anyone with access to the notebook would be able to modify it’s code. If you added authentication, you would then have two separate, parallel auth systems to keep synchronized — one for Observable & one for your sharing system. • In both local + shared situations, you have to add IDs by hand whenever you want to persist something. If you clone a cell, it keeps the same ID, so it edits the same underlying (local or cloud) state. You gotta remember to change them or you’ll clobber stuff. Same for cloning / forking notebooks. In short, I think these approaches all get very awkward, arguably no less awkward than copy-pasting code into cells. I’d be curious to hear whether you’ve heard of them being used successfully in practice, in the face of these problems. Personally, I’d like it if Observable could add persistent UI-editable state, so we wouldn’t have to make up work-arounds. But I don’t see that happening, given that the company is no longer focusing on the notebook product. Perhaps some other notebook product could get in the game, maybe something open-source?

I have build several apps on Observable using these techniques • (commercial) https://observablehq.com/@categorise/surveyslate-filler?username=demoResponder#FnMcjZO1pn1uqmMh%7Ccell-types. <-- magic link authentication to AWS, hosted on a custom domain, persistence is just S3. Whole application is a single cell assembled heirarchically and bound to S3 • (fun) thetarot.online no auth, custom domain, also embedded in a Medium article • (personal) minecraft server front end Firebase login and backend The things you say are true in that there are numerous annoying frictions, CORS being the worse one IMHO, but still, compared to setting up Jest on a Typescript project, or installing Numpy, these are not deal breakers, and the reactive environment makes up for all that, with surplus IMHO. There is a bit of a complexity/performance ceiling, but I would say that is true of Jupyter and Excel too, it doesn't stop them being incredibly useful internal micro-apps. There are several open source Observable-likes in existence already • https://github.com/asg017/dataflow • https://starboard.gg/ Building something that is good enough IDE/Visual Env/Ecosystem is basically too large a project to pull off. Even Observable hasn't really pulled it off! I don't think saving the state of the notebook is the problem.

Level 3 is straight-forward (and “easy”) with currently-available technologies: 1. use diagram editors that save diagrams in some kind of XML or JSON format (I use draw.io, I use Kinopio, I’ve used yEd in the past), then, use XML parsing libraries or OhmJS to inhale the info 2. isolate software components from one another - make each unit of software be completely stand-alone - meaning data and control flow - [hint: closures, queues, avoid function-calling for inter-component communication (level-3 innovation is discouraged by function-calling-think (in fact, at one point, I used mostly Prolog for thinking along with Javascript and Bash and Lisp for clean-up))] 3. think of current GPLs - Haskell, Python, Javascript, Rust, Lisp - as just assembly languages for level 3. (aside: meaning of “easy” == Zac Nowicki of Kagi created a draw.io+Odin based Level 3 DPL for me in less than a week)

ok, interesting, just a prototype but if you click on the plot it adds a point there. The nice thing is that you can work in the coordinate space of the diagram, not pixels or viewport. Maybe plot can be an input 🤔 https://observablehq.com/d/e627aaaaa9857257

FWIW I tend to think the ideal environment allows for all of them: code, dumb diagrams, code backed diagrams, diagrams that are 'code', and everything outside and between. If these coexist in a shared environment, we have the best chance at mutating them and ideally converging on the must useful representations for differing situations. (it is also of my personal opinion that diagrams as code will flourish in this kind of a environment)