Can we extend the definition of structures, like @dataclass in Python, to include links to .png (and .svg?) files? Then, use IntelliSense to show the picture? Something like this

@guitarvydas from the link article "_My contention is that all popular programming languages are essentially designed for write-only use._" I get that! I'm very suspicious of the written form of programming as the complete and final representation. It feels (in my gut) like wired/graphical visualisations are much more readable. But it's only a feeling because I've only ever coded with text and not really with visual node/wire interfaces. These days I'm more thinking that there is no one view that can show everything: the interface, the logic, the data... So, maybe just being able to add additional views onto the map of the application seems like the way to go, and be able to zoom in and out of each. And perhaps this was not exactly what you were getting at. Cheers!

@Daniel Harris That's exactly the premise of Moldable Development. Which in my experience (about five years now) is very powerful.

Thanks @Konrad Hinsen I clearly need to deep dive into Moldable Development. Could these views be created by configurable no-code? [Disclaimer: I want non-developer end-users to be able to build apps using no-code. I'm not interested in AI at the core.] Hope I'm not hijacking this thread.

@Daniel Harris I am not the right person to answer this question, because I have zero experience with no-code environments. Worse, I don't really understand the concept of no-code.

@Daniel Harris I am a proponent of "data visualization driven programming" (which is a term I made up). Certain data are better visualised (e.g. a map, a ui) and so code should be augmented with visual views, and it should definitely support multiple views of the same thing. Specifically visualizing code structure this demo page has 2 reactive code maps on opening https://tomlarkworthy.github.io/lopecode/notebooks/@tomlarkworthy_lopecode-vision.html#view=R100(S50(@tomlarkworthy/[…]S25(@tomlarkworthy/cell-map)). They are not static images, they are live projection of the runtime state. If you change the code they change too. Quite useful for refactoring. Note most coding focuses on the code, not the running program, but in a notebook environment those two things are unified in a single runtime so I feel that that is an important preparatory step. looping back to @guitarvydas, Are we interested in associating an image with a @dataclass schema, or are we interested in associating images of a live system. Operationally I think the 2nd is more useful. I think about code as the control plane, vs the running program as the data plane. For debugging, the dataplane is the real system we are usually interesting in understanding more of.

I think about code as the control plane, vs the running program as the data plane.

That's a nice image! I'd add narratives as the documentation plane. My main gripe with traditional notebooks is that they confound the control and the documentation plane, by forcing the two into a single linear structure. But I fully agree that computation in the data plane is what we ultimately care most about, and should be able to inspect, understand, and debug.

@Tom Larkworthy @Konrad Hinsen In the world that I want these views are not only for just reading/understanding the code model but also for interacting with the model of data, logic, interface. The frustrating part is that the kind of interactive views that we have now, like Excel (for spreadsheets) or Wix (for websites) are tightly bound to their own data and logic stores. And another part of this is: can these interactive views be themselves described by configuration rather than a language? I'm also trying to figure out the difference is between no-code and code, and the best I've come up with is that for code the developer is creating models in their head of what's going on and also they need to know the syntax of how they read/write instructions, whereas for no-code the models (interactive views) are the thing that defines the application/flow.

yeah I think of a data viz as a projection of some information

system -> pixels

. If you want to go the other way

system <- pixels

its a bijection which is reasonable but quite a bit more complicated to develop, so only going one is 20% of the effort for 80% of the gain (i.e. legibility). I did experiment with formalizing directly manipulatable dataviz here https://observablehq.com/@tomlarkworthy/manipulate, which shows promise.

My experience matches what @Tom Larkworthy describes. Moreover, for complex data models, a complete visual representation that could be made bijective is usually unmanageable for users. You want multiple simpler partial projections instead, but that also makes inverting the relation difficult to impossible.

Perhaps I've found my life's work! 😉 Thanks for the input.

I love this conversation! It is inevitable that source code is going to transcend text and become UX. Just exactly how that happens is the great challenge and it is inspiring to see all the passion and creativity shared in this community.

Can you do

system <- textual code

? If so, my knee-jerk reaction (probably based on a misunderstanding of what I read) is why not do

textual code <- pixels

, or, better yet,

textual code <- [SVG | XML | graphML]

then

system <- textual code <- graphical representation

? It's only syntax, after all. Reduce the problem to one of syntax as a skin stretched over formalization. Converting

[SVG | XML | graphML]

to executable, textual code is a trivial operation. Reducing visual representations to

[SVG | XML | graphML]

is an already-solved problem. Easy.

yeah but text is an infinite dimensioned variadic datatype that can represent anything so its not that helpful except for concrete implementation. When visualizing systems we want to break down the control surfaces into a system configuration space. I think a strongly typed text is 1 way of doing it. For numerical and spatial domains even that can be a bit free-form and saying its a fixed dimensioned system can be very helpful because then you can do calculus on it. I like the pixel/scale formulation because pixels are in a 2D configuration space (plus another 3D for their color), and then you can talk about linear projections of system space to display space in a formal way. Its not for everyone but I find it quite a useful way of expressing what we are actually doing when visualizing something on a screen.

Oh cool, I’m interested in working on something like this or related to if. visuals + code / flexible representation.

I propose that textual code can be understood as a form of strongly typed text, and that the software-engineering community has advanced the formalization of such text further than any other representational paradigm. From this perspective, it is reasonable to investigate “narrow-waist” architectures and staged-computation techniques that leverage these existing formal foundations. Notably, contemporary visual editors—such as draw.io and Excalidraw—already serialize diagrammatic structures into XML-like textual formats. Through tools such as OhmJS or conventional XML-processing libraries, these serialized representations can be translated into strongly typed textual forms, thereby enabling existing compilers to perform type checking and code generation. My earlier engagement with compiler construction led me to study Denotational Semantics. I initially dismissed the approach due to its inefficiency and conceptual complexity. However, Peter Lee’s Realistic Compiler Generation reframed my understanding. Although Lee does not present it explicitly in these terms, his method effectively applies a divide-and-conquer—or “narrow-waist”—strategy, decomposing the monolithic denotational framework into two smaller, more tractable, and substantially more efficient formal layers. Whereas naïve applications of Denotational Semantics historically produced compilers that were five to ten times larger than their conventional counterparts, Lee’s structured decomposition yields implementations that execute efficiently on small personal computers and achieved size and performance much closer to that of other production compilers.

Text is pretty lossy. If you could keep info about intent without degrading to plain text, that would be neat.

Doesn't

var x : int;

retain (type) intent while using text? For stuff like

IF...THEN...ELSE

, the intent is simply assumed. It is possible to formalize that intent (e.g. with Denotational Semantics), but we don't feel the need to expose the intent in some formal way in the text - we express it in the formalization (e.g. compilers, or Denotational Semantics). We pick and choose certain words from the English language when forming programming languages, we don't bother to formalize all of War and Peace. We might treat graphic representations in the same way - only a few combinations of pixels (boxes, arrows) need to be formalized, the rest are "syntax errors" or "comments".

You still have to parse the text. I’ve seen some prototypes out there trying to make it so you’d edit the code in a form that’s already similar the AST, which reduces the steps towards compilation. Just a thought. I think / I realized we’re thinking about different things though. By the way, there are a lot of cultural biases encoded in modern languages that are hard to get away from. Some researchers I know are working to understand how we could get around that.

Very thought-provoking. At the day job, our backend is all moderately-documented services, and unit/component-level testing falls short of covering any “happy paths” across service interactions. Debugging issues in happy paths requires diving into the code bases (different organizations, different authors, different style) and reverse engineering. I'm fast-forwarding in my head to a state where my team, all working on separate services with documented api/gRPC contracts, manages to maintain the visuals describing model and system interactions. In my xp it's the interactions that take much of my time to reverse engineer, and I'm usually good on the model at hand. In my world, I could see It being useful to bake in sequence diagrams or something. Of course, everyone's gotta agree on some consistency and stick to it. The architects/implementers analogy rings very true for me. To address some of the concern above, there are comments sprinkled through out the codebase here and there with citations to specs architects maintain as business cases evolve.