Hey everyone, as someone who's new to language oriented programming (e.g. language workbenches, projectional editors, etc), are there any commonly used libraries or tooling? For context, I'm trying to make a very simple projectional editor that has a 1-1 mapping between basic GLSL-esque language and a visual nodegraph representation. Normally I just reach for a handrolled parser & LLVM and lots of scaffolding/boilerplate making the visual editor. What I'm trying to do now is just make it be one editor

I'm working on a tool that I think if as being at the intersection of being able to "roll your own", but getting to use language-oriented tools and interfaces to do so. It's being made "in itself", so it's self-moldable. Let's talk and see if we have common ground to work from.

@Dan Cook I've been thinking of making a "WYSIWYT" (what you see is what you type) editor for (the newer version of) LES. The advantage of this approach would be that LES is not designed as a programming language, but rather as a foundation for building languages (especially DSLs), and so the editor could be re-used for various languages. Also, presumably while building the editor I could figure out how to generalize parts of it so that those parts could be used to build WYSIWYT editors for somewhat different-looking languages... anyway, this is just a thing I'd like to do, but I might never find the time for it unless I pair up with someone, so I thought I'd mention it. I am also curious to hear about what you're building.

Firstly, I think you're not the only one on here building a language-agonistic editor. Someone described an generic AST editor of some sort? If you could all find each other, you might be able to form a collaboration.

@Dan Cook fuck yes; so much this

I see what your saying about combinatorial complexity! People get gung-ho about things (languages, tools, patterns, practices, programming paradigms) that are "better" along one dimension, and ignore that maybe they're worse along other dimensions, so the material improvement overall is actually negligible or negative (of you include all costs -- which you must, or you're cooking the data). Now sure, the incidental (accidental) complexity in software/code is just a sub-space of the total complexity. You cannot tool-away or language-away any of the complexity of the problem-space -- that just is what it is. ...Well, sort of. There can be unnecessary complexity on that side, too. You need a software system to run your factory, and even if you had the perfect language and code for that kind of thing, the factory is very complicated to manage! ... But wouldn't you know it, you could have accomplished better productivity with 3 people and a small toolbox! No amount of software solutioning for that factory had any hope of fixing that situation! But here's the thing: That should be the level that software professionals think at! Software is just one tool in your toolbox. (And that's not secret knowledge; it got pounded into our heads at a SCRUM bootcamp last week). Not just should, but that's the entire thing that a software professional is being paid for! So it's a disaster that the status quo is not even at that level! And why isn't it? I think it's because there's so much complexity at all the layers below that, that that's what software developers thing their job is about. But none of the focus on languages and patterns and tools has any real value if it doesn't improve our ability work and reason about (and in) the problem space. What's super frustrating to me, is we've had sufficient means for decades (in terms of programming language) to outline the problem space in code, and (for the most part) none of the "advancements" (aka recycling) in PLs have improved that in any significant way. And developers still don't know how to do it. Let me explain: Let's say the problem space can be well represented by a picture or diagram, on terms of looking at it and thinking about it, pointing at it and discussing it, changing it, etc. It doesn't significantly matter whether you use a pencil, a whiteboard, arrange pebbles on the ground, or toothpicks on a table-top ... In all cases, you can look at the darn thing, and see what it is! Sure, you can see the pebbles, or the ink, but that's not the thing you're actually looking at. So long as the medium is sufficient for creating a picture of the thing so that that's what you are looking at, that's what matters. Now an insufficient medium would be boulders or colony of ants. An insufficient representation would be throwing toothpicks on the ground and arbitrarily assigning meaning to it; or writing down a list of coordinate-pairs or a crap ton of prose (🤐) describing a diagram. If you get that wrong, then it doesn't matter how much "better" your medium is! Now sure, you could argue that pen is more fine-grained then sand (🙂) for writing text annotations, or that pencil is eraseable, or that text-files are easier to share and track changes ... And of course those things matter; but they matter far far less than a coherent model -- which is much more a matter of discipline (or in a world where it's been done enough times to know better, common [freaking] sense) than anything else. From where I stand, the status quo of software is more at the level of debating whether red ink is better than blue ink; or pencil vs keyboard; or how much better one can diagram with an ergonomically shaped pen (oh thank goodness you solved that one!). Or maybe it's that some people diagram with axially-aligned rectangles, and others with circles and ovals, and there's a never-ending debate as to which is better or more "geometrically pure". But for crying out loud, a crayon with your non-dominant hand is better if you know what a coherent picture of the thing would even look like! So what is "sufficient"? Functions, variables, conditions, abstraction; and making those be proper subsets of the human model that the software IS. (This topic is hard to discuss, due to deeply ingrained ideas about how a "computer thinks" -- We "have to" do it that way ... NO, you HAVE TO deliver human value, and be and to reason about your code in terms of the human model ... I'll leave it at that for now). ... But alas, I'm only half-right on this. Yes, if people knew how to make sane human models in code (which is not that hard to understand, but unfortunately stunted by tons of the wrong thinking that's painfully difficult to challenge / unlearn) ... then things would already be 100x better on so many levels. We'd be thinking way closer to the human model; that would actually be the thing that is being designed in the code and the thing that developers see when looking at the code. The code (and language, etc.) is just the medium for expressing it! ... I'm half-right because code (as currently done) is only borderline sufficient. If developers had the right mindset, and it was used properly -- and a (freaking reasonable) expectation of a professional (or having earned a college degree in the stuff), then maybe it would be obvious enough. But alas, this is not the state of things, and a pile of text is really hard to point at and discuss this matter. Even if developers could work with that though, not just any human could look at that and make sense of it. That's a barrier that I think can be breached, if there's something malleable enough to be / become a more obvious representation. I think both these "halves" go hand-in-hand. When you can see the model for what it is, it will be much easier to see sanity / insanity. And when things can be sane, then YES, the ability to model things more fluidly will be invaluable and people will be able to manage it -- because the thing they are designing is something that makes sense on a human level!

@Dan Cook 100% in agreement

We should discuss this further over a call sometime

I've got a bit of a sprint crunch this week but maybe next weekend?

Sure. This upcoming weekend, or the one after?

Dan, It's hard to know what you're getting at with your prose explanation. It reminded me at first of Kartik Agaram's work, which is all about bootstrapping, but then, his solution ends up being steeped in x86 assembly language and I think yours is somehow higher-level? I'm afraid I have to push back on what you began to describe. You're saying "Why not put (the IDE's) code and YOUR code all in the same bucket?" Okay, so let's say Visual Studio Code is in "the same bucket" as the program I'm writing. So what? What does this buy me? I don't understand VS Code's code, since it's new to me. I've never seen it, but more importantly, I don't care about seeing it either. Based on the number of extensions people have written for VS code, I imagine it's less of a confusing mess than the old Visual Studio (whose extension model I've had to deal with) but still, it's a big mass of code that someone else wrote, that I don't understand. If it's like most codebases, it doesn't come with any explanations ("literate programming" is something most devs haven't heard of, and wouldn't aspire to if they had - at many companies, even the most important classes have zero documentation). If my editor is editable, I can edit it. So far, so good. But weren't there entire Smalltalk or Lisp operating systems in the 1980s that worked exactly this way? If so, why not start from there? Why start from scratch? But also, what does this buy me? I really don't want to modify the core editor, because if I do, I will be the only person with this version of it. If I change VS Code's core in a way that makes it more useful to me, I now cannot download updated versions of VS Code. This is a really fundamental problem that pops up again and again, and not just in the realm of code - I used to work at a company that decided to modify metadata on some road map files, but then the map provider produced a new version of the maps, but wait, we modified the old version, so how do we merge the old and the new? I spent months solving problems like that. "presents you with a custom interface / model for the code, including the code of the whole running program & interface that it presents. No more dependency on some other interface!" But... how? The OS doesn't understand your code, it expects raw machine code. What machine code will you provide, where did it come from, and can I edit this machine code? (My suggestion: check if WebAssembly works for you as a starting point, so you can run in a browser. And it's easiest if the kernel is not modifiable.) "Well, the tool is written in itself, and you already made a generate-the-equivalent-Java-code functionality; use that on the tool itself, and bootstrap the tool right into the JVM (or what have you)" I've done bootstrapping myself - the parser for the language that my parser generator LLLPG understood, LES version 1, was written in itself, which meant at first that I had to parse it some other way (I came up with a scheme to construct grammars in C# in order to bootstrap it), and then LES v1 evolved to LES version 2, and then I wrote a parser for Enhanced C# in LESv2, and then eventually I decided to support a new syntax for the parser generator which was written in Enhanced C#. Bootstrapping is cool, but it's kinda hard and time-consuming. Anyway, so let's say bootstrapping is done AND you can "generate-the-equivalent-Java-code". What you've written is an interpreter, so you can generate a Java interpreter that is able to run your "code that is written in itself". But the code is still written in itself, not in JVM bytecode, so... you can run the interpreter in the JVM, but you still have a long way to go if you want something that fits reasonably well into the Java ecosystem... certainly you can't just start calling out into Java libraries, let alone write Java libraries that call into your code, let alone pass rich data structures seamlessly between you and ordinary Java code. And what if performance sucks, because it's an interpreter after all? I guess that's okay for what you're trying to do, but how broad of an audience do you want with your software? "most software is horrendously more complex than it needs to be." On this we are in complete agreement. Software in industry is generally awful. Even my own open-source software, elegant compared to most people's, has a bunch of ugly hacks and limitations because of flaws in the .NET platform it is built on. But I chose .NET because it was the only major platform designed for multiple programming languages to interoperate, and amazingly that's still true, though I guess the new WASI standard will do... something... about this (as we speak they're probably inventing all-new design flaws for me to work around later.) "I think a tool like this could get a lot of that junk or if the way, and a program that does x y and z would look like whatever the most immediately way to humanly depict that in simple terms would be." I assume you mean "out of the way". But I don't see how your tool would be able to suddenly make code simple. It does not follow from "I can see and edit all the code in my system! And convert the interpreter to Java!" to "now everything is easy!" "Despite the power of abstraction and recursive generation that code offers, source code itself is still more or less a hand-written 1-to-1 representation of the running program..." I imagine you're talking about how we should be able to describe the properties I want the program to have, or describe a domain-specific solution in domain-specific declarative terms? I agree with that... but again it just doesn't seem to follow from what you've written before. I don't see a path from A to B. And similarly for items (2) and (3). So you have good principles, but I don't see how they follow from your starting point. They are also hard problems - each one is something a person could spend their careers perfecting. And that's pretty much what I would advocate for people: specializing our work. BUT ALSO I would advocate working together by combining our stuff, not just spiritually copying but coming up with a plan to literally re-use and combine code where possible. But literal re-use, of course, leads to ideas like Loyc - because we don't all want to work in the same programming language, what we really need to work on right now is better interoperability, OR code conversion between languages. At minimum there should be a common-but-still-high-level way to represent different languages, hence my LES proposal. But advocating code re-use seems to make me unusual around here and I don't know why. Anyway, while I've been specializing by playing with homoiconic language designs and thinking about interoperability & code conversion, Paul Chiusano's been working on sending code from one system to execute in another (with content-indexed addressing for efficiency), others have been working on projectional editing (Lamdu uses the WYTIWYS = what you type is what you see principle)... I think if we were better at specializing and working together, we might actually get somewhere. Especially if funding were available so it didn't have to be a weekends-only-until-you-burn-out-from-writing-code-7-days-a-week thing.

I've made a mess trying to respond to this on my touch screen phone, and realized I didn't even properly answer some of the questions. So now I've got slack on my desktop, and better notes, and I'm just going to try this again while sitting and focusing better. I'll probably delete this reply too, just before I post my responses.

@Dan Cook free anytime this week. how about yourself? @David Piepgrass you're more than welcome to join too

This weekend turned out very busy (I'll answer David's questions once I can squeeze time back into my schedule -- working from home with kids out of school and a bad internet connection, and taking time out to fix it, has been interesting). How about this upcoming weekend?

👍

You're proposing a meeting tomorrow or Sunday? Sure.

@David Piepgrass @Dan Cook 👍 What times work for ya? Sat/Sun post 2 pm is best for me

My schedule is open. Sat 3pm?

works for me @Dan Cook?

That works for me! Mountain time, or Pacific?

sweet; how about Pacific time

OK 4pm mountain time.

sorry guys but it's like 50/50 i may not make it to 3 pm

Ok, so FINALLY, here's my reply to some of David's questions: 1. High-level bootstrapping, UI to edit itself - How? Where's the machine code? There's a set of low-level operations (+, -, get, set, etc.), end everything else (interpreter, compiler, bootstrapper, UI) is built on top of that. The interpreter is just a function that takes a "context" (dictionary) and an expression, and recursively looks up each operation by name in the given context. Each operation is either a high-level function, which is fed back into the interpreter, or a native-function which is invoked directly. Native code will either be directly editable, or (re)generatable from high level (or other) code. The system can "save itself" by generating a native executable that contains everything in the system stored in a statically initialized structure, with the native base operations nested within it, and a native version of the interpreter (generated by feeding the interpreter into the compiler), and a bit of code to start interpreting whatever is needed to "start" it up. A compiler (for some target) basically just needs to generate native code for calling a native function with arguments, and native-coded versions of the base operations. 2. Why put the IDE in your code? What about compatibility & updates? I think the devision between "code" and "programming language" and also IDE is completely artifcial & unnecesary. There's more "language" describing a software system itself -- every function or API or UI has its own language (grammar and rules of application) that higher logic is composed from -- than is nececcary to describe the few fundamental building blocks of programming (functions, variables, conditionals, etc). So instead of an IDE and editor for a "programming language", I'm looking to make it easy to bootstrap whatever language or editor or model for the specific software system, with PL being a relatively tiny, or using regular old programming abstractions to hide it's complexity away. It's a similar philosophy to this: https://ngnghm.github.io/blog/2015/08/02/chapter-1-the-way-houyhnhnms-compute/ But also, I can stretch these ideas into the UI itself: In the same way that functions + data can replace "grammar" and code "written in" that grammar, so can a user interface and some other model. 3. Why not start with SmallTalk, or a LISP OS? Maybe it could be done there, but that's also a full OS and not a thing that people use it for. In practice, it's still a specific language that locks you in like any other (notwithstanding its late-binding philosophy). What about JetBrains MPS? That's closer, but has similar problems: It's turned LOP into a specific locked down formalization of languages and tools (ironic), that I also think is far more complex than it needs to be. Do you need all that just to be and to say "when I say x y, I mean foo(y(x))". You can do that with ordinary programming paraphernalia (that's what programming is!). What about Racket or Scheme? Maybe that solves a lot, but is restricted once again to dead text, not a living model that may (for one example) have a visual representation instead of a textual syntax. 4. What about interop with a different runtime? Boostrapping this whole system into another runtime would create bloated and inefficient programs. Can I just make a Java program in this? Bootstrapping the system to another runtime was more about using the system itself as a replacement / reinvention for the computer or OS itself: A truly open-ended moldable thing that you can tap into and do whatever with, as needed. And for that, I'd want to be able to boostrap the system elsewhere. I'm starting this thing in JavaScript in a browser, but once it gets far enough, and given an implementation of some low-level ops somewhere else, I could take the exact thing I've developed and load it somewhere else, and keep working. And then have more capability than JavaScript in a browser. There's also other potential applications, like a system being able to "send itself" somewhere. But using the system to create a product for another runtime, is a different idea. You would only model your product in this system, and have other code to generate-out a built-product. The advantage is all the stuff I described in #2 (similar to those offered by JetBrains MPS). Also you could do something similar to what Dark lang is doing, and have a full system modeled in one place, with conditionals that effect parts of code that will end up on different machines or layers, but created in one place with one set of logic. For example, a single variable to control one aspect of everything, without having to have that extra logic embedded or repeated in any of the generated components of the built product. Also there's being able to version-control the whole thing as a single entity (so a feature change equates to exactly what changed in the WHOLE system being built). 5. "Getting the junk out of the way" will not suddenly make code simple. It doesn't follow from "I can see everything, can now convert to and interp with Java" to "now everything is easy" Code is as simple as we make it to be. There is a LOT wrong with how "professional" (and other) software code is composed. You could have the perfect language and tools etc. and still write horrible code. But I think much more of that is blamed on "software is just so complex!" than on all the overhead of language and code and tooling than is reality. I think that if it can be more easily possible to have the business logic be the thing you are building and seeing, then this gets a lot easier. I also think that this stuff is much easier than it's often MADE to be, because of all the other noise, and because it's the LAST thing that programmers are thinking about. If we can make it easy to see, then this problem can become more visible, and it can be demonstrated how straight-forward programming CAN be. Yes, there are hard problems and circumstances where the logic is just not as easy as one would think; but that's more like 10% than like 90% of software. Even the simplest things are made horribly complex! But also, If code is something that can be generated or specified more fluidly than basically giving a one-to-one specification for all the instructions in the code, than that can also help to make it much simpler.

@Dan Cook @David Piepgrass is now still good? i'm good enough to hop on

I'm not sure; I need to check on some things, so I'll let you know in a few minutes, if that's ok

sure np; i'll loop back in a couple of minutes and make coffee in the meantime

Ok, I'm good to go when you are, but I should note that if things get chaotic here, I'll need to jump off.

sure

It appears I can't join the call.