I may be misunderstanding you, @Gary Trakhman. Building languages in themselves is (sadly) very common. Gcc is in C. The Hotspot JikesRVM compiler for Java is built in Java. Lisp is particularly prone to this sort of misguided fascination with recursion since McCarthy's original paper showed a one-page Lisp interpreter written in itself. The security risks of building languages in themselves have been known since Ken Thompson's Turing Award lecture in 1977. And such implementations are also hard to understand. The answer to "what is the spec/semantics?" is often "whatever the code says". But what does the code do? You end up with a circular dependency. Both these issues are also present with mutual rather than self recursion, like if you built Python in Ruby, or vice versa. Modern software, obsessed with interfaces and convenience, fails to attend to implementation properties and long-term externalities. My Mu project builds each layer of abstraction strictly out of lower layers. It's a slow, inconvenient process, but not as slow or inconvenient as I'd feared going into it. Prototyping languages is a great thing to do, but we should all be concerned with the depth of the dependency tree and the presence of cycles in it before we deploy something to production. It's a very implicit and pervasive form of technical debt.

Nit: hotspot is not built in Java: https://en.wikipedia.org/wiki/HotSpot The bytecode compiler,

javac

is

Sorry, I got the name wrong. Let me try to dig up what I mean.

It's an interesting point, but I think there some wins from only learning one language to hack on the language. I do think lisps are particularly bad at this, you sort of just have to go really slowly at the beginning until you find all the problems and learn the discipline to avoid them. I think proper types can specify an interface that prevents people from having to dig as deep as often for basic tasks.

@Kartik Agaram Can you elaborate on the security issues you mention?

@tbabb The full story is at https://www.win.tue.nl/~aeb/linux/hh/thompson/trust.html, but in brief: the whole world runs on C, so a malicious backdoor in a C compiler would be very bad. Perhaps we can just audit C compilers for back doors? Just check their sources, monitor PRs as they come through. But that doesn't provide a guarantee because C compilers are written in C. What C compiler do they use? Does it have any backdoors? Oh, it's the same C compiler, just a binary generated by a previous version. Where was that version built? Oh, by the version before it. And on, and on, back to some version written in Assembly that ran on a PDP-11 or something whose sources are not available to us. The whole world runs on software that we have no audit trail for. Anybody who inserted a binary without a record of its sources somewhere along the way could have done anything. And compilers are so large that auditing their disassembly is intractable. Nobody's done it. There's a scifi story that explores this idea in a more visceral manner: https://www.teamten.com/lawrence/writings/coding-machines. Perhaps the association with a scifi story makes it seem less serious. Perhaps it seems like I'm wearing a tinfoil hat. You'll have to make up your mind on that one 🙂

Thanks, makes sense. and not tinfoil hat!

@Gary Trakhman My belief system is definitely out on a limb so you are more likely to be right. Let me try to bridge the gap in case you're able to pull me back in to the mainstream.

there some wins from only learning one language to hack on the language

That is narrowly true. But if you zoom out a bit, your computer relies on many languages. To hack on your computer you need to know many languages. So the narrow win turns into a loss in the long term.

Is SQL too confusing? You don't exactly know everything it'll do, but you don't also stare at query plans for every query.

Most people don't need to stare at their compilers' internals, yes. But there's a very long, very steep funnel here that the world relies on: users of software become programmers, programmers learn more about the internals of their dependencies, down and down until a tiny fraction of people have some sort of global picture of what the computer is doing. (Not what the computer should be doing. I'm not a reductionist, and I don't claim that system software is the most important or anything like that.) Better interfaces definitely reduce the need to inspect implementations (and SQL's interfaces like

explain

are among the best). But humans don't know how to eliminate the need entirely. And Jonathan Blow would claim we shouldn't. In the big picture and the long term implementation quality matters.

Declarativeness and recursion go hand in hand.

I like both! The reason I called recursion misguided here is that it's not actually recursion. Recursion in a single function or spanning a few functions has a clear base case. Here the base case spans codebases and unbounded spans of time. Most of the time our computers don't have it.

Is it so different from a function call? I have to run through a mental model of the data structures and what they'll do.

To me the essential dissatisfaction in this group seems to be that we should rely less on working things out in our heads and offload as much of that load to our computers as possible. When you're tracking down a bug in a fundamental compiler that affects your whole computer, that burden can be quite large. So yes, there are similarities to function calls but they're sirens. Focus on the differences.

would you say a runtime library implemented in the same language as user code is too implicit?

I actually don't have a problem with using a subset of a language to build the whole language. All my projects for the last 6 years (since http://akkartik.name/post/wart-layers or so) do this. There are no cycles and it's perfectly reasonable. In the end my goal is for the software that makes up a computer to reward curiosity. That's my North Star, and I think many of the problems we discuss in this group would be resolved if we hew to this North Star. https://www.reddit.com/r/ProgrammingLanguages/comments/cybbd5/september_2019_monthly_what_are_you_working_on/eyubr43 Hopefully this kinda pointillistic set of responses helps convey where I'm coming from.

It sounds a little extreme to totally avoid this, but it might be interesting to categorize these abstraction violations by distance, time, artifacts involved, into an overall measure of brittleness. For example, I think clojure macros depending on functions that might depend on other macros is kind of bad? Each function is a 'compilation unit', and in order to run a macro at compile-time, the functions it calls has to be already available in the compile pass's run-time. It's a little clearer to have the macro language run in a totally separate context, though maybe less convenient for the implementation of an individual macro. But it's all still just a program running, not creating artifacts outside of its memory space. That might be worse. It is worse, when you consider the practical problems of clojure AOT compiled artifacts (classfiles, jars containing both compiled code and source code).

Interestingly, Racket macros enforce a deep notion of linearity: macros can call functions but not functions that call them. The compiler constructs a topological sort for you and gives pretty decent error messages if it encounters cycles. ...and I don't like it 😄 I think macros should be pretty unconstrained. I'm more concerned about linearity when it spans multiple codebases. Mutual recursion within a single codebase is fine. Hopefully this shows that I don't "totally avoid this". But yes I think the places to avoid cycles are precisely the places where it's most difficult today.

To me the essential dissatisfaction in this group seems to be that we should rely less on working things out in our heads and offload as much of that load to our computers as possible.

I think it's a moving target. It's not clear to me that computing can ever get totally accessible, but maybe we'll focus attention on more interesting problems.

@Kartik Agaram When you define a language operationally through its implementation ("a valid C program is what my C compiler accepts"), then no language is implemented in itself. Maybe that's a good point of view to explain the problem. It immediately leads to the notion that the implementation language of X can at best be an earlier implementation of X, but not the very same language X that is being implemented. Pragmatically, the Guix package manager is a nice way to arrive at the same conclusion, since it traces the build dependency tree back to a ground-zero set of binaries. The Guix community is working on reducing this set to a minimum (see [this talk](https://av.tib.eu/media/42193) for details).

@Konrad Hinsen Well put! Your comment reminds me to link to http://bootstrappable.org. There's also a #bootstrappable channel on Freenode IRC if anybody's interested.

that's ok 🙂 it looks pretty cool. I wish I had a reason to use it.