Worth rewatching and reflecting on what we had vs what we have now. Can be pretty chilling depending on how you frame the comparison

the most import part is at 13:40 ... https://programmingsimplicity.substack.com/p/functions-and-pipes?r=1egdky

Yeah! I personally think that, idiosyncrasies apart, programming directly in the shell is really elegant. I still prefer doing simple automation stuff on the shell because it has unconstrained access to the system. It's aways a bit awkward to do plumbing tasks in a programming language, too much bureaucracy. From time to time I dream about a sort of stream/iterator based concatenative programming language with a rich standard library. Hard to beat the simplicity of programs as data pipelines

Imagine:

A

produces a text string that looks like some sort of "language" (machine readable, not necessarily human-readable, kind of like assembler vs. "C").

A

sends the string - as data - through the pipe to

B

.

B

"parses" the string and calls functions based on what it finds in the parse. This might be called "syntax directed serialization". FP is hinting at this kind of thing, calling it "pattern matching". The key, again, is that A does not CALL B, it simply sends data to B. In UNIX, B already does something like this, but at a feeble level. Add PEG parsing to B's inhalation process to improve on its feeble parsing abilities. It's been done before to build non-trivial programs like compilers.

If you squint a little you see that's very similar to how compiled code is actually executed at the low-level, where procedures put their return values on a stack, to then be picked up by the next procedure in the queue. There's no direct coupling except for the ordering of procedures. I think lifting this paradigm so we could program like this at a higher level would be tremendously fun! We sort of have it with languages like Factor, but I personally find this breed of languages awkward to work with. Once you have the high level words defined they're ok, but to get there you have to code lots of small procedures, which IMO imposes some heavy cognitive load

code lots of small procedures, which IMO imposes some heavy cognitive load

The cognitive loading does not come from the small-ness of the procedures, but, is due to the flat-ness of the namespace, i.e. the "infinite canvas" mentality. The UNIX shell gets around this problem by allowing layering. A shell script can invoke commands or other shell scripts to an infinite depth (as opposed to infinite breadth). The shell and functional programming, fail to restrict this concept. There should be one kind of part to choreograph parts and another kind of part to do the work. Functions allow you to do this, but, functions don't restrict you from doing something broader, too. It's kinda like the GOTO problem in the early days, you could write structured programs using GOTOs but the existence of GOTOs tempted one to break structuring. Here, it's the same, you can write layered programs, but, you tend not to. Just try to understand someone else's code, 90% of the time it's hard, only a few programmers actually shine through as being capable of "writing good code". Another thing to note: UNIX worker commands cannot directly invoke other commands [*]. The UNIX kernel provides a privileged routine called the Dispatcher which decides which command gets to run and when. Again, it is easy to do this with closures choreographed by connecting layers, but, one tends not to structure code this way due to lack of enforced structuring constraints. A piece of worker code that CALLs another piece of code breaks the UNIX-y data-flow (message-sending) paradigm. One must be careful of stack-based ideas. LIFO-based code works on single computer systems, but breaks down in distributed systems, due to under-the-hood coupling caused by the global-ness of the stack. Code that works on single computers, doesn't necessarily scale up when distributed across many computers. [Elsewhere, I argue that context-switching is a crutch that doesn't scale well. We'd be better off using closures and message-sending using FIFO queues. Using functions works on paper for mathematics, but, isn't such a good paradigm for distributed computing]. [*] Modulo tricky uses of system calls, etc.

Very interesting remark regarding stack vs queue for coordination. On this context, doesn't using queues for inter-procedure messaging implies the need for "actors"? I mean, if there're per-procedure queue you'll always end up in Erlang land, no? Now you need addresses and lightweight processes, since all procedures need to be "live". I'm not saying it's bad, but definitely an explosion in complexity. I wonder if using something like a global tuple-space as a communication backbone isn't a simpler mechanism of equal capability, which further decouples components. Since retrieving a tuple from the space requires a query, the addressing scheme is lifted to the domain of application conventions, giving much more flexibility in exchange for more discipline. I'd be fine with that trade-off. The advantage of tuple-spaces is that the whole system can be made heterogeneous, you're not forced to use a single language and environment.

@Jon Secchis I have the same concerns about tuple spaces as I do with stacks. I would be interested in your further thoughts about "complexity explosion". Some more of my thoughts ... Extreme Decoupling

I have read both essays, and I think I somewhat understand the general critique and idea. I can't help but think that what you're proposing can already be done with intentional design and architecture, as a matter of "style", let's say. But I do recognize that having such "style" cemented in the environment, say a in PL, makes it much more manageable and enforceable. Now the clash between "things good for serial programs" vs "things good for parallel/distributed programs", I'm not sure that there's a broad enough domain intersection enabling a single paradigm to shine at both fronts. Tight coupling is inevitably better suited for single-cpu programs, and the opposite for loose coupling and distributed programs on multi-cpu.

"complexity explosion"

Surely you know that definitely assessing complexity is sort of impossible to do impartially, as it's always context-dependent. In the context of per-process queues, and particularly in the Erlang environment, I was using "complexity" as a proxy for the size of the domain ontology needed to establish the architecture. By embracing Erlang's model, you are bound to accept everything that comes within it, you gotta eat it all. Now, for something like tuple spaces, the vocabulary is significantly simpler. It becomes a tool you can integrate however you like, or rather it's more like a material, a foundational component, not the entire blueprint and framework. The complexity and solidity of what you build with it will depend on your ability to adapt it to your necessity, how well you employ it as an architect. The baseline complexity is low. That's my perspective, and I tend to like this trade-off better. I think it's also the preference of most people, that's why we don't have FP and full-blown actor systems as mainstream technologies. These things are binding, unforgiving, all-encompassing, long-term commitments. Turns out soundness is not that big of a deal; reality and the markets are more amenable to heterogeneity, even if it translates to a mess most of the time. The mess allows for moving faster and achieving good-enough quality at a fair price. Worse is better, as they say.

@Jon Secchis Thanks. Why is this model equivalent to the Erlang model? Assume that I don't know what the Erlang model is.

I am using Erlang as a benchmark since it's the canonical model of decoupling functional components via message passing with per-process message queues, where programs are modeled as choreographed distributed systems

When I look at Erlang all I see are it's faults and its relationship to functional programming. Maybe I should look again? Let me know if you have a suggestion of a good overview / starting place.