Why any fundamental improvement in software has to be a generalisation https://blog.metaobject.com/2020/04/why-any-fundamental-improvement-in.html

The only way out I see is that change has to be both radical while also including the status quo, and the only way I can see of achieving that is if it is a generalisation, sort of like quantum mechanics generalised classical mechanics, superseding classical mechanics but still including it as a special case. (Or how circles were generalised to ellipses etc.)

I didn't really get this post. Isn't generalization the status quo? Software makes generalization easy; I think that's where a lot of our problems arise.

I agree with the author that generalization is the correct path. I think what he means is that a grand simplification of the current programming Complexity would be an excellent way forward. That is the thrust of my Beads project and even Early in the total Process it is working very well. There’s so much unnecessary underlying complexity in the Intel/arm Hardware And operating systems that can be generalized away, and you can reduce the APIs you have to learn by a factor of 10 easily enough. Generalizing computation is one part of the project but the other is convincing people to recognize the generalization and simplification as forward progress. Not only do you have the not invented here syndrome that engineers always fall prey to, you also have entrenched incumbents who With their power and groups of followers maintain the status quo. So it’s actually quite hostile environment that structurally prevents change.

He first (correctly) makes the observation that the world isn't black (radical change) or white (incremental only), but its a subtle grey (it depends). But then he immediately makes the same mistake again by heralding one extreme (generalization) as "the solution". Much like with radical vs incremental, an extreme (generalization) can never be the solution. I find most applications of generalization extremely damaging, in terms of complexity and inefficiency it brings. People are scared of specialized software because they're scared it won't be able to do all things for all people, but the benefits it brings in terms of simplicity and efficiency can be huge. That's not to say that I would claim "specialized" is now the holy grail, but rather that the truth is again.. grey.

All of modern technology is basically incrementalism. Every year we improve some process, and in the case of computer hardware the steady improvement led to a million to 1 price/performance improvement when compared to 30 years ago. However, the same cannot be said of software. Incrementalism has been very ineffective in software.

C++

has been through what 20 versions, each time adding more crap to its already bloated design, and if anything it is getting worse and the complexity level of

C++

is absurd at this point.

Perl

is another language whose accretion process had led to

Perl 6

where reportedly random strings have a high probability of being valid code. That is another clear example of incrementalism being not better. Computer languages are one of the few areas where conceptual integrity, simplicity, elegance and generality confer great benefits, and I assert that languages represent a quantum level jump when they are introduced, and that many of them are damaged over time as the original author leaves and a committee of lesser minds takes over and throws in too much. The productivity difference between machine language and assembler was huge, and FORTRAN was a big jump over Assembler. The question is what is next in the

Algol

family, Transformation languages family (

LISP

), and the other lesser known families? Calling something radical is introducing a political concept that doesn't really apply. As for your point about specialized languages and tools, they are indeed very practical and useful, and the fears that people have about using them are mostly unfounded. Specialized languages can make short work out of what might otherwise be a difficult task. I once did some extremely complex text processing, and used the

Icon

language, and I don't think the task could have been accomplished in another language, because Icon is the king of text, being Griswold's evolution over 20 years of

SNOBOL

. Yet hardly anyone has heard of Icon or used it. So i agree that people are superstitious and keep using the top five languages even when contraindicated. Basically they have a few hand tools and when they need to do some specialized task they reach for a hammer, saw and chisel instead of the transmogrified reverse entabulator tool that is perfect for the job.

I'd say it all depends on what aspect of software you look at. At the most global level, incrementalism is the only way to go: lots of software systems have become critical infrastructure for modern societies, and they won't go away, nor can they be replaced overnight. At a more focused level (as in @Edward de Jong / Beads Project’s explanation about programming languages), incrementalism mainly leads to accumulation of cruft. Many people work on better foundations for future work, but neglect building bridges to the legacy systems that we all depend on. I tend to agree with those who say that more focused (and thus simpler) languages and tools are the way to go, but unless they cooperate with the technology of the past and with each other, they won't even be an option for people who deal with information processing on the decade-or-above time scale.

@Konrad Hinsen good points that is an issue of scale.. being radical is easier in isolated circumstances 🙂