This was fantastic. At least 20 entirely new ideas in here. Thank you!
But to get on a pet hobby horse for a second and dwell on a single, throwaway paragraph:
"A programmer is ideally an essayist, who works with traditional aesthetic and literary forms as well as mathematical concepts to communicate the way that an algorithm works, and to convince the reader that it is correct." -- Donald Knuth (1992)
The fundamental reason programmers can't be essayists is that essayists don't get paid. The niche of essayists who get paid a living is
miniscule.
I'm always torn when I hear this quote and others like it. On the one hand it violently feeds my biases, but on the other hand the person quoting it is usually describing Literate Programming or the value of readable code to an (at this point imaginary?) audience that hasn't heard of it, and is content to point out the gap between where we are and where we want to be, without delving into the reasons for the gap. It makes it seem like pointing out the problem is the hard part, and now that it's pointed out everyone can go forth and write glorious essays in their code.
There were no essayists when writing happened on clay tablets. Its only purpose then was tracking shopping lists [1] and at best the occasional bit of business correspondence [2]. Programming is in a similar state at the moment.
[1]
https://archive.org/stream/sumeriantabletsf00bedauoft/sumeriantabletsf00bedauoft_djvu.txt
[2]
https://en.wikipedia.org/wiki/Complaint_tablet_to_Ea-nasir