I disagree strongly - there is a debate needed what progress even means! https://twitter.com/jonathoda/status/1345858339343884288

The comparison with musicians is interesting. There's a limited sense in which music is inherently open source. You can just listen to it and, with enough skill, recreate it. It does take a lot more skill and equipment than compiling an open source project, or even lifting a technique from said project, but I think it lifts some of the pressure that inclines software people towards total transparency. Closed source software hides things we really need to look at. While I enjoy having free software all over the place, I can't see where Jonathan is wrong. You get what you inventivize. Neither "CEOs" (rather broad brush there, btw) nor open source's simulated annealing approach are going to get us where we want to be. I don't have anything like an answer, but here's one random idea: different financial/openness regimes for different layers of the stack, as well as different stages of development. Open source is working pretty well for basic libraries, for instance. We should keep doing that.

I don't find either of Jonathan's arguments particularly convincing. On the first article, I don't see anything special about 1996. The "pre-1996" list includes LISP - a language from 1958! If we are allowed to reach 40 odd years in the past, then a lot of the 1996 list can be included in a 2020 list. It also oddly excludes C++ from the description; "useful incremental improvements on top of the foundational technologies that came before", despite it's name being a literal pun on incrementally improving C. The second article, I don't really follow what type of organisation is expected to be producing groundbreaking research. Small software houses making UI widgets? From my understanding, most significant steps forward have come from monopolies, academia, or military. (this is where most of the 1996 list items came out of). He says research can't be done at a startup, so what research is actually being hindered by OSS? I read a lot of what @jonathoda writes, but I can't follow a coherent line of reasoning here. Perhaps they are more just meant to stimulate debate 🤷

People don’t want to debate. I’ve gotten so much hate from those posts. And lots of nitpicking. But almost no actual arguments.

I am very sympathetic to the argument

@jonathoda sorry about the heat. One standard argument in favor Open Source has been escaping vendor dependence, but what with the Cloudy world we live in, I'm not so sure things have worked out that way at all.

Charlie Stross has got a great essay about how a lot of golden age sci-fi had reasonable expectations about progress in spaceflight if you looked only at the technological domain, and not the political one. This sort of came to a head with Project Orion, the first (and so far only?) proposal for routine travel within the inner solar system; it failed for lack of a political framework capable of organizing the construction of a ton of nuclear rockets in space while maintaining mutual trust that their payloads were going to stay peaceful.

Meh, I've seen great strides in these things since 1996: in ML (deep), dependent types, session types, macro systems, CRDTs, probabilistic data structures and algs, graphics (shaders and parallel ray tracing), schema management (CQL, datalog, and others). My understanding is that the low-hanging fruit is gone rather than there is a lack of creativity. For example, my side project is maintaining system level safety properties under composition while maintaining some level of productivity. The math is super difficult and it may not even work within the heat death of universe.

I think I get the point of this post, and consider it correct, but perhaps the way it is formulated makes the post repelling to people whose personal contact with Open Source is different. For example, the term "Open Source ideology" only applies to a small fundamentlist group of people, best exemplified by Richard Stallman, who however would loudly protest against the use of the term "Open Source" rather than "Free software". There are various arguments for Open Source, and all of them are valid in some context, but don't adapt well to others. In my own domain, computational science, Open Source is increasingly seen as well aligned with how publicy funded research works, or should work. But that argument doesn't cover Linux, Firefox, or VSCode, to name a few well-known projects with very different social structures behind them. I see two important aspects in the Open vs. Closed Source debate. @jonathoda addresses one of them: lack of incentives for significant innovation. The other one has been mentioned by @Andrew F: "Closed source software hides things we really need to look at." I don't think comparisons to music and other creative endeavors outside of technology are very useful in that debate. Technology is special because of its high impact on our lives ("...and then our tools shape us."). The opposition between Open and Closed Source is what has led us into a situation where we have to choose between performant tools and tools that respect our need for personal agency and privacy. My current view is that we can remove this contradiction only by accepting that technology is a political topic. Given that our tools shape us, we can't just leave their design to small interest groups, be they techno-elitists or anarchists.

I think @jonathoda and @Konrad Hinsen bring up a few points. Yes we need a debate. We might actually need the ability to debate! What values should be expressed through our software? When OPen Source means a Liability towards maintaining a certain fixed developmental state , is this gaianing in resilience for society or losing it? What are the values we want to further and which we want to keep in check? Trying really to articulate, making the effort in creating distinctions in necessary IMO.

We need a debate indeed, but not just we in this minuscule community, but we as citizens in industrial and post-industrial societies. And as you say, we probably don't even have the ability to organize that debate. None of the institutions of our societies seems to be able and willing to do it. Next question: can we get that debate going in a bottom-up fashion? In parallel, we as technophiles can start thinking about technological middleways between all-open and complete opacity. One intermediate is audited code, examined by trusted specialists who report on its potential risks to the public. But can we make code auditable? Ensure that those trusted specialists can examine it with reasonable effort? Can we prevent the obfuscation of undesirable features by technical means? I'd love to see type system experts working on such questions. After-lunch addition: my lunchbreak podcast is very related to these issues, and offers some ideas to get things going: https://www.jimruttshow.com/clayton-banks/

I'm glad @jonathoda wrote these. Open source (or free software) has this image of being generally good, something about giving value away for free, but it's just a broken model which idealizes the creation process without any concern for maintenance. There are great things about open source — transparency, for instance, as discussed above — but its solution for funding value creation through privileged creators with enough skill and time on their hands who hopefully stay motivated long enough is… suboptimal.

Now that I read "value creation through privileged creators with enough skill and time on their hands", the parallel with science strikes me even more: that's exactly the profile of the first scientists, in the 17th and 18th century.

Yes, the gentleman scientist shoe definitely fits the open source developer. And the correspondence sometime reads very much the same.

I was about to review the post before commenting, but it's been retracted. So, this is more topically related than a direct response. I've often been puzzled by how so much of the modern computing world was built by companies like xerox and bell, and how that doesn't seem to be happening in the same way now. But reading David Graeber's The Utopia of Rules over the break offered a pretty compelling explanation:

First of all, the amount of really innovative research being done in the private sector has actually declined since the heyday of Bell Labs and similar corporate research divisions in the fifties and sixties. Partly this is because of a change of tax regimes. The phone company was willing to invest so much of its profits in research because those profits were highly taxed—given the choice between sinking the money into higher wages for its workers (which bought loyalty) and research (which made sense to a company that was still locked in the old mind-set that said corporations were ultimately about making things, rather than making money), and having that same money simply appropriated by the government, the choice was obvious. After the changes in the seventies and eighties described in the introduction, all this changed. Corporate taxes were slashed. Executives, whose compensation now increasingly took the form of stock options, began not just paying the profits to investors in dividends, but using money that would otherwise be directed towards raises, hiring, or research budgets on stock buybacks, raising the values of the executives' portfolios but doing nothing to further productivity. In other words, tax cuts and financial reforms had almost precisely the opposite effect as their proponents claimed they would.

— https://theanarchistlibrary.org/library/david-graeber-the-utopia-of-rules

The WayBack machine has the retracted articles. I basically agree with the first one regarding the reduction in the speed of progress over the named period, and I certainly remember the change in tech culture after the Netscape IPO: it brought many MBA/Wall Street types into the industry and shifted the culture of the Valley from nerd concerns about doing cool things to greed-head concerns about getting rich. My only disagreements, really, are: (a) that startups do sometimes manage to do research and ship new stuff, though that has become increasingly rare because of the aforementioned cultural factors (short runway/fail fast/MVP thinking); and (b) there's been a fair bit of interesting research over the intervening decades that mostly hasn't made any difference to end users--or even to most of the industry. For example, Racket's language-oriented programming stuff is great! That said, there are other things that people rely on every day that are sort of invisible, like advances in distributed systems that have allowed reliability at scales that would have been hard to imagine in '96 and huge engineering improvements around non-parametric regression (which too many call "AI"), plus miniaturization and manufacturing improvements that allow all of us to carry around touchscreen supercomputers everywhere we go. The second article is much more hit-or-miss. Many important pre-1996 improvements were done as open source projects. Some that I myself worked on include bind (DNS), sendmail (SMTP), converting EAY's open source SSL implementation to the still open source openssl in the context of the _mod_ssl_ module for the open source web server Apache, the BSD kernel, and so on. Anyone reading this post probably used at least two of these technologies today, MacOS/iOS users have likely benefitted from all four. There's definitely a cultural problem around how modern OSS work is funded. The idea that individuals should fund the projects they use out of after tax salary while companies carry on building empire on the backs of OSS contributors is grotesque, and I'm still fairly bitter that I wasn't able to get Google to put any money (beyond Summer of Code projects) into improving emacs even though half their engineers were using it daily.

it also killed the market for development of better software tools. [...] You can’t compete with free.

Jetbrains might argue with this assertion: https://www.bloomberg.com/news/articles/2020-12-18/czech-startup-founders-turn-billionaires-without-vc-help

Jonathan has undeleted the articles now. https://twitter.com/jonathoda/status/1346485120488771585

Ideas of progress also depend on the kind of society you want to have, and people will disagree here. For example, we could say that the kinds of advancement in tracking and machine learning when it comes to web browsing has resulted in great progress for advertising and the bottom line of the affected corporations. If you value the profit maximization of such corporations -- or the kind of economism that goes hand in hand with doing so -- then this is certainly progress

Physicist's snarky comment: progress is measured by the increase of entropy. Just get more computers running and heat up the universe, that's progress 😉

On the topic of science there was a good question on Alan Kay's AMA once, paraphrasing: "We know we're making progress in science because we can explain the world better, more easily. How do we know we're making progress in computing?" I don't think Alan was able to answer it. Will see if I can find the original.

I fully disagree with @jonathoda. I don't know where he gets funding for his own research, but for me there is no funding. First of all, I happen to be one of those that would prefer music and movies be as free as FOSS, because a economy based on artificial scarcity is bad in a world of impoverished people. But whatever, the current model for music/movies isn't that bad - those who can afford to pay usually pay, and those who can't, don't. I've lived in the Philippines so I've seen how this works - in practice it doesn't work too badly, as filmmakers get paid and poor people still get entertainment. Programming languages is a totally different kettle of fish, where convincing people to use a new tool is already difficult. Why would you want to hinder adoption further by charging money for each copy? Why would you make a tool that OSS fans (myself included) are likely to refuse to use regardless of the merits? The Wolfram language is maybe the best example. It's an excellent tool with an huge library of useful tools, but you have to pay to use it and I think that's why it has gained very little mindshare among developers. Open source suffers from quality issues for an obvious reason: people can only afford to do it in their free time, and almost no one is hiring teams to do open source software. But it does not follow that freedom or "free riding" is the problem. Charging for each copy made far more sense before the computer era, and that's why we used that system. In the computer era it makes far more sense to shift to a public funding model where people doing PL research and open-source engineering can apply for grants or UBI or something. I, for one, would gladly accept minimum wage to work on my PL research and open-source engineering. It sure beats the zero income I'm getting now. I once paid $300 for a compiler when I was a teenager - I had a paper route and could earn that money. But the paid model only worked because (1) there weren't good open source alternatives at the time and (2) I was buying a "proven" solution (a C++ compiler) which everybody was telling me was the best way to do software development (I eventually realized this wasn't true and now recognize C++ as a pretty shitty language). If your goal is to do experimental work and really challenge the status quo, how does the paid model work in 2021?

With conceptual tools, you pay with your attention?

@David Piepgrass I'd like to take up one of your examples, which I happen to know well: the Wolfram language. It's a combination of two things, both very well done: a computational infrastructure, and a huge library of machine-readable knowledge encoded using this infrastructure. Compare with Wikipedia, which is also a huge library of knowledge, but much less machine-readable (in spite of Wikidata, which is a great project), and in particular not executable (something that @Denny Vrandečić is working on). There have been many attempts to make an Open Source clone of Mathematica, which have all failed. Wikipedia is the model of open collaboration that everybody points to. So what's the essential difference? My analysis: the Wolfram language is a top-down design with a genius in the dictator position (I don't use the term "genius" lightly, but Stephen Wolfram is one of the few people I consider derserving it). Wikipedia is a bottom-up effort to which everyone with minimal digital literacy can contribute. We'll get the Open Source equivalent of the Wolfram language when a large enough part of the population acquires sufficient digital literacy to be at ease working with executable knowledge and its strict semantics. Unless of course AI-based mediation makes Wikipedia knowledge robustly executable before that happens. My crystal ball isn't very clear on that question. The problem with computing is the requirement of managing precise semantics over multiple level of abstraction. That's a rare competence today. Open Source projects work when they have enough people with that competence (called "maintainers") participating over long time spans, whatever their motivation may be. But maintainers can't adopt the right mindset for innovation. Stephen Wolfram can.

@Konrad Hinsen Um, I'm not sure how your comments relate to mine or to the question of "is open source stifling progress?" If we restrict our discussion to geniuses, the question of how they should earn a living and whether their work should be free to use remains open.

@David Piepgrass There is the eternal issue with the "Open Source" label that it is used for two different things: the legal status of source code, and the social structures of its development (the famous "bazaar"). My example was meant to illustrate why the bazaar model doesn't encourage technical innovation. The legal and social perspectives of Open Source are orthgonal issues at first sight, but in practice, most Open Source projects rely on both the legal and the social part. One way to frame recent criticisms of Open Source, including @jonathoda's as I understand it, is the lack (so far) of other social structures that can predictably and reliably produce software that is legally Open Source. In particular structures that are more compatible with innovation.

@David Piepgrass "I would further argue that people like me shouldn't have to be businessmen and shouldn't have to found "startups" and spend our time fundraising from VCs and wealthy "angels". My passion is software, not business" while I can sympathise with this idea, I really do think its quite a selfish statement. Niklas Luhman has this idea of System Theory and Sociology , in which he says the following about structures and functions within societies: a function is needed in a society so a structure will emerge to support that function. So you always have to ask yourself, what can be a good balance between your needs and the needs of society. I do thikn the idea of a universal basic income could be a major driver for positive change in society. But there are many people in most societies who do not think this way.

@Konrad Hinsen Re: The Cathedral And The Bazaar, I'd say bazaars are best built atop well-designed cathedrals, but that there's little funding for the latter so you can't expect too much. We have cathedrals of greater and lesser quality - WebAssembly, CSS, React, Haskell, Mathematica, etc. - but rarely does good design coincide with popularity. I do not see the utility of taking a moral tone. What's the point in saying someone is "selfish" for wanting to build a cathedral instead of becoming a businessman? Even if I were a businessman I don't see how it would be financially viable to build what I want to build. Are you suggesting that the market value of a thing is equal to the benefit to society of that thing? That's obviously not true, otherwise people would not rely so much on software whose market value is zero.

@David Piepgrass I agree with all that, and I wonder if analyzing software production by market mechanisms is perhaps a big part of the problem. Software is a product with high up-front investment requirements but near-zero marginal cost when used. That's not incompatible with market mechanisms, but very different from the markets that economists have been studying for a few centuries. Moreover, well-designed foundations are best thought of as enabling infrastructure, and thus a public good.

@David Piepgrass "atop"? Fix the metaphor please, "bazaars are best built beside well-designed cathedrals." It makes for a nice image. A person is more likely to pattern match and find truth more truth to it.

I will try to clarify my perspective. I think that most existing software artifacts give "the user" - the society something to reflect on. Now since software can be very flexible. The question is , what kind of relationship forms this between the people creating the software and the people using the software. What kind of software should be created by others/3rd parties and what should be created by people themselves. Abstraction comes at a cost. The cost in how to think in this model of software and all the effort involved to be able to think in this model are real. This kind of complexity is not very well understood by society. Instead society is happy to bounce off this responsibility towards "the computer industry" - the market? Its very hard if people in society can't tell what they want , let alone talk to each other,

@wtaysom No, not beside. One builds things on top of React or Mathematica. Sometimes we get a bazaar atop another bazaar, like shell scripts vaguely trying to be compatible with "a unix shell" or something that was built in bazaar style (like C++ or GNU utilities), but I prefer bazaars that build upon well-designed cathedrals.

I mean "beside" on the concrete side of the metaphor. 😉 A physical bazaar is liable to be beside a physical cathedral — though the cyberpunk vertical arrangement would make for a nice illustration.