There's an (IMO) utterly cringe-worthy mobbing going on at the moment regarding a Github repo for the epidemiologists who produced an influential paper regard Covid-19 https://github.com/mrc-ide/covid-sim/issues/165

What's next? We'll learn that it's unethical for any software with an SLA to be programmed in a dynamically-typed language?

One point of contention is that due to fairly complicated technical reasons, the code is not deterministic. The scientists point out that the entire simulation is stochastic, so what does it matter? When they are preparing their results they run it 1000s of times and just note the results somewhere (like - gasp - on paper maybe?!).

Wait 'til these programmers learn the meaning of the word "simulation". Oh, to see the looks on their faces! What is a programmer? A miserable pile of secrets.

Upsets me no end to see these guys basically bullied by trolls during what must be the most anxiety inducing period of their lives. Very heartened to see John Carmack take a stand against the mob - https://twitter.com/ID_AA_Carmack/status/1254872368763277313

The implication that all of these folks see their tests pass and “know” they’ve written correct code is also horrifying.

Unfortunately these models were weaponized as instruments of politics; the run of the model that predicted hundreds of thousands of deaths caused the UK to pivot abruptly into major lockdown, and on the day the lockdown was announced the model was rerun and the death estimate lowered to 20k. That is what i read somewhere. With 280 parameters to tweak, this model can be coerced into yielding any desired result, and as a predictive tool it is frankly worthless. Models like this are better for helping build understanding of the factors than generating predictions. This isn't fluid dynamics, which at this point is extremely accurate. The fallout from this politicization of modeling, whereby you release a subsequently retracted non-peer reviewed paper that has major consequences, with no chance for others to validate or double check before decisions were made upon it, was a derail of the normally calm scientific process, and outrageous overestimates in the original paper, which were calculated to spur action, probably with foreknowledge they were exaggerated, will tarnish computer modeling for a long time to come. Per the body victorious, a virus can "create 10^72 copies in 12 hours", so how does one model a process that is like an atomic bomb?

I think there’s a lot of interesting questions about the reliability of scientific code, that responsible people in scientific computing struggle with, and make valiant efforts to improve. And I think this github issue contributes nothing of value to that effort, public understanding, or anything other than some commentators’ sense of superiority.

will tarnish computer modeling for a long time to come

Or to frame it in a positive way, it'll help the public gradually learn that models are not fortune tellers, but rather tools to assist reasoning — and that the models themselves are more valuable than the individual results they produce. All people are going to gradually learn this, and that sort of societal literacy takes a long time to develop, and is not a clean and straightforward process. This is a good thing. We're going to have to go through the same process with ML, and it's going to be even messier. That's not a knock against ML or the people misusing ML. The cybernetic human is in its infancy.

from an educator’s perspective, this is one of the reasons i’ve been thinking a lot lately about how to teach reflective simulation design & simulation epistemology. one of the weird holes in computer science education right now is the lack of a critical pedagogy around modeling real-world phenomena in code. it’s my hope that giving people tools to craft simulations of phenomena they care about for themselves, and putting them in a workshop or studio environment where they can collectively critique one another’s models, might be useful for building this kind of simulation literacy – but i haven’t gotten a chance to put any of this into practice yet

@Edward de Jong / Beads Project This is somewhat off topic but just to correct some misinformation, their report was not changed nor retracted. You can read it here https://www.imperial.ac.uk/media/imperial-college/medicine/sph/ide/gida-fellowships/Imperial-College-COVID19-NPI-modelling-16-03-2020.pdf (see pg 13) - one of the team addresses this misconception here - https://github.com/mrc-ide/covid-sim/issues/175#issuecomment-625377867

@Edward de Jong / Beads Project sounds like you're describing (what is it called?) math washing — using equations and algorithms to give the apparence of certainty. I don't see how we can ever have any math without the potential of misusing it. I guess it's just that the sooner we all know this, he better.

@Ivan Reese made a terrific point, maybe the fallout from this $20 billion a day cost in the USA lockdown will be that people stop blindly obeying unverified computer models. People have always been gullible and believed soothsayers, the computer is the modern equivalent. That they are claiming that 500k would have died had Britain not gone full lockdown is an unproven statement. In fact social distancing as has been practiced has very little evidence that it affects final outcomes. This 2020 study published in nature (https://www.nature.com/articles/s41598-020-58588-1) on the Flu virus shows that viruses become so prevalent in the atmosphere that they are basically unavoidable. The fact that they overestimated deaths in UK by more than factor of 10 is to the modelers excusable, because it spurred action. But is that responsible science? My objection is that exponential functions buried inside a formula almost always result in a runaway result, and from the Club of Rome onward, people have been predicting global catastrophes from famine, seas rising, ice ages, ice melting. Soothsaying merged with the computer holding the sign saying "the end is near".

@Edward de Jong / Beads Project we need a term that incorporates "soothsaying".... Digital Soothsaying... Data Soothsaying... Algorithmic Soothsaying?

Machine learning soothsaying MLS, or Artificial Soothsaying, or Augmented Prediction

The one conclusion I draw from this event that a crisis is not a good time for a serene discussions of how to do science, or how to make science-based policy decisions. There is an ongoing discussion in many scientific disciplines about how to do better computational science. Software engineering is one aspect in this discussion, but only one among many. So far, different disciplines adopt different priorities, for good reasons. I don't expect the debate to be settled any time soon, and I don't expect mobbing to be helpful in any way.

Scientific results are usually more reliable than the code that produced them. This seems like a paradox, but it relates to code being a model that is just one input to the process of understanding.

Or the code is a measurement of a model (or of a model of a model). The measurement can be approximate or noisy while still confirming the clarity of the fundamental model. Consider Newtonian mechanics (a clear, but imperfect, symbolic model of nature) vs observations of the motions of planets (noisy) vs a miniature-scale physical model (approximate). Good science can be built of rough data and imprecise recreation. Good science also doesn't need to be perfectly predictive, just helpfully predictive.

Yes, there are many layers of verification and validation in the process of accreting scientific knowledge, so there is a lot of redundancy that makes for robust results. But you have to be a domain expert to judge how robust any given quoted result actually is. In highly visible domains, which epidemiology has suddenly become, there are always people who oversell unreliable small-scale findings to the public, and others who criticize robust results with superficial arguments. Science works best when politics doesn’t care about its outcomes.

Here is a very hard-hitting blog article on the code quality of the Ferguson model. "First of all, the elephant in the room: code quality. It is very difficult to look at the Ferguson code with any understanding of software engineering and conclude that this is good, or even tolerable..for thirteen years, taxpayer funding from the MRC went to Ferguson and his team, and all it produced was code that violated one of the most fundamental precepts of good software development – intelligibility." https://chrisvoncsefalvay.com/2020/05/09/imperial-covid-model/ Ease of reading is a major objective of my Beads project, and i feel that it has been achieved. I am contemplating adding a feature where you can insist on a certain percentage of commenting in critical areas like function formal parameters, which when stripped down to single letters can be hard to read.

As so often, that's a comment from a software professional who seems quite ignorant about what he criticizes: research. The statement that "*for thirteen years, taxpayer funding from the MRC went to Ferguson and his team, and all it produced was code..."* is very wrong. Computational scientists all around the world still struggle to get funding for software development. Public research funding is attributed for producing research papers, not software. And people who invest too much time on software development don't get jobs in academia. And that's a big part of the problem. Decision makers in public research have only very recently begun to understand that code is both and important tool for doing science and an important output of research. And I hope that the story of this model will help to accelerate this process.

His bio said he was a virologist and did computational work so I think the useful content of that post is “someone doing very similar work has concerns about Ferguson’s model”, which is more interesting than “software professionals in an unrelated area do”.