Eli Mellen07/08/2023, 10:47 PM
What this story tells us is that software has consequences. It's easy to look at a missile guidance system or High Frequency Trading and say "that's unethical!", but far more mundane software performing far more mundane tasks can also have dangerous or even lethal failure modes.
For example, banks are notoriously bad at updating names, and deadnames can resurface at inopportune moments that risk outing the user to housemates. Parental spyware will out a kid to their parents, risking homelessness or suicide.
As engineers we must keep the whole system in mind, including its users and their wider lives and situations. We must respond to our products' worst failure modes, no matter how unlikely we believe them to be. You cannot rollback a corpse.