Coroutines also have the advantage (depending on how they are implemented) of having far smaller (and extendable) stacks than preemptive threads, which allows them to scale to several orders of magnitude more concurrency than preemptive/OS threads.

coroutines allow us to avoid the complexity of state machines (by using stacks) without making in impossible to write correct code (as preemption does).

...

This is unlike multiple threads preemptively writing on the same memory at the same time, which is an environment where even the world's top experts have been shown to be unable to write correct code.

Sorry, I don't buy it. Async lets you write more efficient code around IO scheduling. It solves a particular problem, that is completely orthogonal to correctness or memory safety. Coroutines are about "when", not "where". Corountines do not solve multithreaded memory safety. Async and Parallelism are not the same thing. You can still have race conditions with coroutines (Go has had plenty of bugs showing this). Multiple Coroutines can still be scheduled on different CPU cores and write to the same memory. You need something like the Rust borrow checker to solve that problem. Not coroutines.

Coroutines also have the advantage (depending on how they are implemented) of having far smaller (and extendable) stacks than preemptive threads, which allows them to scale to several orders of magnitude more concurrency than preemptive/OS threads.

This is the problem coroutines solve. Coroutines deal with optimization, not correctness.

Most Lisps implements coroutines the same way btw: async, yield, and iterators

Look. You can do some cool stuff with async and coroutines. But it's just another tool in the toolbox imo. In fact, in my experience, show that async and coroutines make debugging more difficult not less. Coroutines expose more things for the programmer to think about, not less. Standard sequential architecture hides all scheduling from the programmer. The OS or runtime pre-empts your code at arbitrary points. It's invisible to you. You don't think about it. It's like Virtual Memory. Virtual Memory creates the illusion that you have infinite RAM (and the OS in the background pages memory in and out of your process without your program having any knowledge of it at all.) But, there are downsides. A database really needs control over when things are paged in and out of memory. But I don't want all programs to do this. It makes life harder on the programmer. I want the option to pick the right tool for the job. For databases Virtual memory is the wrong tool, for some other application it's the right one. Pre-emptive scheduling is the same idea. You write your code as if there was a single CPU core and your program is the only thing running. Then the OS will sneakily pause your program and let some other program run, you being non the wiser. This is a great abstraction! Similarly, coroutines explicitly expose the scheduling points to the user. Preemptive scheduling has the disadvantage of only being able to proceed as a fast as it's slowest I/O call. If you are building a web server, preemptive scheduling is the wrong tool. You want to process a lot of I/O and not have have to wait for the slowest one, you really want something like coroutines. But! now you have many possible orders of execution that you have to keep track of when debugging. The illusion of a single linear order of execution is gone. You have added more states to your DFA, not less. I have some experience in this area. I am actually a big fan of coroutines! I spent a year working with the async team working to add coroutines and CSP to the Nim language. I read a lot of research on this stuff, and I have experience with the inner workings of both the language transforms and the schedulers. CSP has some cool properties. I'm not denying that. It certainly isn't some magic paradigm shifting solution to modern software architecture though.

Here is an example where (formal) CSP works really well. I would like to point out that it is used as an explicit tool to model state machines, explicitly modeling all the states, not hiding them: https://www.reaktor.com/blog/why-csp-matters-ii-how-do-i-know-sync-works/

Another really cool use of CSP principles is delimited continuations, as seen in Racket: https://docs.racket-lang.org/more/index.html#(part._.Continuations) It's a way to do AJAX style stateful request/response, using only a single handler. It combines server routing and I/O scheduling of the network call together. IMO this results in much more readable and clean code. Iirc Hacker News uses this trick, still to this day. I wish more modern languages and servers supported the technique.

here is a difference between what is possible and what is practical. For example, if I try to use some open source JS code, will everything be written in async/await? Or will I have to rewrite it and potentially maintain the async/await version will future updates of the module? What about all of its dependencies? Do I update and maintain all of those too?

That's a good argument, though as much of a political one as a technical one. It's more practical from a technical standpoint than you might think though. It's possible at the compiler level to use program transforms. i.e. lisp style macros (what Nim and lisp do) or compiler passes (what Zig and Go do) to automate a lot of that, at least from the point of the standard library. Just make the standard library I/O calls async, and automatically transform the callers into delimited continuation closures. Admittedly that is fairly invasive and not foolproof (depending on how the caller code was written).

You could run some "macro expander" program on top of your JS code and all dependencies. I guess during the "bundler" build step in the javascript world. I think something like this is what Vue.js does for their component language? But at that point it's not JS anymore is it? You are basically building a new language that compiles to JS... Are DSLs that get macro expanded in lisp still lisp? 😛

I've been out of the JS world a long time, looks like React already added coroutines (specialized for react components): https://blog.logrocket.com/deep-dive-react-fiber/

That said, if some API was added to allow the programmer to control the ordering of events/callbacks, that could be used to effectively eliminate those yield points (by always ordering the most recent call first, unless it's an explicitly declared yield point).

@Ray Imber "Why is that a problem? Isn't that controllable by the scheduler? I.e the scheduler (iterator) controls the order of execution and the switching of promise chains." Yes, but AFAIK we have no control over the scheduler so using async/await is more like preemptive OS threads except the context switching is limited to all call boundaries instead of individual instructions. This is not how cooperative threading (coroutines) work. They are cooperative because the programmer explicitly chooses the yield points. Preemptive threads make it extremely difficult to write correct code. "I.e the scheduler (iterator) controls the order of execution and the switching of promise chains." I don't understand your suggestion that a scheduler and iterator are the same thing. Can you explain? If JS did give us control over the scheduler, we could get coroutine like behavior by reordering the callback/event queue to avoid any switching on call boundaries that was not requested via an explicit yield or resumed by an explicit resume.

Continuing to resurrect this thread (sorry lol), but have you seen the news? The JVM is getting lightweight threads and coroutines 😁 https://ales.rocks/notes-on-virtual-threads-and-clojure