Folks, can you please point me to recent efforts in Distributed Reactive Programming? How can we build whole systems (spanning multiple simultaneous users as well as the database) in React/Vue/Svelte- like way, instead of just UI components? What are the typical challenges and the unsolved problems?

I thought of this https://dl.acm.org/doi/10.1145/1807167.1807184 but its for large graphs and not reactive. Pregel: a system for large-scale graph processing

@Tom Larkworthy Nice! 🙏 It's from 2012 so feels a bit dated. React/Vue/Svelte all happened afterwards. I came across this one from late 2024, though not a survey paper: https://dl.acm.org/doi/pdf/10.1145/3689492.3690048

not sure if I understand the connection to React/Vue/Svelte. They are not network technologies. I worked on the Firebase realtime database in 2014 which was a causally consistent event driven database. We now have convex.dev or zero or livestore which are successors aimed at application building. None of them are really distributed in the fault tolerant sense. I also worked on Firestore which was actually geographically fault tolerant (5 nines availability!) but it is so complex it requires a mega corp to cosntruct (its built on some of the same infra as Google Spanner). I feel like Firestore probably is the state of the art in true distributed reactive databases but its not necessarily the one I would choose to build an app with, but technologically it is impressive.

Is there a difference between reactive programming and programming a system of asynchronous reactive components?

I don't think so. Components just emphasize modularity and reusability, in addition to reactivity.

Its actually something I'm working on myself - although in a way, I guess a lot of people probably are in bespoke sort of ways. I'm glad to see @Tom Larkworthy chime in, I had a feeling they'd have some good insight/links in this department. There's also https://rxdb.info/ which I've checked out but never used.

Electric Clojure seems relevant here https://github.com/hyperfiddle/electrichttps://github.com/hyperfiddle/electric

Some recent work on propagators might be interesting? For example: • An implementation of FRP with propagators ◦ And see Alexey Radul's PhD dissertation on propagators, which that was based on • Implementing propagators with Spritely Goblins, which means they can be hooked up to each other over the network via CapTP Now, I haven't tried composing those two things yet...

Hrm, I think that "distributed reactive programming" describes what I am trying to invent at work. The ingredients that led me to this are: • A complicated domain entity at the heart of our app, with complex internals and a lot of business logic tightly connected • We want some interactions to be as-you-type in the browser, and in general we want all results to be calculated "as you edit" rather than by clicking a "run simulation" button • Some calculated results can take multiple seconds to complete due to intensive algorithms • Some calculations we don't want to distribute to the browser so we can keep our implementation private This all has led me towards something like: • Read "out of the Tarpit" and decide to crack down on storing only essential state and calculating all the derived state, and relying on caching for performance • Read https://lord.io/spreadsheets/ and try to implement invalidation algorithms • Build in some kind of networking protocol so that nodes in the data flow graph can be transparently "replaced" with implementations on the other side of a network • Try to make the API "look like" Vue's implicit dependency tracking I'm not finished with this and have had several goes at it. I vacillate between thinking this is all over complicated and I should give up on the dream of a fancy in-house data flow engine, and thinking this is going to be incredible if I can just solve cache invalidation.

@Daniel Yeah, I'm in the same boat. 🙂 I came across an interesting paper that explored mixing of imperative code and reactive code which you might find interesting: https://arxiv.org/abs/2306.12313

and thinking this is going to be incredible if I can just solve cache invalidation.

https://github.com/observablehq/runtime/blob/622a1974087f03545b5e91c8625b46874e82e4df/src/runtime.js#L82 marks variables as dirty, marks a subset as reachable (they are on the page). Then does the transitive closure of the dirty set and updates in topological order. If a cell has not been marked dirty, it is not recomputed. Because it computes all its tracking stuff up front, I think it can support dynamic mutation of the dataflow graph. Glitches get deduped out coz its a batch algorithm.

c.f. 'glitch free observable reactivity semantics' which demonstrates deduping of updates, as defined above https://observablehq.com/@tomlarkworthy/observable-notes

Its best to view the Observable dataflow graph as monotonically converging to the latest state rather than a stream processing engine.

Oh I like this, this names something important and why I was wary about pulling e.g. RxJS off the shelf.

yeah I tried to like RxJS in 2016 but hit quite a wall with glitching, memory leaks and stuff, it felt extremely fragile to build a large system out of. I was quite skeptical that Observable could make something usable with no explicit operators but they seem to have solved all the main problems and improved in some unexpected areas like returning generators. There are some gaps: to make something like fine-grained reactive UI components, you need dynamic dataflow construction, as each component instantiation has an identical dataflow pattern you want repeated, and with the notebooks that is not possible. Notebook let you build a static dataflow graph dynamically, but there is no paved path to do it programatically. This is more of a limitation of the frontend notebook programming model and not the underlying runtime though. The runtime is pretty general in my opinion.

React developer expressing similar ideas: https://overreacted.io/the-two-reacts/

How can we build whole systems (spanning multiple simultaneous users as well as the database) in React/Vue/Svelte- like way, instead of just UI components? What are the typical challenges and the unsolved problems?

So I'll be honest, while this question strikes really close to home for me, I wasn't really sure how I wanted to respond at first. The request was for literature on recent efforts. I don't really have that. However, I've actually already built a system somewhat like you're describing. It's still in production right now and powers billions of dollars of loans for a national mortgage lender. I don't work there anymore, but I know people that do. I didn't set out to build a "distributed reactive system", and so its a little more ad hoc that an idealized distributed reactive library or framework might be, but it also does things I haven't seen any other system do, so it might be worth sharing a little information about it. Would anyone here value a little write-up about it?

I would!

@Russell Leggett Yes, would really appreciate if you could find the time to write about it. While there is a lot of writing on distributed systems, this question adds two new complexities to distsys: reactivity and protection boundaries (eg: client and server). I have not seen any writing on this yet.

ok, here's what I'll do. I'll write something up without going into tremendous detail, and then hopefully that'll be a good launching point for any more detailed questions

wow, good luck, sounds challenging.

I've just been working on this stuff a really long time, starting from 2007. There's a lot of ground to cover to describe all the relevant aspects of a distributed reactive system, especially when a lot of it was custom and working from different reference points than OP and most readers would be familiar with (since it has a totally different genealogy). I've never really done a public facing writeup of it before - we never did company whitepapers or anything. So, its been interesting just the act of writing about it, especially all this time later. I still wanted to share a more appropriate briefer writeup response more like I intended - I mostly just need to organize and edit from this larger document into that. And then I can share the larger work later if anyone is interested.