There’s a recurring theme in my career, going back to my very first job, that goes under the mental headline of “wide logic” problems… That’s where you have a somewhat narrow stream (conceptually, not necessarily streaming per se) of data, and you need to evaluate a large number of functions against each item--at least a handful, often dozens, sometimes hundreds! The functions are basically always provided by the user, although normally in a declarative, non-Turing-complete DSL. I’m curious whether others have noticed this pattern, and whether you know of any more mainstream labels for it? The closest I’ve found is papers that reference publish/subscribe and indexing techniques for complex boolean expression trees, e.g.: • An efficient publish/subscribe index for e-commerce databases (VLDB ’14) • Analysis and optimization for boolean expression indexing [BE-Tree] (ACM Trans. DB. Sys. ’13) • A-Tree: A Dynamic Data Structure for Efficiently Indexing Arbitrary Boolean Expressions (SIGMOD/PODS ’21 … actually this is a new one for me, I need to read it now… 🙂)

Sounds like streaming analytics to me.

Sixth job was a SaaS app for monitoring compliance rules at brokerages… There were hundreds of handcrafted rules that needed to be evaluated against every Order, Trade and Account record after a nightly ETL job. Before I got involved, every rule was run one at a time, sequentially, and some customers’ nightly jobs were starting to run up against the clock.

Differential dataflow should handle it well I would think.

I’ve definitely been keeping a close eye on materialize.io 🙂 … IIUC they don’t do a great job with datasets that are significantly larger than main memory

ah yeah, if you need something out of core, I don't think DD (which is lower level than the materialize offering) will give you that, though it may be something you could add. Relational.ai is probably almost exactly what you want (very powerful variant of datalog that can handle very large datasets and does incremental evaluation), but they're a bit early yet.

Lately I’ve been more focused on teasing out the 3-4 very different cost models for the sub-expressions that queries are composed of: • pure functions that don’t even look at data or the environment; that can be evaluated whenever • functions that need to look at the record, or cheap environment stuff like the current time of day • stateful functions, usually aggregates but sometimes more complex • things that need to do IO: lookups, external service calls, etc. Caching becomes mission-critical

yeah I have seen this a lot too, e.g. achievements on fitness trackers (tell me when I have done 10,000 steps over 10K customer base). The specification is declarative but the data arrives incrementally. Due to the huge maintenance issues of trying to upgrade state on an incremental streaming solution, I tend to recommend just recomputing from scratch every X mins and throw horizontal compute at it (e.g. BigQuery). The system remains stateless then, plus you always need the from scratch computation anyway to recover from disaster, so it's the first thing to do anyway.

These are all Complex Event Processing (CEP) use cases. We built a system at a startup called Aleri in the mid-00s that handled this sort of thing with continuous queries implemented over a dataflow engine. We sold the company to Sybase, which sold to SAP, which -- so far as I know -- still sells the product today.