There is a missing semantic in Dataflow programming, something like function calling but not exactly that. After many attempts, I think I finally nailed it: Dataflow Templating. While I have got it working for Observable Dataflow, I think this equally applies to its forefathers like FrTime. I could reexplain it here but its in the article. I saw @Dennis Hansen grappling with the same thing for Artifact. I don't use diagrammatic representations of dataflow, but if I did, I would be trying to figure out what the graphical representation of this is. https://observablehq.com/@tomlarkworthy/dataflow-templating

i think this is a very useful semantic. In general, blocking loops are necessary when the subsequent loop requires the output of the prior loop, but when there isn't that dependency then doing things in parallel is encouraged and should be easy

For The Record, I don't actually dislike functional notation. It has its place, but is being over-used IMO. Pure dataflow notation is the underdog. I favour underdogs and tend to emphasize underdogs. Apparently, there was a PDP-8 manual that gave equal footing to CALL/RETURN and to co-routining. That's the way I see "functions" - there is something else that is on equal footing with functional CALL/RETURN. Modern ABIs insist that compilers must insert opcodes as scaffolding for functions. Forth and Prolog and ??? don't insist on such requirements. ... I approach the "template" and dataflow issues from a different perspective. In my world, arrows are one-way tubes through which data packets flow,,, and, arrows must not cross boundaries of boxes (going only from port to port),,, and arrows don't just represent ephemeral "dependencies" but actual flow paths. Given that mindset, "templates" are just natural outcomes (assuming that I understand what you mean). Boxes can contain boxes all the way down until they hit bottom. Forth does this (threaded words can contain threaded words and/or native words), /bin/bash does this (scripts can invoke other scripts or compiled commands), etc. Functions, though, are just box wannabes. Functions don't satisfy the principles of complete isolation, hence, we need to buttress them with things called "operating systems". The principle that arrows must not cross box boundaries leads to the conclusion that arrows cannot be represented by simple pairs (\{sender,receiver}, but must be represented by triples {direction,sender,receiver} where direction=[down|across|up|through]. To me, the "elephant in the room" is the ability to bundle operations (which I think you call "templates"). Forth does it. Shell does it. PBP/0D does it. Functions promise it, but don't succeed (because functions cause blocking control flow behaviour). The idea of 'shared variables' and 'views' is just a special case. Such 'variables' are just units of software that react to incoming change messages and they emit messages when they change. What's inside doesn't actually matter - maybe it's just "state" or maybe it's something more complicated, as long as it reacts to incoming changes and emits appropriately. [I go on to argue that you can go way beyond such simple 1-in:1-out concepts. Fan-in and fan-out give you the ability to do things that are inconvenient to express in text-only languages. Does the provided blog already contain fan-out?].

Does the provided blog already contain fan-out

yeah, the chart's "startDay" and "endDay" define the x-axis interval that is shared across all dashboard widget instances. Maybe slightly confusing is I have drawn the arrows in the opposite way around to the direction of data flow. It is missing a symmetrical fan-in though

when the subsequent loop requires the output of the prior loop

Can the sequential dependancies be put inside the template? I think it should be possible to instantiate templates on demand, so you could recursively add the next section of track as one finishes maybe? Not sure how to garbage collect the prior template though. Resource management is now an issue with dynamically instantiated dataflow, which was not even a consideration before with a single static dataflow.

Nice! Next step: a refactoring tool for working dataflow templates. Or "parametrizable dataflow" as a new abstraction, where the parameters are the variables to be cloned.

@Tom Larkworthy > Can the sequential dependancies be put inside the template? Probably for most things, unless you want it to pass through some thing that you want to keep decoupled (like some HTTP request or something) > I think it should be possible to instantiate templates on demand, so you could recursively add the next section of track as one finishes maybe I think this is probably useful too. To me the idea of templating is especially useful when you have excess memory or compute available but granted there are some constraints so in my mind its handled based on some user defined rules or* heuristics. You could spin up x instances and load balance or you could dynamically build up and tear down if hardware resources are limited. My thinking is you set useful defaults and allow the user to adjust behavior as desired. In artifact world i anticipate* spinning up and tearing down is basically metaprogramming (graph self modifying-i.e. nodes have entity CRUD capability)

https://github.com/nteract/papermill does parametrization (and distributed execution) on whole-notebook basis. Netflix had blog, posts and some lectures on it; dunno if they still use it internally.

Running parallel dataflow is not useful just for exploiting hardware. If a task is doing async IO (e.g. a http request) you get parallel IO which has value even on a single CPU in languages. I am using papermill for retraining machine learning models! the parameter is the S3 file location to load the (latest) data! Very useful technique for productionizing notebooks.

@Tom Larkworthy for sure I’m referring to the async io example just to illustrate there may be cases where you want sequential dependencies to pass through something that lives elsewhere in your codebase