Does anyone know the current state of the art in functional reactive animation? I‘ve been reading about Conal Elliott’s Fran DSL but it seems there hasn’t been any recent progress in this space. I’m considering writing a JavaScript implementation of that hasn’t been done already http://conal.net/fran/

What would you constitute as progress?

My project, Curv, is a pure functional graphics language. Most of the effort right now is being spent on providing a powerful set of composable graphics primitives. The pure functional animation and reactive parts are very underdeveloped, but matching what is shown in the Fran tutorial is one of my goals.

The goal of Fran as I understand it was to make animation more accessible to beginner programmers. I see this concept eventually replacing Processing for generative art and animation. Progress is anything that brings us closer to mass adoption of FRP in the computer art world. Fran may have the perfect API for that already (I haven’t actually used it to create anything) but the project is currently deprecated with no clear successors

I'm working on this. Or rather, I'm working on an animation-programming hybrid tool that takes some inspiration from Conal's FRP, Fran, etc. I'd be excited to talk to you about FRP, your project, my project, anything! As an aside: I ran into a wall when it came to representing numbers in the way Conal advocates — he wants everything to be exactly precise real numbers, rather than approximations using rational numbers. But of course you can't store infinite-precision real numbers in a computer. I had a number of intense twitter thread conversations with him (and others) and arrived at the conclusion that there's no practical way to commit to real numbers once your animations pass a certain threshold of complexity. For example, there's probably no way to make something like a modern realtime 3d game where all your object positions are stored as arbitrary-precision real numbers AND you have complex collisions, since your number representations will likely explode in size as they accrete terms from all their interactions. So instead, in my system, I'm actually using only integers, since that puts me in control over when and how to throw away precision. /shrug/

@jonas Did you run into any major roadblocks implementing this?

Lots of things. • GPU programming is terrible. • I wrote a multi-stage compiler that performs partial evaluation and optimization. It is difficult to write, and I'm still working on it. • I use reference counting to manage memory. I don't know any online resources that explain how to create recursive function closures without cycles, but I figured it out. • I use signed distance fields and implicit functions to represent geometric shapes. That's a niche technology, and I'm doing original research and inventing new techniques to overcome some of the limitations I have run into. • Conal's design for the Reactive part of FRP is looks complicated to implement. I'm not working on that bit right now, but I expect to encounter some challenges when I do.

Conditional looping typically isn’t available in shaders for example

My oldest test machine has a 2010 Nvidia GPU: the GPU driver will accept

for

loops with a compile time bound on the number of iterations, but it will not accept

while

loops, so a subset of the Curv language doesn't work on that machine. Modern GPUs have no problem with looping, and loops are required for general 3D fractals and noise based procedural modelling. My eventual plan is to drop support for GPU hardware that predates 2012, and migrate to a modern, cross-platform GPU API (probably WebGPU).

@Doug Moen

I use signed distance fields and implicit functions to represent geometric shapes. That's a niche technology [...]

Is it niche? I'm pretty sure both implicits and SDF are pretty common in games. If I recall correctly, it's common to use implicit surfaces for physics and SDF for some lighting effects, for instance. Am I missing something?

@Ivan Reese Conal wants everything to be exactly precise real numbers, rather than approximations using rational numbers.

I could do cool things with that technology if it was practical, but obviously it isn't. Besides being too slow to actually use, you also can't compare exact real numbers (computational reals) for equality--that's the same as solving the halting problem. On a GPU, 32 bit floating point arithmetic is never slower than 32 bit int arithmetic, and is typically faster. So on the GPU, I just use 32 bit floats for everything, unless I actually need an int. I honestly didn't put a lot of thought into that, it just seemed like the only good choice. Sounds like you are doing something more clever than that.

I'm not doing any logic on the GPU, just rendering, so representations there don't matter as much. The logic is all done in JS, so I'm forced to use floats, but I stick to whole numbers (what I would call "shitty ints"). I choose the granularity of time/space representation to be fine enough that, in practice, you don't care. I don't care about average case perf (JS, what can you do) so much as I care about perf cliffs (which arbitrary precision representations seem to dance along the edge of)

@Ivan Reese I'm not a game programmer, so I may be missing something. I know the SDF technique is used for some particular techniques in games, but I've only heard of a few games that use SDFs to model all of their geometry. I am also not aware of any professional 3D modelling tools that output SDFs. Maya, Z-Brush, etc, all output triangle meshes, as far as I am aware.

Yeah, that's also my understanding. Games that do SDF will generate the SDF representation from the triangle meshes. They then use the SDF representation for things like cone tracing, to do raytrace-esq reflections. I've also seen some games do the poly mesh -> SDF conversion so that they can do more interesting deformations of the geometry as part of gameplay.

I am in the market for a good open source library for triangle mesh -> SDF conversion. Any recommendations?

No clue, sorry. I know from watching their twitch stream that the indie dev XRA does this in their game (http://www.brokendimension.com), but I believe they wrote their own code to do it.

A lot of demoscene stuff is all SDFs right? I’m not sure whether that counts as niche or not but I know there’s a lot of crossover into the games industry. I’ve only seen them used for volumetric effects like clouds in games but I’m not an expert here

@jonas John Hart made the important breakthrough in the late 1990's when he invented Sphere Tracing. The demoscene did a lot of the important subsequent research on SDF technology. Almost everything on shadertoy.com uses SDFs. The other important community is Fractal Forums (general 3D fractals like the Mandelbulb).

The Apple Core animation library is pretty good, and offers much of what Conal did in 1998. It uses the GPU behind the scenes to do the animations, and if orbiting things around and changing them as they fly is your objective, Core Animation is a very clean system. Apple merged it into OSX/IOS so that one could avoid directly programming the GPU. It is one of Apple's notable advantages over Windows OS, along with the Sound APIs which are also baked into the kernel for real-time audio work. No other OS to my knowledge has audio stuff built in (probably invented by Amiga though).

I haven't programmed against CoreAnimation yet. What is the API like? Is it focused on function composition? If not, then what about it makes it comparable to FRP? How about something like Greensock or the Web Animation API?

Core Animation is a low level imperative API for programming the kinds of animations that appear in MacOS and iOS GUIs. Apple GUIs uses the Model/View/Controller paradigm. Every window has one or more Views, and every view has one or more Layers, which are Core Animation objects. A Layer has a draw() function supplied by the application that draws the contents using Core Graphics. The drawn contents are cached in a bitmap in the GPU. Looks like you can composite layers together using transparency, apply affine transformations and perform simple animations, and CA orchestrates this on the GPU. It's not FRP because it's not Functional and it isn't Reactive (no input handling). I'd say that GreenSock is higher level.

@Doug Moen gave a nice sketch of core animation, however he is vastly understating its power. It is what is powering all the built-in controls of IOS; the buttery smooth kinetic scrolling is all animated using core animation, and the fact that the frame rate can stay so high even under heavy CPU load is a very clever feat of the OS Kernel, which has more of a real-time flavor than anyone gives it credit for. Android took many years to catch up to the buttery smooth animation tricks that Apple used. Of course nowadays phones are 10x what they were so these clever tricks aren't so necessary because of the horsepower. I don't know what you mean by functional. It is call scheduled via an animation queue which you can control slightly, and you can respond to inputs. Core Animation is a 2.5 dimensional system; you can put a rectangle into a perspective transformation, but it isn't a 3D world. For that people use Unreal Engine, Unity, etc.

I’m also a little surprised that Core Animation popped up in a thread about FRP. It’s a very pragmatic imperative framework using some functional techniques. It’s great trick is that you specify start and end states of an animation (which the UI framework could basically do for you implicitly, and you got default animations without specifying anything), commit it to the render server, and all the interpolation per frame is then done on the GPU. That cleared up the CPU to take care of other things, mostly shoving memory around to manage textures and create the raster image data for these from the UI framework. Apart from much slower chips, memory bandwidth was one of the biggest issues of the early iPhones. Core Animation worked around that very effectively. (A design that fell out of constraints of the time — in case you’re following other threads about creativity. ;-) Sharing the work between CPU and GPU made the first iPhone possible, or rather its smooth kinetic scrolling capabilities which separated generating the textures from moving and scaling or otherwise transforming them. If you ever used an early iPhone, you might remember the checkerboard pattern that showed up whenever you were scrolling into areas that hadn’t texture memory loaded yet — still it always felt responsive because you could see the inertia of the scrolling thanks to the checkerboard. That was a design decision realizing that it’s more important to have the animation be absolutely smooth than to see the actual content. Its greatest benefit is also its greatest weakness: once you commit an animation it just runs and you have little control over it. For animations that might be exactly what you want, but as Apple moved on to interruptible animation and gestures and now that chips are much more capable, Core Animation is likely on the way out to be replaced with something much more resembling FRP, although I wouldn’t count on Apple to deliver a pure functional approach. You can get a glimpse of what it will look like with SwiftUI, which includes — apart from classic UI components still built on top the classic frameworks (but for how long?) — a simple graphics API for shapes and animations that is likely to eventually succeed Core Graphics (Quartz) and Core Animation and will be a more efficient implementation directly leveraging Metal. That might be what you may want to look at for inspiration: https://developer.apple.com/tutorials/swiftui/drawing-paths-and-shapes https://developer.apple.com/tutorials/swiftui/animating-views-and-transitions

My vision for state-of-the-art functional reactive animation (FRAN): • Provide a rich set of 2D and 3D operators for constructing graphical values (shapes and colour patterns). For example, instead of just the endlessly recycled 2D vector graphics operators of Postscript/SVG, support the full power of resolution-independent implicit function representation, where anything that can be described using mathematics can be expressed: algebraic surfaces, set union intersection difference, fractals, non-affine transformations, noise based procedural modelling, and so on. • A pure functional program, which uses these primitives, is evaluated to yield a graphical value, which may be time varying and reactive. A graphical value is represented by tree of graphics operators (call this the CSG tree). • An optimizing graphics compiler walks the CSG tree, compiling it into code that runs partly on the CPU and partly on the GPU.

Not to pile on, but Edward also said..

I don't know what you mean by functional.

FRP is "functional" in the sense of Functional Programming — it's an approach to specifying animations using function composition, where each function describes how to transform some input graphic according to some input time. What the animation does is captured by the combination of functions and their return values as you vary the time input. It sounds like CA works a bit like Greensock or Web Animations and other OO animation tools, where you create some state object to represent the animation, specify properties and values on the object, and then submit that object to some engine. What the animation does is captured by the data in that object and what the engine does based on it. Then there are the procedural animation tools, like Processing or the HTML Canvas, where you write code that is executed every frame. What the animation does is a side effect of your code directly manipulating or generating the graphic.

In the case of Core Animation, there are various properties that you can specify for beginning and ending value, and it does the tricky job of interpolating. You can also chain animations, and run sequences of them in parallel, with triggers. It is a very complete and capable animation system, and far ahead of Windows at the time as MS was lagging in graphics except in DirectX ,which was basically a rogue team inside MS that wanted to make 3D games fast, and realized the only way was to bypass the normal Windows graphical kernel, so they built they own drivers for the hardware. Now we see Apple trying to beat MS's DirectX with their own 3D interface they call Metal. There are also capabilities if i recall correctly to have a callback function called to generate parameters. The most unusual aspect of Core Animation is the it has a 4x4 matrix, but you are pinned down in some of the parameters so you can only put a plane in perspective. They stopped short of 3D. I call it 2 and 1/2 dimensions. Anyway supporting some animation is a very important aspect of building modern interfaces, if you want to go past simple 2D dragging of say images onto a flat surface. I would imagine some of the efforts in the FoC group are using WebGL and just going full bore into 3D. 3D comes with a lot of baggage, and it sure isn't friendly for beginners. There was an add-on library for Adobe AIR called Starling, the work of one really great german programmer, and it gives you the GPU for 2D sprite sheets, and many other accelerated things. A really great compromise, and many popular Mobile games are using Starling, because it leverages the machine independent virtual GPU that underlies Adobe AIR. I know it is a fading technology, but getting great accelerate 2D rendering is a terrific thing, and it takes a huge amount of work to build.

Honestly animation via code feels really strange to me-- hard to write and harder to read. I worked on an animation prototyping tool for a design tool and it felt so much better to use than direct code. We leveraged Greensock, had start/end states constrained by the design (or current page state), and provided tooling to visually modify the transition (basically just leveraging interpolation over the merged tree during playback).

Most of the classic 3D animation tools, like Autocat 3D studio max, and the Maya product, had graphical interactive interfaces, which automatically generated commands in their command line language, which you could then tweak. Or use as inspiration for generating code by computers. Animation has always existed, even back in Macromind Director which spawned the whole CDROM revolution, in concert with the ability to tweak the animation. In hollywood they do motion capture, and then tweak it a little bit by hand to make it just right. The human eye and hand can get you very close, but there is inevitably some massaging, and generating sequences by computer of course is done all the time by the masters at Pixar, which use every technique possible to get things to look good. Classic Disney style cel animation is the "hard way", and it is pretty much extinct due to its cost. It took over 1200 people to draw Snow White.

Wow there's a lot of stuff to respond to here, but here's a summary of what I'm thinking. My hypothesis is that the barrier to entry for generative art / creative coding / reactive animations is very high because the medium is naturally difficult to map onto imperative code. FRP (or more specifically DCTP) was originally invented to solve this exact problem but hasn't taken off for many reasons: meta-level thinking required for higher order streams, lack of interfaces (text or otherwise) that allow one to manipulate streams in an easily-digestible way, and performance to name a few challenges. None of these seem insurmountable and I think that a tool like this has the potential to blur the line between programmer and artist like Processing intended nearly two decades ago

My main goals for Curv are: • Artistic expressiveness. You can make any kind of 2D or 3D art, using a wide variety of techniques. • Ease of use. This covers a lot of ground, including the IDE and the language itself. Mathematical purity is not a goal, but there are lots of areas in the design where mathematical purity makes important contributions to artistic expressiveness, ease of use, and ease of engineering. So I use mathematical purity as a tool, not as an end in itself: • Pure functional programming. This contributes greatly to ease of use, and simplifies the engineering (generating data-parallel rendering code). The semantics of the language are much simpler. A pure functional API for constructing shapes by composing functions is so much nicer than a 1990's style API where graphical primitives paint directly onto a canvas and you have to manage a set of global variables containing the current colour, the current line width, the current transformation matrix, etc. • Abstract algebra, what Haskell calls "equational reasoning". Unlike most programming languages, the equality operator

a==b

is an equivalence relation: `a==a`;

a==b

implies `b==a`;

a==b

and

b==c

implies

a==c

. Unlike most dynamic languages, the boolean operations obey all the axioms of a boolean algebra. For example, unlike in Python Javascript Lisp, the boolean and operator is commutative. This all falls under ease of use: simple language semantics, no surprises. However, there are some kinds of mathematical purity in Haskell culture that are detrimental to my goals. • Lazy function calls are a net loss; Curv uses strict argument evaluation in function calls. • Static typing is a big barrier to learning a language, and is incompatible with live programming, so Curv is dynamically typed. • I haven't built a Turing complete reactive API yet, but DCTP looks really hard to understand. The Elm language transitioned from "mathematically pure" FRP to "the Elm architecture", which is much easier to understand, and it is still "pure functional" and "reactive". So I'll probably take my lead from Elm, begin with the Elm architecture, and iterate from there.

@Doug Moen to defend Haskell culture — a bit of it and not to be confused with Haskell itself. To make types consistent with liveness, you need holes: this part isn't filled in yet or these two parts are at odds because I'm reconciling them. Now laziness strike me as nearly as important as purity: being able to ignore the timing of when a function gets evaluated. Then treat change over time as a first class entity rather than a side effect function evaluation order.

Hazel has holes, and it is dynamically typed for this reason. It's a terminology issue. "Dynamic typing" is not academically respectable, so they had to use different terminology in the paper. But it's dynamic typing because they store type information in the data and check the type information at runtime, which Haskell doesn't do. I make a distinction between Haskell's lazy evaluation of function calls, and lazy data structures. No other languages I know have copied Haskell's lazy function calls, due to ease of use issues for the programmer (unpredictable performance and memory consumption), and due to high cost and complexity of implementation. Even Idris, which is even purer than Haskell, with its dependent types and total functions, uses strict evaluation order. Lazy function calls would be disastrous for Curv, but other forms of laziness will be necessary. Curv is a pure functional language. All functions are pure (have no side effects). The language guarantees that you can't use side effects to determine the order of evaluation of function calls within an expression. So yes, I do want to treat change over time as a first class entity. Curv totally works this way right now. However, the Haskell virtual machine (the way it implements laziness) is totally incompatible with running code on a GPU, so I may need to find new ways to model change over time in a pure functional language as Curv becomes more powerful. I feel that I cannot simply follow in Haskell's footsteps. Haskell just is not designed for GPUs. Haskell's model of laziness is intimately tied to a single-threaded execution model, while GPUs are data-parallel : that is part of the problem.

I'll readily agree that in practice laziness has many drawbacks. As for GPU programming, I don't know how to accomplish more than a basic shader, so no idea how do something serious.

GPU programming is extremely difficult and weird. They typically have very limited instructions sets with all sorts of irregularities, and the fact that they often cannot access regular RAM but instead have their own very fast memory makes it the specialty of game programmers and ML number crunchers. Thankfully core counts on regular CPU's are coming up fast due to AMD finally giving Intel some competition again after languishing for so many years. Perhaps some day it will be just one processor type again.

@Edward de Jong / Beads Project every few years I do stick my nose in to check whether it still stinks. 🔥😱🔥