"Programmable Ink" by Szymon Kaliski (Strange Loop 2022) Sketching with pen & paper is a powerful way to think through ideas, but ink on paper is static. Dynamic behavior, on the other hand, requires programming -- which lacks the immediacy of the canvas: it's indirect, abstract, symbolic, and full of bureaucracy. What if we could harness the power of modern tablet computers to combine the intimacy of ink with the power of computation? Note: @Ivan Reese @szymon_k and @wolkenmachine were involved in this project and a new iteration has started with @Patrick Dubroy

@Joshua Horowitz was also involved, and a ton of other people who are not (afaik) in this Slack; thanks everyone! 🙂

Congrats @szymon_k! I'm excited to watch this when I get home.

Holy shit that is cool, and I'm only at the InkBase part. 🙂

Wow. That's how I want to work with a computer!

What tool are you using under the hood to do the model finding and indicate if the constraints hold?

this is built using a very small part of z3; if we hit

unsat

we permute the set of constraints progressively removing more and more of them until we hit some results -- is that what you were asking?

the presentation of those three tools suggests that some common underlying technology is a commodity not worth discussing any more: the recognition of hand-drawn objects and their subsequent manipulation by code. Is that true? Are there well-understood algorithms for this? Maybe even ready-made implementations?

to the contrary; there's pretty much no "recognition" happening in any of the demos (the very small exception is the delta symbol and star symbol recognition in the "sketchy charts" part) -- everything else happens by spatial relations from known objects -- this is definitely a big technical challenge, especially since "AI" doesn't vibe well with what we're after (it's an unpredictable black-box)

I was thinking of recognition at a lower level of abstraction. Such as recognizing a box or a tick mark as an object distinct from the other pen strokes on the page, and then moving it around.

Such as recognizing a box or a tick mark as an object distinct from the other pen strokes on the page, and then moving it around.

yes, this is what I was trying to answer; we don't do that, and I don't think this is really possible to be 100% perfect -- for example a box can be drawn in 1,2,3,4 strokes (or more if you're sloppy), and then it's easy to make one that's slightly rotated, or skewed, or.... - I don't believe there's a heuristic that can cover all of that, and work out all the time -> so one direction from here is to do the best we can and accept that there can be errors/miscategorizations (which I think is not great -- because then you have to adjust how you draw things to make the computer understand you, and you're one step closer to just filling forms again), and another direction (that I'm more interested in) is to figure out different systemic approach to the problem so 'recognition' is not needed

Notes on this talk in my digital garden (on computer-aided research): https://science-in-the-digital-era.khinsen.net/#Programmable%20ink

Awesome @szymon_k

Yeah, z3, plus permutations on constraints was the answer I was looking for, thanks!

Untangle has a ton of potential. There's something subtly delightful in its circle-and-arrow approach to describing constraints. I'm keen on making an explorer for solution spaces since the prototype really doesn't go there beyond some care in the way solutions are ordered.

My favourite part is the generated list of possible queries which match your selection. It is quite similar to Wrangler/Trifacta http://vis.stanford.edu/wrangler/ It pushes a lot of the difficulty of programming onto the computer, but at the same time it doesn't introduce any black box/ML/woo that will inevitably go wrong.