Title
#linking-together
Mariano Guerra

Mariano Guerra

10/05/2022, 1:56 PM
"Programmable Ink" by Szymon Kaliski (Strange Loop 2022) Sketching with pen & paper is a powerful way to think through ideas, but ink on paper is static. Dynamic behavior, on the other hand, requires programming -- which lacks the immediacy of the canvas: it's indirect, abstract, symbolic, and full of bureaucracy. What if we could harness the power of modern tablet computers to combine the intimacy of ink with the power of computation? Note: @Ivan Reese @szymon_k and @wolkenmachine were involved in this project and a new iteration has started with @Patrick Dubroy

https://www.youtube.com/watch?v=ifYuvgXZ108

s

szymon_k

10/05/2022, 2:46 PM
@Joshua Horowitz was also involved, and a ton of other people who are not (afaik) in this Slack; thanks everyone! 🙂
i

Ivan Reese

10/05/2022, 3:16 PM
Congrats @szymon_k! I'm excited to watch this when I get home.
j

Jason Morris

10/05/2022, 6:18 PM
Holy shit that is cool, and I'm only at the InkBase part. 🙂
6:50 PM
Yeah, untangle overlaps considerably with the constrained, multiple-world tools I'm using, but with a much more human visual interface. What tool are you using under the hood to do the model finding and indicate if the constraints hold?
6:57 PM
I love that if they don't hold, it tells you which don't, instead of telling you nothing.
Konrad Hinsen

Konrad Hinsen

10/06/2022, 5:29 AM
Wow. That's how I want to work with a computer!
5:31 AM
Wondering.. the presentation of those three tools suggests that some common underlying technology is a commodity not worth discussing any more: the recognition of hand-drawn objects and their subsequent manipulation by code. Is that true? Are there well-understood algorithms for this? Maybe even ready-made implementations?
s

szymon_k

10/06/2022, 5:43 AM
What tool are you using under the hood to do the model finding and indicate if the constraints hold?
this is built using a very small part of z3; if we hit
unsat
we permute the set of constraints progressively removing more and more of them until we hit some results -- is that what you were asking?
the presentation of those three tools suggests that some common underlying technology is a commodity not worth discussing any more: the recognition of hand-drawn objects and their subsequent manipulation by code. Is that true? Are there well-understood algorithms for this? Maybe even ready-made implementations?
to the contrary; there's pretty much no "recognition" happening in any of the demos (the very small exception is the delta symbol and star symbol recognition in the "sketchy charts" part) -- everything else happens by spatial relations from known objects -- this is definitely a big technical challenge, especially since "AI" doesn't vibe well with what we're after (it's an unpredictable black-box)
Konrad Hinsen

Konrad Hinsen

10/06/2022, 6:17 AM
I was thinking of recognition at a lower level of abstraction. Such as recognizing a box or a tick mark as an object distinct from the other pen strokes on the page, and then moving it around.
6:20 AM
Unrelated thought: these tools explore two roughly orthogonal aspects of computing: visual vs. symbolic, and formal/rigid vs. informal/fuzzy. I love the expression "tyranny of formalisms". I'll use it next time I talk to a static type system missionary 😉
s

szymon_k

10/06/2022, 6:45 AM
Such as recognizing a box or a tick mark as an object distinct from the other pen strokes on the page, and then moving it around.
yes, this is what I was trying to answer; we don't do that, and I don't think this is really possible to be 100% perfect -- for example a box can be drawn in 1,2,3,4 strokes (or more if you're sloppy), and then it's easy to make one that's slightly rotated, or skewed, or.... - I don't believe there's a heuristic that can cover all of that, and work out all the time -> so one direction from here is to do the best we can and accept that there can be errors/miscategorizations (which I think is not great -- because then you have to adjust how you draw things to make the computer understand you, and you're one step closer to just filling forms again), and another direction (that I'm more interested in) is to figure out different systemic approach to the problem so 'recognition' is not needed
Konrad Hinsen

Konrad Hinsen

10/06/2022, 6:45 AM
Notes on this talk in my digital garden (on computer-aided research): https://science-in-the-digital-era.khinsen.net/#Programmable%20ink
s

Scott Antipa

10/06/2022, 3:55 PM
Awesome @szymon_k
j

Jason Morris

10/06/2022, 4:01 PM
Yeah, z3, plus permutations on constraints was the answer I was looking for, thanks!
w

wtaysom

10/14/2022, 5:29 AM
Untangle has a ton of potential. There's something subtly delightful in its circle-and-arrow approach to describing constraints. I'm keen on making an explorer for solution spaces since the prototype really doesn't go there beyond some care in the way solutions are ordered.
Chris Knott

Chris Knott

10/14/2022, 9:37 AM
My favourite part is the generated list of possible queries which match your selection. It is quite similar to Wrangler/Trifacta http://vis.stanford.edu/wrangler/ It pushes a lot of the difficulty of programming onto the computer, but at the same time it doesn't introduce any black box/ML/woo that will inevitably go wrong.