I’ve been playing around with hand tracking, inspired by Bret Victor’s Future of Interaction Design article from ages back, and it’s come together in this little toy I made, combining fluid simulation: Magic Hands 🙌 (blog post) Very much a starting point, but quite keen to see what else can be done with hand tracking in the browser - also keen to hear more examples of hand tracking work 🙂

That’s fun! I’m curious about how you were inspired by Bret Victor’s article. IIUC, that article is a rant against touchscreens as “Pictures Under Glass”, in favor of giving hands back tactility and physicality. By itself, your experiment is even less tactile & physical than a touchscreen, right? Are you planning on moving in a more tangible direction?

edit; works with Chrome on my MacBook ; seems to want my open palm and not my closed fist or back of my hand - very interesting ; and Firefox ; different on Safari

Thanks for checking again @guitarvydas - I’m still debugging the issue on safari. The open palm/back of hand detection is deliberate. I wanted to create the effect of a delay of nothing happening when you raise your hands, but then showing the visuals when your hand opens - like 🙌. It means you can control when you want the visuals to show, even while your hands are on screen. This may change in future, but still figuring out what I want to do next with this.

datapoint: on iPhone8, swipe works but camera doesn’t (Safari and Chrome)

@Joshua Horowitz sorry, missed your original msg in my notifications, but you are right, that the demo doesn’t make it more tactile, but I’d argue that it is more physical, as you are required to move more of your body. I also like the augmented-ness of it, where you can see yourself and the glowing fluids overlayed, as if it’s more of an embodied experience vs it being just your finger on touch devices (or more disconnected, as a mouse experience). That’s where it feels more theatrical to me. Tangible is def an area I’m very interested in, and I want to explore some gestures that might use some natural feedback mechanics eg, if you tap your finger on your thumb you get tactile feedback for free. I rmb seeing Google Soli doing it ages ago, but it doesn’t seem like they followed through with it - dunno why. Part of my point with Magic Hands is to stimulate more ideas for getting away from standard keyboard, mouse and touch interactions, as Bret was trying to encourage in his article. Definitely a lot more work to be done there, but it’s a good time to play with the tech 🙂