Visualizing data structures (e.g. in PostgreSQL) When we think about the data structures our programs use or the execution flow of our programs, I’m pretty sure we all think visually and if we had to explain it to someone we would be drawing diagrams on a whiteboard. Yet pretty much all our coding and debugging is text-based. We are forced to visualize things in our minds. I’ve been debugging the PostgreSQL codebase to better understand how things work. I think a lot of people treat it like a scary magical black box, but it’s quite easy to debug a query’s execution path, and start to understand how it works. When we learn about databases, say from Andy Pavlo’s CMU database course, there are a bunch of core diagrams used to explain things. Like the parse tree, logical plan tree, physical plan tree, disk storage layout, btree/bitmap/hashmap index, etc. So what I was thinking was to instrument the PostgreSQL C codebase (via LLVM-IR), and then visualize some of the key data structures and stores. Imagine how easy it would be to teach database internals, if you could type in a query, and then visually step through the actual execution of the query with visual diagrams. I would imagine that any PowerPoint slides and diagrams could be replaced by this visual interface with a slider and some filtering options for what to show.

Vaughan, I'd like to personally offer my services to help with this! I'm writing an AR/VR application PoC for macOS/iOS that reads, analyzes, renders, traces, and plays back an executed path through code. It sounds very much like the raw set of tools that might help to put this together. Is this something you might find interesting?

Showing traces and visual structures is great for understanding 🙂

The hard part seems to be knowing the right level of detail to show. This is the bit that seems to need human-level intelligence. A lot of visualization tools produce unintelligible spaghetti when run against real data.

@Jack Rusher @Chris Knott I’m happy to say I’ve finally made the repo public and am excited to share with anyone willing to lend an eye or ear or hand or other useful appendage. https://github.com/tikimcfee/LookAtThat

I am hoping to achieve something similar with a Firebase server implementation (https://observablehq.com/@tomlarkworthy/firebase-server-prototype-1) It leaves its execution trace in the notebook, and you can interact with the request processing live using Chrome dev tools in the browser, but unfortunately at the moment you have to be the owner of the notebook to have authority to attach the live coding hooks. There is an embedded video of it working though. The bit I am working on at the moment is explaining causal consistency & at-least-once semantics. But I hit a fairly hard wall there as creating bespoke diagrams for each concept is quite a lot of work and each is quite a difficult concept. Consistency is super important for a DB but it does not so much manifest in data-structures but why components are connected in the order they are. Checking for causal consistency is kinda hard but it would be a visualization of each participants history on some load testing data. Anyway, my broader point is that I am super interested in collaboration with visualizing DB concepts. Very interested in what topics you will focus on.