🕐🎢 If you were going to create a "timeline" view that showed the progress of execution of a program through time, what would you make it look like?

I don't use a visual language, but I emit a log as the program executes, and each line (or group of lines) in the log maps back to a specific line of code in the source (file+line number). If this approach is palatable to you, it should work for a visual system as well. You just need to replace file+line with a co-ordinate system both sides can agree on.

In an effort to ground trace/log manipulation spatially, I use a long ribbon as the sequence of events, which is folded in various ways for emphasis. My hope is for the grounding in geometric transformations to help keep a person keep oriented. For example, instead of picturing a filter as pulling matching items out of the list, we bring them forward pushing the non-matching items back. From one point of view, it's the normal, filtered list. From another, literally from the side, you can see the distribution of matches. Are they all in the middle? Periodic? Erratic? A call/return tree is an example of a different kind of fold.

That reminds me of one issue I've wrestled with without a satisfactory resolution: whether to highlight logs that are salient to the domain or just at the top of the call graph. My logs use a zooming interface that renders lines above a certain depth. Currently the depth is just the stack depth. But I've found myself wishing for another point of view, as you put it.

I imagine a timeline for a visual language like a comic strip. Each panel shows a static picture of a specific region and the strip shows a sequence of events of interest. These could either follow a piece of data (follow a person as they go from location to location), or show a single site and data passing through (pin to a location). If we do time strictly on one axis (no wrapping) the other axis could be used to add different views of the same timeline. Some kind of zomability seems possible too - a panel expands into a strip, etc.

That's what I'm getting at. Start with a strip of events. Embed in a higher space (say time on the horizontal and place on the vertical) so you'll see the panels zigzagging up and down (unless two things happen at the same time). Then focusing on a place means examining the panels left and right all on the same level. I'm still working on the higher dimensional cases person, place, time. One view that's really nice for imperative programming is to line up all the steps that modify a variable along a horizontal with the immediate context for each modification shown above and below the assignment.

I guess it depends on the actual visual semantic of the language and on what patterns you want to catch. So maybe the solution is not in a timeline view but several ones.

Another timeline I've longed for is a UML like sequence diagram where I specify the sites and the system just shows me the sequence of events between those sites.

i can’t find them at the moment but the leisure programming environment has a bunch of ideas—if no one has them at hand i can look harder

@shalabh I’ve worked on this, but got distracted. Most of what we use at work to communicate about communication is sequence diagrams, so I think it would be great to be able to create them automatically from built systems.

@alltom Interesting! What kind of tech stack did this work with?

Corp internal. :( I was using static analysis for my prototype so that you could construct the diagram primarily by clicking function names.

It was really fun to read everyone's thoughts in this problem space — thanks for weighing in! I'm not done designing my solution, but here's my current sense of my problem. I want to make something that feels similar to the timeline view/control in a video editing or 3d animation program, to leverage people's experience with that kind of interface. I can spare space in my GUI about a hundred pixels tall and as wide as the window. I'd like to show something inside this view that serves as a visual landmark based on what happened in the execution at each human-scale moment in time. Behind the scenes, my environment logs an event whenever the state of an entity is changed. I already use that for time travel, and it feels natural to use it for a timeline too. The open questions are: what should be drawn in the timeline? How do those drawings relate to the code and its execution? Given that context, here are some of my thoughts on your thoughts. @wtaysom said

Instead of picturing a filter as pulling matching items out of the list, we bring them forward pushing the non-matching items back. From one point of view, it's the normal, filtered list. From another, literally from the side, you can see the distribution of matches. Are they all in the middle? Periodic? Erratic?

[...]

One view that's really nice for imperative programming is to line up all the steps that modify a variable along a horizontal with the immediate context for each modification shown above and below the assignment.

That sounds very similar to how I'm planning to handle a few common features such as find & replace, showing all call sites of a function, and even viewing function definitions via their call sites. I'd use multiple cameras to render different parts of the 2d code plane, stacking the views from those cameras at different depth layers vaguely like a "card" interface, to make it quick to flip between different locations in the code. This is similar to what you get with conventional windows/tabs, but by putting things together in the same space you can draw dynamic connections between them and play with the in-between depth space. It's neat to think about leveraging depth for a timeline view, though it butts up against two of my constraints — the "human-scale" sense of time will sometimes mean showing thousands of events from many different objects all at once; and the default timeline GUI is something like 2000x100 pixels big. Not sure what ideas I could adapt from William that'd fit within these constraints, but feel free to make suggestions. @shalabh and @ogadaki offered a bunch of excellent ideas for visualizing things, many of which I already plan to do in the main editor view — but they are visualizations that integrate or aggregate across time rather than differentiate to distinguish the moment-to-moment changes. @Garth Goldwater said

the leisure programming environment has a bunch of ideas—if no one has them at hand i can look harder

I've never heard of this, and I'd love a link if you can find it, Garth. (You should throw it in #C5T9GPWFL in case anyone else is interested.) Thanks again for the thoughts. More are welcome. If I've done a bad job of articulating my problem and you are interested in understanding it better, please let me know and I can post some drawings or something.

I like a lot the idea of a deck of views from different cameras on different parts of your 2D code. The cameras views bounding box are chosen by the user? Or some views are selected automatically based on activity? Maybe a combination of the two?

@Ivan Reese chaining filters is essential when dealing with lots of events. Take out one source of noise after another until you find the signal. And, though it really hasn't arrived yet, not for reading text, spacial computing gives you so much more space and intuitive movement that it's silly.

leisure seems to mostly live in small videos: https://m.youtube.com/watch?v=LQB8XHkvIJg

@Ivan Reese interesting - looks like you want a scrubber bar at the bottom? One question is should this be customized? Finding an automatically useful one when there are many variables seems hard. Maybe a horizontal strip chart where each dot represents a state changed or some processing happend (but no values)? The different horizontal lines (Y axis) would represent different variables or nodes. But if everything is changing all the time, this will just be a homogeneous mess so maybe you can pick what variables go on the Y axis.

Thanks for the links @Garth Goldwater !

@alltom yes kythe is open source but there's no sequence diagram afaik? or is that unrelated?

@shalabh That’s the static analysis library I was using that I thought was internal-only. So maybe I could be working on it outside the walled garden. Looks pretty tied to bazel, though, so maybe not very useful externally.

This is a much nicer version of what I've implemented so far: https://twitter.com/mmalex/status/1159526969790349317