Hey y'all! The past few months I've been working o...
# share-your-work
e
Hey y'all! The past few months I've been working on a system I call Notebooks-On-Demand (repo linked) — its a python library + JupyterLab extension (I don't have a video yet, sorry! but the readme has good screenshots). The short description is "think of putting a notebook interface inside a breakpoint" — it lets you call a
notebook()
function anywhere in a regular python program, which will pause the program execution, create an embedded IPython kernel on your current stack frame, and connect a Jupyter Notebook interface to it. To avoid a wall of text I'll make this a thread:
There was a ton of little problems involved with getting all this to work smoothly, but now that it does work (besides some bugs I'm sure), I'm most curious about what programming is like with this kind of tool, so I'll say a bit about why I think this system is interesting. Personally, I think one big thing that separates pleasant and joyous notebook experiences from depressing and confusing ones is scale. When notebooks stay small (<15-20 cells usually) they can be lovely environments for writing and inspecting programs. When notebooks get big, they get hard to navigate, hard to reorganize, and hard to build on. Lots of new notebook interfaces have tried to redesign notebooks in order to make them better at these last three. The idea of this system came about when I realized we already have an interface that does these latter things pretty well: regular IDEs and text files! Rather than try to make notebooks more like text files (e.g. by fixing the order of the cell execution, having variables be by default scoped to each cell like psuedo-functions, etc), I wanted to explore whether just making it as easy as possible to transition between a notebook and a text file would address many of the problems notebook users have. If they could keep their programs in python files, modules, functions, and classes, but be able to jump into a notebook as the need arose without any overhead, maybe notebooks would never need to get big enough to run into the problems that we see users have. Beyond that, building this system also showed me that there are some interesting things notebooks can do once they can just be ephemeral interfaces on some parts of a program. Because the program's stack frames are the data your notebook-on-demand interface is built on, rather than the organization of your program text in a file, we support navigating your code by traversing up and down your stack, filling in the right state to your notebook kernel. We can also let users save values while the program executes to inject later (to test out arguments from multiple invocations of a function interactively, for example, without having to re-run the whole program). I think both of these would be horribly confusing in a regular notebook, but because the user's program can live safely in a text file, the notebook-on-demand interface can (I want to argue) focus on being a good inspection/composition tool for small bits of a program, which opens up some interesting possibilities I think! Would love any thoughts/feedback/reactions! I'm also running a study on how folks program with this tool for my PhD (which mostly involves just using it for a few weeks and then nerding out about programming interfaces with me for an hour or two, with a $60-$100 gift card!). If that sounds interesting or you know a python programmer who might find this useful, I would really appreciate the help! All the info is in the readme. Thanks ❤️
❤️ 4
j
Hi Eric! Sounds interesting. I’m definitely a fan of bringing in notebooks as a piece of larger development workflows, which I think is what you’re doing. But from my read of your description, I don’t quite understand the role NOD plays. Is this a tool for debugging, analogous to the way some stepping debuggers let you evaluate expressions in an active scope when paused on a line? Or are there other workflows it’s supposed to support? Is the notebook persisted anywhere?
☝️ 1
t
very cool. I am also into this. My notebook system can self serialize so you can "copy as Javascript" and unpack it into 3rd party websites where you can do some more coding and data gathering and then self-serialize it back out again. I think bidirectional self-serialization is an under-explored superpower.
❤️ 2
k
This looks like a reinvention of the Smalltalk debugger for Python. Which is great! For those not familiar with Smalltalk, it's important to understand that the word "debugger" is a misnomer. It's a much more general tool for inspecting system behavior.
g
aside: It has become my opinion that "debugger" in the days of REPLs (lisp, smalltalk, forth, etc) meant something more than just "debugger of code". REPLs were used to debug ideas (I've been using words like Software Architecture and Design). My perspective comes from the bias of thinking that "code" is just a building material like, say a piece of wood. You can build things with wood or you can merely concentrate on how to use (hand saw vs. power saw) and treat the wood. We've sloughed off the concept of designing architectures by developing tools like Continuous Deployment (the CD part of CI/CD) and waiting for paying customers to (very slowly) debug and give us feedback about our concepts.
👍 2
e
@Joshua Horowitz Hey Josh! Apologies — I think I'm still figuring out how to communicate what the system actually does. Personally, I think of Nod as more of a composition tool than a debugging tool (or a debugging tool if we use @Konrad Hinsen and @guitarvydas’s more expansive conceptions of "debugger"). Specifically, the simple loop is this: 1. I have a big python program, spread over many functions, classes, and modules. 2. Inside a function, I insert
notebook()
and run my python program. A notebook interface is generated for me from that function body, and the current stackframe is put into my notebook kernel state. 3. I make changes to my function, having the function arguments (and whatever else) available to me in my interactive session. 4. I send my changes back to the original source files with a button at the top of the Nod pane, where my cells are converted back to plaintext, and I see the changes appear in my text editor. The way I have started to use it is that I'll plan out a dummy function and call it with arguments, then in the body of that empty function I write
notebook()
and run my program, then write the whole function in the notebook editor. If I need a range of values, I can write a few lines which will call
nodLog(functionArgs)
up until the nth call of the function, then will call
notebook()
, and the arguments from previous function calls will be available to me in a list in Jupyterlab to test between (this is very helpful for working with underspecified JSON APIs, for example).
@Konrad Hinsen Yes! I think the basic idea is very reminiscent of how smalltalk does it. Most of the implementation challenge was figuring out how to make this work smoothly within Jupyter's architecture, but it wasn't a mystery that giving python users a way to insert an interactive environment in the middle of their program would probably have some good uses. Personally, I'm most interested in thinking about how python programs change once we also know that we can open that 'same' program as a notebook. Using nod for my own work, I find myself making little bespoke debugging and visualization environments for individual functions, since markdown cells are converted to comments and cell breaks are converted to new lines. I can leave in code which generates a rich output (like a data table), but just comment it out when I send it back to my source file (I'm thinking of more elegant ways to do this, but for now commenting isn't too painful). So if Nod is doing anything new (and maybe it isn't!), it might be the combination of a smalltalk-style debugging/composition tool with a notebook interface, and thinking about functions as little micro-notebooks.
❤️ 2
@Tom Larkworthy oh cool! Yes, totally agree — I think theres a real "whole is greater than the sum of the parts" effect here — being able to move flexibly between environments gives you a way of interacting with and thinking about your program that feels different than either environment alone.