Hi everyone! I'm Josh Sunshine, an Assistant Professor at Carnegie Mellon University. I lead the Penrose team. Penrose is a platform for conceptual diagramming. In Penrose, you declare what you want to be true of your diagram and Penrose automatically lays out your diagram to meet your goals. Under the hood, we use some interesting differentiable geometry, numeric optimization, and compiler tricks to make it work. I'm broadly interested in research at the intersection of human-computer interaction (HCI) and programming languages (PL). That intersection seems closely connected to this group. I also run an annual research workshop called PLATEAU that focuses on PL+HCI which may interest some of you.

Welcome! Penrose is a fantastic project — long something I've had my eye on. Glad to have you here.

😊

Just curious, because it is a question I get a lot in my area, and I would like to hear what you usually say: doesn't AI upend your whole research topic? I.e. can't you "just refine" an existing foundational model to create the diagrams, and then use your symbolic methods to ensure it actually fulfills the required conditions? (I am not trying to be mean)

@Denny Vrandečić I certainly get asked about LLMs all the time and we've used and finetuned them in our work. However, I don't understand your specific question. What do you mean by "create the diagrams"? What is the input to the model? Where do the "required conditions" come from in your scenario?

With "required conditions" I meant the code in your platform, that ensures that the diagrams do what they are supposed to do -- but then use an LLM to generate the diagrams (in whatever format it is that you have diagrams). Does this make a bit more sense?

Unfortunately I'm still not following. A users needs to specify somehow what the diagram should contain. In Penrose, the user writes some domain specific code. That code is declarative and tells the system what they need/want of the diagram. Perhaps that is what you mean by "required conditions." We then compile the domain-specific code to a numeric optimization problem. Penrose then optimizes the energy function. Given all that background, what are you suggesting the LLM should do? Generate the specification? Generate the diagram when given a specification? Solve the optimization problem? Something else?

Ah, sorry, I misunderstood. With the background, I would say, a) generate the specification.

Sure, that is certainly something an LLM can sometimes do. In particular, we've found LLMs are actually much better at generating our domain-specific code that SVGs. This is particularly interesting given that our languages / libraries barely appear in their training data.

Uhh, that's interesting! We had a similar surpise when even GPT-2 was really good at creating SPARQL queries. Actually, better than I am in many cases...

Hi Josh, the "PL and HCI: better together" paper that you co-authored is a significant part of the causation-chain that led to me joining this community here today. Very inspiring!

@Thomas van Binsbergen thank you so much for sharing that with me! You made my day!