That said, I agree about the nice perspective of AI for glue. Or, more generally, for the "outer", most user-facing aspects of software. This echoes the structure of pre-computing work based on good old mathematics: plain-language reasoning with embedded formal systems.

Oh, one final advice to techies working on AI integration: wait for Open Source AIs. If you all jump on OpenAI's offerings, you will probably regret it when OpenAI tightens the screws (as it invariably will).

The direction that excites me is LLaMa/Alpaca + Langchain. But the direction that I'm fearing all of this will take (and that I currently use and even pay for, to be honest) is the corporate capture that OpenAI and Microsoft are currently executing.

Also, another problem I have: I was a Machine Learning Engineer in the past, at the height of the CNN hype shortly before transformers hit the scene. And when you're not one of the handful of people doing foundational work/research, I feel like ML engineering is super boring. It's basically - ironically - all glue code, all of the time.

But maybe the curve we're on really is so steep that there won't even be a real transition period where all programmers "have to become" ML engineers for a while, and we're going straight for whatever it is that follows? 😃

We’ve spent the last ~8 years trying to make something like this happen; it’ll be really wild to see how much of what we came up with becomes real.

That’s unlikely to be a change overnight or anything, but perhaps more pertinent to this group is that it destroys a lot of the nascent opportunities.

For example, the ChatGPT plugins release nuked a good portion of the current YC class a week and a half before demo day. Who knows what the implications of that will be.

So then I'm left wonder if human imagination will be a limiting factor — but then I spend a few minutes seeing what imaginative humans try out. And so I cannot find the ceiling at the moment.

Here, I'm discussing the case where ChatGPT "is" the programming language: what formalism will still be needed? And will that be graphical?

In https://www.geoffreylitt.com/2023/03/25/llm-end-user-programming.html#opening-up-the-programming-bottleneck our @Geoffrey Litt is talking about this issue. He says for example (I'm still reading the article so may find better quotes!):

There’s a lot of value to seeing the spreadsheet as an alternate view of the underlying data of the website, which we can directly look at and manipulate. Clicking around in a table and sorting by column headers feels good, and is faster than typing “sort by column X”. Having spreadsheet formulas that the user can directly see and edit gives them more control.

The basic point here is that user interfaces still matter. We can imagine specific, targeted roles for LLMs that help empower users to customize and build software, without carelessly throwing decades of interaction design out the window.

Just finished the article and I think it's hitting all the interesting points for me.

@Duncan Cragg And I assume your point about @Geoffrey Litt’s article is that old-school programming isn’t going away completely?

Could be that everyone else is (also) struggling to see /my/ points, which seems more likely! I do Think Differently, to my cost as well as benefit 😁

I think insight has to be something close to generalization: in many situations, find a common pattern or rule. It probably has to be predictive, if only probabilistically... It really is hard to tease apart from intuitive pattern matching. If you, a human, can predict what some weird system or situation will do, is it because you have insight or just because you're used to the pattern? Maybe it has to do with his explicit (communicable?) the mental model is.... It will not be easy to tell when/if AI systems cross the line into "insight", assuming they haven't already.

For those who’d like to dig deeper: This is perhaps good as an introduction to attention, which is deeply interwoven with insight (There is a reason why the foundational transformer paper was called “_Attention_ is all you need”): https://stefanlesser.substack.com/p/attention-and-insight In the post I mostly link to Vervaeke’s explanation in episode 9 of Awakening from the Meaning Crisis, which is about Insight: This is all still pretty basic stuff, but should already give you a good enough sense that you don’t need to run around scared that current LLMs might be sentient. But of course there’s more. Episode 32 of that series connects more recent research about self-organizing criticality as a dynamic component to insight (It’s probably necessary, for sure helpful, to watch episode 31 first). That is not the only component, but is interwoven with a structural component of small-world networks in a framework called relevance realization. That is all relatively recent theory, which means it’s a hypothesis that needs more validation, but if you follow the whole argument and it’s connections (which is a lot of work, so I don’t really expect anyone reading this actually spending the time and effort, but if you do, let me know!) it seems quite plausible and well-grounded in prior research. The details may not turn out to work exactly as described, but the overall connections made seem to already have good converging evidence. The gist is: the relatively static LLMs we use at the moment have some of the structural capability but are missing the dynamic capability for insight. It could be argued that some of that happens during the learning phase (and I don’t understand the AI research side deeply enough to know if that is actually the case, but probably not), but as long as we have separate learning phases and not fully integrated live learning (and the tricks we use for fine tuning models and using custom embeddings don’t do that) we don’t have to be worried that what cognitive science understands insight to be to occur in current LLMs. It is, however, not that far away, and both AI research and cognitive science are tightly interwoven to make progress on this. This is why I find what is outlined in this Relevance Realization paper quite fascinating: http://www.ipsi.utoronto.ca/sdis/Relevance-Published.pdf The good news is that as we get closer to creating consciousness, we also figure out how it works, and vice versa. The bad news is, by the time we fully understand what consciousness is and does, we will likely also have already created it.

Let me elaborate a tiny bit on what I mean by characteristics of consciousness. Three come immediately to mind, which I'll call (those others probably are trying to standardize names for these): • Sentience: the experience of things, sense experience. • Awareness: knowing what you're doing, being able to reflect on your thinking. Intentionality (having rich concepts in mind rather than just manipulating symbols) being the beginnings of awareness. • Selfhood: having the notion of being a thinking thing separate from others with some continuity over time. One surprise with LLMs is that I now see the possibility of awareness without sentience. Though now I begin to wonder whether vast combined descriptions of sense experience can end up amounting to something similar. Consider GTP-4 being able to output SVG drawings of unicorns from only their descriptions. This capability lives in an awkward in-between. Or, I don't know if you've had this experience, but if you've read a lot, lot about a place and then visit, there is a sense of having been there before mediated by the indirect observation,