Has anyone here spent much time thinking about how the business of software changes as AI models, techniques, and devtools improve? In particular, I've been spending a lot of time thinking about the implications of: 1) what happens when things that previously took 6mo-1y to build out are quickly turning in to $0.10 API calls today and once models are small enough, architectures optimize more, and hardware improves, transform into a low level OS primitive tomorrow 2) do we need to start accounting for software as an asset differently when the payback period gets measured on a much shorter timeline? 3) internally, how much time do we need to focus on reducing our Time To Rebuild vs letting external forces reduce it for us?

I’m worried about the environmental impact few people seem to care about. Also AI models are currently deceptively hard to make. They’re still error-prone and a good design using them implicitly automates LESS because a human should be able to correct errors. So I’d argue the models don’t cover a vast majority of use cases as much as everyone seems to thinks so. As with every tool, there are niches and drawbacks. My opinion, also, is that no one should “wait around.” This probably doesn’t cover most of the question, but I thought I’d add my 2¢.

I'm curious to see how the energy usage/environmental impact changes as different model generations and architectures are developed. For example, with deepseek's latest release you could download a 32b parameter model that runs fine on last generation's macbook pro that's roughly gpt4/sonnet3.5 level which is high enough quality for most tasks

My concern is that people will just consume more and more to compensate for higher/efficiency, meaning that unless people and organizations are actively aware, it won’t matter. I hear deepseek on-device is a lie. Actually just llama fine tuned on deepseek data. Actual deepseek is not able to run locally.

Yeah, they're the llama and qwen models, but the techniques they used still improve them enough to hit those benchmarks running locally, but that's also not the stopping point...its more that it shows its possible and we haven't hit a wall yet

I think we still need to treat these carefully even if the power consumption per request is reduced, but yes.

The cost should be included in the price (not that is). That way decisions can be made based on only one metric.

Money is more abstract that environmental impact though. People will pay. hmmm

Ooh @George if I'm understanding your comment - this new "cost" I'm talking about could actually cause software prices to rise and be more volatile? Because if you're doing something that has a high Time To Rebuild (at first) that means you're probably one of the only ones offering it for a time and then have a monopoly over it...so earn as much as you can until your advantage is gone

Good question! I've been recently thinking about this as well and I wrote a couple of short articles [1] [2] From the perspective of your question, what I see happening is: • The moat of models trained in-house is shrinking. I am experiencing that some glue code and prompting can go quite far. • AI through UI, based on text. A lot of UX just becomes text prompts. • Near zero vendor lock-in with LLMs: they all just understand natural language and produce either that or a structured language that you instruct them to use.

I’ll push back a tiny bit. There’s basically total vendor lock-in if you’re relying on anyone’s API to a model whose training data are behind a blackbox. We don’t know what’s going on behind the scenes.

Ah I think the point Federico was making there was that everyone has basically standardized on the OpenAI compatible api for the basic conversation call…so the only thing you need to do to switch is change the url and api key

Just some interesting augmentations to the text https://arxiv.org/abs/2310.03691

I won't be surprised if the main purpose of AI code tools ends up as a way for ketamine billionaire technocrats to control what code gets written https://github.com/orgs/community/discussions/72603 In the meantime I reserve the right to not be interested in AI code tools

Karl, indeed there's a vendor risk in that if you cannot run a good LLM on your own, then you're completely dependent on a vendor's LLM. But what I mean is that the nature of the LLM is to accept and produce natural language (or a structured output of your choosing), so you can switch from one provider to the next with ease.

Sort of. A lot of the LLMs respond well or poorly to different versions of the prompt, so it’s not as universal as it sounds, but yes in concept I see.

eh, for the large majority of the cases, especially if you're using them as generic functions of string to some sort of structured output and not building a chatbot, there's not much of a noticeable difference

Indeed, in practice it might be trickier, but in principle, with really good LLMs it should work OK. If there was only one good LLM in town, then my point would make no sense. An oligopoly is a more stable situation. Open source LLMs like DeepSeek really tilt the balance towards the rest of us.