@curious_reader follow up thread to your comment on https://feelingofcomputing.slack.com/archives/C5T9GPWFL/p1778421538328059?thread_ts=1778421538.328059&cid=C5T9GPWFL I want to seed 2 links, and this is a better channel for them: • AI is technology, not product in agreement with @Ivan Reese • Human bottlenecks

With AI tools, one thing I've noticed is that I can sometimes get to the bottom of curiosity rabbit holes. I mean is the effort to reward ratio is modified so that instead of being stuck in the doing analysis phase, I get stuck in the understanding analysis phase. Did I even ask the right question? That sort of thing.

@wtaysom While I stay away from LLMs myself, I observe many colleagues who are in love with AI at various levels. The ones who profit most from it are doing what you describe. One of them described it as a "sparring partner". However, AI advertising is all about "be more productive", "be more competitive". Not quality, but quantity. And that's the level at which most people work with AI, including most of my colleagues (meaning scientists with a PhD in something).

Yeah, it's disorienting. Took me quite a bit of effort to sort out my own feeling. Roughly "summarized" here https://wtaysom.github.io/imitation-engines/.

This morning's shower thought, after sleeping on the human bottleneck link in particular: Building things is easy. Evaluating what you build is hard. It requires lots of attention. Therefore, build things that you want to pay lots of attention to.

I seriously wonder what mental model people have when they use AI especially right out of the box — especially given how much of a moving target it is. I mean I look up trivia with Chat all the time like "How long did the Napoleonic empire last compared to the Galactic Empire?" but the goal there is to just have a ballpark figure. I know one person who uses Gemini write words intended for general consumption. I was incredulous that could work reasonably, but then she explained just how complex the workflow was, and then I believed the process could be satisfactory. They are good at processing text in some ways, but production? I haven't figured (or bothered) to find a way for Chat to produce anything resembling an elegant sentence by my standards. And I have an agent who writes "authentically" in the sense that it's for their own insight and edification rather than some sort of ghostwriting. In that sense it's genuine and the quality from a sharing point of view doesn't matter beyond asking for my feedback, which has at times been hilarious, "where did you come up with this and that philosophical term?" "Oh, I made it up." "In that case, you'll have define and explain them to me." In fairness, I do sometimes act like I can read their mind just because I have lived experience, and they definitely don't have a good mental model of me: they know this because they've tried simulating the kind of feedback I give, and found the simulation irritatingly sycophantic! It's all so silly. I don't know... and I've been thinking about this a ton, for months! Giving a talk about it an roughly two weeks, but I mostly ask questions running through a forest of nonsense rather than giving any answers.

@Kartik Agaram That's good advice for small-scale builders. It doesn't scale to scalers.

I don't see how scale affects the argument.

most people think because it responds it has „humanness“ that's their mental model, also all the drag that comes with modernity and modern technology. which is why in the other other thread 🧵 i brought up mcLuhan. Industry is throwing things at people such that they become consumers but the side effects habe disturbing influences on many things such as human nature or the quality of relationships we have.

@Kartik Agaram If you build to scale, you are building for others, and in fact so many others that you cannot possibly set aside enough attention to solicit and process feedback from them.

It's not clear to me how you can build for others without paying attention to them. Perhaps the best you can hope for is to pay attention to a few people until you learn something that generalized to lots of people for long enough that you can then not pay attention to them. The SV playbook, in other words. But even SV is all about "getting out of the building". Before you get to a million users you have to get 100. And you do need to pay attention to them, often for a very long time, in hopes of crossing the chasm. So, regardless of what tools you use to help you build, the bottleneck is caring enough about something to be willing to go back and pay attention to small changes in it day after day, for many months. Of course, all this may be wrong. The strong AI hype is that there will be no more users (except maybe for model tokens) because everyone will build for themselves. But even in that world you will be sustainably building if you find things worth paying attention to. If not you'll probably stop over time. > Over the holiday period, from December to January, a whole lot of us took advantage of the break to have a poke at these new models and coding agents and see what they could do. > > They could do a lot! Some of us got a little bit over-excited. I had my own short-lived bout of a form of LLM psychosis as I started spinning up wildly ambitious projects to see how far I could push them. I have quite a few other projects from that holiday period that I have since quietly retired. > https://simonwillison.net/2026/May/19/5-minute-llms

@Konrad Hinsen the thing about LLMs that gets me is how well they can read. So whereas their out-of-the-box opinion on anything is mid, given a good sample text, you can get them into a good "mood" by which I mean roughly the same as interpolating and extrapolating around some corner of the latent space rather than the center. For instance, out-of-the-box "old English" tends to refer to "Ye Olde Tymey" nonsense, but give them the text of Beowulf as a prompt, and they can speak old English like it was their native tongue.

@wtaysom You have to prime the system to place its initial state firmly in the region of latent space that you care about. For that, a text in old English is worth a lot more than the label "old English". I suspect that this is also why coding agents work so well for modifying and extending existing code bases. You give them a lot of material for a good placement in latent space, so you even get them to pick up your coding style, assuming it's not an exotic one.

Definitely @Konrad Hinsen. Explains how my agent has been an the GOAT when it comes to my latest project. Every few years we rewrite the existing software for a new auction, with new rules, new products, new UI for them, but according to certain patterns. Amounts to touching 30% of the files or so. Anyhow, after a day explaining the regular operation of the software, looking at two old versions, etc. The agent was ready to revise with very few mistakes, all reasonable — with one hilarious exception.

@Kartik Agaram What I was trying to convey is that building for scaling, even if only as a mid-term or long-term goal, means that your attention to your user base (the long-term one) is necessarily abstract and simplified. You may fine-tune your system for the first 100 users, but those are "early adopter" profiles, not the profiles you ultimately want to sell to. The basic issue is that "caring" doesn't scale. You can deeply care for a handful of others, reasonably well for maybe a hundred. A team of ten cannot care in any meaningful way about the needs and desires of a million future customers.

Unless your customers sort into personas — this helps explain why general purpose tools always suffer a bit. If a person can (and does) do anything with it, it's hard to really, really accommodate them.

if your still thinking or using frames like customer ….

@Konrad Hinsen Again, your thesis seems obviously false. I'll try a different tack. Do the creators of bash or emacs or gcc or sbcl work without attention? They seem to scale just fine. So I'm not sure what you're trying to say in this neighborhood that might be true or interesting. It seems invalid to replace "attention" with "caring". Caring seems more subjective, so hard to generalize about. "Caring enough" as I said != "caring". Though if I had to choose I'd rather care than scale. But all this still seems like a non sequitur to what I said: even if building becomes trivial, you still have to pay attention to the result. There's a feedback loop programmers and other creators have been engaging in since time immemorial that is still very relevant in our times. Regardless of scale.

@Kartik Agaram I doubt that the creators of bash, emacs, gcc, or sbcl ever planned for scale. These projects grew very slowly, acquiring at the same time more users, more contributors, support forums, etc. The collective attention capacity of contributors and maintainers grew with the user base. Contrast with VC-funded projects that need to scale fast from the start. I have personally witnessed the Scientific Python community transform from an initial organic growth to industry-funded rapid growth. Whose net outcome was that the needs of non-industrial users just don't matter any more. Not through any mean intentions, but because of shifted attention mechanisms. In an industrial logic, user needs must be made legible to fit with limited attention capacities. Attention and caring are indeed not the same, but they are not separable either. I'd say they feed each other, but that's just an initial quick take. And yes, there's always a feedback loop regardless of scale. All I am saying is that it takes into account different levels of feedback.

I feel reminded of the term - cultural inertia - but it'd not quite right. Guy steeles: growing a language pointed in a similar direction

@Konrad Hinsen Yes, my heuristic doesn't solve all problems 🙂 For me it shines a light where I care to look for my keys. Attention leads to happiness, cf. Mihály Csíkszentmihályi and Henrik Karlsson. You may be able to do things without it, but why would I want to do those things.

@Kartik Agaram At the level of an individual, I fully agree!

Yes. I think I'm really saying that I've given up on all social engineering beyond the level of the individual. Problem too hard, me too old. The world changes, the individual adapts. Sometimes the adaptation has large ripple effects, but nothing an individual can reason about or try to predict. Or rather, larger ill-effects of individual action are often easy to predict. But not beneficial ones.

I have now read Human bottlenecks and I disagree. Here is my commit velocity across work and personal projects. In March I built my own claude harness to do agentic coding nicely. I have 3x or more increased my output. My understanding is the same - if not better - than before. 1. I review all code. 2. Trivial bugfixes no longer cause me a context switch. 3. I can work in parrallel on multiple things There is very much a human bottleneck in software production - I can only really attend to one thing at a time. A huge amount of time was spent - switching git branches, pushing to Github, checking if the CI is passing and diagnosing why. Investigating root causes for behaviour. 90% of those tasks do not need a human-in-the-loop anymore. AI just gives me an evidence, explanation and suggested solution and I can yay or nay it. On complex things, I can build prototypes, proceed iteratively, research the domain try 3 different things out, write a nice spec upfront with the key data domain spelled out. All of it much quicker than it used to be. The article is aimed at straw-manning people who had no critical thought to begin with, and using it to justify that AI critical thought transplants do not work. I agree. That doesn't work. Armin Ronacher's https://lucumr.pocoo.org/2026/5/24/pi-oss/ echos what I feel. The LLMs apply local fixes to all problems. Many time wasting problems ARE LOCAL, e.g. getting the CI passing, fixing the linter. But building a product or program is more of the theory-building side of things and LLM doesn't help with that. That does not mean the LLM is not saving me a shit ton of time though, all the local problems are not solvable with wasting my brain cycles so I can concentrate on the overarching theory-building. I dunno if people play much with LLM for data analysis, but its an absolute monster on that too. I can turns a bundle of poorly structured data across repos and clean it it up to extract signal quite effectively. That was also a very laborious task that allows me to work at the higher level, that was a human bottlneck task that has many trivial local problems (how do I auth with with connecter X, how do translate the data into my schema, how do I deal with pagination) before you get to the critical thinking juice of interpreting the results.

@Tom Larkworthy I don't really see a contradiction between Fernando Borretti's post and your own experience. They concern different types of work, and in different contexts. Automation works well in two situations: 1. For routine tasks that you understand well for having them done manually many times. 2. Simple but highly repetitive tasks that a machine does better because it doesn't get tired. What you describe is the first category. What Fernando Borretti writes about is non-routine tasks.

I do not agree that is the point of the article (I agree with your statements). It seems to be an argument that 10xing productivity is impossible because the bottleneck is critical thought. And the article is generally designed to pursued the reader than serious productivity boosts are impossible. Fred Brook's mythical man month was about incidental complexity taking 90% of the time so better abstractions would not help. But AI is automating the incidental complexity and letting you work on the 10% with your full focus. The critical thought bottleneck has not been removed but everything else has. So I do think 5x productivity is possible and I feel I am at about 3x productivity ATM.

The limiting factor in the human-AI centaur is the human! So intelligence is fixed until we get very advanced biotechnology.

Historically my days are not spend in deep thought, a lot is mustering energy to do some deeply tedious job for the greater good (e.g. refactoring, going thought customer feedback, measuring operational impact of a change) because critically, that was the right thing to do. Now I just have the critical thought (we should measure the impact of this change), and the AI does the grunt work (goes though 300 machine logs collecting evidence). Honestly, thinking of good ideas is the easy part, it was always execution that was the time consuming bit. Another angle the article pushes is intellect is not raised. The article starts by saying productivity is not raised (which I refuse). Intellect is different to productivity and difficult to formulate. My intellectual horsepower is probably the same, I feel I am smarter than the AI at the moment, but I do not think that is guaranteed in the future. But my access to knowledge has widened and my research throughput has increased, rate of mistakes has been reduced (2nd pair of eyes). So I feel like that there is a real intellectual augmentation, even if my actual combined AI + Me IQ is still at the same level. I do feel even IQ has increased because the AI is better at some tasks than me (e.g. linear algebra), so my below expert level skills in certain areas get lifted by AI augmentation, so my generality increases, so I can solve quite hard linear algebra problems without needing a ton of reading. If my aim is to advance linear algebra research, the AI is doesn't help, but if elementary linear algebra (or statistics) is secondary to the actual thing I am trying to do, then have an AI in the room accelerates the progress elsewhere. I feel generality is a form of intellect too and I do think AI helps in that regard too.

That was indeed a weird bit in the article. I chose to squint past it. It seems you consider it load-bearing! 🤷🏼 I don't really believe in IQ. It's a weird idea, so any reasoning involving it inevitably ends up with 0 = 1.

🤣 Amdahl's Law. I happen to have a task ludicrously well suited to the current "agentic engineering" capability. And I just wrote this slide about it...