An LLM-using society should develop a taboo for calling what they do "thinking" (inclusive of of adjacent terms) because anthropomorphism distorts our thinking. Saying "thinking" has long been a useful shorthand for waiting for a deterministic algorithm or explaining the output of small-scale heuristic tools like Markov chains, but LLMs are in the uncanny valley of non-determinism/pattern-matching/recursive natural language through an mind-boggling large search space pushes it past an inflection point for me where it's too misleading in too many subtle ways. The extreme case of people taking LLMs as friends and romantic partners is obvious. But it's important to remember even just for programming and debugging that its internal models are fabulously alien despite using fluent English. Curious if others see where I am with this.

I've been very, very (may too very) focused on this topic and find myself living awkwardly in the middle between AI as new best friend and fancy autocomplete with strong sensitivity to subtle details. I mean one LLM can part of systems on either end and the difference between, let's say, romantic partner and romantic fanfic is not that big. So at times coding agent thinking is human thinking, sometimes its superhuman, and other times it resembles drunken thinking and all three modes can occur in rapid succession. My practical approach is to assume less and allow for more possibilities.

I still try to think of the output text tokens as ultimately human origin, and the sentence/paragraph structures etc. Mentally prepend “Here’s a collage of relevant text snippets cut and paste from the internet!” to every answer. Maybe I should make it output in ransom note style with words cut out of magazines. It's fine to feel like you are reading human text because you sort of are, in a similar way to reading machine translated tweets.

Often, it is like collage. Then the quality is in the cutting and the glue. As the size of the pieces decreases, we see more of the effect of the cutting and the glue.