My latest blog post, Automating Science, is primarily about the use of LLMs in scientific research, but much of what I write applies for knowledge work more generally.

I like the article. And I fully agree on the remark that (in my words) science is not (and in my opinion should never be) settled. Whenever we learn or experience something new, existing science might need to be adapted. This also holds for general knowledge. The older I get the more the feeling grows that there is so much I do not know. LLMs on the other hand seem to act contrary, everything seems a hard fact in the way it is told. And I think/seem to see that a lot of people put a lot of trust in these ‘facts’. So the best option for me remains: keep an open mind and think for myself. Not always easy though to not get caught in existing believes and previous experiences.

In fact, one of the best uses for LLMs I have seen so far in science is as a critic of one's own beliefs. State an hypothesis and ask for evidence against it.

Indeed. Be aware that it has enough input to be able to do so of course. An unpopular world view might not be part of its learning set. But that is what you mentioned in to article too I guess.

You have to know quite a bit about LLMs to do a good job - as with other powerful tools. And it would help a lot if we had more transparent LLMs, whose learning set could be scrutinized. Perhaps the two biggest issues with LLMs, other than their huge negative externalities, are opacity and the exceptional ease for using them badly.

There are software developers using LLMs without even reading the code they produce, they just manually see if the app works. Are scientists doing this too with just testing some end product? I hope not. Seems like a different problem from software where it's more about the output than finding some novel new way to do things, atleast in app development.

Just about everything possible is happening with LLMs in science at the moment. Many people use them as tools to help them with their research, or just with writing their reports. Some check the output, others don't. There are plenty of scientific publications with hallucinated references, or telltale phrases such as "Certainly, here is the paragraph you asked for:". Publications completely written by LLMs (based on no human research work at all) remain very rare, but a few made it into some conference proceedings.