Video / Short
Breaking An LLM
Large language models can answer confidently even when they are guessing. This short explains why AI output should support, not replace, human judgement.
Video summary
Pete Cooper explains that large language models can hallucinate: they may produce confident answers while guessing or making things up. He says attempts to train that behaviour away can create another problem if a model becomes unable to generate the useful content people expect.
The short also considers the commercial incentives behind highly engaging AI products, including the pressure to keep people interacting with a particular model. Cooper’s main caution is practical: treat an LLM’s output as support for human judgement, not an authority that can be trusted without checking.
Video transcript
Read the spoken content without loading the YouTube player.
So how does an LLM break? Well, we've all seen this problem with LLMs: they hallucinate. They make things up. When they try to train that out, it actually breaks the model. Why? Because it doesn't do the thing we know it needs to do: generate content for you. But it goes deeper than that. It goes to the fundamental business model. What are all these companies trying to do right now? They're trying to get us to interact, stay sticky, stay loyal, and use their LLMs. So ChatGPT is competing with Claude in the same way Facebook used to compete with MySpace.
Facebook got more interaction, got more people, and became successful, while MySpace died out. And we're seeing the same thing. If the model doesn't reward the user and keep them using it, whether it's right or wrong, their business model is going to fail, and they will be the ones left behind.