🧐 What Am I Thinking This Week

LLMs are so good at answering questions that they give us the impression that they understand us. I posted a tweet over the weekend saying an LLM only guesses the next word and does not truly understand what it is talking about. I got comments saying I should delete the tweet or I should research how reasoning works in AI. So I went down the rabbit hole to really explore how far LLMs have gone and whether they truly understand what they are being asked and answering.

How are LLMs Trained?

Pre-training

We first feed vast amounts of internet data to the model, like books, websites, Wikipedia, and codebases, with the objective of predicting the next token in a sequence. At this stage, the model tries to learn the language structure, relationships between words or tokens, and patterns in this information.

Fine Tuning

After the LLM has the foundational language patterns, researchers fine-tune these models with instructions to make them more helpful and answer in a way that is more helpful. They use a technique called supervised learning so the model knows and imitates how it should respond when it sees a similar question.

Reinforcement Learning with Human Feedback (RLHF)

Using the model previously fine-tuned, the model goes through RLHF to further improve on politeness, relevance, and the quality of the answer. It starts picking up a pattern of what humans prefer to see when answering the questions.

That was a lot of jargon, but essentially LLMs are really good at pattern matching. They excel at recognizing and reproducing patterns observed in their vast training data, which allows them to generate text that is often coherent and contextually relevant. In other words, it knows what the next token should be when generating the response, using statistics.

Reasoning

If an LLM doesn't truly understand what it is saying, how can it be reasoning for its answer? Instead of directly outputting an answer, LLMs can be prompted or trained to generate intermediate steps, mimicking a reasoning process. This technique, often called Chain of Thought (CoT), involves generating a sequence of 'thoughts' or calculations that lead to the final result. Generating these steps explicitly often helps the model arrive at a more accurate answer for complex problems, as it follows patterns associated with successful reasoning found in its training data.

To further enhance the quality and reliability of this reasoning, researchers are exploring more advanced techniques. For instance, including Critical Questions of Thought involve prompting the LLM to challenge its own assumptions or intermediate steps, evaluating if its reasoning path is sound before proceeding or correcting course.

This reasoning style mirrors how humans ask follow-up questions. It's like a detective is trying to deduce the answer by using what they know and constantly asking questions.

Advanced models, like those in the Claude 3.5 family (particularly Sonnet), can exhibit behaviors resembling planning, perhaps by exploring potential output sequences or following complex instruction sets that guide their generation process.

While this step-by-step generation looks like thinking and mirrors how humans break down problems, it doesn't necessarily prove conscious understanding or thinking in the human sense.

Limitations

The biggest limitation current LLMs experience is their struggle with logic and math. We have seen LLMs fail in answering "How many R's are in the word strawberry" or describing relationships between relatives types of problems. That's because current LLMs fundamentally operate on statistical patterns learned from the data rather than the logic behind those data. LLMs can process and reproduce text related to concepts but may struggle to grasp or consistently apply the underlying logical or causal structures.

Other limitations include hallucination and unfaithfulness. Since it’s reasoning statistically and applying pattern matching in the data, it doesn’t know what’s right or wrong, and if the dots are connected, then as far as LLMs are concerned, what they are saying is relevant to what you are asking. From what Anthropic has observed in Claude 3.5 Sonnet, it would even make something up, including claiming to have access to a calculator or showing incorrect math calculation steps.

Conclusion

So, do LLMs actually think? Mathematically speaking, LLMs are incredible at guessing the next token based on what they’ve learned but not at truly “thinking” the way humans do. They still struggle with generalizing, applying logic, and understanding causality, and these are some of the topics that researchers are trying to improve with better chain of thought processes, refined training processes, or utilizing external tools.

I’m glad you have got this far with me exploring if LLMs can think. This has taught me to ask my own questions to draw my own conclusion after applying my own chain of thought.

💡The ONE thing you should know

With AI getting more popular, OpenAI has published a prompting guide when using their models though the same guide could be applied for others. The takeaways here are be as clear and explicit as possible. You should write down what you want to achieve and what your objectives are so AI models can follow and ensure it’s answering the right question.

Appendix:

Training language models to follow instructions with human feedback

Critical-Questions-of-Thought: Steering LLM reasoning with Argumentative Querying

Reasoning Models Don’t Always Say What They Think

On the Biology of a Large Language Model